Why Leadership Thinks Copilot Is Useless (And Where the Numbers Back Them Up) cover art

Why Leadership Thinks Copilot Is Useless (And Where the Numbers Back Them Up)

Why Leadership Thinks Copilot Is Useless (And Where the Numbers Back Them Up)

Listen for free

View show details

About this listen

 Here’s something spicy: most organizations think Copilot is underused… not because users hate it, but because no one’s checking the right dashboard. Subscribe at m365.show if you want the survival guide, not the PPT bingo. We’ll check which Copilot telemetry matters, where users actually click, and how prompts reveal who’s using it for real work. Often the signals you need live in a different pane—let’s show you where to look in your tenant. This isn’t a pep rally; it’s a reality check with the data points that count. And once you’ve seen that, we need to talk about the reports leadership is already waving in their hands.The CFO’s Report Doesn’t LieEver had that moment when the CFO barges in, waving a glossy admin report like it’s courtroom evidence, and asks why the company shelled out for a Copilot license nobody seems to use? Your stomach drops, because you’re not just defending an IT budget line—you’re defending your job. And here’s the kicker: the chart they’re holding isn’t wrong, but it’s not telling the story they think it is. The leadership bind is simple: licenses cost real money, so execs want hard proof that Copilot isn’t just another line item in the finance system. Microsoft does provide reports, but what those charts measure isn’t what most people assume. Log into the admin center and you’ll see nice graphs of sign-ins and “active users.” Sounds impressive until you realize it’s basically counting how many times someone opened Word, not whether they actually touched Copilot once they were in there. This is where the data trips people up. That report showing 2,000 Word sign-ins? Leadership reads that as 2,000 instances of Copilot lighting up productivity. Reality: it just means 2,000 people still have Word pinned to their taskbar and clicked it once. No one tells them that Copilot activity is captured in separate telemetry. So while the chart says “adoption,” in truth Copilot might be sitting unused like an expensive treadmill doubling as a coat rack. Now, to be fair, Entra AD does exactly what it promises. It focuses on identity and sign-in telemetry—it tells you who walked through the door and which app they technically opened. What it does not do, by default, is surface the action-level data that proves Copilot adoption. Put simply: it’ll show you that John launched Word, but it won’t show you that John asked Copilot to crank out a three-page summary to save himself an afternoon. Always check your tenant’s docs or Insights pane for what’s actually available, because the defaults don’t go that far. Here’s one clean metaphor you can safely use with leadership: those reports are counting swipes of a gym membership card. They don’t show whether anyone touched the treadmill, lifted weights, or just grabbed a chocolate bar from the vending machine. That one line paints the picture without drowning in analogies. So what do you say when finance is breathing down your neck with pretty graphs? Here’s the leadership-ready soundbite: “Sign-in counts show who opened the apps. Copilot adoption means showing who actually used prompts and actions. I can pull behavioral reports for that—if our tenant has telemetry enabled.” That’s a safe, honest line that doesn’t oversell anything but tells executives you can provide a real answer once you’ve got the right data enabled. And this is where your action item comes in. Don’t waste time trying to prove adoption from identity numbers. Instead, verify whether your tenant has Copilot Insights or usage reports that surface prompts and actions. If it does, prep a side-by-side demo for the CFO or CIO: slide one shows a bland graph of “sign-ins,” slide two shows actual prompts being used in Outlook, Word, or Teams. The contrast makes your point in about 30 seconds flat. Because at the end of the day, raw sign-in and license charts will always frame the wrong narrative. They’re a door count, not a usage log. What leadership really needs to see are the actions that prove value—the moments where Copilot shaved hours off real workflows, not just opened an application window. And that sets up the bigger story. Because Microsoft doesn’t give you just one way to see Copilot activity—they give you two different dashboards. One is the guard at the door. The other is the camera inside the building. And only one of them will tell you whether Copilot is actually changing how people work.Entra AD vs. Insights: The Tale of Two DashboardsMicrosoft gives you two main dashboards tied to Copilot adoption: Entra AD and Insights. At first glance they both look polished enough to screenshot into a slide deck, but they don’t measure the same thing. If you confuse them, you’ll end up telling execs a story that sounds great but falls apart the minute someone asks, “Okay, but did anyone actually use Copilot to get work done?” Here’s the split. Entra AD primarily records identity ...
No reviews yet
In the spirit of reconciliation, Audible acknowledges the Traditional Custodians of country throughout Australia and their connections to land, sea and community. We pay our respect to their elders past and present and extend that respect to all Aboriginal and Torres Strait Islander peoples today.