GitHub, Azure DevOps, or Fabric—Who’s Actually in Charge? cover art

GitHub, Azure DevOps, or Fabric—Who’s Actually in Charge?

GitHub, Azure DevOps, or Fabric—Who’s Actually in Charge?

Listen for free

View show details

About this listen

 Here’s a statement that might sting: without CI/CD, your so‑called Medallion Architecture is nothing more than a very expensive CSV swamp. Subscribe to the M365.Show newsletter so you i can reach Gold Medallion on Substack!Now, the good news: we’re not here to leave you gasping in that swamp. We’ll show a practical, repeatable approach you can follow to keep Fabric Warehouse assets versioned, tested, and promotable without midnight firefights. By the end, you’ll see how to treat data pipelines like code, not mystery scripts. And that starts with the first layer, where one bad load can wreck everything that follows.Bronze Without Rollback: Your CSV GraveyardPicture this: your Bronze layer takes in corrupted data. No red lights, no alarms, just several gigabytes of garbage neatly written into your landing zone. What do you do now? Without CI/CD to protect you, that corruption becomes permanent. Worse, every table downstream is slurping it up without realizing. That’s why Bronze so often turns into what I call the CSV graveyard. Teams think it’s just a dumping ground for raw data, but if you don’t have version control and rollback paths, what you’re really babysitting is a live minefield. People pitch Bronze as the safe space: drop in your JSON files, IoT logs, or mystery exports for later. Problem is, “safe” usually means “nobody touches it.” The files become sacred artifacts—raw, immutable, untouchable. Except they’re not. They’re garbage-prone. One connector starts spewing broken timestamps, or a schema sneaks in three extra columns. Maybe the feed includes headers some days and skips them on others. Weeks pass before anyone realizes half the nightly reports are ten percent wrong. And when the Bronze layer is poisoned, there’s no quick undo. Think about it: you can’t just Control+Z nine terabytes of corrupted ingestion. Bronze without CI/CD is like writing your dissertation in one single Word doc, no backups, no versions, and just praying you don’t hit crash-to-desktop. Spoiler alert: crash-to-desktop always comes. I’ve seen teams lose critical reporting periods that way—small connector tweaks going straight to production ingestion, no rollback, no audit trail. What follows is weeks of engineers reconstructing pipelines from scratch while leadership asks why financials suddenly don’t match reality. Not fun. Here’s the real fix: treat ingestion code like any other codebase. Bronze pipelines are not temporary throwaway scripts. They live longer than you think, and if they’re not branchable, reviewable, and version-controlled, they’ll eventually blow up. It’s the same principle as duct taping your car bumper—you think it’s temporary until one day the bumper falls off in traffic. I once watched a retail team load a sea of duplicated rows into Bronze after an overnight connector failure. By the time they noticed, months of dashboards and lookups were poisoned. The rollback “process” was eight engineers manually rewriting ingestion logic while trying to reload weeks of data under pressure. That entire disaster could have been avoided if they had three simple guardrails. Step one: put ingestion code in Git with proper branching. Treat notebooks and configs like real deployable code. Step two: parameterize your connection strings and schema maps so you don’t hardwire production into every pipeline. Step three: lock deployments behind pipeline runs that validate syntax and schema before touching Bronze. That includes one small but vital test—run a pre-deploy schema check or a lightweight dry‑run ingestion. That catches mismatched timestamps or broken column headers before they break Bronze forever. Now replay that earlier horror story with these guardrails in place. Instead of panicking at three in the morning, you review last week’s commit, you roll back, redeploy, and everything stabilizes in minutes. That’s the difference between being crushed by Bronze chaos and running controlled, repeatable ingestion that you trust under deadline. The real lesson here? You never trust luck. You trust Git. Ingestion logic sits in version control, deployments run through CI/CD with schema checks, and rollback is built into the process. That way, when failure hits—and it always does—you’re not scrambling. You’re reverting. Big difference. Bronze suddenly feels less like Russian roulette and more like a controlled process that won’t keep you awake at night. Fixing Bronze is possible with discipline, but don’t take a victory lap yet. Because the next layer looks polished, structured, and safe—but it hides even nastier problems that most teams don’t catch until the damage is already done.Silver Layer: Where Governance Dies QuietlyAt first glance, Silver looks like the clean part of the Warehouse. Neat columns, standard formats, rows aligned like showroom furniture. But this is also where governance takes the biggest hit—because the mess ...
No reviews yet
In the spirit of reconciliation, Audible acknowledges the Traditional Custodians of country throughout Australia and their connections to land, sea and community. We pay our respect to their elders past and present and extend that respect to all Aboriginal and Torres Strait Islander peoples today.