Insights from the data engineering frontline
Expert articles and project deep-dives — covering Azure, Databricks, Microsoft Fabric and the future of data engineering.
Latest articles
The Medallion Architecture: Bronze, Silver, Gold Explained
The medallion architecture has become the standard pattern for organising lakehouse data. We break down what each layer does and how to implement it correctly.
Building a Production-Grade Databricks Lakehouse: 5 Lessons Learned
After delivering dozens of Databricks lakehouses, we share the architectural patterns and gotchas that separate a toy project from a production platform.
Retrieval-Augmented Generation: A Practical Architecture for Enterprise RAG
RAG is the most production-ready pattern for LLMs in the enterprise. Here's a battle-tested architecture with the gotchas we've learned.
Data Contract Testing: Why It Matters and How to Start
Data quality isn't a dashboard metric — it's an engineering discipline. Here's how to implement contract testing that catches issues before they reach production.
Azure Data Factory vs. Synapse Pipelines: Which Should You Choose in 2026?
The lines between ADF and Synapse pipelines have blurred. We break down when to use each, and why the answer increasingly depends on your broader Azure strategy.
Azure Synapse vs. Snowflake: A 2026 Comparison
Both platforms promise scalable analytics, but the decision is rarely straightforward. We compare them across cost, performance, ecosystem and strategic fit.
OneLake: Understanding Fabric's Unified Storage Layer
OneLake is the single storage layer underpinning all of Microsoft Fabric. Here's how it differs from a traditional data lake and why it matters.
Unity Catalog: The Cornerstone of Databricks Governance
If you're using Databricks without Unity Catalog, you're missing the single most important governance feature. Here's why it matters and how to adopt it.
Securing Your Azure Data Platform: A Checklist
Security is not a phase — it's a continuous discipline. Here's a practical checklist for securing your Azure data estate end to end.
Optimising Databricks Costs Without Sacrificing Performance
Databricks costs can spiral quickly. Here are the levers that actually move the needle — from cluster policies to Delta Lake optimisations.
DirectLake Mode: Power BI Without the Refresh Schedules
DirectLake is Fabric's most exciting feature for Power BI users — direct access to OneLake Delta tables with no import and no scheduled refresh. Here's what you need to know.
Feature Stores: Bridging the Gap Between Data Engineering and ML
Feature stores solve the training-serving skew problem and give data teams a clean interface to ML. Here's when you need one and how to choose.
Healthcare Data Engineering: Challenges and Best Practices
Healthcare data is complex, regulated and high-stakes. Here's how to architect platforms that are compliant, reliable and actually useful to clinicians.