The prep layer, from light to heavy
| Tool | What it does | When to reach for it |
|---|---|---|
| Dataflows | Power Query (the same 'M' transforms) run in the SERVICE, output stored for reuse | Centralise and reuse cleaned tables across many models |
| Data pipelines (Fabric) | Orchestration — copy, transform, schedule multi-step data movement | ETL/ELT at scale, moving data into a lakehouse |
| Lakehouse / Warehouse (Fabric) | Full data-engineering stores (files + tables) in OneLake | The governed data estate models source from |
Dataflows — reusable prep in the cloud
A dataflow runs Power Query transformations in the Power BI/Fabric service (not on the desktop) and persists the result, so multiple semantic models can reuse the same cleaned, conformed tables instead of each re-implementing the same logic. This is the DRY principle for BI: prep once, consume many times. Gen2 dataflows in Fabric can write their output directly into OneLake destinations.
OneLake — one storage layer for everything
OneLake is Fabric's single, tenant-wide data lake (built on ADLS Gen2, using open Delta/Parquet formats). Every Fabric workload — lakehouses, warehouses, dataflows, and Direct Lake semantic models — reads and writes the same OneLake, so data is stored once and consumed in place rather than copied between silos. 'Shortcuts' let OneLake reference external data without moving it. This is the convergence that makes Direct Lake possible.
How it fits together
Source → (pipeline/dataflow prep) → OneLake lakehouse/warehouse → semantic model (often Direct Lake) → report. The BI estate and the data estate are now the same estate.
Failure modes
- Dataflow sprawl. Dozens of overlapping dataflows re-deriving the same tables — the reuse benefit inverted into duplication and confusion.
- Broken dependency chains. A model depends on a dataflow that depends on a pipeline; one upstream failure cascades into stale reports downstream, often silently.
- OneLake as an ungoverned dumping ground. Files landed without lineage, ownership, or sensitivity labels — a lake that becomes a swamp.
- Refresh ordering. A model refreshing before its upstream dataflow completes serves yesterday's data with today's timestamp.
- Credential/gateway breakage in dataflows pulling from on-prem sources (same failure surface as model refresh).
What to watch (proofs)
- Dataflow inventory:
Get-PowerBIDataflow -Scope Organization(andadmin/dataflows) lists every dataflow and its workspace — the prep-layer census. - Lineage: the admin scanner (
admin/workspaces/getInfo) returns model→dataflow→source dependencies — the chain that breaks. - Refresh health of dataflows: dataflow refresh history alongside model refresh, to catch upstream failures feeding stale data downstream.
- OneLake governance: which lakehouses/warehouses exist, their workspaces, and whether sensitivity labels are applied — the swamp check.