Do You Need a Data Lake? DuckDB and MotherDuck as the Pragmatic Alternative

The pitch you’ve probably heard

At some point a consultant, a vendor, or a well-meaning new hire tells you that your business needs a data lake. Or a lakehouse. The pitch usually arrives with an architecture diagram containing a dozen logos and a phrase like “single source of truth.”

Here is my position, as someone who builds data platforms for a living: most South African businesses that get this pitch do not need a data lake. They need their data in one queryable place, with someone accountable for its quality. Those are different problems, and the second one is much cheaper to solve than the first.

Let me unpack what a data lake actually is, what it actually costs, and what I use instead on real production systems.

What a data lake actually is

Strip away the branding and a data lake (or its tidier successor, the lakehouse) is four things:

1. Object storage: files sitting in something like S3, Azure Blob or GCS.
2. Open file formats: usually Parquet, often wrapped in a table format like Iceberg or Delta that adds transactions and schema evolution.
3. A catalog: something that knows which files belong to which table, so engines can find them.
4. Query engines: Spark, Trino, Snowflake, Databricks and friends, reading those files on demand.

The genuinely good idea in there is the decoupling: storage is cheap and open, and any engine can read it. Nobody is locked into one vendor’s proprietary format.

The problem is the org size at which this pays off. The architecture was designed for companies where many teams with different tools need shared access to very large data. If you are one team, with one or two analytical workloads, over gigabytes-to-low-terabytes of data (which describes most mid-sized businesses I meet), you are buying a shipping port to move a bakkie-load of goods.

The hidden costs nobody puts on the diagram

The architecture diagram never shows the payroll. In practice a lakehouse comes with:

• A platform team. Someone has to run the catalog, manage the table formats, tune the engines, and upgrade all of it. That is one to three engineers before a single business question gets answered. In South Africa, that is a seven-figure annual rand commitment in salaries alone, before cloud spend.
• Governance overhead. Access control across object storage, catalog and engines is genuinely hard. Most small deployments quietly end up with “everyone can read everything,” which defeats a stated purpose of the exercise.
• Pipeline sprawl. Once the lake exists, everything must flow into it. You accumulate ingestion jobs, orchestrators, dbt projects and monitoring for all of the above. Each piece is reasonable; the sum is a full-time maintenance burden.
• Cluster economics. Distributed engines bill you for coordination. A significant slice of your compute spend goes to machines talking to each other about work, not doing it. Paid in dollars, felt in rand.

None of this is wasted at genuine scale. All of it is wasted below it.

The middle path: Parquet files and DuckDB

Here is what changed the calculus over the last few years: single machines got very big, and single-node analytical engines got very good.

DuckDB is an open-source, in-process analytical database: think “SQLite for analytics.” It reads and writes Parquet natively, queries files on object storage directly, and executes the kind of aggregations, joins and window functions that used to demand a cluster. It runs inside your Python script, your Go binary, or a plain CLI. There is no server to operate.

The practical consequence is that the useful core of the data lake idea (cheap open storage plus a fast engine) survives without the platform around it. Land your data as Parquet, partitioned sensibly, in object storage or on disk. Point DuckDB at it. That’s the stack.

And it handles far more data than people assume. A modern engine on a single decent VM comfortably works through tens or hundreds of gigabytes of compressed columnar data: as of writing, workloads well into the hundreds-of-millions-of-rows range are routine on hardware that costs less per month than a junior analyst’s daily rate. Most businesses dramatically overestimate how “big” their data is. Your transactional history since inception is probably smaller than a season of a Netflix show.

MotherDuck: when you want it shared and serverless

The gap in the plain-DuckDB story is sharing. DuckDB is in-process: brilliant for a pipeline or a single analyst, awkward when five people and three services need the same live tables.

MotherDuck is DuckDB’s managed cloud service, and it fills exactly that gap: your DuckDB databases live in the cloud, multiple clients connect to shared data, and compute is serverless: you are not administering a warehouse cluster. Your local DuckDB can even query cloud tables and local files in the same statement, which is a genuinely pleasant way to work.

This is not theoretical for me. The Decarb.Earth case study runs on this stack in production: a Go ingestion service pulls telemetry from fifteen inverter-vendor ecosystems and lands normalised time-series into MotherDuck, covering 24,350 solar plants across 44 client projects. The same warehouse powers a fleet operations dashboard that surfaces silent plants, negative-consumption rows and capacity outliers. One engineer owns the whole platform. There is no platform team, no Spark cluster, no catalog service, and it produces registry-grade carbon accounting for a continental fleet.

There are real constraints to respect. MotherDuck is single-writer at the storage layer, so on Decarb I serialise writes while keeping reads concurrent (a design consideration, not a dealbreaker), but you should know it going in. It is also a young platform compared to the incumbent warehouses; evaluate it against your workload rather than my word.

When Postgres alone is still the right answer

Honesty demands the step below DuckDB too. If your analytical questions are “monthly sales by branch” over a few million rows, the Postgres (or MySQL) database you already run will answer them in milliseconds with an index and a view. Adding any second system (DuckDB included) costs you a sync pipeline and a second source of truth.

Stay on Postgres alone while all of these hold:

• Analytical queries don’t visibly slow the production database.
• Data volumes are in the low gigabytes.
• One team consumes the reports.
• You don’t need to join across multiple source systems.

The moment reporting queries start fighting transactions, or you’re joining three systems’ exports in spreadsheets, that’s the trigger for the next rung, not a lakehouse, just the next rung.

When you genuinely do need a lakehouse

The big platforms exist for good reasons. In my view you have a real case when several of these are true, not one:

• Multiple autonomous teams need governed, concurrent access to shared data with different tools: Spark here, SQL there, a BI layer on top.
• Volume is genuinely large: sustained tens of terabytes to petabytes, where single-node reading stops being physically reasonable.
• Streaming is core, with many always-on pipelines feeding near-real-time consumers.
• ML at scale: feature platforms, training sets rebuilt across huge histories, model teams sharing curated data.
• Regulatory-grade governance (lineage, audit, fine-grained access control) is a hard external requirement, not an aspiration.

If that’s you, hire the platform team and build it properly. Just notice the pattern: every item on that list is about organisational scale as much as data scale.

The decision ladder

Climb only when the current rung demonstrably fails, and expect each rung to last longer than the vendors selling the next one suggest. Decarb (a production platform spanning a continent’s solar fleet) sits happily on rung three.

Start with the question, not the platform

If you’re weighing this decision for your own business (whether that’s rung one or rung four), I’d rather help you find the cheapest architecture that answers your actual questions than sell you a bigger one. Run your project through the scoping tool for a quick structured estimate, or contact me directly and we’ll talk through where on the ladder you belong.