Skip to content
Free since 2015. Built for the engineers who keep things running.

cat /articles/ducklake-11-solves-the-absurd-complexity-of-data-lakes-at-ai-scale.md

AI & MLOps

DuckLake 1.1 solves the absurd complexity of data lakes at AI scale

A New Stack podcast interview with DuckLabs' Pedro Holanda on DuckLake 1.1, an open-source lakehouse format that keeps table metadata in a standard SQL database instead of many manifest files.

The New Stack14 Nov 2026 · 4 min read · summary

article --summary

The summary

The piece explains how DuckLake stores lakehouse metadata in an ordinary SQL catalog rather than scattering manifest files across object storage, so queries can locate the right Parquet files directly. Readers learn how features like file compaction and inlining/flushing help with the small-files problem in streaming workloads. Holanda also describes why Parquet's layout resembles DuckDB's storage, which keeps conversion work low and lets DuckLake run without large clusters. The article additionally covers Amazon's comments on the DuckLabs acquisition and the DuckDB and DuckLake open-source licensing under the MIT license.

Editorial note: This is a summary of an article first published by The New Stack. The full piece is on their site.

Reading now

Topic
AI & MLOps
Source
The New Stack
Published
14 Nov 2026
Format
Curated summary

Have a take on this?

Write it up and submit it. It goes on the list once an editor approves it.

Submit an article

articles --latest

Latest articles

New from the blogs we follow and the community.

All articles

newsletter --weekly

DevOps Weekly, minus the noise.

The best reads, new jobs and upcoming events, in one email a week.