Architecture .md

LaminDB is a distributed data management system like git that can be run or hosted anywhere. It just needs a SQLite or Postgres database and at least one storage location (file system, S3, GCP, Hugging Face, …). Creating a local LaminDB instance after pip install lamindb is as easy as:

lamin init

import lamindb as ln
ln.setup.init()

library(laminr)
lamin_init()

Or you connect to an existing remote database:

lamin connect --here account/instance  # --here localizes your connection to the current directory

import lamindb as ln
ln.connect("account/instance")

library(laminr)
ln <- import_module("lamindb")
ln <- ln$connect("account/instance")

For more configuration, see Install & setup. LaminDB instances work standalone but can optionally be managed by LaminHub.

On a high level, LaminDB’s architecture has the following properties:

This document provides more background and details for several of the bullets.

Distributed architecture

Within a single team, collaborators often share a central database as their source of truth. However, across teams and organizations, LaminDB is typically used as a distributed system — databases and storage locations are distributed across different clouds or physical machines, with objects seamlessly shared and transferred between them. For example, one team might work in a LaminDB instance on AWS, while another team operating on GCP imports and builds upon those assets. Yet another team might run LaminDB entirely on-prem.

Distributing data across LaminDB instances is meaningful even on the same machine or on the same cloud: it allows teams to completely decouple their work, enabling them to move faster without aligning on schema and naming conventions.

Transferring data between LaminDB instances has three noteworthy properties:

  • it’s lineage-aware: that is, information about the upstream instance is stored in the downstream instance

  • it defaults to zero-copy: rather than copying terabytes of data, a downstream instance only receives a metadata record; this makes re-using data fast and lightweight

  • it’s idempotent: a repeated transfer does not duplicate data, making it safe to sync updates

For details, see Transfer & sync across databases .

One can also create full “clones” of databases by running lamin io export or lamin io snapshot, which is particularly useful to expose metadata in a serverless manner via an SQLite instance.

The distributed architecture allows organizations to build a global, interconnected data ecosystem without requiring centralized infrastructure.

Lakehouse architecture

Working with a high number of raw files across different sources almost inevitably leads to fragile data organization. This brittleness is amplified when working with agents: they prioritize solving the immediate task over long-term maintainability, they make frequent mistakes, and their concurrent read/write patterns can quickly corrupt a purely file-based architecture. Lakehouse frameworks solve these problems with ACID transactions to prevent partial writes, with schema enforcement to prevent inconsistent datasets, and with time travel to easily restore erroneous written datasets. They also make agents more efficient.[1] Let’s briefly review available options.

*File layout of an Iceberg table.*

Today’s most popular framework is Iceberg.[2] Like Delta Lake[3][4] and Apache Hudi,[5] Iceberg provides ACID transactions and “time travel” by organizing parquet files into snapshots, managed by manifest and metadata files (Figure 4). However, this file-based metadata introduces costs (snapshot creation is expensive dictating large, infrequent writes), optimistic concurrency leads to conflicts between simultaneous writers, and coordinating updates on S3 requires an external catalog like AWS Glue or Nessie.[6]

Feature

Raw S3

Iceberg

DuckLake

LaminDB

Data lake (file management & annotation)

✅

❌

❌

✅

ACID transactions

❌

✅

✅

✅ ¹

Time travel / snapshot version isolation

❌

✅

✅

✅ ²

Schema evolution without rewriting data

❌

✅ ³

✅ ³

✅ ³

Write-Audit-Publish workflow

❌

✅

❌

✅ ⁴

Automatic maintenance

❌

❌

✅ ⁵

✅ ⁵

Dataset formats beyond tables

✅

❌

❌

✅

Data lineage

❌

❌

❌

✅

Registries/ontologies

❌

❌

❌

✅
A high-level overview of lakehouse technologies.

¹ LaminDB provides snapshot isolation and time travel by managing dataset state as transactional metadata records in Postgres/SQLite rather than mutating existing files. While it does not perform in-place row-level mutations like a SQL database, operations like Collection.append() atomically create new collection versions pointing to new, immutable artifacts. This extends core lakehouse ACID guarantees to multimodal datasets, explicitly rejecting conflicting concurrent revisions. For more, see Will data & metadata stay in sync? .

² See the Developer experience section for examples.

³ Adding a nullable/optional column without rewriting existing files.

⁴ In LaminDB, via branches (draft, review, merge).

⁵ Less excessive or no need for cleaning orphaned files like in Iceberg.

An increasingly popular approach to addressing Iceberg’s limitations is DuckLake,[7][8] developed by the DuckDB team. Rather than storing metadata in files, DuckLake keeps all metadata in a relational database, leaving only parquet files in storage. This gives it cheap writes that can be more frequent, transactions with true concurrent writer support, automatic maintenance via the database’s native mechanisms, and native multi-table transactions — all things that are difficult or impossible with Iceberg’s file-based metadata. A complementary development in operational workloads is Lakebase, which decouples Postgres database compute and storage via Write-Ahead Logs in object storage. This brings serverless, transactional Postgres to live applications and agents, while continuously syncing operational row changes into analytical lakehouses like Delta Lake.

LaminDB shares DuckLake’s core architectural pattern — using a relational database for metadata and object storage for data — but extends it beyond tables to support any format, in particular, parquet, zarr, AnnData, HDF5, and others. This enables unified schema management, data lineage, and registry annotations across complex, multimodal datasets (Table 1).

While Iceberg & DuckLake are based on the parquet format, and LaminDB is format-agnostic, LanceDB manages datasets in the Lance format, a columnar format inspired by parquet that’s optimized for arrays.[9] To use LanceDB, you need to convert your data into the Lance format. While LanceDB fits the lakehouse architecture, non-lakehouse architectures for managing array-like data exist, too, in particular, arraylake & tensorstore for .zarr arrays, and tiledb for .tiledb arrays.[10] These non-lakehouse technologies are out of scope for this post given the established query engines don’t apply to them.

Decoupled compute

Rather than locking metadata resolution inside a dedicated query engine or custom SQL driver, LaminDB acts as an independent semantic orchestration layer.

When executing analytical queries, LaminDB first resolves metadata in Postgres or SQLite to yield precise object storage paths. You then pass these paths directly to modern open-source engines like DuckDB, Polars, or PySpark. Because these engines read native Parquet and Zarr files directly over object storage, query execution retains full optimization benefits:

  • Projection Pushdowns: Only downloading requested columns.

  • Filter Pushdowns: Reading file footers to execute row-group pruning and skip irrelevant data blocks before fetching them.

This decoupled design ensures that using LaminDB for provenance, lineage, and ACID governance introduces minimal performance overhead during data processing and analytics. See Query tables in storage for implementation details.

Branching & idempotency

To safely delegate tasks to autonomous agents and distributed teams, data infrastructure must support non-destructive experimentation and repeatable execution.

  • Git-like branching (Write-Audit-Publish): LaminDB provides database-level branching (stage, review, merge) to support isolated workflows, similar to tools like git or Nessie.[6] Rather than spinning up a new isolated metadata database clone, the underlying database and its schema remain shared. Isolation is achieved by letting all objects in LaminDB live on a branch via SQLRecord’s branch field. This means that newly created objects are isolated on their creation branch. Similarly, changes to versioned objects (artifacts, collections, transforms, and blocks/notes) are isolated on the creation branch of a new version of an object. For example, modifying an artifact or note block creates a new version isolated to the current branch. What LaminDB’s branching does not achieve is isolating in-place updates to existing objects, which remain shared across all branches. Achieving that, in the context of Postgres or SQLite, would typically be resolved by creating new SQL databases and much overhead. For more, see Manage changes.

  • Idempotent execution: Re-running Python scripts, workflow tasks (Nextflow, Snakemake), or agent traces is inherently safe. LaminDB validates content hashes and metadata prior to writing, preventing duplicate artifacts or dangling storage objects within the scope of content-hash reuse. For details, see Will data get duplicated upon re-running code? .

Schema evolution & time travel

To see how these concepts translate into developer experience, let’s compare the code required to perform these essential agentic operations—appending data, evolving schemas, and time-traveling.

The first type of write operation we need to perform is adding new data to the system. Rather than just dropping a raw file into a bucket, the following code snippets ensure that a new dataset complies with the schema of the existing dataset, and that it’s added in an ACID fashion.

Atomic and snapshot-isolated. A new parquet file creates a new collection version.

collection.append(batch)  # batch is an artifact

Atomic and snapshot-isolated. New Parquet files and a snapshot manifest are written to S3; concurrent readers see a consistent state throughout.

table.append(batch)  # batch is a pyarrow dataset

add() writes new rows to S3 and automatically increments the table version.

table.add(batch)  # batch is a pyarrow dataset

Similarly, when an analysis requires new features, the following snippets ensure that columns are updated consistently across the entire dataset, and future incoming datasets.

LaminDB registers the feature in its schema registry, validating all future artifacts instance-wide.

feature = ln.Feature(name="QC_PASS", dtype=bool).save()
collection.schema.add(feature)

A new metadata file records the updated schema. Existing Parquet files are not modified; reads of old files return null for the new column.

from pyiceberg.types import BooleanType
with table.update_schema() as update:
    update.add_column("QC_PASS", BooleanType())

add_columns takes a per-column SQL value expression — hence the CAST(NULL AS BOOLEAN) string, which supplies both the value and its type for existing rows.

table.add_columns({"QC_PASS": "CAST(NULL AS BOOLEAN)"})

Finally, because agents inevitably make mistakes, we look at how to retrieve a previous version of a dataset via “time travel”.

collection.versions.get(version="1")  # get a previous version

first_snapshot = table.history()[0].snapshot_id  # access version 0
table.scan(snapshot_id=first_snapshot)

table.checkout(1)             # checkout a previous version

Permissions

See Manage access permissions.

Metadata schema & API

LaminDB provides a SQL schema for common metadata entities: Artifact, Collection, Transform, Feature, Record etc. - see the API reference or the source code.

The core metadata schema is extendable through modules, e.g., with basic biological (Gene, Protein, CellLine, etc.) & operational entities (Biosample, Techsample, Treatment, etc.).

Data models are defined in Python using the Django ORM. Django translates them to SQL tables. Django is one of the most-used & highly-starred projects on GitHub and has been robustly maintained for 15 years. While the SQLAlchemy ORM has some advantages, Django has been the most popular choice for building metadata management systems in the life sciences for over a decade.

On top of the metadata schema, LaminDB is a Python API that models datasets as artifacts, abstracts storage & database access, data transformations, and ontologies.

Modules

LaminDB can be extended with modules building on the Django ecosystem. Examples are:

  • bionty: Basic biological ontologies, with easy import from >20 public ontologies

  • pertdb: Registries for perturbations (compounds, biologics, genetic interventions, etc.)

If you’d like to create your own module:

  1. Create a git repository with registries similar to pertdb

  2. Create & deploy migrations via lamin migrate create and lamin migrate deploy

For more information, see Install & setup.

Repositories

LaminDB and its plugins consist in open-source Python libraries & publicly hosted metadata assets:

Tightly integrated dependencies are available as git submodules here, for instance,

Use cases / domain-specific repos:

For a comprehensive list of open-sourced software, browse our GitHub account, for instance,

There is a public repository for LaminHub:

  • laminhub-public: Make issues and follow releases of LaminHub, no source code.