#### Architecture

LaminDB is a distributed data management system like git that can be
run or hosted anywhere. It just needs a SQLite or Postgres database
and at least one storage location (file system, S3, GCP, Hugging Face,
...). Creating a local LaminDB instance after "pip install lamindb" is
as easy as:

-[ Shell ]-

 lamin init

-[ Py ]-

 import lamindb as ln
 ln.setup.init()

-[ R ]-

 library(laminr)
 lamin_init()

Or you connect to an existing remote database:

-[ Shell ]-

 lamin connect --here account/instance  # --here localizes your connection to the current directory

-[ Py ]-

 import lamindb as ln
 ln.connect("account/instance")

-[ R ]-

 library(laminr)
 ln <- import_module("lamindb")
 ln <- ln$connect("account/instance")

For more configuration, see Install & setup. LaminDB instances work
standalone but can optionally be managed by LaminHub.

On a high level, LaminDB's architecture has the following properties:

This document provides more background and details for several of the
bullets.

### Distributed architecture

Within a single team, collaborators often share a central database as
their source of truth. However, across teams and organizations,
LaminDB is typically used as a distributed system — databases and
storage locations are distributed across different clouds or physical
machines, with objects seamlessly shared and transferred between them.
For example, one team might work in a LaminDB instance on AWS, while
another team operating on GCP imports and builds upon those assets.
Yet another team might run LaminDB entirely on-prem.

Distributing data across LaminDB instances is meaningful even on the
same machine or on the same cloud: it allows teams to completely
decouple their work, enabling them to move faster without aligning on
schema and naming conventions.

Transferring data between LaminDB instances has three noteworthy
properties:

* it's **lineage-aware**: that is, information about the upstream
  instance is stored in the downstream instance

* it defaults to **zero-copy**: rather than copying terabytes of data,
  a downstream instance only receives a metadata record; this makes
  re-using data fast and lightweight

* it's **idempotent**: a repeated transfer does not duplicate data,
  making it safe to sync updates

For details, see Transfer & sync across databases .

One can also create full "clones" of databases by running "lamin io
export" or "lamin io snapshot", which is particularly useful to expose
metadata in a serverless manner via an SQLite instance.

The distributed architecture allows organizations to build a global,
interconnected data ecosystem without requiring centralized
infrastructure.

### Lakehouse architecture

Working with a high number of raw files across different sources
almost inevitably leads to fragile data organization. This brittleness
is amplified when working with agents: they prioritize solving the
immediate task over long-term maintainability, they make frequent
mistakes, and their concurrent read/write patterns can quickly corrupt
a purely file-based architecture. Lakehouse frameworks solve these
problems with ACID transactions to prevent partial writes, with schema
enforcement to prevent inconsistent datasets, and with time travel to
easily restore erroneous written datasets. They also make agents more
efficient.[1] Let's briefly review available options.

Today's most popular framework is **Iceberg**.[2] Like Delta
Lake[3][4] and Apache Hudi,[5] Iceberg provides ACID transactions and
"time travel" by organizing parquet files into snapshots, managed by
manifest and metadata files (**Figure 4**). However, this file-based
metadata introduces costs (snapshot creation is expensive dictating
large, infrequent writes), optimistic concurrency leads to conflicts
between simultaneous writers, and coordinating updates on S3 requires
an external catalog like AWS Glue or Nessie.[6]

| --- | --- | --- | --- | --- |
| Feature | Raw S3 | Iceberg | DuckLake | LaminDB |
| ====================== | ====================== | ====================== | ====================== | ====================== |
| Data lake (file | ✅ | ❌ | ❌ | ✅ |
| management & |
| annotation) |
| --- | --- | --- | --- | --- |
| ACID transactions | ❌ | ✅ | ✅ | ✅ ¹ |
| --- | --- | --- | --- | --- |
| Time travel / | ❌ | ✅ | ✅ | ✅ ² |
| snapshot version |
| isolation |
| --- | --- | --- | --- | --- |
| Schema evolution | ❌ | ✅ ³ | ✅ ³ | ✅ ³ |
| without rewriting |
| data |
| --- | --- | --- | --- | --- |
| Write-Audit-Publish | ❌ | ✅ | ❌ | ✅ ⁴ |
| workflow |
| --- | --- | --- | --- | --- |
| Automatic | ❌ | ❌ | ✅ ⁵ | ✅ ⁵ |
| maintenance |
| --- | --- | --- | --- | --- |
| Dataset formats | ✅ | ❌ | ❌ | ✅ |
| beyond tables |
| --- | --- | --- | --- | --- |
| Data lineage | ❌ | ❌ | ❌ | ✅ |
| --- | --- | --- | --- | --- |
| Registries/ontologi | ❌ | ❌ | ❌ | ✅ |
| es |
| --- | --- | --- | --- | --- |

-[ *A high-level overview of lakehouse technologies.* ]-

¹ LaminDB provides snapshot isolation and time travel by managing
dataset state as transactional metadata records in Postgres/SQLite
rather than mutating existing files. While it does not perform in-
place row-level mutations like a SQL database, operations like
"Collection.append()" atomically create new collection versions
pointing to new, immutable artifacts. This extends core lakehouse ACID
guarantees to multimodal datasets, explicitly rejecting conflicting
concurrent revisions. For more, see Will data & metadata stay in sync?
.

² See the Developer experience section for examples.

³ Adding a nullable/optional column without rewriting existing files.

⁴ In LaminDB, via branches (draft, review, merge).

⁵ Less excessive or no need for cleaning orphaned files like in
Iceberg.

An increasingly popular approach to addressing Iceberg's limitations
is **DuckLake**,[7][8] developed by the DuckDB team. Rather than
storing metadata in files, DuckLake keeps all metadata in a relational
database, leaving only parquet files in storage. This gives it cheap
writes that can be more frequent, transactions with true concurrent
writer support, automatic maintenance via the database's native
mechanisms, and native multi-table transactions — all things that are
difficult or impossible with Iceberg's file-based metadata. A
complementary development in operational workloads is **Lakebase**,
which decouples Postgres database compute and storage via Write-Ahead
Logs in object storage. This brings serverless, transactional Postgres
to live applications and agents, while continuously syncing
operational row changes into analytical lakehouses like Delta Lake.

**LaminDB** shares DuckLake's core architectural pattern — using a
relational database for metadata and object storage for data — but
extends it beyond tables to support any format, in particular,
parquet, zarr, AnnData, HDF5, and others. This enables unified schema
management, data lineage, and registry annotations across complex,
multimodal datasets (Table 1).

While Iceberg & DuckLake are based on the parquet format, and LaminDB
is format-agnostic, **LanceDB** manages datasets in the Lance format,
a columnar format inspired by parquet that's optimized for arrays.[9]
To use LanceDB, you need to convert your data into the Lance format.
While LanceDB fits the lakehouse architecture, non-lakehouse
architectures for managing array-like data exist, too, in particular,
"arraylake" & "tensorstore" for ".zarr" arrays, and "tiledb" for
".tiledb" arrays.[10] These non-lakehouse technologies are out of
scope for this post given the established query engines don't apply to
them.

### Decoupled compute

Rather than locking metadata resolution inside a dedicated query
engine or custom SQL driver, LaminDB acts as an independent semantic
orchestration layer.

When executing analytical queries, LaminDB first resolves metadata in
Postgres or SQLite to yield precise object storage paths. You then
pass these paths directly to modern open-source engines like DuckDB,
Polars, or PySpark. Because these engines read native Parquet and Zarr
files directly over object storage, query execution retains full
optimization benefits:

* **Projection Pushdowns**: Only downloading requested columns.

* **Filter Pushdowns**: Reading file footers to execute row-group
  pruning and skip irrelevant data blocks before fetching them.

This decoupled design ensures that using LaminDB for provenance,
lineage, and ACID governance introduces minimal performance overhead
during data processing and analytics. See Query tables in storage  for
implementation details.

### Branching & idempotency

To safely delegate tasks to autonomous agents and distributed teams,
data infrastructure must support non-destructive experimentation and
repeatable execution.

* **Git-like branching (Write-Audit-Publish):** LaminDB provides
  database-level branching ("stage", "review", "merge") to support
  isolated workflows, similar to tools like git or Nessie.[6] Rather
  than spinning up a new isolated metadata database clone, the
  underlying database and its schema remain shared. Isolation is
  achieved by letting all objects in LaminDB live on a branch via
  "SQLRecord"'s "branch" field. This means that newly created objects
  are isolated on their creation branch. Similarly, changes to
  versioned objects (artifacts, collections, transforms, and
  blocks/notes) are isolated on the creation branch of a new version
  of an object. For example, modifying an artifact or note block
  creates a new version isolated to the current branch. What LaminDB's
  branching does not achieve is isolating in-place updates to existing
  objects, which remain shared across all branches. Achieving that, in
  the context of Postgres or SQLite, would typically be resolved by
  creating new SQL databases and much overhead. For more, see Manage
  changes.

* **Idempotent execution:** Re-running Python scripts, workflow tasks
  (Nextflow, Snakemake), or agent traces is inherently safe. LaminDB
  validates content hashes and metadata prior to writing, preventing
  duplicate artifacts or dangling storage objects within the scope of
  content-hash reuse. For details, see Will data get duplicated upon
  re-running code? .

### Schema evolution & time travel

To see how these concepts translate into developer experience, let's
compare the code required to perform these essential agentic
operations—appending data, evolving schemas, and time-traveling.

The first type of write operation we need to perform is adding new
data to the system. Rather than just dropping a raw file into a
bucket, the following code snippets ensure that a new dataset complies
with the schema of the existing dataset, and that it's added in an
ACID fashion.

-[ LaminDB ]-

Atomic and snapshot-isolated. A new parquet file creates a new
collection version.

 collection.append(batch)  # batch is an artifact

-[ Iceberg ]-

Atomic and snapshot-isolated. New Parquet files and a snapshot
manifest are written to S3; concurrent readers see a consistent state
throughout.

 table.append(batch)  # batch is a pyarrow dataset

-[ LanceDB ]-

"add()" writes new rows to S3 and automatically increments the table
version.

 table.add(batch)  # batch is a pyarrow dataset

Similarly, when an analysis requires new features, the following
snippets ensure that columns are updated consistently across the
entire dataset, and future incoming datasets.

-[ LaminDB ]-

LaminDB registers the feature in its schema registry, validating all
future artifacts instance-wide.

 feature = ln.Feature(name="QC_PASS", dtype=bool).save()
 collection.schema.add(feature)

-[ Iceberg ]-

A new metadata file records the updated schema. Existing Parquet files
are not modified; reads of old files return "null" for the new column.

 from pyiceberg.types import BooleanType
 with table.update_schema() as update:
 update.add_column("QC_PASS", BooleanType())

-[ LanceDB ]-

"add_columns" takes a per-column SQL value expression — hence the
"CAST(NULL AS BOOLEAN)" string, which supplies both the value and its
type for existing rows.

 table.add_columns({"QC_PASS": "CAST(NULL AS BOOLEAN)"})

Finally, because agents inevitably make mistakes, we look at how to
retrieve a previous version of a dataset via "time travel".

-[ LaminDB ]-

 collection.versions.get(version="1")  # get a previous version

-[ Iceberg ]-

 first_snapshot = table.history()[0].snapshot_id  # access version 0
 table.scan(snapshot_id=first_snapshot)

-[ LanceDB ]-

 table.checkout(1) # checkout a previous version

### Permissions

See Manage access permissions.

### Metadata schema & API

LaminDB provides a SQL schema for common metadata entities:
"Artifact", "Collection", "Transform", "Feature", "Record" etc. - see
the API reference or the source code.

The core metadata schema is extendable through modules, e.g., with
basic biological ("Gene", "Protein", "CellLine", etc.) & operational
entities ("Biosample", "Techsample", "Treatment", etc.).

Data models are defined in Python using the Django ORM. Django
translates them to SQL tables. Django is one of the most-used &
highly-starred projects on GitHub and has been robustly maintained for
15 years. While the SQLAlchemy ORM has some advantages, Django has
been the most popular choice for building metadata management systems
in the life sciences for over a decade.

On top of the metadata schema, LaminDB is a Python API that models
datasets as artifacts, abstracts storage & database access, data
transformations, and ontologies.

### Modules

LaminDB can be extended with modules building on the Django ecosystem.
Examples are:

* bionty: Basic biological ontologies, with easy import from >20
  public ontologies

* pertdb: Registries for perturbations (compounds, biologics, genetic
  interventions, etc.)

If you'd like to create your own module:

1. Create a git repository with registries similar to pertdb

2. Create & deploy migrations via "lamin migrate create" and "lamin
 migrate deploy"

For more information, see Install & setup.

### Repositories

LaminDB and its plugins consist in open-source Python libraries &
publicly hosted metadata assets:

* lamindb: Core library.

Tightly integrated dependencies are available as git submodules here,
for instance,

* lamindb-setup: Setup & configure LaminDB.

* lamin-cli: The CLI.

* lamin-skills: Agent skills.

Use cases / domain-specific repos:

* nf-lamin: Nextflow integration with LaminDB.

* laminr: R client for LaminDB.

* lamin-usecases: Mixed use cases.

* lamin-agents: Agentic use cases.

* redun-lamin: Track redun workflow runs with LaminDB.

* lamin-mlops: MLOps use cases (MNIST, W&B, MLflow, Croissant).

* cellxgene-lamin: CELLxGENE data and curation.

* lamin-spatial: Spatial data (RxRx, Vitessce).

* snakemake-lamin: Track Snakemake runs with LaminDB.

For a comprehensive list of open-sourced software, browse our GitHub
account, for instance,

* readfcs: FCS artifact reader.

There is a public repository for LaminHub:

* laminhub-public: Make issues and follow releases of LaminHub, no
  source code.

---

[1] Pillai R et al. (2026). Agentic variant analysis of the 1000
 Genomes Project using Polars, DuckDB, and lakehouses. Lamin Blog.

[2] Apache Software Foundation. Apache Iceberg: The open table format
 for analytic datasets. Apache Iceberg.

[3] Linux Foundation. Delta Lake: An open-source storage framework
 that enables building a Lakehouse architecture. Delta Lake.

[4] Databricks (2020). Accurately Building Genomic Cohorts at Scale
 with Delta Lake and Spark. Databricks Blog.

[5] Apache Software Foundation. Apache Hudi: Streaming data on data
 lakes. Apache Hudi.

[6] Project Nessie. Nessie: Transactional Catalog for Data Lakes.
 Project Nessie.

[7] Raasveldt M & Mühleisen H (2025). DuckLake: SQL as a Lakehouse
 Format. DuckLake Blog.

[8] Raasveldt M & Holanda P (2026). DuckLake v1.0: The Lakehouse
 Format Built on SQL Reaches Production-Readiness. DuckLake Blog.

[9] LanceDB (2024). Lance Format v2.2 Benchmarks: Half the storage,
 none of the slowdown. LanceDB Blog.

[10] TileDB (2020). Population Genomics Data with TileDB. TileDB Blog.