Introduction¶

LaminDB is an open-source data framework for biology.

LaminHub is a data collaboration hub built on LaminDB similar to how GitHub is built on git.

Quickstart¶

You’ll ingest a small dataset while tracking data lineage, and see how to validate, annotate, query & search.

Install the lamindb Python package.

# install with notebook support & biological entities
!pip install 'lamindb[jupyter,bionty]'

Initialize a LaminDB instance that stores data locally and mounts plugin bionty.

# store artifacts in local directory `./lamin-intro`
!lamin init --storage ./lamin-intro --schema bionty
# (optional) make Django's unnecessary functionality private for clean auto-complete
!lamin settings set private-django-api true

Data transformations¶

Call track() to register a data transformation and start tracking inputs & outputs of a run. You will find your notebook in the Transform registry along with scripts, pipelines & functions. Run stores executions.

import lamindb as ln

# --> `ln.track()` generates a uid for your code
# --> `ln.track(uid)` initiates a tracked run
ln.track("FPnfDtJz8qbE0000")  

Artifacts & versioning¶

An Artifact stores a dataset or model as a file, folder or array.

import pandas as pd

# a sample dataset
df = pd.DataFrame(
    {"CD8A": [1, 2, 3], "CD4": [3, 4, 5], "CD14": [5, 6, 7], "perturbation": ["DMSO", "IFNJ", "DMSO"],},
    index=["sample1", "sample2", "sample3"],
)

# create & save an artifact from a DataFrame
artifact = ln.Artifact.from_df(df, description="my RNA-seq").save()

# artifacts come with typed, relational metadata
artifact.describe()

Load the dataset into memory.

artifact.load()

	CD8A	CD4	CD14	perturbation
sample1	1	3	5	DMSO
sample2	2	4	6	IFNJ
sample3	3	5	7	DMSO

Looking at this: "IFNJ" should have been "IFNG". 🙈 Let’s create a new version by revising this dataset.

# keep the dataframe with a typo around - we'll need it later
df_typo = df.copy()

# fix the typo "IFNJ" typo
df.loc["sample2", "perturbation"] = "IFNG"

# create a new version by revising the artifact
artifact = ln.Artifact.from_df(df, revises=artifact).save()

# see all versions of an artifact
artifact.versions.df()

Show code cell output Hide code cell output

	uid	version	is_latest	description	key	suffix	type	size	hash	n_objects	n_observations	_hash_type	_accessor	visibility	_key_is_virtual	storage_id	transform_id	run_id	created_at	created_by_id
id
1	g4Zwebh0RdRSjegY0000	None	False	my RNA-seq	None	.parquet	dataset	4091	MUAn2flgfgEAhH8UayfKkQ	None	None	md5	DataFrame	1	True	1	1	1	2024-10-19 08:03:37.159513+00:00	1
2	g4Zwebh0RdRSjegY0001	None	True	my RNA-seq	None	.parquet	dataset	4091	GDTjY7XSC6E2k9d_ZpLCnw	None	None	md5	DataFrame	1	True	1	1	1	2024-10-19 08:03:37.206096+00:00	1

Similar to tagging a git commit, you can label a revision.

artifact.version = "1.0"
artifact.save()
artifact.versions.df()

Show code cell output Hide code cell output

	uid	version	is_latest	description	key	suffix	type	size	hash	n_objects	n_observations	_hash_type	_accessor	visibility	_key_is_virtual	storage_id	transform_id	run_id	created_at	created_by_id
id
1	g4Zwebh0RdRSjegY0000	None	False	my RNA-seq	None	.parquet	dataset	4091	MUAn2flgfgEAhH8UayfKkQ	None	None	md5	DataFrame	1	True	1	1	1	2024-10-19 08:03:37.159513+00:00	1
2	g4Zwebh0RdRSjegY0001	1.0	True	my RNA-seq	None	.parquet	dataset	4091	GDTjY7XSC6E2k9d_ZpLCnw	None	None	md5	DataFrame	1	True	1	1	1	2024-10-19 08:03:37.206096+00:00	1

View data lineage.

artifact.view_lineage()

Datasets & labels¶

Label an artifact with a ULabel and a bionty.CellType. The same works for any entity in any custom schema module.

import bionty as bt

# create & save a ulabel record
candidate_marker_study = ln.ULabel(name="Candidate marker study").save()

# label the artifact
artifact.ulabels.add(candidate_marker_study)

# repeat for a bionty entity
cell_type = bt.CellType.from_source(name="effector T cell").save()
artifact.cell_types.add(cell_type)

# describe the artifact
artifact.describe()

Registries, records & fields¶

LaminDB’s central classes are related records that inherit from Record. We’ve already seen how to create new artifact, transform and ulabel records.

The easiest way to see all existing records of a given type is to call the class method df.

ln.ULabel.df()

Show code cell output Hide code cell output

	uid	name	description	reference	reference_type	run_id	created_at	created_by_id
id
1	ljZdn94n	Candidate marker study	None	None	None	1	2024-10-19 08:03:37.424652+00:00	1

Existing records are stored in the record’s registry (metaclass Registry), which maps 1:1 to on a SQL table in the SQLite or Postgres backend.

A record and its registry share the same fields, which define the metadata you can query for. If you want to see them, look at the class or auto-complete.

ln.Artifact

Query & search¶

You can write arbitrary relational queries using the class methods get and filter. The syntax for it is Django’s query syntax, one of the two most popular ORMs in Python (the other is SQLAlchemy).

# get a single record by uid (here, the latest version of the current notebook)
transform = ln.Transform.get("FPnfDtJz8qbE")

# get a single record by matching a field
transform = ln.Transform.get(name="Introduction")

# get a set of records by filtering on description
ln.Artifact.filter(description="my RNA-seq").df()

# query all artifacts ingested from the current notebook
artifacts = ln.Artifact.filter(transform=transform).all()

# query all artifacts ingested from a notebook with "intro" in the name and labeled "Candidate marker study"
artifacts = ln.Artifact.filter(
    transform__name__icontains="intro", ulabels=candidate_marker_study
).all()

The class methods search and lookup help finding sets of approximately matching records.

# search in a registry
ln.Transform.search("intro").df()

# look up records with auto-complete
ulabels = ln.ULabel.lookup()
cell_types = bt.CellType.lookup()

Datasets & features¶

What fields are to metadata records, features are to datasets. You can annotate datasets by the features they measure.

But because LaminDB validates all user input against its registries, annotating with a "temperature" feature doesn’t work right away.

import pytest

with pytest.raises(ln.core.exceptions.ValidationError) as e:
    artifact.features.add_values({"temperature": 21.6})

print(e.exconly())

Following the hint in the error message, create & save a Feature.

# create & save the "temperature" feature (only required once)
ln.Feature(name="temperature", dtype="float").save()

# now we can annotate with the feature & the value
artifact.features.add_values({"temperature": 21.6})

# describe the artifact
artifact.describe()

We can also annotate with categorical features:

# register a categorical feature
ln.Feature(name="study", dtype="cat").save()

# add a categorical value
artifact.features.add_values({"study": "Candidate marker study"})

# describe the artifact with type information
artifact.describe(print_types=True)

This is how you query artifacts by features.

ln.Artifact.features.filter(study__contains="marker study").df()

Show code cell output Hide code cell output

	uid	version	is_latest	description	key	suffix	type	size	hash	n_objects	n_observations	_hash_type	_accessor	visibility	_key_is_virtual	storage_id	transform_id	run_id	created_at	created_by_id
id
2	g4Zwebh0RdRSjegY0001	1.0	True	my RNA-seq	None	.parquet	dataset	4091	GDTjY7XSC6E2k9d_ZpLCnw	None	None	md5	DataFrame	1	True	1	1	1	2024-10-19 08:03:37.206096+00:00	1

Features organize labels by how they’re measured in datasets, independently of how labels are stored in metadata registries.

Key examples¶

Understand data lineage¶

Understand where a dataset comes from and what it’s used for (background).

artifact.view_lineage()

You don’t need a workflow manager to track data lineage (if you want to use one, see Pipelines – workflow managers). All you need is:

import lamindb as ln

ln.track()  # track your run

# your code

ln.finish()  # mark run as finished, save execution report, source code & environment

Below is how a single transform (a notebook) with its run report looks on the hub.

To create a new version of a notebook or script, run lamin load on the terminal, e.g.,

$ lamin load https://lamin.ai/laminlabs/lamindata/transform/13VINnFk89PE0004
→ connected lamindb: laminlabs/lamindata
→ updated uid: 13VINnFk89PE0004 → 13VINnFk89PE0005
→ notebook is here: mcfarland_2020_preparation.ipynb

Curate datasets¶

In the quickstart, you just saw how to ingest & annotate datasets without validation. This is often enough if you’re prototyping or working with one-off studies. But if you want to create a big body of standardized data, you should invest a more time and curate your datasets.

Let’s use the high-level Curator class to curate a DataFrame.

# construct a Curator object to validate & annotate a DataFrame
curator = ln.Curator.from_df(
    df,
    # define validation criteria as mappings
    columns=ln.Feature.name,  # map column names
    categoricals={"perturbation": ln.ULabel.name},  # map categories
)

# validate the dataset
curator.validate()

The validation did not pass because LaminDB’s registries don’t yet know about the features "CD8A", "CD4", "CD14", "perturbation" and labels "DMSO", "IFNG", "DMSO" in this dataset. Hence, we need to initially populate them.

# add non-validated features based on the DataFrame columns
curator.add_new_from_columns()

# add non-validated labels based on the perturbation column of the dataframe
curator.add_new_from("perturbation")

# see the updated content of the ULabel registry
ln.ULabel.df()

Show code cell output Hide code cell output

✓ added 3 records with Feature.name for columns: 'CD14', 'CD8A', 'CD4'

✓ added 2 records with ULabel.name for perturbation: 'DMSO', 'IFNG'

	uid	name	description	reference	reference_type	run_id	created_at	created_by_id
id
4	T90NjUnC	IFNG	None	None	None	1	2024-10-19 08:03:40.246905+00:00	1
3	uGIXp43y	DMSO	None	None	None	1	2024-10-19 08:03:40.246854+00:00	1
2	uDbAVfo7	is_perturbation	None	None	None	1	2024-10-19 08:03:40.172354+00:00	1
1	ljZdn94n	Candidate marker study	None	None	None	1	2024-10-19 08:03:37.424652+00:00	1

With the ULabel and Feature registries now containing meaningful reference values, validation passes & and we can automatically parse features & labels to save an annotated & curated artifact.

# given the updated registries, the validation passes
curator.validate()

# save curated artifact
artifact = curator.save_artifact(description="my RNA-seq")

# see the parsed annotations
artifact.describe()

# query for a ulabel that was parsed from the dataset
ln.Artifact.get(ulabels__name="IFNG")

Had we used ln.Cuartor from the beginning, we would have caught the typo.

# construct a Curator object to validate & annotate a DataFrame
curator = ln.Curator.from_df(
    df_typo,
    columns=ln.Feature.name,
    categoricals={"perturbation": ln.ULabel.name},
)

# validate the dataset
curator.validate()

Manage biological registries¶

The generic Feature and ULabel registries will get you pretty far.

But let’s now look at what you do can with a dedicated biological registry like Gene.

Every bionty registry is based on configurable public ontologies (>20 of them).

cell_types = bt.CellType.public()
cell_types

cell_types.search("gamma delta T cell").head(2)

Show code cell output Hide code cell output

	ontology_id	definition	synonyms	parents	__ratio__
name
gamma-delta T cell	CL:0000798	A T Cell That Expresses A Gamma-Delta T Cell R...	gammadelta T cell\|gamma-delta T-cell\|gamma-del...	[CL:0000084]	100.000000
CD27-negative gamma-delta T cell	CL:0002125	A Circulating Gamma-Delta T Cell That Expresse...	gammadelta-17 cells	[CL:0000800]	86.486486

Validate & annotate with typed features.

import anndata as ad

# store the dataset as an AnnData object to distinguish data from metadata
adata = ad.AnnData(
    df[["CD8A", "CD4", "CD14"]], obs=df[["perturbation"]]
)

# create an annotation flow for an AnnData object
curate = ln.Curator.from_anndata(
    adata,
    # define validation criteria
    var_index=bt.Gene.symbol,  # map .var.index onto Gene registry
    categoricals={adata.obs.perturbation.name: ln.ULabel.name},
    organism="human",  # specify the organism for the Gene registry
)
curate.validate()

# save curated artifact
artifact = curate.save_artifact(description="my RNA-seq")
artifact.describe()

Query for typed features.

# get a lookup object for human genes
genes = bt.Gene.filter(organism__name="human").lookup()
# query for all feature sets that contain CD8A
feature_sets = ln.FeatureSet.filter(genes=genes.cd8a).all()
# write the query
ln.Artifact.filter(feature_sets__in=feature_sets).df()

Show code cell output Hide code cell output

	uid	version	is_latest	description	key	suffix	type	size	hash	n_objects	n_observations	_hash_type	_accessor	visibility	_key_is_virtual	storage_id	transform_id	run_id	created_at	created_by_id
id
3	ukIRl4tRpNMKn4sy0000	None	True	my RNA-seq	None	.h5ad	dataset	19240	nLH34gqty3-5c2eGF6deOA	None	3	md5	AnnData	1	True	1	1	1	2024-10-19 08:03:43.260005+00:00	1

Update ontologies, e.g., create a cell type record and add a new cell state.

# create an ontology-coupled cell type record and save it
neuron = bt.CellType.from_source(name="neuron").save()

# create a record to track a new cell state
new_cell_state = bt.CellType(name="my neuron cell state", description="explains X").save()

# express that it's a neuron state
new_cell_state.parents.add(neuron)

# view ontological hierarchy
new_cell_state.view_parents(distance=2)

Show code cell output Hide code cell output

✓ created 1 CellType record from Bionty matching name: 'neuron'

✓ created 3 CellType records from Bionty matching ontology_id: 'CL:0000393', 'CL:0000404', 'CL:0002319'

! records with similar names exist! did you mean to load one of them?

	uid	name	ontology_id	abbr	synonyms	description	source_id	run_id	created_at	created_by_id
id
1	3nfZTVV4	effector T cell	CL:0000911	None	effector T lymphocyte\|effector T-cell\|effector...	A Differentiated T Cell With Ability To Traffi...	32	1	2024-10-19 08:03:38.628858+00:00	1
2	2OTzqBTM	mature T cell	CL:0002419	None	CD3e-positive T cell\|mature T-cell	A T Cell That Expresses A T Cell Receptor Comp...	32	1	2024-10-19 08:03:39.461393+00:00	1
3	4bKGljt0	cell	CL:0000000	None	None	A Material Entity Of Anatomical Origin (Part O...	32	1	2024-10-19 08:03:39.861906+00:00	1

_images/398f04f27aa3b02ffd807b663c1069c5695782e4e6da61cd2357d0047da5debb.svg

Scale up data & learning¶

How do you integrate new datasets with your existing datasets? Leverage Collection.

# a new dataset
df = pd.DataFrame(
    {"CD8A": [2, 3, 3], "CD4": [3, 4, 5], "CD38": [4, 2, 3], "perturbation": ["DMSO", "IFNG", "IFNG"],},
    index=["sample4", "sample5", "sample6"],
)
adata = ad.AnnData(df[["CD8A", "CD4", "CD38"]], obs=df[["perturbation"]])

# validate, curate and save a new artifact
curate = ln.Curator.from_anndata(
    adata,
    var_index=bt.Gene.symbol,
    categoricals={adata.obs.perturbation.name: ln.ULabel.name},
    organism="human",
)
curate.validate()
artifact2 = curate.save_artifact(description="my RNA-seq dataset 2")

Create a collection using Collection.

collection = ln.Collection([artifact, artifact2], name="my RNA-seq collection").save()
collection.describe()
collection.view_lineage()

# if it's small enough, you can load the entire collection into memory as if it was one
collection.load()

# typically, it's too big, hence, iterate over its artifacts
collection.artifacts.all()

# or look at a DataFrame listing the artifacts
collection.artifacts.df()

Show code cell output Hide code cell output

	uid	version	is_latest	description	key	suffix	type	size	hash	n_objects	n_observations	_hash_type	_accessor	visibility	_key_is_virtual	storage_id	transform_id	run_id	created_at	created_by_id
id
3	ukIRl4tRpNMKn4sy0000	None	True	my RNA-seq	None	.h5ad	dataset	19240	nLH34gqty3-5c2eGF6deOA	None	3	md5	AnnData	1	True	1	1	1	2024-10-19 08:03:43.260005+00:00	1
4	l96tMNWkZiTZxvu60000	None	True	my RNA-seq dataset 2	None	.h5ad	dataset	19240	K95PcyOoxIxtlytXMr6AVg	None	3	md5	AnnData	1	True	1	1	1	2024-10-19 08:03:46.924515+00:00	1

Directly train models on collections of AnnData.

# to train models, batch iterate through the collection as if it was one array
from torch.utils.data import DataLoader, WeightedRandomSampler
dataset = collection.mapped(obs_keys=["perturbation"])
sampler = WeightedRandomSampler(
    weights=dataset.get_label_weights("perturbation"), num_samples=len(dataset)
)
data_loader = DataLoader(dataset, batch_size=2, sampler=sampler)
for batch in data_loader:
    pass

Read this blog post for more on training models on sharded datasets.

Design¶

Why?¶

Objects like pd.DataFrame are at the heart of many data science workflows but there hasn’t been a tool to manage these objects in the rich context that collaborative biological research requires:

data lineage: data sources, data transformations, models, users
domain knowledge & experimental metadata: the features & labels derived from domain entities

In this blog post, we discuss how the complexity of modern R&D data often blocks realizing the scientific progress it promises.

Assumptions¶

Teams need to have enough freedom to initiate work independently but enough structure to easily integrate datasets later on
Batched datasets from physical instruments are transformed (Transform) into useful representations (Artifact)
Learning needs features (Feature, CellMarker, …) and labels (ULabel, CellLine, …)
Insights connect representations to experimental metadata and knowledge (ontologies)

Distributed¶

LaminDB is a distributed system like git. Collaborators can connect to your instance via:

ln.connect("account-handle/instance-name")

Or via the command line for auto-connecting in a Python session:

lamin connect account-handle/instance-name

Or you create your new instance:

lamin init --storage ./my-data-folder

Metada schema & API¶

LaminDB provides a SQL schema for common metadata entities: Artifact, Collection, Transform, Feature, ULabel etc. - see the API reference or the source code.

The core metadata schema is extendable through plugins (see green vs. red entities in graphic), e.g., with basic biological (Gene, Protein, CellLine, etc.) & operational entities (Biosample, Techsample, Treatment, etc.).

On top of the metadata schema, LaminDB is a Python API that models datasets as artifacts, abstracts over storage & database access, data transformations, and (biological) ontologies.

Note that the datasets schema (e.g., .parquet files or .h5ad arrays) is modeled through the Feature registry and does not require migrations to be updated.

Custom schemas and plugins¶

LaminDB can be customized & extended with schema & app plugins building on the Django ecosystem. Examples are:

bionty: Registries for basic biological entities, coupled to public ontologies.
wetlab: Exemplary custom schema to manage samples, treatments, etc.

If you’d like to create your own schema or app:

Create a git repository with registries similar to wetlab
Create & deploy migrations via lamin migrate create and lamin migrate deploy

Repositories¶

LaminDB and its plugins consist in open-source Python libraries & publicly hosted metadata assets:

lamindb: Core package.
bionty: Registries for basic biological entities, coupled to public ontologies.
wetlab: Default wetlab schema.
guides: Guides.
usecases: Use cases.

All immediate dependencies are available as git submodules here, for instance,

lnschema-core: Core schema.
lamindb-setup: Setup & configure LaminDB.
lamin-cli: CLI for lamindb and lamindb-setup.

For a comprehensive list of open-sourced software, browse our GitHub account.

lamin-utils: Generic utilities, e.g., a logger.
readfcs: FCS artifact reader.
nbproject: Light-weight Jupyter notebook tracker.
bionty-assets: Assets for public biological ontologies.

LaminHub is not open-sourced.

Influences¶

LaminDB was influenced by many other projects, see Influences.