Stars pypi

Introduction

LaminDB is an open-source data framework for biology.

LaminDB features

Manage data & metadata with a unified Python API (“lakehouse”).

Track data lineage across notebooks, scripts, pipelines & UI.

  • Track run context with a simple method call: track()

  • A unified registry for all your notebooks, scripts & pipelines: Transform

  • A unified registry for all data transformation runs: Run

  • Manage execution reports, source code and Python environments for notebooks & scripts

  • Integrate with workflow managers: redun, nextflow, snakemake

Manage registries for experimental metadata & in-house ontologies, import public ontologies.

Validate, standardize & annotate.

Organize and share data across a mesh of LaminDB instances.

  • Create & load instances like git repos: lamin init & lamin load

  • Zero-copy transfer data across instances

Integrate with analytics tools.

Zero lock-in, scalable, auditable.

  • Zero lock-in: LaminDB runs on generic backends server-side and is not a client for “Lamin Cloud”

    • Flexible storage backends (local, S3, GCP, anything fsspec supports)

    • Two SQL backends for managing metadata: SQLite & Postgres

  • Scalable: metadata registries support 100s of millions of entries, storage is as scalable as S3

  • Auditable: data & metadata records are hashed, timestamped, and attributed to users (full audit log to come)

  • Secure: embedded in your infrastructure (Lamin has no access to your data & metadata)

  • Tested, typed, idempotent & ACID

LaminHub is a data collaboration hub built on LaminDB similar to how GitHub is built on git.

LaminHub features

See for yourself by browsing the demo instances in the hub UI or lamin connect owner/instance them via the CLI.

See the pricing page. Basic LaminHub features are free.

Secure & intuitive access management.

Rather than configuring storage & database permissions directly on AWS or GCP, LaminHub allows you to manage collaborators for databases & storage locations in the same way you manage access to repositories on GitHub. See Access management.

A UI to work with LaminDB instances.

See validated datasets in context of ontologies & experimental metadata.

Query & search.

See scripts, notebooks & pipelines with their inputs & outputs.

Track pipelines, notebooks & UI transforms in one place.

Quickstart

You’ll ingest a small dataset while tracking data lineage, and see how to validate, annotate, query & search.

Install the lamindb Python package.

# install with notebook support & biological entities
!pip install 'lamindb[jupyter,bionty]'

Initialize a LaminDB instance that stores data locally and mounts plugin bionty.

# store artifacts in local directory `./lamin-intro`
!lamin init --storage ./lamin-intro --schema bionty
# (optional) make Django's unnecessary functionality private for clean auto-complete
!lamin settings set private-django-api true
Hide code cell output
! using anonymous user (to identify, call: lamin login)
→ connected lamindb: anonymous/lamin-intro

Data transformations

Call track() to register a data transformation and start tracking inputs & outputs of a run. You will find your notebook in the Transform registry along with scripts, pipelines & functions. Run stores executions.

import lamindb as ln

# --> `ln.track()` generates a uid for your code
# --> `ln.track(uid)` initiates a tracked run
ln.track("FPnfDtJz8qbE0000")  
Hide code cell output
→ connected lamindb: anonymous/lamin-intro
→ notebook imports: anndata==0.10.9 bionty==0.51.1 lamindb==0.76.10 pandas==2.2.3 pytest==8.3.3
→ created Transform('FPnfDtJz'), started new Run('RBj9cOYM') at 2024-10-04 16:17:31 UTC
Is this compliant with OpenLineage?

Yes. What OpenLineage calls a “job”, LaminDB calls a “transform”. What OpenLineage calls a “run”, LaminDB calls a “run”.

What is the uid?

To tie a piece of code to a record in a database in a way that survives name and content changes, you need to attach it to an immutable identifier, e.g., LaminDB’s uid.

git, by comparison, identifies code by its content hash & file name. If you rename a notebook or script file and change the content, you lose the identity of the file. Notebook platforms like Google Colab and DeepNote support renaming and changing content of a given notebook, but they do not support versioning in a simple queryable way: every notebook version comes with the same notebook id.

To enable versioning, LaminDB auto-generates uid = f"{suid}{vuid}" so that different versions of a transform are grouped by a random “stem uid” suid (the first part of the uid) while the last four characters encode a version in a vuid (an auto-incrementing base62 number). You can optionally tag a version using the .version field.

All versioned entities in LaminDB are versioned in this way, including artifacts and collections.

Artifacts & versioning

An Artifact stores a dataset or model as a file, folder or array.

import pandas as pd

# a sample dataset
df = pd.DataFrame(
    {"CD8A": [1, 2, 3], "CD4": [3, 4, 5], "CD14": [5, 6, 7], "perturbation": ["DMSO", "IFNJ", "DMSO"],},
    index=["sample1", "sample2", "sample3"],
)

# create & save an artifact from a DataFrame
artifact = ln.Artifact.from_df(df, description="my RNA-seq").save()

# artifacts come with typed, relational metadata
artifact.describe()
Hide code cell output
Artifact(uid='W7W19wttLVBxaq2q0000', is_latest=True, description='my RNA-seq', suffix='.parquet', type='dataset', size=4091, hash='MUAn2flgfgEAhH8UayfKkQ', _hash_type='md5', _accessor='DataFrame', visibility=1, _key_is_virtual=True, created_at=2024-10-04 16:17:31 UTC)
  Provenance
    .storage = '/home/runner/work/lamin-docs/lamin-docs/docs/lamin-intro'
    .transform = 'Introduction'
    .run = 2024-10-04 16:17:31 UTC
    .created_by = 'anonymous'

Load the dataset into memory.

artifact.load()
CD8A CD4 CD14 perturbation
sample1 1 3 5 DMSO
sample2 2 4 6 IFNJ
sample3 3 5 7 DMSO

Looking at this: "IFNJ" should have been "IFNG". 🙈 Let’s create a new version by revising this dataset.

# keep the dataframe with a typo around - we'll need it later
df_typo = df.copy()

# fix the typo "IFNJ" typo
df.loc["sample2", "perturbation"] = "IFNG"

# create a new version by revising the artifact
artifact = ln.Artifact.from_df(df, revises=artifact).save()

# see all versions of an artifact
artifact.versions.df()
Hide code cell output
uid version is_latest description key suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
1 W7W19wttLVBxaq2q0000 None False my RNA-seq None .parquet dataset 4091 MUAn2flgfgEAhH8UayfKkQ None None md5 DataFrame 1 True 1 1 1 1 2024-10-04 16:17:31.760837+00:00
2 W7W19wttLVBxaq2q0001 None True my RNA-seq None .parquet dataset 4091 GDTjY7XSC6E2k9d_ZpLCnw None None md5 DataFrame 1 True 1 1 1 1 2024-10-04 16:17:31.761618+00:00

Similar to tagging a git commit, you can label a revision.

artifact.version = "1.0"
artifact.save()
artifact.versions.df()
Hide code cell output
uid version is_latest description key suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
1 W7W19wttLVBxaq2q0000 None False my RNA-seq None .parquet dataset 4091 MUAn2flgfgEAhH8UayfKkQ None None md5 DataFrame 1 True 1 1 1 1 2024-10-04 16:17:31.760837+00:00
2 W7W19wttLVBxaq2q0001 1.0 True my RNA-seq None .parquet dataset 4091 GDTjY7XSC6E2k9d_ZpLCnw None None md5 DataFrame 1 True 1 1 1 1 2024-10-04 16:17:31.782428+00:00

View data lineage.

artifact.view_lineage()
Hide code cell output
_images/473ff7d410225d4e46ec338aa0cbd163fe6b71bf356079b94ca6d7a441c6f6f2.svg
I’d rather control versioning through a key or file path like on S3.

That works, too, and you won’t need to pass an old version via revises:

artifact_v1 = ln.Artifact.from_df(df, key="my_datasets/my_study1.parquet").save()
# below automatically creates a new version of artifact_v1 because the `key` matches
artifact_v2 = ln.Artifact.from_df(df_updated, key="my_datasets/my_study1.parquet").save()

The good thing about passing revises: Artifact is that it works for entities that don’t come with a file path and you don’t need to worry about coming up with naming conventions for paths. You’ll see that LaminDB makes it easy to organize data by entities, rather than file paths.

How does this look for a file or folder?

Source path is local:

ln.Artifact("./my_data.fcs", description="my flow cytometry file")
ln.Artifact("./my_images/", description="my folder of images")

Upon artifact.save(), the source path will be copied (uploaded) into your default storage.

If the source path is remote, artifact.save() won’t trigger data duplication but register the existing path.

ln.Artifact("s3://my-bucket/my_data.fcs", description="my flow cytometry file")
ln.Artifact("s3://my-bucket/my_images/", description="my folder of images")

You can also use other remote file systems supported by `fsspec`.
How does LaminDB compare to a AWS S3?

LaminDB is a layer on top of a storage backend (AWS S3, GCP storage, local filesystem, etc.) and a database (Postgres, SQLite) for managing metadata.

Similar to organizing files in file systems & object stores with paths, you can organize artifacts using the key parameter of Artifact.

However, LaminDB encourages you to not rely on semantic keys but instead organize your data based on metadata.

Rather than memorizing names of folders and files, you find data via the entities you care about: people, code, experiments, genes, proteins, cell types, etc.

LaminDB embeds each artifact into rich relational metadata and indexes them in storage with a universal ID (uid).

This scales much better than semantic keys, which lead to deep hierarchical information structures that can become hard to navigate.

Because metadata is typed and relational, you can work with more structure, more integrity, and richer queries compared to leveraging S3’s JSON-like metadata. You’ll learn more about this below.

Are artifacts aware of array-like data?

Yes.

You can make artifacts from paths referencing array-like objects:

ln.Artifact("./my_anndata.h5ad", description="curated array")
ln.Artifact("./my_zarr_array/", description="my zarr array store")

Or from in-memory objects:

ln.Artifact.from_df(df, description="my dataframe")
ln.Artifact.from_anndata(adata, description="annotated array")

You can open large artifacts for slicing from the cloud or load small artifacts directly into memory.

Datasets & labels

Label an artifact with a ULabel and a bionty.CellLine. The same works for any entity in any custom schema module.

import bionty as bt

# create & save a ulabel record
candidate_marker_study = ln.ULabel(name="Candidate marker study").save()

# label the artifact
artifact.ulabels.add(candidate_marker_study)

# repeat for a bionty entity
cell_line = bt.CellLine.from_source(name="HEK293").save()
artifact.cell_lines.add(cell_line)

# describe the artifact
artifact.describe()
Hide code cell output
Artifact(uid='W7W19wttLVBxaq2q0001', version='1.0', is_latest=True, description='my RNA-seq', suffix='.parquet', type='dataset', size=4091, hash='GDTjY7XSC6E2k9d_ZpLCnw', _hash_type='md5', _accessor='DataFrame', visibility=1, _key_is_virtual=True, created_at=2024-10-04 16:17:31 UTC)
  Provenance
    .storage = '/home/runner/work/lamin-docs/lamin-docs/docs/lamin-intro'
    .transform = 'Introduction'
    .run = 2024-10-04 16:17:31 UTC
    .created_by = 'anonymous'
  Labels
    .cell_lines = 'HEK293'
    .ulabels = 'Candidate marker study'

Registries, records & fields

LaminDB’s central classes are related records that inherit from Record. We’ve already seen how to create new artifact, transform and ulabel records.

The easiest way to see all existing records of a given type is to call the class method df.

ln.ULabel.df()
Hide code cell output
uid name description reference reference_type run_id created_by_id updated_at
id
1 qO5DQ1FW Candidate marker study None None None 1 1 2024-10-04 16:17:31.970297+00:00

Existing records are stored in the record’s registry (metaclass Registry), which maps 1:1 to on a SQL table in the SQLite or Postgres backend.

A record and its registry share the same fields, which define the metadata you can query for. If you want to see them, look at the class or auto-complete.

ln.Artifact
Hide code cell output
Artifact
  Simple fields
    .uid: CharField
    .description: CharField
    .key: CharField
    .suffix: CharField
    .type: CharField
    .size: BigIntegerField
    .hash: CharField
    .n_objects: BigIntegerField
    .n_observations: BigIntegerField
    .visibility: SmallIntegerField
    .version: CharField
    .is_latest: BooleanField
    .created_at: DateTimeField
    .updated_at: DateTimeField
  Relational fields
    .storage: Storage
    .transform: Transform
    .run: Run
    .created_by: User
    .ulabels: ULabel
    .input_of_runs: Run
    .feature_sets: FeatureSet
    .collections: Collection
  Bionty fields
    .organisms: bionty.Organism
    .genes: bionty.Gene
    .proteins: bionty.Protein
    .cell_markers: bionty.CellMarker
    .tissues: bionty.Tissue
    .cell_types: bionty.CellType
    .diseases: bionty.Disease
    .cell_lines: bionty.CellLine
    .phenotypes: bionty.Phenotype
    .pathways: bionty.Pathway
    .experimental_factors: bionty.ExperimentalFactor
    .developmental_stages: bionty.DevelopmentalStage
    .ethnicities: bionty.Ethnicity

Datasets & features

What fields are to metadata records, features are to datasets. You can annotate datasets by the features they measure.

But because LaminDB validates all user input against its registries, annotating with a "temperature" feature doesn’t work right away.

import pytest

with pytest.raises(ln.core.exceptions.ValidationError) as e:
    artifact.features.add_values({"temperature": 21.6})

print(e.exconly())
Hide code cell output
lamindb.core.exceptions.ValidationError: These keys could not be validated: ['temperature']
Here is how to create a feature:

  ln.Feature(name='temperature', dtype='float').save()

Following the hint in the error message, create & save a Feature.

# create & save the "temperature" feature (only required once)
ln.Feature(name="temperature", dtype="float").save()

# now we can annotate with the feature & the value
artifact.features.add_values({"temperature": 21.6})

# describe the artifact
artifact.describe()
Hide code cell output
Artifact(uid='W7W19wttLVBxaq2q0001', version='1.0', is_latest=True, description='my RNA-seq', suffix='.parquet', type='dataset', size=4091, hash='GDTjY7XSC6E2k9d_ZpLCnw', _hash_type='md5', _accessor='DataFrame', visibility=1, _key_is_virtual=True, created_at=2024-10-04 16:17:31 UTC)
  Provenance
    .storage = '/home/runner/work/lamin-docs/lamin-docs/docs/lamin-intro'
    .transform = 'Introduction'
    .run = 2024-10-04 16:17:31 UTC
    .created_by = 'anonymous'
  Labels
    .cell_lines = 'HEK293'
    .ulabels = 'Candidate marker study'
  Features
    'temperature' = 21.6

We can also annotate with categorical features:

# register a categorical feature
ln.Feature(name="study", dtype="cat").save()

# add a categorical value
artifact.features.add_values({"study": "Candidate marker study"})

# describe the artifact with type information
artifact.describe(print_types=True)
Hide code cell output
Artifact(uid='W7W19wttLVBxaq2q0001', version='1.0', is_latest=True, description='my RNA-seq', suffix='.parquet', type='dataset', size=4091, hash='GDTjY7XSC6E2k9d_ZpLCnw', _hash_type='md5', _accessor='DataFrame', visibility=1, _key_is_virtual=True, created_at=2024-10-04 16:17:31 UTC)
  Provenance
    .storage: Storage = '/home/runner/work/lamin-docs/lamin-docs/docs/lamin-intro'
    .transform: Transform = 'Introduction'
    .run: Run = 2024-10-04 16:17:31 UTC
    .created_by: User = 'anonymous'
  Labels
    .cell_lines: bionty.CellLine = 'HEK293'
    .ulabels: ULabel = 'Candidate marker study'
  Features
    'study': cat[ULabel] = 'Candidate marker study'
    'temperature': float = 21.6

This is how you query artifacts by features.

ln.Artifact.features.filter(study__contains="marker study").df()
Hide code cell output
uid version is_latest description key suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
2 W7W19wttLVBxaq2q0001 1.0 True my RNA-seq None .parquet dataset 4091 GDTjY7XSC6E2k9d_ZpLCnw None None md5 DataFrame 1 True 1 1 1 1 2024-10-04 16:17:31.782428+00:00

Features organize labels by how they’re measured in datasets, independently of how labels are stored in metadata registries.

Key examples

Understand data lineage

Understand where a dataset comes from and what it’s used for (background).

artifact.view_lineage()
I just want to see the transformations.
transform.view_lineage()

You don’t need a workflow manager to track data lineage (if you want to use one, see Pipelines – workflow managers). All you need is:

import lamindb as ln

ln.track()  # track your run

# your code

ln.finish()  # mark run as finished, save execution report, source code & environment

Below is how a single transform (a notebook) with its run report looks on the hub.

To create a new version of a notebook or script, run lamin load on the terminal, e.g.,

$ lamin load https://lamin.ai/laminlabs/lamindata/transform/13VINnFk89PE0004
→ connected lamindb: laminlabs/lamindata
→ updated uid: 13VINnFk89PE0004  13VINnFk89PE0005
→ notebook is here: mcfarland_2020_preparation.ipynb

Curate datasets

In the quickstart, you just saw how to ingest & annotate datasets without validation. This is often enough if you’re prototyping or working with one-off studies. But if you want to create a big body of standardized data, you should invest a more time and curate your datasets.

Let’s use the high-level Curator class to curate a DataFrame.

# construct a Curator object to validate & annotate a DataFrame
curate = ln.Curator.from_df(
    df,
    # define validation criteria
    columns=ln.Feature.name,  # map column names
    categoricals={"perturbation": ln.ULabel.name},  # map categories
)

# validate the dataset
curate.validate()
Hide code cell output
✓ added 1 record with Feature.name for columns: 'perturbation'
3 non-validated values are not saved in Feature.name: ['CD14', 'CD8A', 'CD4']!
      → to lookup values, use lookup().columns
      → to save, run add_new_from_columns
• mapping perturbation on ULabel.name
!    2 terms are not validated: 'DMSO', 'IFNG'
      → fix typos, remove non-existent values, or save terms via .add_new_from('perturbation')
False

The validation did not pass because LaminDB’s registries don’t yet know about the features "CD8A", "CD4", "CD14", "perturbation" and labels "DMSO", "IFNG", "DMSO" in this dataset. Hence, we need to initially populate them.

# add non-validated features based on the DataFrame columns
curate.add_new_from_columns()

# add non-validated labels based on the perturbation column of the dataframe
curate.add_new_from("perturbation")

# see the updated content of the ULabel registry
ln.ULabel.df()
Hide code cell output
✓ added 3 records with Feature.name for columns: 'CD14', 'CD8A', 'CD4'
✓ added 2 records with ULabel.name for perturbation: 'DMSO', 'IFNG'
uid name description reference reference_type run_id created_by_id updated_at
id
4 rYRlcTSG is_perturbation None None None 1 1 2024-10-04 16:17:34.785598+00:00
3 Vsw7x8Sc IFNG None None None 1 1 2024-10-04 16:17:34.780375+00:00
2 yQdgIOcN DMSO None None None 1 1 2024-10-04 16:17:34.780341+00:00
1 qO5DQ1FW Candidate marker study None None None 1 1 2024-10-04 16:17:31.970297+00:00

With the ULabel and Feature registries now containing meaningful reference values, validation passes & and we can automatically parse features & labels to save an annotated & curated artifact.

# given the updated registries, the validation passes
curate.validate()

# save curated artifact
artifact = curate.save_artifact(description="my RNA-seq")

# see the parsed annotations
artifact.describe()

# query for a ulabel that was parsed from the dataset
ln.Artifact.get(ulabels__name="IFNG")
Hide code cell output
✓ perturbation is validated against ULabel.name
→ returning existing artifact with same hash: Artifact(uid='W7W19wttLVBxaq2q0001', version='1.0', is_latest=True, description='my RNA-seq', suffix='.parquet', type='dataset', size=4091, hash='GDTjY7XSC6E2k9d_ZpLCnw', _hash_type='md5', _accessor='DataFrame', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-04 16:17:31 UTC)
Artifact(uid='W7W19wttLVBxaq2q0001', version='1.0', is_latest=True, description='my RNA-seq', suffix='.parquet', type='dataset', size=4091, hash='GDTjY7XSC6E2k9d_ZpLCnw', _hash_type='md5', _accessor='DataFrame', visibility=1, _key_is_virtual=True, created_at=2024-10-04 16:17:31 UTC)
  Provenance
    .storage = '/home/runner/work/lamin-docs/lamin-docs/docs/lamin-intro'
    .transform = 'Introduction'
    .run = 2024-10-04 16:17:31 UTC
    .created_by = 'anonymous'
  Labels
    ._previous_runs = '2024-10-04 16:17:31.091019+00:00'
    .cell_lines = 'HEK293'
    .ulabels = 'Candidate marker study', 'DMSO', 'IFNG'
  Features
    'perturbation' = 'DMSO', 'IFNG'
    'study' = 'Candidate marker study'
    'temperature' = 21.6
  Feature sets
    'columns' = 'perturbation', 'CD8A', 'CD4', 'CD14'
Artifact(uid='W7W19wttLVBxaq2q0001', version='1.0', is_latest=True, description='my RNA-seq', suffix='.parquet', type='dataset', size=4091, hash='GDTjY7XSC6E2k9d_ZpLCnw', _hash_type='md5', _accessor='DataFrame', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-04 16:17:31 UTC)

Had we used ln.Cuartor from the beginning, we would have caught the typo.

# construct a Curator object to validate & annotate a DataFrame
curate = ln.Curator.from_df(
    df_typo,
    # define validation criteria
    columns=ln.Feature.name,  # map column names
    categoricals={"perturbation": ln.ULabel.name},  # map categories
)

# validate the dataset
curate.validate()
Hide code cell output
• mapping perturbation on ULabel.name
!    1 terms is not validated: 'IFNJ'
      → fix typos, remove non-existent values, or save terms via .add_new_from('perturbation')
False

Manage biological registries

The generic Feature and ULabel registries will get you pretty far.

But let’s now look at what you do can with a dedicated biological registry like Gene.

Every bionty registry is based on configurable public ontologies (>20 of them).

cell_types = bt.CellType.public()
cell_types
Hide code cell output
PublicOntology
Entity: CellType
Organism: all
Source: cl, 2024-05-15
#terms: 2931
cell_types.search("gamma delta T cell").head(2)
Hide code cell output
ontology_id definition synonyms parents __ratio__
name
gamma-delta T cell CL:0000798 A T Cell That Expresses A Gamma-Delta T Cell R... gammadelta T cell|gamma-delta T-cell|gamma-del... [CL:0000084] 100.000000
CD27-negative gamma-delta T cell CL:0002125 A Circulating Gamma-Delta T Cell That Expresse... gammadelta-17 cells [CL:0000800] 86.486486

Validate & annotate with typed features.

import anndata as ad

# store the dataset as an AnnData object to distinguish data from metadata
adata = ad.AnnData(
    df[["CD8A", "CD4", "CD14"]], obs=df[["perturbation"]]
)

# create an annotation flow for an AnnData object
curate = ln.Curator.from_anndata(
    adata,
    # define validation criteria
    var_index=bt.Gene.symbol,  # map .var.index onto Gene registry
    categoricals={adata.obs.perturbation.name: ln.ULabel.name},
    organism="human",  # specify the organism for the Gene registry
)
curate.add_validated_from_var_index()
curate.validate()

# save curated artifact
artifact = curate.save_artifact(description="my RNA-seq")
artifact.describe()
Hide code cell output
✓ var_index is validated against Gene.symbol
✓ perturbation is validated against ULabel.name
• path content will be copied to default storage upon `save()` with key `None` ('.lamindb/YvsYQFbBeiKrNRTC0000.h5ad')
✓ storing artifact 'YvsYQFbBeiKrNRTC0000' at '/home/runner/work/lamin-docs/lamin-docs/docs/lamin-intro/.lamindb/YvsYQFbBeiKrNRTC0000.h5ad'
• parsing feature names of X stored in slot 'var'
3 unique terms (100.00%) are validated for symbol
✓    linked: FeatureSet(uid='6mDx8YOWqqntOQj4cxjR', n=3, dtype='int', registry='bionty.Gene', hash='f2UVeHefaZxXFjmUwo9Ozw', created_by_id=1, run_id=1)
• parsing feature names of slot 'obs'
1 unique term (100.00%) is validated for name
✓    linked: FeatureSet(uid='SDPMyGRRY5iXzCxC0uxG', n=1, registry='Feature', hash='7wf1mTYEWzamWiP8axpQ0Q', created_by_id=1, run_id=1)
✓ saved 2 feature sets for slots: 'var','obs'
Artifact(uid='YvsYQFbBeiKrNRTC0000', is_latest=True, description='my RNA-seq', suffix='.h5ad', type='dataset', size=19240, hash='nLH34gqty3-5c2eGF6deOA', n_observations=3, _hash_type='md5', _accessor='AnnData', visibility=1, _key_is_virtual=True, created_at=2024-10-04 16:17:39 UTC)
  Provenance
    .storage = '/home/runner/work/lamin-docs/lamin-docs/docs/lamin-intro'
    .transform = 'Introduction'
    .run = 2024-10-04 16:17:31 UTC
    .created_by = 'anonymous'
  Labels
    .ulabels = 'DMSO', 'IFNG'
  Features
    'perturbation' = 'DMSO', 'IFNG'
  Feature sets
    'var' = 'CD8A', 'CD4', 'CD14'
    'obs' = 'perturbation'

Query for typed features.

# get a lookup object for human genes
genes = bt.Gene.filter(organism__name="human").lookup()
# query for all feature sets that contain CD8A
feature_sets = ln.FeatureSet.filter(genes=genes.cd8a).all()
# write the query
ln.Artifact.filter(feature_sets__in=feature_sets).df()
Hide code cell output
uid version is_latest description key suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
3 YvsYQFbBeiKrNRTC0000 None True my RNA-seq None .h5ad dataset 19240 nLH34gqty3-5c2eGF6deOA None 3 md5 AnnData 1 True 1 1 1 1 2024-10-04 16:17:39.256989+00:00

Update ontologies, e.g., create a cell type record and add a new cell state.

# create an ontology-coupled cell type record and save it
neuron = bt.CellType.from_source(name="neuron").save()

# create a record to track a new cell state
new_cell_state = bt.CellType(name="my neuron cell state", description="explains X").save()

# express that it's a neuron state
new_cell_state.parents.add(neuron)

# view ontological hierarchy
new_cell_state.view_parents(distance=2)
Hide code cell output
✓ created 1 CellType record from Bionty matching name: 'neuron'
✓ created 3 CellType records from Bionty matching ontology_id: 'CL:0000393', 'CL:0000404', 'CL:0002319'
! records with similar names exist! did you mean to load one of them?
uid name ontology_id abbr synonyms description source_id run_id created_by_id updated_at
id
1 3QnZfoBk neuron CL:0000540 None nerve cell The Basic Cellular Unit Of Nervous Tissue. Eac... 32 1 1 2024-10-04 16:17:39.701033+00:00
2 2qSJYeQX electrically responsive cell CL:0000393 None None A Cell Whose Function Is Determined By Its Res... 32 1 1 2024-10-04 16:17:40.549784+00:00
3 5NqNmmSr electrically signaling cell CL:0000404 None None A Cell That Initiates An Electrical Signal And... 32 1 1 2024-10-04 16:17:40.549824+00:00
_images/a3a84730e32e8ed1d74fe024ae2c24524062f8939fafd65e41e48494a3848f63.svg

Scale up data & learning

How do you integrate new datasets with your existing datasets? Leverage Collection.

# a new dataset
df = pd.DataFrame(
    {"CD8A": [2, 3, 3], "CD4": [3, 4, 5], "CD38": [4, 2, 3], "perturbation": ["DMSO", "IFNG", "IFNG"],},
    index=["sample4", "sample5", "sample6"],
)
adata = ad.AnnData(df[["CD8A", "CD4", "CD38"]], obs=df[["perturbation"]])

# validate, curate and save a new artifact
curate = ln.Curator.from_anndata(
    adata,
    var_index=bt.Gene.symbol,
    categoricals={adata.obs.perturbation.name: ln.ULabel.name},
    organism="human",
)
curate.add_validated_from_var_index()
curate.validate()
artifact2 = curate.save_artifact(description="my RNA-seq dataset 2")
Hide code cell output
✓ var_index is validated against Gene.symbol
✓ perturbation is validated against ULabel.name
• path content will be copied to default storage upon `save()` with key `None` ('.lamindb/QdomVNW1Q0thhv2l0000.h5ad')
✓ storing artifact 'QdomVNW1Q0thhv2l0000' at '/home/runner/work/lamin-docs/lamin-docs/docs/lamin-intro/.lamindb/QdomVNW1Q0thhv2l0000.h5ad'
• parsing feature names of X stored in slot 'var'
3 unique terms (100.00%) are validated for symbol
✓    linked: FeatureSet(uid='JOBhNyuV826ZeH2ew7KR', n=3, dtype='int', registry='bionty.Gene', hash='QW2rHuIo5-eGNZbRxHMDCw', created_by_id=1, run_id=1)
• parsing feature names of slot 'obs'
1 unique term (100.00%) is validated for name
✓    linked: FeatureSet(uid='SDPMyGRRY5iXzCxC0uxG', n=1, registry='Feature', hash='7wf1mTYEWzamWiP8axpQ0Q', created_by_id=1, run_id=1, created_at=2024-10-04 16:17:39 UTC)
✓ saved 1 feature set for slot: 'var'

Create a collection using Collection.

collection = ln.Collection([artifact, artifact2], name="my RNA-seq collection").save()
collection.describe()
collection.view_lineage()
Hide code cell output
Collection(uid='rJpxdVFrlRhz6d3M0000', is_latest=True, name='my RNA-seq collection', hash='xDjhklRxArFHharWMZPEzw', visibility=1, created_at=2024-10-04 16:17:43 UTC)
  Provenance
    .created_by = 'anonymous'
    .transform = 'Introduction'
    .run = 2024-10-04 16:17:31 UTC
_images/247bf14b97335aba7c86a7b44d945487f8d9fb743348029ed746e898ff740e82.svg
# if it's small enough, you can load the entire collection into memory as if it was one
collection.load()

# typically, it's too big, hence, iterate over its artifacts
collection.artifacts.all()

# or look at a DataFrame listing the artifacts
collection.artifacts.df()
Hide code cell output
uid version is_latest description key suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
3 YvsYQFbBeiKrNRTC0000 None True my RNA-seq None .h5ad dataset 19240 nLH34gqty3-5c2eGF6deOA None 3 md5 AnnData 1 True 1 1 1 1 2024-10-04 16:17:39.256989+00:00
4 QdomVNW1Q0thhv2l0000 None True my RNA-seq dataset 2 None .h5ad dataset 19240 K95PcyOoxIxtlytXMr6AVg None 3 md5 AnnData 1 True 1 1 1 1 2024-10-04 16:17:43.052070+00:00

Directly train models on collections of AnnData.

# to train models, batch iterate through the collection as if it was one array
from torch.utils.data import DataLoader, WeightedRandomSampler
dataset = collection.mapped(obs_keys=["perturbation"])
sampler = WeightedRandomSampler(
    weights=dataset.get_label_weights("perturbation"), num_samples=len(dataset)
)
data_loader = DataLoader(dataset, batch_size=2, sampler=sampler)
for batch in data_loader:
    pass

Read this blog post for more on training models on sharded datasets.

Design

Why?

Objects like pd.DataFrame are at the heart of many data science workflows but there hasn’t been a tool to manage these objects in the rich context that collaborative biological research requires:

  • data lineage: data sources, data transformations, models, users

  • domain knowledge & experimental metadata: the features & labels derived from domain entities

In this blog post, we discuss how the complexity of modern R&D data often blocks realizing the scientific progress it promises.

Assumptions

  1. Teams need to have enough freedom to initiate work independently but enough structure to easily integrate datasets later on

  2. Batched datasets from physical instruments are transformed (Transform) into useful representations (Artifact)

  3. Learning needs features (Feature, CellMarker, …) and labels (ULabel, CellLine, …)

  4. Insights connect representations to experimental metadata and knowledge (ontologies)

Distributed

LaminDB is a distributed system like git. Collaborators can connect to your instance via:

ln.connect("account-handle/instance-name")

Or via the command line for auto-connecting in a Python session:

lamin connect account-handle/instance-name

Or you create your new instance:

lamin init --storage ./my-data-folder

Metada schema & API

LaminDB provides a SQL schema for common metadata entities: Artifact, Collection, Transform, Feature, ULabel etc. - see the API reference or the source code.

The core metadata schema is extendable through plugins (see green vs. red entities in graphic), e.g., with basic biological (Gene, Protein, CellLine, etc.) & operational entities (Biosample, Techsample, Treatment, etc.).

What is the metadata schema language?

Data models are defined in Python using the Django ORM. Django translates them to SQL tables. Django is one of the most-used & highly-starred projects on GitHub (~1M dependents, ~73k stars) and has been robustly maintained for 15 years.

On top of the metadata schema, LaminDB is a Python API that models datasets as artifacts, abstracts over storage & database access, data transformations, and (biological) ontologies.

Note that the datasets schema (e.g., .parquet files or .h5ad arrays) is modeled through the Feature registry and does not require migrations to be updated.

Custom schemas and plugins

LaminDB can be customized & extended with schema & app plugins building on the Django ecosystem. Examples are:

  • bionty: Registries for basic biological entities, coupled to public ontologies.

  • wetlab: Exemplary custom schema to manage samples, treatments, etc.

If you’d like to create your own schema or app:

  1. Create a git repository with registries similar to wetlab

  2. Create & deploy migrations via lamin migrate create and lamin migrate deploy

Repositories

LaminDB and its plugins consist in open-source Python libraries & publicly hosted metadata assets:

  • lamindb: Core package.

  • bionty: Registries for basic biological entities, coupled to public ontologies.

  • wetlab: Default wetlab schema.

  • guides: Guides.

  • usecases: Use cases.

All immediate dependencies are available as git submodules here, for instance,

For a comprehensive list of open-sourced software, browse our GitHub account.

LaminHub is not open-sourced.

Influences

LaminDB was influenced by many other projects, see Influences.