Curate datasets¶

Curating a dataset with LaminDB means three things:

Validate that the dataset matches a desired schema
In case the dataset doesn’t validate, standardize it, e.g., by fixing typos or mapping synonyms
Annotate the dataset by linking it against metadata entities so that it becomes queryable

In this guide we’ll curate common data structures. Here is a guide for the underlying low-level API.

Curate a DataFrame¶

# pip install 'lamindb[bionty]'
!lamin init --storage ./test-curate --modules bionty

Let’s start with a DataFrame that we’d like to validate.

import lamindb as ln
import bionty as bt
import pandas as pd


df = ln.core.datasets.small_dataset1(
    with_cell_type_synonym=True, with_cell_type_typo=True
)
df

Show code cell output Hide code cell output

→ connected lamindb: testuser1/test-curate

	ENSG00000153563	ENSG00000010610	ENSG00000170458	perturbation	sample_note	cell_type_by_expert	cell_type_by_model	assay_oid	concentration	treatment_time_h	donor
sample1	1	3	5	DMSO	was ok	B-cell	B cell	EFO:0008913	0.1%	24	D0001
sample2	2	4	6	IFNG	looks naah	CD8-pos alpha-beta T cell	T cell	EFO:0008913	200 nM	24	D0002
sample3	3	5	7	DMSO	pretty! 🤩	CD8-pos alpha-beta T cell	T cell	EFO:0008913	0.1%	6	None

Define a schema to define the minimal columns we expect in such a dataset.

schema = ln.Schema(
    name="My immuno schema",
    features=[
        ln.Feature(name="perturbation", dtype=ln.ULabel).save(),
        ln.Feature(name="cell_type_by_model", dtype=bt.CellType).save(),
        ln.Feature(name="cell_type_by_expert", dtype=bt.CellType).save(),
        ln.Feature(name="assay_oid", dtype=bt.ExperimentalFactor.ontology_id).save(),
        ln.Feature(name="donor", dtype=str, nullable=True).save(),
        ln.Feature(name="concentration", dtype=str).save(),
        ln.Feature(name="treatment_time_h", dtype=float, coerce_dtype=True).save(),
    ],
).save()
# display the associated features as a dataframe
schema.features.df()

Show code cell output Hide code cell output

	uid	name	dtype	is_type	unit	description	array_rank	array_size	array_shape	proxy_dtype	synonyms	_expect_many	_curation	space_id	type_id	run_id	created_at	created_by_id	_aux	_branch_code
id
1	FAwDqDmJehnC	perturbation	cat[ULabel]	None	None	None	0	0	None	None	None	True	None	1	None	None	2025-03-16 20:57:45.935000+00:00	1	{'af': {'0': None, '1': True, '2': False}}	1
2	PYnOIEn8zcSt	cell_type_by_model	cat[bionty.CellType]	None	None	None	0	0	None	None	None	True	None	1	None	None	2025-03-16 20:57:46.217000+00:00	1	{'af': {'0': None, '1': True, '2': False}}	1
3	EZzWJLMxuVEd	cell_type_by_expert	cat[bionty.CellType]	None	None	None	0	0	None	None	None	True	None	1	None	None	2025-03-16 20:57:46.223000+00:00	1	{'af': {'0': None, '1': True, '2': False}}	1
4	8nMeyS4dM9M9	assay_oid	cat[bionty.ExperimentalFactor.ontology_id]	None	None	None	0	0	None	None	None	True	None	1	None	None	2025-03-16 20:57:46.229000+00:00	1	{'af': {'0': None, '1': True, '2': False}}	1
5	8ZTLDeWt1Qbe	donor	str	None	None	None	0	0	None	None	None	True	None	1	None	None	2025-03-16 20:57:46.235000+00:00	1	{'af': {'0': None, '1': True, '2': False}}	1
6	cO1g6Ht9piw3	concentration	str	None	None	None	0	0	None	None	None	True	None	1	None	None	2025-03-16 20:57:46.240000+00:00	1	{'af': {'0': None, '1': True, '2': False}}	1
7	o8qclzwfRg5W	treatment_time_h	float	None	None	None	0	0	None	None	None	True	None	1	None	None	2025-03-16 20:57:46.246000+00:00	1	{'af': {'0': None, '1': True, '2': True}}	1

Create a Curator using the dataset & the schema.

curator = ln.curators.DataFrameCurator(df, schema)

The validate() method validates that your dataset adheres to the criteria defined by the schema. It identifies which values are already validated (exist in our registries) and which are potentially problematic (do not yet exist in our registries).

try:
    curator.validate()
except ln.errors.ValidationError:
    pass

# check the non-validated terms
curator.cat.non_validated

For cell_type, we saw that “cerebral pyramidal neuron”, “astrocytic glia” are not validated.

First, let’s standardize synonym “astrocytic glia” as suggested

curator.cat.standardize("cell_type_by_expert")

# now we have only one non-validated cell type left
curator.cat.non_validated

For “cerebral pyramidal neuron”, let’s understand which cell type in the public ontology might be the actual match.

# to check the correct spelling of categories, pass `public=True` to get a lookup object from public ontologies
# use `lookup = curator.cat.lookup()` to get a lookup object of existing records in your instance
lookup = curator.cat.lookup(public=True)
lookup

# here is an example for the "cell_type" column
cell_types = lookup["cell_type_by_expert"]
cell_types.cd8_positive_alpha_beta_t_cell

# fix the cell type name
df["cell_type_by_expert"] = df["cell_type_by_expert"].cat.rename_categories(
    {"CD8-pos alpha-beta T cell": cell_types.cd8_positive_alpha_beta_t_cell.name}
)

For perturbation, we want to add the new values: “DMSO”, “IFNG”

# this adds perturbations that were _not_ validated
curator.cat.add_new_from("perturbation")

# validate again
curator.validate()

Save a curated artifact.

artifact = curator.save_artifact(key="my_datasets/my_curated_dataset.parquet")

artifact.describe()

Curate an AnnData¶

Here we additionally specify which var_index to validate against.

import anndata as ad

X = pd.DataFrame(
    {
        "ENSG00000081059": [1, 2, 3],
        "ENSG00000276977": [4, 5, 6],
        "ENSG00000198851": [7, 8, 9],
        "ENSG00000010610": [10, 11, 12],
        "ENSG00000153563": [13, 14, 15],
        "ENSGcorrupted": [16, 17, 18],
    },
    index=df.index,  # because we already curated the dataframe above, it will validate
)
adata = ad.AnnData(X=X, obs=df)
adata

# define var schema
var_schema = ln.Schema(
    name="my_var_schema",
    itype=bt.Gene.ensembl_gene_id,  # identifier type
    dtype=int,
).save()

# define composite schema
anndata_schema = ln.Schema(
    name="small_dataset1_anndata_schema",
    otype="AnnData",  # object type
    components={"obs": schema, "var": var_schema},
).save()

Check the slots of a schema:

anndata_schema.slots

curator = ln.curators.AnnDataCurator(adata, anndata_schema)
try:
    curator.validate()
except ln.errors.ValidationError as error:
    print(error)

Subset the AnnData to validated genes only:

adata_validated = adata[:, ~adata.var.index.isin(["ENSGcorrupted"])].copy()

Now let’s validate the subsetted object:

curator = ln.curators.AnnDataCurator(adata_validated, anndata_schema)
curator.validate()

The validated AnnData can be subsequently saved as an Artifact:

artifact = curator.save_artifact(key="my_datasets/my_curated_anndata.h5ad")

Show code cell output Hide code cell output

✓ "perturbation" is validated against ULabel.name

✓ "cell_type_by_model" is validated against CellType.name

✓ "cell_type_by_expert" is validated against CellType.name

✓ "assay_oid" is validated against ExperimentalFactor.ontology_id

! no run & transform got linked, call `ln.track()` & re-run

• path content will be copied to default storage upon `save()` with key 'my_datasets/my_curated_anndata.h5ad'

✓ storing artifact 'QvRVF7drw5qTvIdq0000' at '/home/runner/work/lamindb/lamindb/docs/test-curate/.lamindb/QvRVF7drw5qTvIdq0000.h5ad'

! run input wasn't tracked, call `ln.track()` and re-run

• parsing feature names of X stored in slot 'var'

✓    5 unique terms (100.00%) are validated for ensembl_gene_id

✓    linked: Schema(uid='D6Z7pHSctTHeawGW9OBx', n=5, dtype='int', itype='bionty.Gene', is_type=False, hash='nmFTQkXy239ruKDl8gDLSw', minimal_set=True, ordered_set=False, maximal_set=False, space_id=1, created_by_id=1, created_at=<django.db.models.expressions.DatabaseDefault object at 0x7f2d9c72ce50>)

• parsing feature names of slot 'obs'

✓    7 unique terms (63.60%) are validated for name

!    4 unique terms (36.40%) are not validated for name: 'ENSG00000153563', 'ENSG00000010610', 'ENSG00000170458', 'sample_note'

✓    loaded 7 Feature records matching name: 'perturbation', 'cell_type_by_expert', 'cell_type_by_model', 'assay_oid', 'concentration', 'treatment_time_h', 'donor'

!    did not create Feature records for 4 non-validated names: 'ENSG00000010610', 'ENSG00000153563', 'ENSG00000170458', 'sample_note'

→    returning existing schema with same hash: Schema(uid='w5vFPFAWKul4xgzITGx5', name='My immuno schema', n=7, itype='Feature', is_type=False, hash='7xSKmcRDwl2GnTv9sG_ivQ', minimal_set=True, ordered_set=False, maximal_set=False, space_id=1, created_by_id=1, created_at=2025-03-16 20:57:46 UTC)

!    updated otype from None to DataFrame

✓    linked: Schema(uid='w5vFPFAWKul4xgzITGx5', name='My immuno schema', n=7, itype='Feature', is_type=False, otype='DataFrame', hash='7xSKmcRDwl2GnTv9sG_ivQ', minimal_set=True, ordered_set=False, maximal_set=False, space_id=1, created_by_id=1, created_at=2025-03-16 20:57:46 UTC)

✓ saved 1 feature set for slot: 'var'

Access the schema for each slot:

artifact.features.slots

The saved artifact has been annotated with validated features and labels:

artifact.describe()

Show code cell output Hide code cell output

Artifact .h5ad/AnnData
├── General
│   ├── .uid = 'QvRVF7drw5qTvIdq0000'
│   ├── .key = 'my_datasets/my_curated_anndata.h5ad'
│   ├── .size = 31400
│   ├── .hash = 'DqsPrPqnKBCg7zrD5IPBtw'
│   ├── .n_observations = 3
│   ├── .path = /home/runner/work/lamindb/lamindb/docs/test-curate/.lamindb/QvRVF7drw5qTvIdq0000.h5ad
│   ├── .created_by = testuser1 (Test User1)
│   └── .created_at = 2025-03-16 20:57:54
├── Dataset features/.feature_sets
│   ├── var • 5                     [bionty.Gene]                                                       
│   │   TCF7                        int                                                                 
│   │   CD3E                        int                                                                 
│   │   CD4                         int                                                                 
│   │   PDCD1                       int                                                                 
│   │   CD8A                        int                                                                 
│   └── obs • 7                     [Feature]                                                           
│       assay_oid                   cat[bionty.ExperimentalF…  single-cell RNA sequencing               
│       cell_type_by_expert         cat[bionty.CellType]       B cell, CD8-positive, alpha-beta T cell  
│       cell_type_by_model          cat[bionty.CellType]       B cell, T cell                           
│       perturbation                cat[ULabel]                DMSO, IFNG                               
│       donor                       str                                                                 
│       concentration               str                                                                 
│       treatment_time_h            float                                                               
└── Labels
    └── .cell_types                 bionty.CellType            T cell, B cell, CD8-positive, alpha-beta…
        .experimental_factors       bionty.ExperimentalFactor  single-cell RNA sequencing               
        .ulabels                    ULabel                     IFNG, DMSO

Standardize an AnnData¶

If you need more control, you can access DataFrameCurator objects for the "var" and "obs" slots, respectively.

curator.slots

# revert the previous cell type standardization
df["cell_type_by_expert"] = df["cell_type_by_expert"].cat.rename_categories(
    {"B cell": "B-cell"}
)
# an AnnData where a cell type matches a synonym
adata_with_synonym = ad.AnnData(X=adata_validated.X, var=adata_validated.var, obs=df)
adata_with_synonym

curator = ln.curators.AnnDataCurator(adata_with_synonym, anndata_schema)
try:
    curator.validate()
except ln.errors.ValidationError:
    pass

curator.slots["obs"].cat.standardize("cell_type_by_expert")

curator.validate()

Summary¶

We’ve walked through the process of validating, standardizing, and annotating datasets going through these key steps:

Defining validation criteria
Validating data against existing registries
Adding new validated entries to registries
Annotating artifacts with validated metadata

By following these steps, you can ensure your data is standardized and well-curated.

If you have other data structures, read: How do I validate & annotate arbitrary data structures?.