Curate DataFrames and AnnDatas¶

Curating a dataset with LaminDB means three things:

Validate: ensure the dataset meets predefined validation criteria
Standardize: transform the dataset so that it meets validation criteria, e.g., by fixing typos or using standard instead of ad hoc identifiers
Annotate: link the dataset against validated metadata so that it becomes queryable

If a dataset passes validation, curating it takes two lines of code:

curator = ln.Curator.from_df(df, ...)  # create a Curator and pass criteria in "..."
curator.save_artifact()                # validates the content of the dataset and saves it as annotated artifact

Beyond having valid content, the curated dataset is now queryable via metadata identifiers found in the dataset because they have been validated & linked against LaminDB registries.

Definition: valid metadata identifier

An identifier like "Experiment 1" is a valid value for ULabel.name if a record with name "Experiment 1" exists in the ULabel registry.

categoricals = {"experiment": ln.ULabel.name}  # the validation constraint
curator = ln.Curator.from_df(df, categoricals=categoricals)
curator.validate()

The DataFrame validates if

there is a column with name "experiment" in the dataframe whose values are all found in the name field of the ULabel registry
the column name "experiment" is found in the name field of the Feature registry

Beyond validating metadata identifiers, LaminDB also validates data types and dataset schema.

# !pip install 'lamindb[bionty]'
!lamin init --storage ./test-curate --schema bionty

Curate a DataFrame¶

Let’s start with a DataFrame that we’d like to validate.

import lamindb as ln
import bionty as bt
import pandas as pd


df = pd.DataFrame(
    {
        "temperature": [37.2, 36.3, 38.2],
        "cell_type": ["cerebral pyramidal neuron", "astrocytic glia", "oligodendrocyte"],
        "assay_ontology_id": ["EFO:0008913", "EFO:0008913", "EFO:0008913"],
        "donor": ["D0001", "D0002", "D0003"]
    },
    index = ["obs1", "obs2", "obs3"]
)
df

Show code cell output Hide code cell output

→ connected lamindb: testuser1/test-curate

	temperature	cell_type	assay_ontology_id	donor
obs1	37.2	cerebral pyramidal neuron	EFO:0008913	D0001
obs2	36.3	astrocytic glia	EFO:0008913	D0002
obs3	38.2	oligodendrocyte	EFO:0008913	D0003

Define validation criteria and create a Curator object.

# in the dictionary, each key is a column name of the dataframe, and each value
# is a registry field onto which values are mapped
categoricals = {
    "cell_type": bt.CellType.name,
    "assay_ontology_id": bt.ExperimentalFactor.ontology_id,
    "donor": ln.ULabel.name,
}

# pass validation criteria
curate = ln.Curator.from_df(df, categoricals=categoricals)

The validate() method checks our data against the defined criteria. It identifies which values are already validated (exist in our registries) and which are potentially problematic (do not yet exist in our registries).

curate.validate()

# check the non-validated terms
curate.non_validated

{'cell_type': ['cerebral pyramidal neuron', 'astrocytic glia'],
 'donor': ['D0001', 'D0002', 'D0003']}

For cell_type, we saw that “cerebral pyramidal neuron”, “astrocytic glia” are not validated.

First, let’s standardize synonym “astrocytic glia” as suggested

curate.standardize("cell_type")

✓ standardized 1 synonym in "cell_type": "astrocytic glia" → "astrocyte"

# now we have only one non-validated term left
curate.non_validated

{'cell_type': ['cerebral pyramidal neuron'],
 'donor': ['D0001', 'D0002', 'D0003']}

For “cerebral pyramidal neuron”, let’s understand which cell type in the public ontology might be the actual match.

# to check the correct spelling of categories, pass `public=True` to get a lookup object from public ontologies
# use `lookup = curate.lookup()` to get a lookup object of existing records in your instance
lookup = curate.lookup(public=True)
lookup

# here is an example for the "cell_type" column
cell_types = lookup["cell_type"]
cell_types.cerebral_cortex_pyramidal_neuron

# fix the cell type
df.cell_type = df.cell_type.replace({"cerebral pyramidal neuron": cell_types.cerebral_cortex_pyramidal_neuron.name})

For donor, we want to add the new donors: “D0001”, “D0002”, “D0003”

# this adds donors that were _not_ validated
curate.add_new_from("donor")

# validate again
curate.validate()

Save a curated artifact.

artifact = curate.save_artifact(description="My curated dataframe")

! no run & transform got linked, call `ln.track()` & re-run

! run input wasn't tracked, call `ln.track()` and re-run

! 1 unique term (25.00%) is not validated for name: 'temperature'

! did not create Feature record for 1 non-validated name: 'temperature'

artifact.describe(print_types=True)

Artifact .parquet/DataFrame
├── General
│   ├── .uid = 'kqg4mqd2qzm5fJJN0000'
│   ├── .size = 3786
│   ├── .hash = 'LZCfO2VdCCz0bzQ2cGpDEw'
│   ├── .path = /home/runner/work/lamindb/lamindb/docs/test-curate/.lamindb/kqg4mqd2qzm5fJJN0000.parquet
│   ├── .created_by = testuser1 (Test User1)
│   └── .created_at = 2024-12-20 15:03:42
├── Dataset features/.feature_sets
│   └── columns • 3                 [Feature]                                                           
│       assay_ontology_id           cat[bionty.ExperimentalF…  single-cell RNA sequencing               
│       cell_type                   cat[bionty.CellType]       astrocyte, cerebral cortex pyramidal neu…
│       donor                       cat[ULabel]                D0001, D0002, D0003                      
└── Labels
    └── .cell_types                 bionty.CellType            oligodendrocyte, astrocyte, cerebral cor…
        .experimental_factors       bionty.ExperimentalFactor  single-cell RNA sequencing               
        .ulabels                    ULabel                     D0001, D0003, D0002

Curate an AnnData¶

Here we additionally specify which var_index to validate against.

import anndata as ad

X = pd.DataFrame(
    {
        "ENSG00000081059": [1, 2, 3], 
        "ENSG00000276977": [4, 5, 6], 
        "ENSG00000198851": [7, 8, 9], 
        "ENSG00000010610": [10, 11, 12], 
        "ENSG00000153563": [13, 14, 15],
        "ENSGcorrupted": [16, 17, 18]
    }, 
    index=df.index  # because we already curated the dataframe above, it will validate 
)
adata = ad.AnnData(X=X, obs=df)
adata

curate = ln.Curator.from_anndata(
    adata, 
    var_index=bt.Gene.ensembl_gene_id,  # validate var.index against Gene.ensembl_gene_id
    categoricals=categoricals, 
    organism="human",
)
curate.validate()

Non-validated terms can be accessed via:

curate.non_validated

Subset the AnnData to validated genes only:

adata_validated = adata[:, ~adata.var.index.isin(curate.non_validated["var_index"])].copy()

Now let’s validate the subsetted object:

curate = ln.Curator.from_anndata(
    adata_validated, 
    var_index=bt.Gene.ensembl_gene_id,  # validate var.index against Gene.ensembl_gene_id
    categoricals=categoricals, 
    organism="human",
)
curate.validate()

The validated object can be subsequently saved as an Artifact:

artifact = curate.save_artifact(description="test AnnData")

Saved artifact has been annotated with validated features and labels:

artifact.describe()

We’ve walked through the process of validating, standardizing, and annotating datasets going through these key steps:

Defining validation criteria
Validating data against existing registries
Adding new validated entries to registries
Annotating artifacts with validated metadata

By following these steps, you can ensure your data is standardized and well-curated.

If you have datasets that aren’t DataFrame-like or AnnData-like, read: Curate datasets of any format.