Standardize and append a dataset¶
Here, we’ll learn
how to standardize a less well curated dataset
how to append it to the growing versioned collection
import lamindb as ln
import bionty as bt
ln.track("ManDYgmftZ8C")
Show code cell output
→ connected lamindb: testuser1/test-scrna
→ created Transform('ManDYgmftZ8C0000'), started new Run('gdrx82vo...') at 2025-06-03 16:32:02 UTC
→ notebook imports: bionty==1.5.0 lamindb==1.6.1
Let’s now consider a less-well curated dataset:
adata = ln.core.datasets.anndata_pbmc68k_reduced()
# we don't trust the cell type annotation in this dataset
adata.obs.rename(columns={"cell_type": "cell_type_untrusted"}, inplace=True)
adata
Show code cell output
AnnData object with n_obs × n_vars = 70 × 765
obs: 'cell_type_untrusted', 'n_genes', 'percent_mito', 'louvain'
var: 'n_counts', 'highly_variable'
uns: 'louvain', 'louvain_colors', 'neighbors', 'pca'
obsm: 'X_pca', 'X_umap'
varm: 'PCs'
obsp: 'connectivities', 'distances'
Create a curator:
curator = ln.Curator.from_anndata(
adata,
var_index=bt.Gene.symbol,
categoricals={"cell_type_untrusted": bt.CellType.name},
organism="human",
)
curator.validate()
Show code cell output
! organism is ignored, define it on the dtype level
! indexing datasets with gene symbols can be problematic: https://docs.lamin.ai/faq/symbol-mapping
! 1 term not validated in feature 'columns': 'cell_type_untrusted'
→ fix typos, remove non-existent values, or save terms via: curator.cat.add_new_from('columns')
! 9 terms not validated in feature 'cell_type_untrusted': 'Dendritic cells', 'CD19+ B', 'CD4+/CD45RO+ Memory', 'CD8+ Cytotoxic T', 'CD4+/CD25 T Reg', 'CD14+ Monocytes', 'CD56+ NK', 'CD8+/CD45RA+ Naive Cytotoxic', 'CD34+'
→ fix typos, remove non-existent values, or save terms via: curator.cat.add_new_from('cell_type_untrusted')
! using default organism = human
! using default organism = human
! using default organism = human
! 65 terms not validated in feature 'var_index': 'ATPIF1', 'C1orf228', 'CCBL2', 'RP11-782C8.1', 'RP11-277L2.3', 'RP11-156E8.1', 'AC079767.4', 'H1FX', 'SELT', 'ATP5I', 'IGJ', 'CCDC109B', 'FYB', 'H2AFY', 'FAM65B', 'HIST1H4C', 'HIST1H1E', 'ZNRD1', 'C6orf48', 'RP3-467N11.1', ...
54 synonyms found: "ATPIF1" → "ATP5IF1", "C1orf228" → "ARMH1", "CCBL2" → "KYAT3", "AC079767.4" → "LINC01857", "H1FX" → "H1-10", "SELT" → "SELENOT", "ATP5I" → "ATP5ME", "IGJ" → "JCHAIN", "CCDC109B" → "MCUB", "FYB" → "FYB1", "H2AFY" → "MACROH2A1", "FAM65B" → "RIPOR2", "HIST1H4C" → "H4C3", "HIST1H1E" → "H1-4", "ZNRD1" → "POLR1H", "C6orf48" → "SNHG32", "SEPT7" → "SEPTIN7", "WBSCR22" → "BUD23", "RSBN1L-AS1" → "APTR", "CCDC132" → "VPS50", ...
→ curate synonyms via: .standardize("var_index")
for remaining terms:
→ fix typos, remove non-existent values, or save terms via: curator.cat.add_new_from('var_index')
False
Standardize & validate genes 
¶
Let’s convert Gene symbols to Ensembl ids via standardize(). Note that this is a non-unique mapping and the first match is kept because the keep parameter in .standardize() defaults to "first":
adata.var["ensembl_gene_id"] = bt.Gene.standardize(
adata.var.index,
field=bt.Gene.symbol,
return_field=bt.Gene.ensembl_gene_id,
organism="human",
)
# use ensembl_gene_id as the index
adata.var.index.name = "symbol"
adata.var = adata.var.reset_index().set_index("ensembl_gene_id")
# we only want to save data with validated genes
validated = bt.Gene.validate(adata.var.index, bt.Gene.ensembl_gene_id, mute=True)
adata_validated = adata[:, validated].copy()
Here, we’ll use .raw:
adata_validated.raw = adata.raw[:, validated].to_adata()
adata_validated.raw.var.index = adata_validated.var.index
curator = ln.Curator.from_anndata(
adata_validated,
var_index=bt.Gene.ensembl_gene_id,
categoricals={"cell_type_untrusted": bt.CellType.name},
organism="human",
)
Show code cell output
! organism is ignored, define it on the dtype level
curator.validate()
Show code cell output
! 9 terms not validated in feature 'cell_type_untrusted': 'Dendritic cells', 'CD19+ B', 'CD4+/CD45RO+ Memory', 'CD8+ Cytotoxic T', 'CD4+/CD25 T Reg', 'CD14+ Monocytes', 'CD56+ NK', 'CD8+/CD45RA+ Naive Cytotoxic', 'CD34+'
→ fix typos, remove non-existent values, or save terms via: curator.cat.add_new_from('cell_type_untrusted')
False
Standardize & validate cell types ¶
None of the cell type names are valid. We’ll now look up the non-validated cell types using the values of the public ontology and create a mapping.
curator.non_validated["cell_type_untrusted"]
Show code cell output
['Dendritic cells',
'CD19+ B',
'CD4+/CD45RO+ Memory',
'CD8+ Cytotoxic T',
'CD4+/CD25 T Reg',
'CD14+ Monocytes',
'CD56+ NK',
'CD8+/CD45RA+ Naive Cytotoxic',
'CD34+']
ct_public_lo = bt.CellType.public().lookup()
name_mapping = {
"Dendritic cells": ct_public_lo.dendritic_cell.name,
"CD19+ B": ct_public_lo.b_cell_cd19_positive.name,
"CD4+/CD45RO+ Memory": ct_public_lo.effector_memory_cd45ra_positive_alpha_beta_t_cell_terminally_differentiated.name,
"CD8+ Cytotoxic T": ct_public_lo.cd8_positive_alpha_beta_cytotoxic_t_cell.name,
"CD4+/CD25 T Reg": ct_public_lo.cd4_positive_cd25_positive_alpha_beta_regulatory_t_cell.name,
"CD14+ Monocytes": ct_public_lo.cd14_positive_monocyte.name,
"CD56+ NK": ct_public_lo.cd56_positive_cd161_positive_immature_natural_killer_cell_human.name,
"CD8+/CD45RA+ Naive Cytotoxic": ct_public_lo.cd8_positive_alpha_beta_memory_t_cell_cd45ro_positive.name,
"CD34+": ct_public_lo.cd34_positive_cd56_positive_cd117_positive_common_innate_lymphoid_precursor_human.name,
}
We can now standardize cell type names using the lookup-based mapper:
adata_validated.obs["cell_type_untrusted_original"] = adata_validated.obs[
"cell_type_untrusted"
] # copy the original annotations
adata_validated.obs["cell_type_untrusted"] = adata_validated.obs[
"cell_type_untrusted_original"
].map(name_mapping)
Now, all cell types are validated:
curator.validate()
Show code cell output
True
Register 
¶
artifact = curator.save_artifact(description="10x reference adata")
Show code cell output
! 4 unique terms (80.00%) are not validated for name: 'n_genes', 'percent_mito', 'louvain', 'cell_type_untrusted_original'
artifact.view_lineage()
Show code cell output
artifact.describe()
Artifact .h5ad/AnnData ├── General │ ├── .uid = '4bF9qwEBEaC73zEO0000' │ ├── .size = 859840 │ ├── .hash = '8cSIZsvUrKeGfL64-H9RLw' │ ├── .n_observations = 70 │ ├── .path = /home/runner/work/lamin-usecases/lamin-usecases/docs/test-scrna/.lamindb/4bF9qwEBEaC73zEO0000.h5ad │ ├── .created_by = testuser1 (Test User1) │ ├── .created_at = 2025-06-03 16:32:12 │ └── .transform = 'Standardize and append a dataset' ├── Dataset features │ ├── var • 754 [bionty.Gene] │ │ HES4 float │ │ TNFRSF4 float │ │ SSU72 float │ │ PARK7 float │ │ RBP7 float │ │ SRM float │ │ MAD2L2 float │ │ AGTRAP float │ │ TNFRSF1B float │ │ EFHD2 float │ │ NECAP2 float │ │ HP1BP3 float │ │ C1QA float │ │ C1QB float │ │ HNRNPR float │ │ GALE float │ │ STMN1 float │ │ CD52 float │ │ FGR float │ │ ATP5IF1 float │ └── obs • 1 [Feature] │ cell_type_untrusted cat[bionty.CellType] B cell, CD19-positive, CD14-positive mon… └── Labels └── .cell_types bionty.CellType CD8-positive, alpha-beta memory T cell, …
Re-curate¶
We review the dataset and find all annotations trustworthy up there being a 'CD38-positive naive B cell'.
Inspecting the name_mapping in detail tells us 'CD8+/CD45RA+ Naive Cytotoxic' was erroneously mapped on a B cell.
Let us correct this and create a 'cell_type' feature that we can now trust.
name_mapping["CD38-positive naive B cell"] = "cytotoxic T cell"
adata_validated.obs["cell_type"] = adata_validated.obs[
"cell_type_untrusted_original"
].map(name_mapping)
adata_validated.obs["cell_type"].unique()
['dendritic cell', 'B cell, CD19-positive', 'effector memory CD45RA-positive, alpha-beta T..., 'CD8-positive, alpha-beta cytotoxic T cell', 'CD4-positive, CD25-positive, alpha-beta regul..., 'CD14-positive monocyte', 'CD56-positive, CD161-positive immature natura..., 'CD8-positive, alpha-beta memory T cell, CD45R..., 'CD34-positive, CD56-positive, CD117-positive ...]
Categories (9, object): ['CD4-positive, CD25-positive, alpha-beta regul..., 'effector memory CD45RA-positive, alpha-beta T..., 'CD8-positive, alpha-beta cytotoxic T cell', 'CD8-positive, alpha-beta memory T cell, CD45R..., ..., 'B cell, CD19-positive', 'CD34-positive, CD56-positive, CD117-positive ..., 'CD56-positive, CD161-positive immature natura..., 'dendritic cell']
artifact_trusted = ln.Curator.from_anndata(
adata_validated,
var_index=bt.Gene.ensembl_gene_id,
categoricals={
"cell_type": bt.CellType.name,
"cell_type_untrusted": bt.CellType.name,
},
organism="human",
).save_artifact(
description="10x reference adata, trusted cell type annotation",
revises=artifact,
)
! organism is ignored, define it on the dtype level
→ returning existing schema with same hash: Schema(uid='suFSS7E6nE1YeOLL', n=754, is_type=False, itype='bionty.Gene', dtype='float', hash='cFusP7E9awXecG_aP3LCDA', minimal_set=True, ordered_set=False, maximal_set=False, branch_id=1, space_id=1, created_by_id=1, run_id=2, created_at=2025-06-03 16:32:13 UTC)
! 4 unique terms (66.70%) are not validated for name: 'n_genes', 'percent_mito', 'louvain', 'cell_type_untrusted_original'
artifact_trusted.describe()
Show code cell output
Artifact .h5ad/AnnData ├── General │ ├── .uid = '4bF9qwEBEaC73zEO0001' │ ├── .size = 857336 │ ├── .hash = 'GK721a-L-fGDI8kXefKMtA' │ ├── .n_observations = 70 │ ├── .path = /home/runner/work/lamin-usecases/lamin-usecases/docs/test-scrna/.lamindb/4bF9qwEBEaC73zEO0001.h5ad │ ├── .created_by = testuser1 (Test User1) │ ├── .created_at = 2025-06-03 16:32:14 │ └── .transform = 'Standardize and append a dataset' ├── Dataset features │ ├── var • 754 [bionty.Gene] │ │ HES4 float │ │ TNFRSF4 float │ │ SSU72 float │ │ PARK7 float │ │ RBP7 float │ │ SRM float │ │ MAD2L2 float │ │ AGTRAP float │ │ TNFRSF1B float │ │ EFHD2 float │ │ NECAP2 float │ │ HP1BP3 float │ │ C1QA float │ │ C1QB float │ │ HNRNPR float │ │ GALE float │ │ STMN1 float │ │ CD52 float │ │ FGR float │ │ ATP5IF1 float │ └── obs • 2 [Feature] │ cell_type cat[bionty.CellType] B cell, CD19-positive, CD14-positive mon… │ cell_type_untrusted cat[bionty.CellType] B cell, CD19-positive, CD14-positive mon… └── Labels └── .cell_types bionty.CellType CD8-positive, alpha-beta memory T cell, …
Append the dataset to the collection¶
Query the previous collection:
collection_v1 = ln.Collection.get(key="scrna/collection1")
Create a new version of the collection by sharding it across the new artifact and the artifact underlying version 1 of the collection:
collection_v2 = collection_v1.append(artifact_trusted).save()
If you want, you can label the collection’s version by setting .version.
collection_v2.version = "2"
collection_v2.save()
Show code cell output
Collection(uid='c7yFGvbjFzXi8ZtD0001', version='2', is_latest=True, key='scrna/collection1', hash='luH-jPb6eJLsXvc1TWGpUg', branch_id=1, space_id=1, created_by_id=1, run_id=2, created_at=2025-06-03 16:32:15 UTC)
Version 2 of the collection covers significantly more conditions.
collection_v2.describe()
Show code cell output
Collection └── General ├── .uid = 'c7yFGvbjFzXi8ZtD0001' ├── .key = 'scrna/collection1' ├── .hash = 'luH-jPb6eJLsXvc1TWGpUg' ├── .version = '2' ├── .created_by = testuser1 (Test User1) ├── .created_at = 2025-06-03 16:32:15 └── .transform = 'Standardize and append a dataset'
View data lineage:
collection_v2.view_lineage()
Show code cell output
