scrna2/6 Jupyter Notebook lamindata

Standardize and append a batch of data

Here, we’ll learn

  • how to standardize a less well curated collection

  • how to append it to the growing versioned collection

import lamindb as ln
import bionty as bt

ln.settings.verbosity = "hint"
bt.settings.auto_save_parents = False
ln.settings.transform.stem_uid = "ManDYgmftZ8C"
ln.settings.transform.version = "1"
ln.track()
💡 connected lamindb: testuser1/test-scrna
💡 notebook imports: bionty==0.44.0 lamindb==0.74.1
💡 saved: Transform(uid='ManDYgmftZ8C5zKv', version='1', name='Standardize and append a batch of data', key='scrna2', type='notebook', created_by_id=1, updated_at='2024-07-01 13:57:06 UTC')
💡 saved: Run(uid='qEyJkVeYUpejvE3WjMbD', transform_id=2, created_by_id=1)
💡 tracked pip freeze > /home/runner/.cache/lamindb/run_env_pip_qEyJkVeYUpejvE3WjMbD.txt
Run(uid='qEyJkVeYUpejvE3WjMbD', started_at='2024-07-01 13:57:06 UTC', is_consecutive=True, transform_id=2, created_by_id=1)

Let’s now consider a less-well curated dataset:

adata = ln.core.datasets.anndata_pbmc68k_reduced()
adata
Hide code cell output
AnnData object with n_obs × n_vars = 70 × 765
    obs: 'cell_type', 'n_genes', 'percent_mito', 'louvain'
    var: 'n_counts', 'highly_variable'
    uns: 'louvain', 'louvain_colors', 'neighbors', 'pca'
    obsm: 'X_pca', 'X_umap'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'

We are still working with human data, and can globally set an organism:

bt.settings.organism = "human"
annotate = ln.Annotate.from_anndata(adata, var_index=bt.Gene.symbol, categoricals={adata.obs.cell_type.name: bt.CellType.name})
3 non-validated categories are not saved in Feature.name: ['percent_mito', 'n_genes', 'louvain']!
      → to lookup categories, use lookup().columns
      → to save, run add_new_from_columns
✅ added 5 records from public with Gene.symbol for var_index: 'GPX1', 'SOD2', 'RN7SL1', 'SNORD3B-2', 'IGLL5'
11 non-validated categories are not saved in Gene.symbol: ['RP11-782C8.1', 'RP11-277L2.3', 'RP11-156E8.1', 'RP3-467N11.1', 'RP11-390E23.6', 'RP11-489E7.4', 'RP11-291B21.2', 'RP11-620J15.3', 'TMBIM4-1', 'AC084018.1', 'CTD-3138B18.5']!
      → to lookup categories, use lookup().var_index
      → to save, run add_new_from_var_index

Standardize & validate genes

Let’s convert Gene symbols to Ensembl ids via standardize(). Note that this is a non-unique mapping and the first match is kept because the keep parameter in .standardize() defaults to "first":

adata.var["ensembl_gene_id"] = bt.Gene.standardize(
    adata.var.index,
    field=bt.Gene.symbol,
    return_field=bt.Gene.ensembl_gene_id,
)
# use ensembl_gene_id as the index
adata.var.index.name = "symbol"
adata.var = adata.var.reset_index().set_index("ensembl_gene_id")

# we only want to save data with validated genes
validated = bt.Gene.validate(adata.var.index, bt.Gene.ensembl_gene_id, mute=True)
adata_validated = adata[:, validated].copy()
💡 standardized 754/765 terms

Here, we’ll use .raw:

adata_validated.raw = adata.raw[:, validated].to_adata()
adata_validated.raw.var.index = adata_validated.var.index
annotate = ln.Annotate.from_anndata(adata_validated, var_index=bt.Gene.ensembl_gene_id, categoricals={"cell_type": bt.CellType.name})
3 non-validated categories are not saved in Feature.name: ['percent_mito', 'n_genes', 'louvain']!
      → to lookup categories, use lookup().columns
      → to save, run add_new_from_columns
annotate.validate()
✅ var_index is validated against Gene.ensembl_gene_id
💡 mapping cell_type on CellType.name
9 terms are not validated: 'Dendritic cells', 'CD19+ B', 'CD4+/CD45RO+ Memory', 'CD8+ Cytotoxic T', 'CD4+/CD25 T Reg', 'CD14+ Monocytes', 'CD56+ NK', 'CD8+/CD45RA+ Naive Cytotoxic', 'CD34+'
      → save terms via .add_new_from('cell_type')
False

Standardize & validate cell types

Since none of the cell types are validate, let us search the cell type names from the public ontology, and add the name found in the AnnData object as a synonym to the top match found in the public ontology.

bionty = bt.CellType.public()  # access the public ontology through bionty
name_mapper = {}
for name in adata_validated.obs.cell_type.unique():
    # search the public ontology and use the ontology id of the top match
    ontology_id = bionty.search(name).iloc[0].ontology_id
    # create a record by loading the top match from bionty
    record = bt.CellType.from_public(ontology_id=ontology_id)
    name_mapper[name] = record.name  # map the original name to standardized name
    record.save()
    record.add_synonym(name)
Hide code cell output
✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0001087'
✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0000910'
✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0000919'
✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0002397'
✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0002101'

We can now standardize cell type names using the search-based mapper:

adata_validated.obs.cell_type = adata_validated.obs.cell_type.map(name_mapper)

Now, all cell types are validated:

annotate.validate()
✅ var_index is validated against Gene.ensembl_gene_id
✅ cell_type is validated against CellType.name
True

Register

artifact = annotate.save_artifact(description="10x reference adata")
💡 path content will be copied to default storage upon `save()` with key `None` ('.lamindb/0Ex8iQ4KF5ccsaFhsNpO.h5ad')
✅ storing artifact '0Ex8iQ4KF5ccsaFhsNpO' at '/home/runner/work/lamin-usecases/lamin-usecases/docs/test-scrna/.lamindb/0Ex8iQ4KF5ccsaFhsNpO.h5ad'
💡 parsing feature names of X stored in slot 'var'
754 terms (100.00%) are validated for ensembl_gene_id
✅    linked: FeatureSet(uid='dKCCIuUzXuAiI5vUEaG1', n=754, dtype='float', registry='bionty.Gene', hash='j8QkIeLBgJwsscY4vVPx', created_by_id=1, run_id=2)
💡 parsing feature names of slot 'obs'
1 term (25.00%) is validated for name
3 terms (75.00%) are not validated for name: n_genes, percent_mito, louvain
✅    linked: FeatureSet(uid='tygmfcuYan6j9gKbeyon', n=1, registry='Feature', hash='xf9283gZNz1D6jVz0My2', created_by_id=1, run_id=2)
✅ saved 2 feature sets for slots: 'var','obs'
artifact.view_lineage()
_images/5971310d975aacb0d4b25d452e54e105ac31d941b881326a2e46b5de5982db89.svg

Append the dataset to the collection

Query the previous collection:

collection_v1 = ln.Collection.filter(
    name="My versioned scRNA-seq collection", version="1"
).one()

Create a new version of the collection by sharding it across the new artifact and the artifact underlying version 1 of the collection:

collection_v2 = ln.Collection(
    [artifact, collection_v1.artifacts[0]],
    is_new_version_of=collection_v1,
)
collection_v2.save()
Hide code cell output
💡 adding collection ids [1] as inputs for run 2, adding parent transform 1
💡 adding artifact ids [1] as inputs for run 2, adding parent transform 1
✅ saved 1 feature set for slot: 'var'

Version 2 of the collection covers significantly more conditions.

collection_v2.describe()
Collection(uid='TJr1y2nZfcXtGnXDdTcs', version='2', name='My versioned scRNA-seq collection', hash='Umjxg4HR1wkZqKROsyz1', visibility=1, updated_at='2024-07-01 13:57:26 UTC')
  Provenance
    .created_by = 'testuser1'
    .transform = 'Standardize and append a batch of data'
    .run = '2024-07-01 13:57:06 UTC'
  Feature sets
    'obs' = 'donor', 'tissue', 'cell_type', 'assay'
    'var' = 'MIR1302-2HG', 'FAM138A', 'OR4F5', 'None', 'OR4F29', 'OR4F16', 'LINC01409', 'FAM87B', 'LINC01128', 'LINC00115', 'FAM41C'

View data lineage:

collection_v2.view_lineage()
_images/3d31dae8f77bf7e4a62eacf5d73f1d0f9ed4d1b3d6ac3c7794a10138e928739e.svg