Jupyter Notebook Binder

Analysis flow

Here, we’ll track typical data transformations like subsetting that occur during analysis.

If exploring more generally, read this first: Project flow.

Setup

# a lamindb instance containing Bionty schema
!lamin init --storage ./analysis-usecase --schema bionty
Hide code cell output
💡 connected lamindb: testuser1/analysis-usecase
import lamindb as ln
import bionty as bt
from lamin_utils import logger

bt.settings.auto_save_parents = False
💡 connected lamindb: testuser1/analysis-usecase

Register an initial dataset

Here we register an initial artifact with a pipeline script register_example_file.py.

!python analysis-flow-scripts/register_example_file.py
Hide code cell output
💡 connected lamindb: testuser1/analysis-usecase
💡 saved: Transform(uid='K4wsS5DTYdFp6K79', version='0', name='register_example_file.py', key='register_example_file.py', type='script', created_by_id=1, updated_at='2024-06-19 23:20:45 UTC')
💡 saved: Run(uid='fKwMWxEDd8f9oQBsBDoY', transform_id=1, created_by_id=1)
✅ added 3 records with Feature.name for columns: 'cell_type', 'tissue', 'disease'
❗ 1 non-validated categories are not saved in Feature.name: ['cell_type_id']!
      → to lookup categories, use lookup().columns
      → to save, run add_new_from_columns
✅ added 99 records from public with Gene.ensembl_gene_id for var_index: 'ENSG00000000003', 'ENSG00000000005', 'ENSG00000000419', 'ENSG00000000457', 'ENSG00000000460', 'ENSG00000000938', 'ENSG00000000971', 'ENSG00000001036', 'ENSG00000001084', 'ENSG00000001167', 'ENSG00000001460', 'ENSG00000001461', 'ENSG00000001497', 'ENSG00000001561', 'ENSG00000001617', 'ENSG00000001626', 'ENSG00000001629', 'ENSG00000001630', 'ENSG00000001631', 'ENSG00000002016', ...
💡 saving labels for 'cell_type'
✅ added 3 records from public with CellType.name for cell_type: 'T cell', 'hematopoietic stem cell', 'hepatocyte'
❗ 1 non-validated categories are not saved in CellType.name: ['my new cell type']!
      → to lookup categories, use lookup().cell_type
      → to save, run .add_new_from('cell_type')
💡 saving labels for 'tissue'
✅ added 4 records from public with Tissue.name for tissue: 'kidney', 'liver', 'heart', 'brain'
💡 saving labels for 'disease'
✅ added 4 records from public with Disease.name for disease: 'chronic kidney disease', 'liver lymphoma', 'cardiac ventricle disorder', 'Alzheimer disease'
✅ added 1 record with CellType.name for cell_type: 'my new cell type'
✅ var_index is validated against Gene.ensembl_gene_id
✅ cell_type is validated against CellType.name
✅ tissue is validated against Tissue.name
✅ disease is validated against Disease.name
💡 path content will be copied to default storage upon `save()` with key `None` ('.lamindb/YrE1gP61FaZZe8fgC6Nh.h5ad')
✅ storing artifact 'YrE1gP61FaZZe8fgC6Nh' at '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase/.lamindb/YrE1gP61FaZZe8fgC6Nh.h5ad'
💡 parsing feature names of X stored in slot 'var'
✅    99 terms (100.00%) are validated for ensembl_gene_id
✅    linked: FeatureSet(uid='O0sk5XareVwviclbMu8o', n=99, dtype='float', registry='bionty.Gene', hash='-frOq7J0bik-J7Ad9DX7', created_by_id=1, run_id=1)
💡 parsing feature names of slot 'obs'
✅    3 terms (75.00%) are validated for name
❗    1 term (25.00%) is not validated for name: cell_type_id
✅    linked: FeatureSet(uid='ywlbw3DUtRPA4llYeKu0', n=3, registry='Feature', hash='uegnMyjHzK4VMhNY56FO', created_by_id=1, run_id=1)
✅ saved 2 feature sets for slots: 'var','obs'

Pull the registered dataset, apply a transformation, and register the result

Track the current notebook:

ln.settings.transform.stem_uid = "eNef4Arw8nNM"
ln.settings.transform.version = "0"
ln.track()
💡 notebook imports: bionty==0.44.0 lamin_utils==0.13.2 lamindb==0.74.0
💡 saved: Transform(uid='eNef4Arw8nNM6K79', version='0', name='Analysis flow', key='analysis-flow', type='notebook', created_by_id=1, updated_at='2024-06-19 23:21:40 UTC')
💡 saved: Run(uid='Qn3M2HjchMiVGqYXBAT5', transform_id=2, created_by_id=1)
Run(uid='Qn3M2HjchMiVGqYXBAT5', started_at='2024-06-19 23:21:40 UTC', is_consecutive=True, transform_id=2, created_by_id=1)
artifact = ln.Artifact.filter(description="anndata with obs").one()
artifact.describe()
Artifact(uid='YrE1gP61FaZZe8fgC6Nh', description='anndata with obs', suffix='.h5ad', type='dataset', accessor='AnnData', size=46992, hash='IJORtcQUSS11QBqD-nTD0A', hash_type='md5', n_observations=40, visibility=1, key_is_virtual=True, updated_at='2024-06-19 23:21:39 UTC')
  Provenance
    .created_by = 'testuser1'
    .storage = '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase'
    .transform = 'register_example_file.py'
    .run = '2024-06-19 23:20:45 UTC'
  Labels
    .tissues = 'kidney', 'liver', 'heart', 'brain'
    .cell_types = 'T cell', 'hematopoietic stem cell', 'hepatocyte', 'my new cell type'
    .diseases = 'chronic kidney disease', 'liver lymphoma', 'cardiac ventricle disorder', 'Alzheimer disease'
  Features
    'tissue' = 'kidney', 'liver', 'heart', 'brain'
    'cell_type' = 'T cell', 'hematopoietic stem cell', 'hepatocyte', 'my new cell type'
    'disease' = 'chronic kidney disease', 'liver lymphoma', 'cardiac ventricle disorder', 'Alzheimer disease'
  Feature sets
    'var' = 'TSPAN6', 'TNMD', 'DPM1', 'SCYL3', 'FIRRM', 'FGR', 'CFH', 'FUCA2', 'GCLC', 'NFYA', 'STPG1', 'NIPAL3', 'LAS1L', 'ENPP4', 'SEMA3F', 'CFTR', 'ANKIB1', 'CYP51A1', 'KRIT1', 'RAD52'
    'obs' = 'cell_type', 'tissue', 'disease'

Get a backed AnnData object

adata = artifact.backed()
adata
AnnDataAccessor object with n_obs × n_vars = 40 × 100
  constructed for the AnnData object YrE1gP61FaZZe8fgC6Nh.h5ad
    obs: ['_index', 'cell_type', 'cell_type_id', 'disease', 'tissue']
    var: ['_index']

Subset dataset to specific cell types and diseases

cell_types = artifact.cell_types.all().lookup(return_field="name")
diseases = artifact.diseases.all().lookup(return_field="name")

Create the subset:

subset_obs = adata.obs.cell_type.isin(
    [cell_types.t_cell, cell_types.hematopoietic_stem_cell]
) & (adata.obs.disease.isin([diseases.liver_lymphoma, diseases.chronic_kidney_disease]))
adata_subset = adata[subset_obs]
adata_subset
AnnDataAccessorSubset object with n_obs × n_vars = 20 × 100
  obs: ['_index', 'cell_type', 'cell_type_id', 'disease', 'tissue']
  var: ['_index']
adata_subset.obs[["cell_type", "disease"]].value_counts()
cell_type                disease               
T cell                   chronic kidney disease    10
hematopoietic stem cell  liver lymphoma            10
dtype: int64

Register the subsetted AnnData:

annotate = ln.Annotate.from_anndata(
    adata_subset.to_memory(), 
    var_index=bt.Gene.ensembl_gene_id, 
    categoricals={
        "cell_type": bt.CellType.name, 
        "disease": bt.Disease.name, 
        "tissue": bt.Tissue.name,
    },
    organism="human"
)

annotate.validate()
Hide code cell output
1 non-validated categories are not saved in Feature.name: ['cell_type_id']!
      → to lookup categories, use lookup().columns
      → to save, run add_new_from_columns
✅ var_index is validated against Gene.ensembl_gene_id
✅ cell_type is validated against CellType.name
✅ disease is validated against Disease.name
✅ tissue is validated against Tissue.name
/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/anndata/_core/anndata.py:1820: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
True
artifact = annotate.save_artifact(description="anndata with obs subset")
Hide code cell output
💡 path content will be copied to default storage upon `save()` with key `None` ('.lamindb/qUKtuyFwADptX6XJHh0j.h5ad')
✅ storing artifact 'qUKtuyFwADptX6XJHh0j' at '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase/.lamindb/qUKtuyFwADptX6XJHh0j.h5ad'
💡 parsing feature names of X stored in slot 'var'
99 terms (100.00%) are validated for ensembl_gene_id
✅    linked: FeatureSet(uid='O0sk5XareVwviclbMu8o', n=99, dtype='float', registry='bionty.Gene', hash='-frOq7J0bik-J7Ad9DX7', created_by_id=1, run_id=1)
💡 parsing feature names of slot 'obs'
3 terms (75.00%) are validated for name
1 term (25.00%) is not validated for name: cell_type_id
✅    linked: FeatureSet(uid='ywlbw3DUtRPA4llYeKu0', n=3, registry='Feature', hash='uegnMyjHzK4VMhNY56FO', created_by_id=1, run_id=1)
artifact.describe()
Artifact(uid='qUKtuyFwADptX6XJHh0j', description='anndata with obs subset', suffix='.h5ad', type='dataset', accessor='AnnData', size=38992, hash='RgGUx7ndRplZZSmalTAWiw', hash_type='md5', n_observations=20, visibility=1, key_is_virtual=True, updated_at='2024-06-19 23:21:41 UTC')
  Provenance
    .created_by = 'testuser1'
    .storage = '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase'
    .transform = 'Analysis flow'
    .run = '2024-06-19 23:21:40 UTC'
  Labels
    .tissues = 'kidney', 'liver'
    .cell_types = 'T cell', 'hematopoietic stem cell'
    .diseases = 'chronic kidney disease', 'liver lymphoma'
  Features
    'tissue' = 'kidney', 'liver'
    'cell_type' = 'T cell', 'hematopoietic stem cell'
    'disease' = 'chronic kidney disease', 'liver lymphoma'
  Feature sets
    'var' = 'TSPAN6', 'TNMD', 'DPM1', 'SCYL3', 'FIRRM', 'FGR', 'CFH', 'FUCA2', 'GCLC', 'NFYA', 'STPG1', 'NIPAL3', 'LAS1L', 'ENPP4', 'SEMA3F', 'CFTR', 'ANKIB1', 'CYP51A1', 'KRIT1', 'RAD52'
    'obs' = 'cell_type', 'tissue', 'disease'

Examine data flow

Query a subsetted .h5ad artifact containing “hematopoietic stem cell” and “T cell”:

cell_types = bt.CellType.lookup()
my_subset = ln.Artifact.filter(
    suffix=".h5ad",
    description__endswith="subset",
    cell_types__in=[
        cell_types.hematopoietic_stem_cell,
        cell_types.t_cell,
    ],
).first()
my_subset
Artifact(uid='qUKtuyFwADptX6XJHh0j', description='anndata with obs subset', suffix='.h5ad', type='dataset', accessor='AnnData', size=38992, hash='RgGUx7ndRplZZSmalTAWiw', hash_type='md5', n_observations=20, visibility=1, key_is_virtual=True, created_by_id=1, storage_id=1, transform_id=2, run_id=2, updated_at='2024-06-19 23:21:41 UTC')

Common questions that might arise are:

  • What is the history of this artifact?

  • Which features and labels are associated with it?

  • Which notebook analyzed and registered this artifact?

  • By whom?

  • And which artifact is its parent?

Let’s answer this using LaminDB:

print("--> What is the history of this artifact?\n")
artifact.view_lineage()

print("\n\n--> Which features and labels are associated with it?\n")
logger.print(artifact.features)
logger.print(artifact.labels)

print("\n\n--> Which notebook analyzed and registered this artifact\n")
logger.print(artifact.transform)

print("\n\n--> By whom\n")
logger.print(artifact.created_by)

print("\n\n--> And which artifact is its parent\n")
display(artifact.run.input_artifacts.df())
--> What is the history of this artifact?
_images/7292cdad33e4f82e3353a8009884309867e1af531917294c49d080ed70acdd3b.svg
--> Which features and labels are associated with it?
  Features
    'tissue' = 'kidney', 'liver'
    'cell_type' = 'T cell', 'hematopoietic stem cell'
    'disease' = 'chronic kidney disease', 'liver lymphoma'
  Feature sets
    'var' = 'TSPAN6', 'TNMD', 'DPM1', 'SCYL3', 'FIRRM', 'FGR', 'CFH', 'FUCA2', 'GCLC', 'NFYA', 'STPG1', 'NIPAL3', 'LAS1L', 'ENPP4', 'SEMA3F', 'CFTR', 'ANKIB1', 'CYP51A1', 'KRIT1', 'RAD52'
    'obs' = 'cell_type', 'tissue', 'disease'
  Labels
    .tissues = 'kidney', 'liver'
    .cell_types = 'T cell', 'hematopoietic stem cell'
    .diseases = 'chronic kidney disease', 'liver lymphoma'
--> Which notebook analyzed and registered this artifact

Transform(uid='eNef4Arw8nNM6K79', version='0', name='Analysis flow', key='analysis-flow', type='notebook', created_by_id=1, updated_at='2024-06-19 23:21:40 UTC')
--> By whom

User(uid='DzTjkKse', handle='testuser1', name='Test User1', updated_at='2024-06-19 23:20:42 UTC')
--> And which artifact is its parent
uid version description key suffix type accessor size hash hash_type n_objects n_observations visibility key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
1 YrE1gP61FaZZe8fgC6Nh None anndata with obs None .h5ad dataset AnnData 46992 IJORtcQUSS11QBqD-nTD0A md5 None 40 1 True 1 1 1 1 2024-06-19 23:21:39.187143+00:00
Hide code cell content
!lamin delete --force analysis-usecase
!rm -r ./analysis-usecase
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.14/x64/bin/lamin", line 8, in <module>
    sys.exit(main())
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/rich_click/rich_command.py", line 367, in __call__
    return super().__call__(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/rich_click/rich_command.py", line 152, in main
    rv = self.invoke(ctx)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/lamin_cli/__main__.py", line 103, in delete
    return delete(instance, force=force)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/lamindb_setup/_delete.py", line 98, in delete
    n_objects = check_storage_is_empty(
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/lamindb_setup/core/upath.py", line 779, in check_storage_is_empty
    raise InstanceNotEmpty(message)
lamindb_setup.core.upath.InstanceNotEmpty: Storage /home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase/.lamindb contains 4 objects ('_is_initialized' ignored) - delete them prior to deleting the instance
['/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase/.lamindb/GZPopc615T5XwSOKiDKv.txt', '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase/.lamindb/OUl9oZb1V5kP1SXSYz2V.py', '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase/.lamindb/YrE1gP61FaZZe8fgC6Nh.h5ad', '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase/.lamindb/_is_initialized', '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase/.lamindb/qUKtuyFwADptX6XJHh0j.h5ad']