Analysis flow¶

Here, we’ll track typical data transformations like subsetting that occur during analysis.

If exploring more generally, read this first: Project flow.

# !pip install 'lamindb[jupyter,bionty]'
!lamin init --storage ./analysis-usecase --schema bionty

import lamindb as ln
import bionty as bt
from lamin_utils import logger

→ connected lamindb: testuser1/analysis-usecase

Register an initial dataset¶

Here we register an initial artifact with a pipeline script register_example_file.py.

!python analysis-flow-scripts/register_example_file.py

Pull the registered dataset, apply a transformation, and register the result¶

Track the current notebook:

ln.track("eNef4Arw8nNM0000")

artifact = ln.Artifact.get(description="anndata with obs")
artifact.describe()

Get a backed AnnData object¶

adata = artifact.open()
adata

Subset dataset to specific cell types and diseases¶

cell_types = artifact.cell_types.all().lookup(return_field="name")
diseases = artifact.diseases.all().lookup(return_field="name")

Create the subset:

subset_obs = adata.obs.cell_type.isin(
    [cell_types.t_cell, cell_types.hematopoietic_stem_cell]
) & (adata.obs.disease.isin([diseases.liver_lymphoma, diseases.chronic_kidney_disease]))

adata_subset = adata[subset_obs]
adata_subset

adata_subset.obs[["cell_type", "disease"]].value_counts()

Register the subsetted AnnData:

curate = ln.Curator.from_anndata(
    adata_subset.to_memory(),
    var_index=bt.Gene.ensembl_gene_id,
    categoricals={
        "cell_type": bt.CellType.name,
        "disease": bt.Disease.name,
        "tissue": bt.Tissue.name,
    },
    organism="human",
)

curate.validate()

artifact = curate.save_artifact(description="anndata with obs subset")

artifact.describe()

Examine data flow¶

Query a subsetted .h5ad artifact containing “hematopoietic stem cell” and “T cell”:

cell_types = bt.CellType.lookup()

my_subset = ln.Artifact.filter(
    suffix=".h5ad",
    description__endswith="subset",
    cell_types__in=[
        cell_types.hematopoietic_stem_cell,
        cell_types.t_cell,
    ],
).first()
my_subset

Common questions that might arise are:

What is the history of this artifact?
Which features and labels are associated with it?
Which notebook analyzed and registered this artifact?
By whom?
And which artifact is its parent?

Let’s answer this using LaminDB:

print("--> What is the history of this artifact?\n")
artifact.view_lineage()

print("\n\n--> Which features and labels are associated with it?\n")
logger.print(artifact.features)
logger.print(artifact.labels)

print("\n\n--> Which notebook analyzed and registered this artifact\n")
logger.print(artifact.transform)

print("\n\n--> By whom\n")
logger.print(artifact.created_by)

print("\n\n--> And which artifact is its parent\n")
display(artifact.run.input_artifacts.df())

--> What is the history of this artifact?

_images/3bf803593063392fe5de9567ee5f9d416cf23a452cb0d1b490eedaacb7bc5ea5.svg

--> Which features and labels are associated with it?

  Features
    'cell_type' = 'T cell', 'hematopoietic stem cell'
    'disease' = 'chronic kidney disease', 'liver lymphoma'
    'tissue' = 'kidney', 'liver'
  Feature sets
    'var' = 'TSPAN6', 'TNMD', 'DPM1', 'SCYL3', 'FIRRM', 'FGR', 'CFH', 'FUCA2', 'GCLC', 'NFYA', 'STPG1', 'NIPAL3', 'LAS1L', 'ENPP4', 'SEMA3F', 'CFTR', 'ANKIB1', 'CYP51A1', 'KRIT1', 'RAD52'
    'obs' = 'cell_type', 'cell_type_id', 'tissue', 'disease'

  Labels
    .tissues = 'kidney', 'liver'
    .cell_types = 'T cell', 'hematopoietic stem cell'
    .diseases = 'chronic kidney disease', 'liver lymphoma'

--> Which notebook analyzed and registered this artifact

Transform(uid='eNef4Arw8nNM0000', is_latest=True, name='Analysis flow', key='analysis-flow.ipynb', type='notebook', created_by_id=1, created_at=2024-10-04 09:29:22 UTC)

--> By whom

User(uid='DzTjkKse', handle='testuser1', name='Test User1', created_at=2024-10-04 09:29:13 UTC)

--> And which artifact is its parent

	uid	version	is_latest	description	key	suffix	type	size	hash	n_objects	n_observations	_hash_type	_accessor	visibility	_key_is_virtual	storage_id	transform_id	run_id	created_by_id	updated_at
id
1	IqESa1pW1IcGmz6C0000	None	True	anndata with obs	None	.h5ad	dataset	46992	IJORtcQUSS11QBqD-nTD0A	None	40	md5	AnnData	1	True	1	1	1	1	2024-10-04 09:29:21.640501+00:00