Analysis flow¶

Here, we’ll track typical data transformations like subsetting that occur during analysis.

# !pip install 'lamindb[jupyter,bionty]'
!lamin init --storage ./analysis-flow --schema bionty

import lamindb as ln
import bionty as bt

→ connected lamindb: testuser1/analysis-flow

Save an initial dataset¶

register_example_file.py¶

import lamindb as ln
import bionty as bt

ln.track("K4wsS5DTYdFp0000")

# an example dataset that has a few cell type, tissue and disease annotations
adata = ln.core.datasets.anndata_with_obs()

# validate and register features
curate = ln.Curator.from_anndata(
    adata,
    var_index=bt.Gene.ensembl_gene_id,
    categoricals={
        "cell_type": bt.CellType.name,
        "cell_type_id": bt.CellType.ontology_id,
        "tissue": bt.Tissue.name,
        "disease": bt.Disease.name,
    },
    organism="human",
)
curate.add_new_from("cell_type")
curate.validate()
curate.save_artifact(description="anndata with obs")

ln.finish()

!python analysis-flow-scripts/register_example_file.py

Open a dataset, subset it, and register the result¶

Track the current notebook:

ln.track("eNef4Arw8nNM0000")

artifact = ln.Artifact.get(description="anndata with obs")
artifact.describe()

Get a backed AnnData object¶

adata = artifact.open()
adata

Subset dataset to specific cell types and diseases¶

cell_types = artifact.cell_types.all().distinct().lookup(return_field="name")
diseases = artifact.diseases.all().distinct().lookup(return_field="name")

Create the subset:

subset_obs = adata.obs.cell_type.isin(
    [cell_types.t_cell, cell_types.hematopoietic_stem_cell]
) & (adata.obs.disease.isin([diseases.liver_lymphoma, diseases.chronic_kidney_disease]))

adata_subset = adata[subset_obs]
adata_subset

adata_subset.obs[["cell_type", "disease"]].value_counts()

Register the subsetted AnnData:

curate = ln.Curator.from_anndata(
    adata_subset.to_memory(),
    var_index=bt.Gene.ensembl_gene_id,
    categoricals={
        "cell_type": bt.CellType.name,
        "disease": bt.Disease.name,
        "tissue": bt.Tissue.name,
    },
    organism="human",
)
curate.validate()

artifact = curate.save_artifact(description="anndata with obs subset")
artifact.describe()

Examine data lineage¶

Query a subsetted .h5ad artifact containing “hematopoietic stem cell” and “T cell”:

cell_types = bt.CellType.lookup()

my_subset = ln.Artifact.filter(
    suffix=".h5ad",
    description__endswith="subset",
    cell_types__in=[
        cell_types.hematopoietic_stem_cell,
        cell_types.t_cell,
    ],
).first()
my_subset

Common questions that might arise are:

What is the history of this artifact?
Which features and labels are associated with it?
Which notebook analyzed and registered this artifact?
By whom?
And which artifact is its parent?

Let’s answer this using LaminDB:

print("--> What is the lineage of this artifact?\n")
artifact.view_lineage()

print("\n\n--> Which features and labels are associated with it?\n")
print(artifact.features)
print(artifact.labels)

print("\n\n--> Which notebook analyzed and saved this artifact\n")
print(artifact.transform)

print("\n\n--> Who save this artifact?\n")
print(artifact.created_by)

print("\n\n--> Which artifacts were inputs?\n")
display(artifact.run.input_artifacts.df())

--> What is the lineage of this artifact?

_images/e84698b7cbfd3d977e48777d45f55749e831c756bcdba70cd31b265bab02c51e.svg

--> Which features and labels are associated with it?

  Features
    'cell_type' = 'T cell', 'hematopoietic stem cell'
    'disease' = 'chronic kidney disease', 'liver lymphoma'
    'tissue' = 'kidney', 'liver'
  Feature sets
    'var' = 'TSPAN6', 'TNMD', 'DPM1', 'SCYL3', 'FIRRM', 'FGR', 'CFH', 'FUCA2', 'GCLC', 'NFYA', 'STPG1', 'NIPAL3', 'LAS1L', 'ENPP4', 'SEMA3F', 'CFTR', 'ANKIB1', 'CYP51A1', 'KRIT1', 'RAD52'
    'obs' = 'cell_type', 'cell_type_id', 'tissue', 'disease'

  Labels
    .tissues = 'liver', 'kidney'
    .cell_types = 'hematopoietic stem cell', 'T cell'
    .diseases = 'liver lymphoma', 'chronic kidney disease'



--> Which notebook analyzed and saved this artifact

Transform(uid='eNef4Arw8nNM0000', is_latest=True, name='Analysis flow', key='analysis-flow.ipynb', type='notebook', created_by_id=1, created_at=2024-11-21 06:57:28 UTC)


--> Who save this artifact?

User(uid='DzTjkKse', handle='testuser1', name='Test User1', created_at=2024-11-21 06:57:19 UTC)


--> Which artifacts were inputs?

	uid	version	is_latest	description	key	suffix	type	size	hash	n_objects	n_observations	_hash_type	_accessor	visibility	_key_is_virtual	storage_id	transform_id	run_id	created_at	created_by_id
id
1	3ur15azB5SSYkYcc0000	None	True	anndata with obs	None	.h5ad	dataset	46992	IJORtcQUSS11QBqD-nTD0A	None	40	md5	AnnData	1	True	1	1	1	2024-11-21 06:57:27.156126+00:00	1