Analyze a collection in memory

Here, we’ll analyze the growing collection by loading it into memory. This is only possible if it’s not too large. If your data is large, you’ll likely want to iterate over the collection to train a model, the topic of the next page ().

import lamindb as ln
import bionty as bt

💡 connected lamindb: testuser1/test-scrna

ln.settings.transform.stem_uid = "mfWKm8OtAzp8"
ln.settings.transform.version = "1"
ln.track()

💡 notebook imports: bionty==0.44.3 lamindb==0.74.3 scanpy==1.10.2

💡 saved: Transform(uid='mfWKm8OtAzp85zKv', version='1', name='Analyze a collection in memory', key='scrna4', type='notebook', created_by_id=1, updated_at='2024-07-26 12:22:33 UTC')

💡 saved: Run(uid='CtqGrrWtIbzQ5eWgTyPS', transform_id=4, created_by_id=1)

Run(uid='CtqGrrWtIbzQ5eWgTyPS', started_at='2024-07-26 12:22:33 UTC', is_consecutive=True, transform_id=4, created_by_id=1)

ln.Collection.df()

	uid	version	name	description	hash	reference	reference_type	visibility	transform_id	artifact_id	run_id	created_by_id	updated_at
id
2	7hRDhzTTUWLHntihz8tr	2	My versioned scRNA-seq collection	None	Umjxg4HR1wkZqKROsyz1	None	None	1	2	None	2	1	2024-07-26 12:22:23.068212+00:00
1	7hRDhzTTUWLHntihhDN6	1	My versioned scRNA-seq collection	None	exJtsBYH53iiebYH-Qx0	None	None	1	1	None	1	1	2024-07-26 12:21:50.499619+00:00

collection = ln.Collection.filter(
    name="My versioned scRNA-seq collection", version="2"
).one()

collection.artifacts.df()

	uid	version	description	key	suffix	type	accessor	size	hash	hash_type	n_objects	n_observations	visibility	key_is_virtual	storage_id	transform_id	run_id	created_by_id	updated_at
id
2	k67H9jNu2MLlWWa7pkpZ	None	10x reference adata	None	.h5ad	dataset	AnnData	857752	PnpU6XI5Fbzwc49XgrgdNg	md5	None	70	1	True	1	2	2	1	2024-07-26 12:22:19.985907+00:00
1	7jwarJoEsuhyU2IeT2kC	None	Human immune cells from Conde22	None	.h5ad	dataset	AnnData	57612943	9sXda5E7BYiVoDOQkTC0KB	sha1-fl	None	1648	1	True	1	1	1	1	2024-07-26 12:21:49.424710+00:00

If the collection isn’t too large, we can now load it into memory.

Under-the-hood, the AnnData objects are concatenated during loading.

The amount of time this takes depends on a variety of factors.

If it occurs often, one might consider storing a concatenated version of the collection, rather than the individual pieces.

adata = collection.load()

The default is an outer join during concatenation as in pandas:

adata

AnnData object with n_obs × n_vars = 1718 × 36508
    obs: 'cell_type', 'n_genes', 'percent_mito', 'louvain', 'donor', 'tissue', 'assay', 'artifact_uid'
    obsm: 'X_pca', 'X_umap'

The AnnData has the reference to the individual artifacts in the .obs annotations:

adata.obs.artifact_uid.cat.categories

Index(['k67H9jNu2MLlWWa7pkpZ', '7jwarJoEsuhyU2IeT2kC'], dtype='object')

We can easily obtain ensemble IDs for gene symbols using the look up object:

genes = bt.Gene.lookup(field="symbol")

genes.itm2b.ensembl_gene_id

'ENSG00000136156'

Let us create a plot:

import scanpy as sc

sc.pp.pca(adata, n_comps=2)

sc.pl.pca(
    adata,
    color=genes.itm2b.ensembl_gene_id,
    title=(
        f"{genes.itm2b.symbol} / {genes.itm2b.ensembl_gene_id} /"
        f" {genes.itm2b.description}"
    ),
    save="_itm2b",
)

WARNING: saving figure to file figures/pca_itm2b.pdf

We could save a plot as a pdf and then see it in the flow diagram:

artifact = ln.Artifact("./figures/pca_itm2b.pdf", description="My result on ITM2B")
artifact.save()
artifact.view_lineage()

Show code cell output Hide code cell output

But given the image is part of the notebook, we can also rely on the report that we create when saving the notebook:

ln.finish()

To see the current notebook, visit: lamin.ai/laminlabs/lamindata/transform/mfWKm8OtAzp8z8