Project flow

LaminDB allows tracking data lineage on the entire project level.

Here, we walk through exemplified app uploads, pipelines & notebooks following Schmidt et al., 2022.

A CRISPR screen reading out a phenotypic endpoint on T cells is paired with scRNA-seq to generate insights into IFN-γ production.

These insights get linked back to the original data through the steps taken in the project to provide context for interpretation & future decision making.

More specifically: Why should I care about data flow?

Data flow tracks data sources & transformations to trace biological insights, verify experimental outcomes, meet regulatory standards, increase the robustness of research and optimize the feedback loop of team-wide learning iterations.

While tracking data flow is easier when it’s governed by deterministic pipelines, it becomes hard when it’s governed by interactive human-driven analyses.

LaminDB interfaces workflow mangers for the former and embraces the latter.

# !pip install 'lamindb[jupyter,bionty,aws]'
!lamin init --storage ./mydata

Show code cell output Hide code cell output

→ initialized lamindb: testuser1/mydata

Import lamindb:

import lamindb as ln
from IPython.display import Image, display

→ connected lamindb: testuser1/mydata

Steps

In the following, we walk through exemplified steps covering different types of transforms (Transform).

Note

The full notebooks are in this repository.

App upload of phenotypic data

Register data through app upload from wetlab by testuser1:

# This function mimics the upload of artifacts via the UI
# In reality, you simply drag and drop files into the UI
def mock_upload_crispra_result_app():
    ln.setup.login("testuser1")
    transform = ln.Transform(name="Upload GWS CRISPRa result", type="upload")
    ln.track(transform=transform)
    output_path = ln.core.datasets.schmidt22_crispra_gws_IFNG(ln.settings.storage.root)
    output_file = ln.Artifact(
        output_path, description="Raw data of schmidt22 crispra GWS"
    )
    output_file.save()


mock_upload_crispra_result_app()

Show code cell output Hide code cell output

/tmp/ipykernel_3722/3297532022.py:5: FutureWarning: `name` will be removed soon, please pass 'Upload GWS CRISPRa result' to `key` instead
  transform = ln.Transform(name="Upload GWS CRISPRa result", type="upload")

→ created Transform('iU6zGKzpYYxv0000'), started new Run('SZelolI8...') at 2025-07-29 19:28:03 UTC

• recommendation: to identify the script across renames, pass the uid: ln.track("iU6zGKzpYYxv")

Hit identification in notebook

Access, transform & register data in drylab by testuser2 in notebook hit-identification.

Show code cell content Hide code cell content

# the following mimics the integrated analysis notebook
# In reality, you would execute inside the notebook
import nbproject_test
from pathlib import Path

cwd = Path.cwd()
nbproject_test.execute_notebooks(
    cwd / "project-flow-scripts/hit-identification.ipynb", write=True
)

Inspect data flow:

artifact = ln.Artifact.get(description="hits from schmidt22 crispra GWS")
artifact.view_lineage()

Sequencer upload

Upload files from sequencer via script chromium_10x_upload.py:

!python project-flow-scripts/chromium_10x_upload.py

scRNA-seq bioinformatics pipeline

Process uploaded files using a script or workflow manager: Pipelines – workflow managers and obtain 3 output files in a directory filtered_feature_bc_matrix/:

cellranger.py

!python project-flow-scripts/cellranger.py

postprocess_cellranger.py

!python project-flow-scripts/postprocess_cellranger.py

Inspect data flow:

output_file = ln.Artifact.get(description="perturbseq counts")
output_file.view_lineage()

Integrate scRNA-seq & phenotypic data

Integrate data in notebook integrated-analysis.

Show code cell content Hide code cell content

# the following mimics the integrated analysis notebook
# In reality, you would execute inside the notebook
nbproject_test.execute_notebooks(
    cwd / "project-flow-scripts/integrated-analysis.ipynb", write=True
)

Review results

Let’s load one of the plots:

# track the current notebook as transform
ln.track("1LCd8kco9lZU0000")

artifact = ln.Artifact.get(key__contains="figures/matrixplot")
artifact.cache()

Show code cell output Hide code cell output

PosixUPath('/home/runner/work/lamin-usecases/lamin-usecases/docs/mydata/.lamindb/PKHH2MtvLd1mDRZl0000.png')

display(Image(filename=artifact.path))

We see that the image artifact is tracked as an input of the current notebook. The input is highlighted, the notebook follows at the bottom:

artifact.view_lineage()

Alternatively, we can also look at the sequence of transforms:

transform = ln.Transform.search("Project flow").first()
transform.predecessors.df()

	uid	id	key	description	type	source_code	hash	reference	reference_type	space_id	_template_id	version	is_latest	created_at	created_by_id	_aux	branch_id

transform.view_lineage()

Understand runs

We tracked pipeline and notebook runs through lamindb.core.Context.track, which stores a Transform and a Run record within a global context.

Artifact objects are the inputs and outputs of runs.

What if I don’t want a global context?

Sometimes, we don’t want to create a global run context but manually pass a run when creating an artifact:

run = ln.Run(transform=transform)
ln.Artifact(filepath, run=run)

When does an artifact appear as a run input?

When accessing an artifact via cache(), load() or open(), two things happen:

1. The current run gets added to artifact.input_of

2. The transform of that artifact gets added as a parent of the current transform

You can then switch off auto-tracking of run inputs if you set ln.settings.track_run_inputs = False: Can I disable tracking run inputs?

You can also track run inputs on a case by case basis via is_run_input=True, e.g., here:

artifact.load(is_run_input=True)

Query by provenance

We can query or search for the notebook that created the artifact:

transform = ln.Transform.search("GWS CRIPSRa analysis").first()

And then find all the artifacts created by that notebook:

ln.Artifact.filter(transform=transform).df()

	uid	key	description	suffix	kind	otype	size	hash	n_files	n_observations	_hash_type	_key_is_virtual	_overwrite_versions	space_id	storage_id	schema_id	version	is_latest	run_id	created_at	created_by_id	_aux	branch_id
id
2	7PQBwxvxNl27nRUr0000	None	hits from schmidt22 crispra GWS	.parquet	dataset	DataFrame	17075	biAkiZprbg59E06i7wFGOA	None	123	md5	True	False	1	1	None	None	True	2	2025-07-29 19:28:09.615000+00:00	2	{'af': {'0': True}}	1

Which transform ingested a given artifact?

artifact = ln.Artifact.filter().first()
artifact.transform

Transform(uid='iU6zGKzpYYxv0000', is_latest=True, key='Upload GWS CRISPRa result', type='upload', branch_id=1, space_id=1, created_by_id=1, created_at=2025-07-29 19:28:03 UTC)

And which user?

artifact.created_by

<User: User object (1)>

Which transforms were created by a given user?

users = ln.User.lookup()

ln.Transform.filter(created_by=users.testuser1).df()

	uid	key	description	type	source_code	hash	reference	reference_type	space_id	_template_id	version	is_latest	created_at	created_by_id	_aux	branch_id
id
1	iU6zGKzpYYxv0000	Upload GWS CRISPRa result	None	upload	None	None	None	None	1	None	None	True	2025-07-29 19:28:03.894000+00:00	1	None	1
3	qCJPkOuZAi9q0000	chromium_10x_upload.py	chromium_10x_upload.py	script	import lamindb as ln\n\nln.setup.login("testus...	nXWdh475QhVKuoAfToWZTw	None	None	1	None	None	True	2025-07-29 19:28:12.867000+00:00	1	None	1
7	1LCd8kco9lZU0000	project-flow.ipynb	Project flow	notebook	None	None	None	None	1	None	None	True	2025-07-29 19:28:26.560000+00:00	1	None	1

Which notebooks were created by a given user?

ln.Transform.filter(created_by=users.testuser1, type="notebook").df()

	uid	key	description	type	source_code	hash	reference	reference_type	space_id	_template_id	version	is_latest	created_at	created_by_id	_aux	branch_id
id
7	1LCd8kco9lZU0000	project-flow.ipynb	Project flow	notebook	None	None	None	None	1	None	None	True	2025-07-29 19:28:26.560000+00:00	1	None	1

We can also view all recent additions to the entire database:

ln.view()

Show code cell output Hide code cell output

	uid	key	description	suffix	kind	otype	size	hash	n_files	n_observations	_hash_type	_key_is_virtual	_overwrite_versions	space_id	storage_id	schema_id	version	is_latest	run_id	created_at	created_by_id	_aux	branch_id
id
12	PKHH2MtvLd1mDRZl0000	figures/matrixplot_fig2_score-wgs-hits-per-clu...	None	.png	None	None	28815	DF_kMhhFqw-Lad8zK9u4Sw	None	None	md5	True	False	1	1	None	None	True	6	2025-07-29 19:28:25.215000+00:00	2	{'af': {'0': True}}	1
11	PN2ocjfpkbbd0QkP0000	figures/umap_fig1_score-wgs-hits.png	None	.png	None	None	119000	BXPM4Ap5dGUESVHXgs-7Xg	None	None	md5	True	False	1	1	None	None	True	6	2025-07-29 19:28:25.043000+00:00	2	{'af': {'0': True}}	1
10	4ppH2Ba0IqKb8zqM0000	schmidt22_perturbseq.h5ad	perturbseq counts	.h5ad	None	AnnData	20659936	la7EvqEUMDlug9-rpw-udA	None	None	md5	False	False	1	1	None	None	True	5	2025-07-29 19:28:19.836000+00:00	2	None	1
9	XPfI8MjhwjLanZkm0000	perturbseq/filtered_feature_bc_matrix/features...	None	.tsv.gz	None	None	6	MAhSasOy91oTUornnX1WAw	None	None	md5	False	False	1	1	None	None	True	4	2025-07-29 19:28:16.643000+00:00	2	None	1
8	bn3MhiUunX0Fwx000000	perturbseq/filtered_feature_bc_matrix/matrix.m...	None	.mtx.gz	None	None	6	gJSfu3_7bpay78gP6bxtRg	None	None	md5	False	False	1	1	None	None	True	4	2025-07-29 19:28:16.642000+00:00	2	None	1
7	oojkJvziFa9HNqaX0000	perturbseq/filtered_feature_bc_matrix/barcodes...	None	.tsv.gz	None	None	6	3BKVt9yiIEWnIzolQH6vzA	None	None	md5	False	False	1	1	None	None	True	4	2025-07-29 19:28:16.641000+00:00	2	None	1
5	jOGO4OnryYeQwrxc0000	fastq/perturbseq_R2_001.fastq.gz	None	.fastq.gz	None	None	6	AcXlVLs4AvYB7GyqPbYD6g	None	None	md5	False	False	1	1	None	None	True	3	2025-07-29 19:28:13.525000+00:00	1	None	1

	uid	name	started_at	finished_at	reference	reference_type	_is_consecutive	_status_code	space_id	transform_id	report_id	_logfile_id	environment_id	initiated_by_run_id	created_at	created_by_id	_aux	branch_id
id
1	SZelolI8tWwayWCn	None	2025-07-29 19:28:03.897250+00:00	NaT	None	None	None	-1	1	1	NaN	None	NaN	None	2025-07-29 19:28:03.897000+00:00	1	None	1
2	kXbd5Hb1DEpYBasy	None	2025-07-29 19:28:09.182840+00:00	NaT	None	None	None	-1	1	2	NaN	None	NaN	None	2025-07-29 19:28:09.183000+00:00	2	None	1
3	qEFtxbPuhRUGwCB9	None	2025-07-29 19:28:12.869186+00:00	2025-07-29 19:28:13.527019+00:00	None	None	True	0	1	3	6.0	None	3.0	None	2025-07-29 19:28:12.869000+00:00	1	None	1
4	podTtZd99j0d2h7v	None	2025-07-29 19:28:16.256075+00:00	NaT	None	None	None	-1	1	4	NaN	None	NaN	None	2025-07-29 19:28:16.256000+00:00	2	None	1
5	SMXMxwH5Vxk6BRoJ	None	2025-07-29 19:28:18.552843+00:00	NaT	None	None	True	-1	1	5	NaN	None	3.0	None	2025-07-29 19:28:18.553000+00:00	2	None	1
6	EEh519jVEGrb21DJ	None	2025-07-29 19:28:24.139030+00:00	NaT	None	None	None	-1	1	6	NaN	None	NaN	None	2025-07-29 19:28:24.139000+00:00	2	None	1
7	nW1ksuipkyVQiPf2	None	2025-07-29 19:28:26.563787+00:00	NaT	None	None	None	-1	1	7	NaN	None	NaN	None	2025-07-29 19:28:26.564000+00:00	1	None	1

	uid	root	description	type	region	instance_uid	space_id	run_id	created_at	created_by_id	_aux	branch_id
id
1	4ne1S6rLzD6h	/home/runner/work/lamin-usecases/lamin-usecase...	None	local	None	54ZGqgkROOFf	1	None	2025-07-29 19:28:00.982000+00:00	1	None	1

	uid	key	description	type	source_code	hash	reference	reference_type	space_id	_template_id	version	is_latest	created_at	created_by_id	_aux	branch_id
id
7	1LCd8kco9lZU0000	project-flow.ipynb	Project flow	notebook	None	None	None	None	1	None	None	True	2025-07-29 19:28:26.560000+00:00	1	None	1
6	lB3IyPLQSmvt0000	integrated-analysis.ipynb	Perform single cell analysis, integrate with C...	notebook	None	None	None	None	1	None	None	True	2025-07-29 19:28:24.133000+00:00	2	None	1
5	YqmbO6oMXjRj0000	postprocess_cellranger.py	postprocess_cellranger.py	script	import lamindb as ln\n\n\n# Post-process 3 cel...	A7Fg8VHna7y4ZnGWGPEwVw	None	None	1	None	None	True	2025-07-29 19:28:18.551000+00:00	2	None	1
4	tk4fZ2NBtGnR0000	Cell Ranger	None	pipeline	None	None	https://www.10xgenomics.com/support/software/c...	None	1	None	7.2.0	True	2025-07-29 19:28:16.254000+00:00	2	None	1
3	qCJPkOuZAi9q0000	chromium_10x_upload.py	chromium_10x_upload.py	script	import lamindb as ln\n\nln.setup.login("testus...	nXWdh475QhVKuoAfToWZTw	None	None	1	None	None	True	2025-07-29 19:28:12.867000+00:00	1	None	1
2	T0T28btuB0PG0000	hit-identification.ipynb	GWS CRIPSRa analysis	notebook	None	None	None	None	1	None	None	True	2025-07-29 19:28:09.179000+00:00	2	None	1
1	iU6zGKzpYYxv0000	Upload GWS CRISPRa result	None	upload	None	None	None	None	1	None	None	True	2025-07-29 19:28:03.894000+00:00	1	None	1

Show code cell content Hide code cell content

!lamin login testuser1
!rm -r ./mydata
!lamin delete --force mydata