Jupyter Notebook Binder

Project flow

LaminDB allows tracking data lineage on the entire project level.

Here, we walk through exemplified app uploads, pipelines & notebooks following Schmidt et al., 2022.

A CRISPR screen reading out a phenotypic endpoint on T cells is paired with scRNA-seq to generate insights into IFN-γ production.

These insights get linked back to the original data through the steps taken in the project to provide context for interpretation & future decision making.

More specifically: Why should I care about data flow?

Data flow tracks data sources & transformations to trace biological insights, verify experimental outcomes, meet regulatory standards, increase the robustness of research and optimize the feedback loop of team-wide learning iterations.

While tracking data flow is easier when it’s governed by deterministic pipelines, it becomes hard when it’s governed by interactive human-driven analyses.

LaminDB interfaces workflow mangers for the former and embraces the latter.

# !pip install 'lamindb[jupyter,bionty,aws]'
!lamin init --storage ./mydata
Hide code cell output
 connected lamindb: testuser1/mydata

Import lamindb:

import lamindb as ln
from IPython.display import Image, display
 connected lamindb: testuser1/mydata

Steps

In the following, we walk through exemplified steps covering different types of transforms (Transform).

Note

The full notebooks are in this repository.

App upload of phenotypic data

Register data through app upload from wetlab by testuser1:

# This function mimics the upload of artifacts via the UI
# In reality, you simply drag and drop files into the UI
def mock_upload_crispra_result_app():
    ln.setup.login("testuser1")
    transform = ln.Transform(name="Upload GWS CRISPRa result", type="upload")
    ln.track(transform=transform)
    output_path = ln.core.datasets.schmidt22_crispra_gws_IFNG(ln.settings.storage.root)
    output_file = ln.Artifact(
        output_path, description="Raw data of schmidt22 crispra GWS"
    )
    output_file.save()


mock_upload_crispra_result_app()
Hide code cell output
 created Transform('V4RJqYty'), started new Run('0z4nIVfN') at 2024-12-03 08:36:12 UTC

Hit identification in notebook

Access, transform & register data in drylab by testuser2 in notebook hit-identification.

Hide code cell content
# the following mimics the integrated analysis notebook
# In reality, you would execute inside the notebook
import nbproject_test
from pathlib import Path

cwd = Path.cwd()
nbproject_test.execute_notebooks(
    cwd / "project-flow-scripts/hit-identification.ipynb", write=True
)
Executing notebooks in /home/runner/work/lamin-usecases/lamin-usecases/docs/project-flow-scripts/hit-identification.ipynb
Scheduled: ['hit-identification']
hit-identification 
✓ (3.770s)
Total time: 3.771s

Inspect data flow:

artifact = ln.Artifact.get(description="hits from schmidt22 crispra GWS")
artifact.view_lineage()
_images/3186e253606b39297ee606a3b5205961504b4ef854bf5f07140a9b7a35566a7e.svg

Sequencer upload

Upload files from sequencer via script chromium_10x_upload.py:

!python project-flow-scripts/chromium_10x_upload.py
Hide code cell output
 connected lamindb: testuser1/mydata
 created Transform('qCJPkOuZ'), started new Run('UTC51c3h') at 2024-12-03 08:36:18 UTC
 finished Run('UTC51c3h') after 0d 0h 0m 0s at 2024-12-03 08:36:18 UTC

scRNA-seq bioinformatics pipeline

Process uploaded files using a script or workflow manager: Pipelines – workflow managers and obtain 3 output files in a directory filtered_feature_bc_matrix/:

cellranger.py

!python project-flow-scripts/cellranger.py
Hide code cell output
 connected lamindb: testuser1/mydata
 created Transform('NwCqDYAd'), started new Run('IbAqrfud') at 2024-12-03 08:36:21 UTC

postprocess_cellranger.py

!python project-flow-scripts/postprocess_cellranger.py
Hide code cell output
 connected lamindb: testuser1/mydata
 created Transform('YqmbO6oM'), started new Run('98x9DVZ5') at 2024-12-03 08:36:23 UTC

Inspect data flow:

output_file = ln.Artifact.get(description="perturbseq counts")
output_file.view_lineage()
_images/0b6dd9002167419575899d819bdefa05760615841975a95e3616a6fa5da62f2d.svg

Integrate scRNA-seq & phenotypic data

Integrate data in notebook integrated-analysis.

Hide code cell content
# the following mimics the integrated analysis notebook
# In reality, you would execute inside the notebook
nbproject_test.execute_notebooks(
    cwd / "project-flow-scripts/integrated-analysis.ipynb", write=True
)
Executing notebooks in /home/runner/work/lamin-usecases/lamin-usecases/docs/project-flow-scripts/integrated-analysis.ipynb
Scheduled: ['integrated-analysis']
integrated-analysis 
✓ (4.199s)
Total time: 4.200s

Review results

Let’s load one of the plots:

# track the current notebook as transform
ln.track("1LCd8kco9lZU0000")
 created Transform('1LCd8kco'), started new Run('x5aZcXMB') at 2024-12-03 08:36:28 UTC
 notebook imports: ipython==8.30.0 lamindb==0.77.2 nbproject_test==0.5.1
artifact = ln.Artifact.get(key__contains="figures/matrixplot")
artifact.cache()
Hide code cell output
PosixUPath('/home/runner/work/lamin-usecases/lamin-usecases/docs/mydata/.lamindb/QYruahx3C3Cb8RDX0000.png')
display(Image(filename=artifact.path))
_images/539b6c935124adb8cc6e9b9179e15b68fa381c90f891111b6b1362461a875b48.png

We see that the image artifact is tracked as an input of the current notebook. The input is highlighted, the notebook follows at the bottom:

artifact.view_lineage()
_images/ba4145d6296eaed54c2169ea7a744850185e042db635508ca9f4148cbad45ba5.svg

Alternatively, we can also look at the sequence of transforms:

transform = ln.Transform.search("Project flow").first()
transform.predecessors.df()
uid name key description type source_code hash reference reference_type _source_code_artifact_id version is_latest created_at created_by_id
id
6 lB3IyPLQSmvt0000 Perform single cell analysis, integrate with C... integrated-analysis.ipynb None notebook None None None None None None True 2024-12-03 08:36:27.306959+00:00 2
transform.view_lineage()
_images/31d3360b99192dcb46e74d4972da0a867d854b3ceaee4917bbab54c269b6f2a2.svg

Understand runs

We tracked pipeline and notebook runs through track(), which stores a Transform and a Run record within a global context.

Artifact objects are the inputs and outputs of runs.

What if I don’t want a global context?

Sometimes, we don’t want to create a global run context but manually pass a run when creating an artifact:

run = ln.Run(transform=transform)
ln.Artifact(filepath, run=run)
When does an artifact appear as a run input?

When accessing an artifact via cache(), load() or open(), two things happen:

  1. The current run gets added to artifact.input_of

  2. The transform of that artifact gets added as a parent of the current transform

You can then switch off auto-tracking of run inputs if you set ln.settings.track_run_inputs = False: Can I disable tracking run inputs?

You can also track run inputs on a case by case basis via is_run_input=True, e.g., here:

artifact.load(is_run_input=True)

Query by provenance

We can query or search for the notebook that created the artifact:

transform = ln.Transform.search("GWS CRIPSRa analysis").first()

And then find all the artifacts created by that notebook:

ln.Artifact.filter(transform=transform).df()
uid key description suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id version is_latest run_id created_at created_by_id
id
2 2JUq7Vr5kTvxGXkF0000 None hits from schmidt22 crispra GWS .parquet dataset 16948 4OX9zhvxqzWicuZXALgoxw None None md5 DataFrame 1 True 1 2 None True 2 2024-12-03 08:36:16.538590+00:00 2

Which transform ingested a given artifact?

artifact = ln.Artifact.filter().first()
artifact.transform
Transform(uid='V4RJqYtyoFLy0000', is_latest=True, name='Upload GWS CRISPRa result', type='upload', created_by_id=1, created_at=2024-12-03 08:36:12 UTC)

And which user?

artifact.created_by
User(uid='DzTjkKse', handle='testuser1', name='Test User1', created_at=2024-12-03 08:36:10 UTC)

Which transforms were created by a given user?

users = ln.User.lookup()
ln.Transform.filter(created_by=users.testuser1).df()
uid name key description type source_code hash reference reference_type _source_code_artifact_id version is_latest created_at created_by_id
id
1 V4RJqYtyoFLy0000 Upload GWS CRISPRa result None None upload None None None None None None True 2024-12-03 08:36:12.507441+00:00 1
3 qCJPkOuZAi9q0000 chromium_10x_upload.py chromium_10x_upload.py None script import lamindb as ln\n\nln.setup.login("testus... nXWdh475QhVKuoAfToWZTw None None None None True 2024-12-03 08:36:18.613688+00:00 1
7 1LCd8kco9lZU0000 Project flow project-flow.ipynb None notebook None None None None None None True 2024-12-03 08:36:28.817659+00:00 1

Which notebooks were created by a given user?

ln.Transform.filter(created_by=users.testuser1, type="notebook").df()
uid name key description type source_code hash reference reference_type _source_code_artifact_id version is_latest created_at created_by_id
id
7 1LCd8kco9lZU0000 Project flow project-flow.ipynb None notebook None None None None None None True 2024-12-03 08:36:28.817659+00:00 1

We can also view all recent additions to the entire database:

ln.view()
Hide code cell output
Artifact
uid key description suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id version is_latest run_id created_at created_by_id
id
11 QYruahx3C3Cb8RDX0000 figures/matrixplot_fig2_score-wgs-hits-per-clu... None .png None 28814 CXsOQW6HNpVqS3ldTKInRw None None md5 None 1 True 1 6 None True 6 2024-12-03 08:36:28.018355+00:00 2
10 evWSxdUpzOKAOrE30000 figures/umap_fig1_score-wgs-hits.png None .png None 118999 yE6AUQNo3aNGAFQxCbslzQ None None md5 None 1 True 1 6 None True 6 2024-12-03 08:36:27.854481+00:00 2
9 WIR7UavcCUEUmWjr0000 schmidt22_perturbseq.h5ad perturbseq counts .h5ad None 20659936 la7EvqEUMDlug9-rpw-udA None None md5 AnnData 1 False 1 5 None True 5 2024-12-03 08:36:23.935635+00:00 2
8 0JTxFzbYgM4x2Vtu0000 perturbseq/filtered_feature_bc_matrix/barcodes... None .tsv.gz None 6 T4Mex807kGQ5RvPdCDLnEQ None None md5 None 1 False 1 4 None True 4 2024-12-03 08:36:21.451222+00:00 2
7 76DU9x6aECnfjAtG0000 perturbseq/filtered_feature_bc_matrix/matrix.m... None .mtx.gz None 6 yZ0DRLrh08TLqA-9IkEmjw None None md5 None 1 False 1 4 None True 4 2024-12-03 08:36:21.450738+00:00 2
6 0nH2ocIIUmkpSDhL0000 perturbseq/filtered_feature_bc_matrix/features... None .tsv.gz None 6 fTF8A7pwsNPmw2MKO9Q1FA None None md5 None 1 False 1 4 None True 4 2024-12-03 08:36:21.450053+00:00 2
4 hGwx5j9Xwi35M8KF0000 fastq/perturbseq_R2_001.fastq.gz None .fastq.gz None 6 cuUPL9LdSOAnQ_O5fr617A None None md5 None 1 False 1 3 None True 3 2024-12-03 08:36:18.958674+00:00 1
Run
uid started_at finished_at is_consecutive reference reference_type transform_id report_id environment_id parent_id created_at created_by_id
id
1 0z4nIVfNmzSFOGZUQImg 2024-12-03 08:36:12.509838+00:00 NaT True None None 1 None NaN None 2024-12-03 08:36:12.509866+00:00 1
2 OkgEhqYleEj9m5oP9mmg 2024-12-03 08:36:16.150588+00:00 NaT True None None 2 None NaN None 2024-12-03 08:36:16.150616+00:00 2
3 UTC51c3hhjgcgMbjjQ3T 2024-12-03 08:36:18.615934+00:00 2024-12-03 08:36:18.968014+00:00 True None None 3 None 5.0 None 2024-12-03 08:36:18.615960+00:00 1
4 IbAqrfuddHbAcjyowujI 2024-12-03 08:36:21.097538+00:00 NaT None None None 4 None NaN None 2024-12-03 08:36:21.097563+00:00 2
5 98x9DVZ5flU1S7MKyEWL 2024-12-03 08:36:23.108915+00:00 NaT None None None 5 None NaN None 2024-12-03 08:36:23.108942+00:00 2
6 XMjACX0lsJCBIJZXxAVi 2024-12-03 08:36:27.310447+00:00 NaT True None None 6 None NaN None 2024-12-03 08:36:27.310478+00:00 2
7 x5aZcXMBOKvhjGmxNdEk 2024-12-03 08:36:28.821084+00:00 NaT True None None 7 None NaN None 2024-12-03 08:36:28.821115+00:00 1
Storage
uid root description type region instance_uid run_id created_at created_by_id
id
1 jREwktSU7QRJ /home/runner/work/lamin-usecases/lamin-usecase... None local None 54ZGqgkROOFf None 2024-12-03 08:36:10.786988+00:00 1
Transform
uid name key description type source_code hash reference reference_type _source_code_artifact_id version is_latest created_at created_by_id
id
7 1LCd8kco9lZU0000 Project flow project-flow.ipynb None notebook None None None None None None True 2024-12-03 08:36:28.817659+00:00 1
6 lB3IyPLQSmvt0000 Perform single cell analysis, integrate with C... integrated-analysis.ipynb None notebook None None None None None None True 2024-12-03 08:36:27.306959+00:00 2
5 YqmbO6oMXjRj0000 postprocess_cellranger.py postprocess_cellranger.py None script None None None None None None True 2024-12-03 08:36:23.106282+00:00 2
4 NwCqDYAdtC9L0000 Cell Ranger None None pipeline None None https://www.10xgenomics.com/support/software/c... None None 7.2.0 True 2024-12-03 08:36:21.095378+00:00 2
3 qCJPkOuZAi9q0000 chromium_10x_upload.py chromium_10x_upload.py None script import lamindb as ln\n\nln.setup.login("testus... nXWdh475QhVKuoAfToWZTw None None None None True 2024-12-03 08:36:18.613688+00:00 1
2 T0T28btuB0PG0000 GWS CRIPSRa analysis hit-identification.ipynb None notebook None None None None None None True 2024-12-03 08:36:16.147643+00:00 2
1 V4RJqYtyoFLy0000 Upload GWS CRISPRa result None None upload None None None None None None True 2024-12-03 08:36:12.507441+00:00 1
User
uid handle name created_at
id
2 bKeW4T6E testuser2 Test User2 2024-12-03 08:36:16.145227+00:00
1 DzTjkKse testuser1 Test User1 2024-12-03 08:36:10.784404+00:00
Hide code cell content
!lamin login testuser1
!rm -r ./mydata
!lamin delete --force mydata
 logged in with email [email protected] (uid: DzTjkKse)
 deleting instance testuser1/mydata