Jupyter Notebook Binder

Project flow

LaminDB allows tracking data lineage on the entire project level.

Here, we walk through exemplified app uploads, pipelines & notebooks following Schmidt et al., 2022.

A CRISPR screen reading out a phenotypic endpoint on T cells is paired with scRNA-seq to generate insights into IFN-γ production.

These insights get linked back to the original data through the steps taken in the project to provide context for interpretation & future decision making.

More specifically: Why should I care about data flow?

Data flow tracks data sources & transformations to trace biological insights, verify experimental outcomes, meet regulatory standards, increase the robustness of research and optimize the feedback loop of team-wide learning iterations.

While tracking data flow is easier when it’s governed by deterministic pipelines, it becomes hard when it’s governed by interactive human-driven analyses.

LaminDB interfaces workflow mangers for the former and embraces the latter.

Setup

Init a test instance:

!lamin init --storage ./mydata
Hide code cell output
💡 connected lamindb: testuser1/mydata

Import lamindb:

import lamindb as ln
from IPython.display import Image, display
💡 connected lamindb: testuser1/mydata

Steps

In the following, we walk through exemplified steps covering different types of transforms (Transform).

Note

The full notebooks are in this repository.

App upload of phenotypic data

Register data through app upload from wetlab by testuser1:

# This function mimics the upload of artifacts via the UI
# In reality, you simply drag and drop files into the UI
def mock_upload_crispra_result_app():
    ln.setup.login("testuser1")
    transform = ln.Transform(name="Upload GWS CRISPRa result", type="upload")
    ln.track(transform=transform)
    output_path = ln.core.datasets.schmidt22_crispra_gws_IFNG(ln.settings.storage)
    output_file = ln.Artifact(
        output_path, description="Raw data of schmidt22 crispra GWS"
    )
    output_file.save()

mock_upload_crispra_result_app()
Hide code cell output
💡 saved: Transform(uid='2QUzBRcGLiNY6QEg', name='Upload GWS CRISPRa result', type='upload', created_by_id=1, updated_at='2024-06-19 23:21:49 UTC')
💡 saved: Run(uid='4cncFn7mwnvcniFjmiuu', transform_id=1, created_by_id=1)

Hit identification in notebook

Access, transform & register data in drylab by testuser2 in notebook hit-identification.

Hide code cell content
# the following mimics the integrated analysis notebook
# In reality, you would execute inside the notebook
import nbproject_test
from pathlib import Path

cwd = Path.cwd()
nbproject_test.execute_notebooks(cwd / "project-flow-scripts/hit-identification.ipynb", write=True)
Executing notebooks in /home/runner/work/lamin-usecases/lamin-usecases/docs/project-flow-scripts/hit-identification.ipynb
Scheduled: ['hit-identification']
hit-identification 
✓ (4.262s)
Total time: 4.263s

Inspect data flow:

artifact = ln.Artifact.filter(description="hits from schmidt22 crispra GWS").one()
artifact.view_lineage()
_images/9c25329064d968c41bdbed227a662122ff2a9be41f7cb9253ae7dbea2d4914c5.svg

Sequencer upload

Upload files from sequencer via script chromium_10x_upload.py:

!python project-flow-scripts/chromium_10x_upload.py
Hide code cell output
💡 connected lamindb: testuser1/mydata
💡 saved: Transform(uid='qCJPkOuZAi9q5zKv', version='1', name='chromium_10x_upload.py', key='chromium_10x_upload.py', type='script', created_by_id=1, updated_at='2024-06-19 23:21:56 UTC')
💡 saved: Run(uid='jA5XCIDJHSMHaY6wF0AP', transform_id=3, created_by_id=1)

scRNA-seq bioinformatics pipeline

Process uploaded files using a script or workflow manager: Pipelines – workflow managers and obtain 3 output files in a directory filtered_feature_bc_matrix/:

cellranger.py

!python project-flow-scripts/cellranger.py
Hide code cell output
💡 connected lamindb: testuser1/mydata
💡 saved: Transform(uid='97rTO159DwIrwFVU', version='7.2.0', name='Cell Ranger', type='pipeline', reference='https://www.10xgenomics.com/support/software/cell-ranger/7.2', created_by_id=2, updated_at='2024-06-19 23:21:59 UTC')
💡 saved: Run(uid='kN6utGNryMtDwJ4W4KP6', transform_id=4, created_by_id=2)
❗ this creates one artifact per file in the directory - you might simply call ln.Artifact(dir) to get one artifact for the entire directory

postprocess_cellranger.py

!python project-flow-scripts/postprocess_cellranger.py
Hide code cell output
💡 connected lamindb: testuser1/mydata
💡 saved: Transform(uid='YqmbO6oMXjRj65cN', version='2', name='postprocess_cellranger.py', key='postprocess_cellranger.py', type='script', created_by_id=2, updated_at='2024-06-19 23:22:01 UTC')
💡 saved: Run(uid='LaGdvrKKog47fCJhTCMk', transform_id=5, created_by_id=2)

Inspect data flow:

output_file = ln.Artifact.filter(description="perturbseq counts").one()
output_file.view_lineage()
_images/2509b878f832cf3f37e374d1cd8c04d47f8b871edb0b9a2be69da670938d2231.svg

Integrate scRNA-seq & phenotypic data

Integrate data in notebook integrated-analysis.

Hide code cell content
# the following mimics the integrated analysis notebook
# In reality, you would execute inside the notebook
nbproject_test.execute_notebooks(cwd / "project-flow-scripts/integrated-analysis.ipynb", write=True)
Executing notebooks in /home/runner/work/lamin-usecases/lamin-usecases/docs/project-flow-scripts/integrated-analysis.ipynb
Scheduled: ['integrated-analysis']
integrated-analysis 
✓ (4.179s)
Total time: 4.181s

Review results

Let’s load one of the plots:

# track the current notebook as transform
ln.settings.transform.stem_uid = "1LCd8kco9lZU"
ln.settings.transform.version = "0"
ln.track()
💡 notebook imports: ipython==8.25.0 lamindb==0.74.0 nbproject_test==0.5.1
💡 saved: Transform(uid='1LCd8kco9lZU6K79', version='0', name='Project flow', key='project-flow', type='notebook', created_by_id=1, updated_at='2024-06-19 23:22:10 UTC')
💡 saved: Run(uid='PzNjPmKWiL5DbppOrH0U', transform_id=7, created_by_id=1)
Run(uid='PzNjPmKWiL5DbppOrH0U', started_at='2024-06-19 23:22:10 UTC', is_consecutive=True, transform_id=7, created_by_id=1)
artifact = ln.Artifact.filter(key__contains="figures/matrixplot").one()
artifact.cache()
Hide code cell output
PosixUPath('/home/runner/work/lamin-usecases/lamin-usecases/docs/mydata/.lamindb/sit20d3h6LNMKkvLEMJO.png')
display(Image(filename=artifact.path))

We see that the image artifact is tracked as an input of the current notebook. The input is highlighted, the notebook follows at the bottom:

artifact.view_lineage()
_images/3029b8c984ebb6b5bbd3016ef83a3182dc854577fb2875920ad424d9a6c38172.svg

Alternatively, we can also look at the sequence of transforms:

transform = ln.Transform.search("Project flow").first()
transform.parents.df()
uid version name key description type reference reference_type latest_report_id source_code_id created_by_id updated_at
id
6 lB3IyPLQSmvt5zKv 1 Perform single cell analysis, integrate with C... integrated-analysis None notebook None None None None 2 2024-06-19 23:22:08.315721+00:00
transform.view_parents()
_images/e222515fa9f14f3da6fd139b9995142a91787c1da177f7fa37dadc1cb7398561.svg

Understand runs

We tracked pipeline and notebook runs through run_context, which stores a Transform and a Run record as a global context.

Artifact objects are the inputs and outputs of runs.

What if I don’t want a global context?

Sometimes, we don’t want to create a global run context but manually pass a run when creating an artifact:

run = ln.Run(transform=transform)
ln.Artifact(filepath, run=run)
When does an artifact appear as a run input?

When accessing an artifact via cache(), load() or backed(), two things happen:

  1. The current run gets added to artifact.input_of

  2. The transform of that artifact gets added as a parent of the current transform

You can then switch off auto-tracking of run inputs if you set ln.settings.track_run_inputs = False: Can I disable tracking run inputs?

You can also track run inputs on a case by case basis via is_run_input=True, e.g., here:

artifact.load(is_run_input=True)

Query by provenance

We can query or search for the notebook that created the artifact:

transform = ln.Transform.search("GWS CRIPSRa analysis").first()

And then find all the artifacts created by that notebook:

ln.Artifact.filter(transform=transform).df()
uid version description key suffix type accessor size hash hash_type n_objects n_observations visibility key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
2 Zb6S158oeo6jeSCy5gXw None hits from schmidt22 crispra GWS None .parquet dataset DataFrame 18368 5GtKh_v__shMvLryIdXkKA md5 None None 1 True 1 2 2 2 2024-06-19 23:21:54.126882+00:00

Which transform ingested a given artifact?

artifact = ln.Artifact.filter().first()
artifact.transform
Transform(uid='2QUzBRcGLiNY6QEg', name='Upload GWS CRISPRa result', type='upload', created_by_id=1, updated_at='2024-06-19 23:21:49 UTC')

And which user?

artifact.created_by
User(uid='DzTjkKse', handle='testuser1', name='Test User1', updated_at='2024-06-19 23:21:56 UTC')

Which transforms were created by a given user?

users = ln.User.lookup()
ln.Transform.filter(created_by=users.testuser1).df()
uid version name key description type reference reference_type latest_report_id source_code_id created_by_id updated_at
id
1 2QUzBRcGLiNY6QEg None Upload GWS CRISPRa result None None upload None None None NaN 1 2024-06-19 23:21:49.167801+00:00
3 qCJPkOuZAi9q5zKv 1 chromium_10x_upload.py chromium_10x_upload.py None script None None None 5.0 1 2024-06-19 23:21:57.103557+00:00
7 1LCd8kco9lZU6K79 0 Project flow project-flow None notebook None None None NaN 1 2024-06-19 23:22:10.125061+00:00

Which notebooks were created by a given user?

ln.Transform.filter(created_by=users.testuser1, type="notebook").df()
uid version name key description type reference reference_type latest_report_id source_code_id created_by_id updated_at
id
7 1LCd8kco9lZU6K79 0 Project flow project-flow None notebook None None None None 1 2024-06-19 23:22:10.125061+00:00

We can also view all recent additions to the entire database:

ln.view()
Hide code cell output
Artifact
uid version description key suffix type accessor size hash hash_type n_objects n_observations visibility key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
12 sit20d3h6LNMKkvLEMJO None None figures/matrixplot_fig2_score-wgs-hits-per-clu... .png dataset None 28814 vKSAeP8dZnBAPkdOYa_kRQ md5 None None 1 True 1 6 6 2 2024-06-19 23:22:09.369666+00:00
11 87qVTspzsGfh1vJpDUCU None None figures/umap_fig1_score-wgs-hits.png .png dataset None 118999 IlWQvuhi-VqBf1nCqWnYXQ md5 None None 1 True 1 6 6 2 2024-06-19 23:22:09.057239+00:00
10 bHFCB9koc5hBwyeGcUwf None perturbseq counts schmidt22_perturbseq.h5ad .h5ad dataset AnnData 20659936 la7EvqEUMDlug9-rpw-udA md5 None None 1 False 1 5 5 2 2024-06-19 23:22:05.345183+00:00
9 WglTKKh0VYMRXktcT95U None None perturbseq/filtered_feature_bc_matrix/features... .tsv.gz dataset None 6 55Zi4XvR93UluC_rLUSpZw md5 None None 1 False 1 4 4 2 2024-06-19 23:22:00.128137+00:00
8 90eYv3MITQ5ETailog2R None None perturbseq/filtered_feature_bc_matrix/matrix.m... .mtx.gz dataset None 6 I98w2LKKs8bBZR0OZ7LXOQ md5 None None 1 False 1 4 4 2 2024-06-19 23:22:00.127550+00:00
7 xuyiWLuv9ZXIPAuwmMFT None None perturbseq/filtered_feature_bc_matrix/barcodes... .tsv.gz dataset None 6 aVD228uXiyehZeWaQPVyvA md5 None None 1 False 1 4 4 2 2024-06-19 23:22:00.126685+00:00
4 1Jf8TstSndvil6oPVXUv None None fastq/perturbseq_R2_001.fastq.gz .fastq.gz dataset None 6 ecLYJfhKK1sAfAHb4H-88g md5 None None 1 False 1 3 3 1 2024-06-19 23:21:57.087419+00:00
Run
uid started_at finished_at is_consecutive reference reference_type transform_id report_id environment_id created_by_id
id
1 4cncFn7mwnvcniFjmiuu 2024-06-19 23:21:49.171712+00:00 NaT True None None 1 None NaN 1
2 7LqIo88ZtYntoXPguyLy 2024-06-19 23:21:53.619842+00:00 NaT True None None 2 None NaN 2
3 jA5XCIDJHSMHaY6wF0AP 2024-06-19 23:21:56.638721+00:00 2024-06-19 23:21:57.101782+00:00 True None None 3 None 6.0 1
4 kN6utGNryMtDwJ4W4KP6 2024-06-19 23:21:59.658563+00:00 NaT None None None 4 None NaN 2
5 LaGdvrKKog47fCJhTCMk 2024-06-19 23:22:01.518466+00:00 NaT None None None 5 None NaN 2
6 QhBQnG9GMwhP4AjhvByr 2024-06-19 23:22:08.321267+00:00 NaT True None None 6 None NaN 2
7 PzNjPmKWiL5DbppOrH0U 2024-06-19 23:22:10.130907+00:00 NaT True None None 7 None NaN 1
Storage
uid root description type region instance_uid run_id created_by_id updated_at
id
1 d1dRaNwxNytZ /home/runner/work/lamin-usecases/lamin-usecase... None local None 54ZGqgkROOFf None 1 2024-06-19 23:21:46.889454+00:00
Transform
uid version name key description type reference reference_type latest_report_id source_code_id created_by_id updated_at
id
7 1LCd8kco9lZU6K79 0 Project flow project-flow None notebook None None None NaN 1 2024-06-19 23:22:10.125061+00:00
6 lB3IyPLQSmvt5zKv 1 Perform single cell analysis, integrate with C... integrated-analysis None notebook None None None NaN 2 2024-06-19 23:22:08.315721+00:00
5 YqmbO6oMXjRj65cN 2 postprocess_cellranger.py postprocess_cellranger.py None script None None None NaN 2 2024-06-19 23:22:01.513742+00:00
4 97rTO159DwIrwFVU 7.2.0 Cell Ranger None None pipeline https://www.10xgenomics.com/support/software/c... None None NaN 2 2024-06-19 23:21:59.654758+00:00
3 qCJPkOuZAi9q5zKv 1 chromium_10x_upload.py chromium_10x_upload.py None script None None None 5.0 1 2024-06-19 23:21:57.103557+00:00
2 T0T28btuB0PG5zKv 1 GWS CRIPSRa analysis hit-identification None notebook None None None NaN 2 2024-06-19 23:21:53.615120+00:00
1 2QUzBRcGLiNY6QEg None Upload GWS CRISPRa result None None upload None None None NaN 1 2024-06-19 23:21:49.167801+00:00
User
uid handle name updated_at
id
2 bKeW4T6E testuser2 Test User2 2024-06-19 23:21:59.645991+00:00
1 DzTjkKse testuser1 Test User1 2024-06-19 23:21:56.527027+00:00
Hide code cell content
!lamin login testuser1
!lamin delete --force mydata
!rm -r ./mydata
✅ logged in with email [email protected] (uid: DzTjkKse)
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.14/x64/bin/lamin", line 8, in <module>
    sys.exit(main())
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/rich_click/rich_command.py", line 367, in __call__
    return super().__call__(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/rich_click/rich_command.py", line 152, in main
    rv = self.invoke(ctx)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/lamin_cli/__main__.py", line 103, in delete
    return delete(instance, force=force)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/lamindb_setup/_delete.py", line 98, in delete
    n_objects = check_storage_is_empty(
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/lamindb_setup/core/upath.py", line 779, in check_storage_is_empty
    raise InstanceNotEmpty(message)
lamindb_setup.core.upath.InstanceNotEmpty: Storage /home/runner/work/lamin-usecases/lamin-usecases/docs/mydata/.lamindb contains 5 objects ('_is_initialized' ignored) - delete them prior to deleting the instance
['/home/runner/work/lamin-usecases/lamin-usecases/docs/mydata/.lamindb/87qVTspzsGfh1vJpDUCU.png', '/home/runner/work/lamin-usecases/lamin-usecases/docs/mydata/.lamindb/Rg7b74uDYXfAMqRE4QNL.txt', '/home/runner/work/lamin-usecases/lamin-usecases/docs/mydata/.lamindb/Zb6S158oeo6jeSCy5gXw.parquet', '/home/runner/work/lamin-usecases/lamin-usecases/docs/mydata/.lamindb/_is_initialized', '/home/runner/work/lamin-usecases/lamin-usecases/docs/mydata/.lamindb/sWGMUbMt9e8FhZq00HOr.py', '/home/runner/work/lamin-usecases/lamin-usecases/docs/mydata/.lamindb/sit20d3h6LNMKkvLEMJO.png']