Snakemake

Note

This notebook serves as a demo for Python scripting that you could run before and after Snakemake runs. Typically, you would include lamindb logic directly in your Snakemake workflow.

Snakemake is a workflow manager for executing scientific workflows across platforms scalably, portably, and reproducibly.

This guide shows how to track a Snakemake run with inputs & outputs for the example of the snakemake-workflows/rna-seq-star-deseq2 pipeline by running a Python script.

First we clone the Snakemake pipeline with git. Because the test datasets come with the repo and, for simplicity, we want to avoid moving them into another directory, we initialize a LaminDB instance in the same directory.

# pip install lamindb snakemake
!git clone https://github.com/snakemake-workflows/rna-seq-star-deseq2 --single-branch --branch v3.1.0
!lamin init --storage ./rna-seq-star-deseq2

Show code cell output Hide code cell output

Cloning into 'rna-seq-star-deseq2'...

remote: Enumerating objects: 874, done.
remote: Counting objects:   0% (1/256)
remote: Counting objects:   1% (3/256)
remote: Counting objects:   2% (6/256)
remote: Counting objects:   3% (8/256)
remote: Counting objects:   4% (11/256)
remote: Counting objects:   5% (13/256)
remote: Counting objects:   6% (16/256)
remote: Counting objects:   7% (18/256)
remote: Counting objects:   8% (21/256)
remote: Counting objects:   9% (24/256)
remote: Counting objects:  10% (26/256)
remote: Counting objects:  11% (29/256)
remote: Counting objects:  12% (31/256)
remote: Counting objects:  13% (34/256)
remote: Counting objects:  14% (36/256)
remote: Counting objects:  15% (39/256)
remote: Counting objects:  16% (41/256)
remote: Counting objects:  17% (44/256)
remote: Counting objects:  18% (47/256)
remote: Counting objects:  19% (49/256)
remote: Counting objects:  20% (52/256)
remote: Counting objects:  21% (54/256)
remote: Counting objects:  22% (57/256)
remote: Counting objects:  23% (59/256)
remote: Counting objects:  24% (62/256)
remote: Counting objects:  25% (64/256)
remote: Counting objects:  26% (67/256)
remote: Counting objects:  27% (70/256)
remote: Counting objects:  28% (72/256)
remote: Counting objects:  29% (75/256)
remote: Counting objects:  30% (77/256)
remote: Counting objects:  31% (80/256)
remote: Counting objects:  32% (82/256)
remote: Counting objects:  33% (85/256)
remote: Counting objects:  34% (88/256)
remote: Counting objects:  35% (90/256)
remote: Counting objects:  36% (93/256)
remote: Counting objects:  37% (95/256)
remote: Counting objects:  38% (98/256)
remote: Counting objects:  39% (100/256)
remote: Counting objects:  40% (103/256)
remote: Counting objects:  41% (105/256)
remote: Counting objects:  42% (108/256)
remote: Counting objects:  43% (111/256)
remote: Counting objects:  44% (113/256)
remote: Counting objects:  45% (116/256)
remote: Counting objects:  46% (118/256)
remote: Counting objects:  47% (121/256)
remote: Counting objects:  48% (123/256)
remote: Counting objects:  49% (126/256)
remote: Counting objects:  50% (128/256)
remote: Counting objects:  51% (131/256)
remote: Counting objects:  52% (134/256)
remote: Counting objects:  53% (136/256)
remote: Counting objects:  54% (139/256)
remote: Counting objects:  55% (141/256)
remote: Counting objects:  56% (144/256)
remote: Counting objects:  57% (146/256)
remote: Counting objects:  58% (149/256)
remote: Counting objects:  59% (152/256)
remote: Counting objects:  60% (154/256)
remote: Counting objects:  61% (157/256)
remote: Counting objects:  62% (159/256)
remote: Counting objects:  63% (162/256)
remote: Counting objects:  64% (164/256)
remote: Counting objects:  65% (167/256)
remote: Counting objects:  66% (169/256)
remote: Counting objects:  67% (172/256)
remote: Counting objects:  68% (175/256)
remote: Counting objects:  69% (177/256)
remote: Counting objects:  70% (180/256)
remote: Counting objects:  71% (182/256)
remote: Counting objects:  72% (185/256)
remote: Counting objects:  73% (187/256)
remote: Counting objects:  74% (190/256)
remote: Counting objects:  75% (192/256)
remote: Counting objects:  76% (195/256)
remote: Counting objects:  77% (198/256)
remote: Counting objects:  78% (200/256)
remote: Counting objects:  79% (203/256)
remote: Counting objects:  80% (205/256)
remote: Counting objects:  81% (208/256)
remote: Counting objects:  82% (210/256)
remote: Counting objects:  83% (213/256)
remote: Counting objects:  84% (216/256)
remote: Counting objects:  85% (218/256)
remote: Counting objects:  86% (221/256)
remote: Counting objects:  87% (223/256)
remote: Counting objects:  88% (226/256)
remote: Counting objects:  89% (228/256)
remote: Counting objects:  90% (231/256)
remote: Counting objects:  91% (233/256)
remote: Counting objects:  92% (236/256)
remote: Counting objects:  93% (239/256)
remote: Counting objects:  94% (241/256)
remote: Counting objects:  95% (244/256)
remote: Counting objects:  96% (246/256)
remote: Counting objects:  97% (249/256)
remote: Counting objects:  98% (251/256)
remote: Counting objects:  99% (254/256)
remote: Counting objects: 100% (256/256)
remote: Counting objects: 100% (256/256), done.
remote: Compressing objects:   0% (1/112)
remote: Compressing objects:   1% (2/112)
remote: Compressing objects:   2% (3/112)
remote: Compressing objects:   3% (4/112)
remote: Compressing objects:   4% (5/112)
remote: Compressing objects:   5% (6/112)
remote: Compressing objects:   6% (7/112)
remote: Compressing objects:   7% (8/112)
remote: Compressing objects:   8% (9/112)
remote: Compressing objects:   9% (11/112)
remote: Compressing objects:  10% (12/112)
remote: Compressing objects:  11% (13/112)
remote: Compressing objects:  12% (14/112)
remote: Compressing objects:  13% (15/112)
remote: Compressing objects:  14% (16/112)
remote: Compressing objects:  15% (17/112)
remote: Compressing objects:  16% (18/112)
remote: Compressing objects:  17% (20/112)
remote: Compressing objects:  18% (21/112)
remote: Compressing objects:  19% (22/112)
remote: Compressing objects:  20% (23/112)
remote: Compressing objects:  21% (24/112)
remote: Compressing objects:  22% (25/112)
remote: Compressing objects:  23% (26/112)
remote: Compressing objects:  24% (27/112)
remote: Compressing objects:  25% (28/112)
remote: Compressing objects:  26% (30/112)
remote: Compressing objects:  27% (31/112)
remote: Compressing objects:  28% (32/112)
remote: Compressing objects:  29% (33/112)
remote: Compressing objects:  30% (34/112)
remote: Compressing objects:  31% (35/112)
remote: Compressing objects:  32% (36/112)
remote: Compressing objects:  33% (37/112)
remote: Compressing objects:  34% (39/112)
remote: Compressing objects:  35% (40/112)
remote: Compressing objects:  36% (41/112)
remote: Compressing objects:  37% (42/112)
remote: Compressing objects:  38% (43/112)
remote: Compressing objects:  39% (44/112)
remote: Compressing objects:  40% (45/112)
remote: Compressing objects:  41% (46/112)
remote: Compressing objects:  42% (48/112)
remote: Compressing objects:  43% (49/112)
remote: Compressing objects:  44% (50/112)
remote: Compressing objects:  45% (51/112)
remote: Compressing objects:  46% (52/112)
remote: Compressing objects:  47% (53/112)
remote: Compressing objects:  48% (54/112)
remote: Compressing objects:  49% (55/112)
remote: Compressing objects:  50% (56/112)
remote: Compressing objects:  51% (58/112)
remote: Compressing objects:  52% (59/112)
remote: Compressing objects:  53% (60/112)
remote: Compressing objects:  54% (61/112)
remote: Compressing objects:  55% (62/112)
remote: Compressing objects:  56% (63/112)
remote: Compressing objects:  57% (64/112)
remote: Compressing objects:  58% (65/112)
remote: Compressing objects:  59% (67/112)
remote: Compressing objects:  60% (68/112)
remote: Compressing objects:  61% (69/112)
remote: Compressing objects:  62% (70/112)
remote: Compressing objects:  63% (71/112)
remote: Compressing objects:  64% (72/112)
remote: Compressing objects:  65% (73/112)
remote: Compressing objects:  66% (74/112)
remote: Compressing objects:  67% (76/112)
remote: Compressing objects:  68% (77/112)
remote: Compressing objects:  69% (78/112)
remote: Compressing objects:  70% (79/112)
remote: Compressing objects:  71% (80/112)
remote: Compressing objects:  72% (81/112)
remote: Compressing objects:  73% (82/112)
remote: Compressing objects:  74% (83/112)
remote: Compressing objects:  75% (84/112)
remote: Compressing objects:  76% (86/112)
remote: Compressing objects:  77% (87/112)
remote: Compressing objects:  78% (88/112)
remote: Compressing objects:  79% (89/112)
remote: Compressing objects:  80% (90/112)
remote: Compressing objects:  81% (91/112)
remote: Compressing objects:  82% (92/112)
remote: Compressing objects:  83% (93/112)
remote: Compressing objects:  84% (95/112)
remote: Compressing objects:  85% (96/112)
remote: Compressing objects:  86% (97/112)
remote: Compressing objects:  87% (98/112)
remote: Compressing objects:  88% (99/112)
remote: Compressing objects:  89% (100/112)
remote: Compressing objects:  90% (101/112)
remote: Compressing objects:  91% (102/112)
remote: Compressing objects:  92% (104/112)
remote: Compressing objects:  93% (105/112)
remote: Compressing objects:  94% (106/112)
remote: Compressing objects:  95% (107/112)
remote: Compressing objects:  96% (108/112)
remote: Compressing objects:  97% (109/112)
remote: Compressing objects:  98% (110/112)
remote: Compressing objects:  99% (111/112)
remote: Compressing objects: 100% (112/112)
remote: Compressing objects: 100% (112/112), done.
Receiving objects:   0% (1/874)
Receiving objects:   1% (9/874)
Receiving objects:   2% (18/874)
Receiving objects:   3% (27/874)
Receiving objects:   4% (35/874)
Receiving objects:   5% (44/874)
Receiving objects:   6% (53/874)
Receiving objects:   7% (62/874)
Receiving objects:   8% (70/874)
Receiving objects:   9% (79/874)
Receiving objects:  10% (88/874)
Receiving objects:  11% (97/874)
Receiving objects:  12% (105/874)
Receiving objects:  13% (114/874)
Receiving objects:  14% (123/874)
Receiving objects:  15% (132/874)

Receiving objects:  16% (140/874)
Receiving objects:  17% (149/874)
Receiving objects:  18% (158/874)
Receiving objects:  19% (167/874)
Receiving objects:  20% (175/874)
Receiving objects:  21% (184/874)
Receiving objects:  22% (193/874)
Receiving objects:  23% (202/874)
Receiving objects:  24% (210/874)
Receiving objects:  25% (219/874)
Receiving objects:  26% (228/874)
Receiving objects:  27% (236/874)
Receiving objects:  28% (245/874)
Receiving objects:  29% (254/874)
Receiving objects:  30% (263/874)
Receiving objects:  31% (271/874)
Receiving objects:  32% (280/874)
Receiving objects:  33% (289/874)
Receiving objects:  34% (298/874)
Receiving objects:  35% (306/874)
Receiving objects:  36% (315/874)
Receiving objects:  37% (324/874)
Receiving objects:  38% (333/874)
Receiving objects:  39% (341/874)
Receiving objects:  40% (350/874)
Receiving objects:  41% (359/874)
Receiving objects:  42% (368/874)
Receiving objects:  43% (376/874)
Receiving objects:  44% (385/874)
Receiving objects:  45% (394/874)
Receiving objects:  46% (403/874)
Receiving objects:  47% (411/874)
Receiving objects:  48% (420/874)
Receiving objects:  49% (429/874)
Receiving objects:  50% (437/874)
Receiving objects:  51% (446/874)

Receiving objects:  52% (455/874)
Receiving objects:  53% (464/874)
Receiving objects:  54% (472/874)
Receiving objects:  55% (481/874)
Receiving objects:  56% (490/874)
Receiving objects:  57% (499/874)
Receiving objects:  58% (507/874)
Receiving objects:  59% (516/874)
Receiving objects:  60% (525/874)
Receiving objects:  61% (534/874)
Receiving objects:  62% (542/874)
Receiving objects:  63% (551/874)
Receiving objects:  64% (560/874)
remote: Total 874 (delta 197), reused 144 (delta 144), pack-reused 618 (from 3)
Receiving objects:  65% (569/874)
Receiving objects:  66% (577/874)
Receiving objects:  67% (586/874)
Receiving objects:  68% (595/874)
Receiving objects:  69% (604/874)
Receiving objects:  70% (612/874)
Receiving objects:  71% (621/874)
Receiving objects:  72% (630/874)
Receiving objects:  73% (639/874)
Receiving objects:  74% (647/874)
Receiving objects:  75% (656/874)
Receiving objects:  76% (665/874)
Receiving objects:  77% (673/874)
Receiving objects:  78% (682/874)
Receiving objects:  79% (691/874)
Receiving objects:  80% (700/874)
Receiving objects:  81% (708/874)
Receiving objects:  82% (717/874)
Receiving objects:  83% (726/874)
Receiving objects:  84% (735/874)
Receiving objects:  85% (743/874)
Receiving objects:  86% (752/874)
Receiving objects:  87% (761/874)
Receiving objects:  88% (770/874)
Receiving objects:  89% (778/874)
Receiving objects:  90% (787/874)
Receiving objects:  91% (796/874)
Receiving objects:  92% (805/874)
Receiving objects:  93% (813/874)
Receiving objects:  94% (822/874)
Receiving objects:  95% (831/874)
Receiving objects:  96% (840/874)
Receiving objects:  97% (848/874)
Receiving objects:  98% (857/874)
Receiving objects:  99% (866/874)
Receiving objects: 100% (874/874)
Receiving objects: 100% (874/874), 16.97 MiB | 49.10 MiB/s, done.
Resolving deltas:   0% (0/455)
Resolving deltas:   1% (5/455)
Resolving deltas:   2% (10/455)
Resolving deltas:   3% (14/455)
Resolving deltas:   4% (19/455)
Resolving deltas:   5% (23/455)
Resolving deltas:   6% (28/455)
Resolving deltas:   7% (32/455)
Resolving deltas:   8% (37/455)
Resolving deltas:   9% (41/455)
Resolving deltas:  10% (46/455)
Resolving deltas:  11% (51/455)
Resolving deltas:  12% (55/455)
Resolving deltas:  13% (60/455)
Resolving deltas:  14% (64/455)
Resolving deltas:  15% (69/455)
Resolving deltas:  16% (73/455)
Resolving deltas:  17% (78/455)
Resolving deltas:  18% (82/455)
Resolving deltas:  19% (87/455)
Resolving deltas:  20% (91/455)
Resolving deltas:  21% (96/455)
Resolving deltas:  22% (101/455)
Resolving deltas:  23% (105/455)
Resolving deltas:  24% (110/455)
Resolving deltas:  25% (115/455)
Resolving deltas:  26% (120/455)
Resolving deltas:  27% (123/455)
Resolving deltas:  28% (129/455)
Resolving deltas:  29% (132/455)
Resolving deltas:  30% (137/455)
Resolving deltas:  31% (142/455)
Resolving deltas:  32% (147/455)
Resolving deltas:  33% (151/455)
Resolving deltas:  34% (155/455)
Resolving deltas:  35% (160/455)
Resolving deltas:  36% (164/455)
Resolving deltas:  37% (169/455)
Resolving deltas:  38% (173/455)
Resolving deltas:  39% (178/455)
Resolving deltas:  40% (183/455)
Resolving deltas:  41% (187/455)
Resolving deltas:  42% (192/455)
Resolving deltas:  43% (196/455)
Resolving deltas:  44% (201/455)
Resolving deltas:  45% (205/455)
Resolving deltas:  46% (210/455)
Resolving deltas:  47% (214/455)
Resolving deltas:  48% (220/455)
Resolving deltas:  49% (223/455)
Resolving deltas:  50% (228/455)
Resolving deltas:  51% (233/455)
Resolving deltas:  52% (237/455)
Resolving deltas:  53% (242/455)
Resolving deltas:  54% (246/455)
Resolving deltas:  55% (251/455)
Resolving deltas:  56% (255/455)
Resolving deltas:  57% (260/455)
Resolving deltas:  58% (264/455)
Resolving deltas:  59% (269/455)
Resolving deltas:  60% (273/455)
Resolving deltas:  61% (278/455)
Resolving deltas:  62% (283/455)
Resolving deltas:  63% (287/455)
Resolving deltas:  64% (292/455)
Resolving deltas:  65% (296/455)
Resolving deltas:  66% (301/455)
Resolving deltas:  67% (305/455)
Resolving deltas:  68% (310/455)
Resolving deltas:  69% (315/455)
Resolving deltas:  70% (319/455)
Resolving deltas:  71% (324/455)
Resolving deltas:  72% (328/455)
Resolving deltas:  73% (334/455)
Resolving deltas:  74% (337/455)
Resolving deltas:  75% (342/455)
Resolving deltas:  76% (346/455)
Resolving deltas:  77% (351/455)
Resolving deltas:  78% (355/455)
Resolving deltas:  79% (360/455)
Resolving deltas:  80% (364/455)
Resolving deltas:  81% (369/455)
Resolving deltas:  82% (375/455)
Resolving deltas:  83% (379/455)
Resolving deltas:  84% (384/455)
Resolving deltas:  85% (388/455)
Resolving deltas:  86% (392/455)
Resolving deltas:  87% (396/455)
Resolving deltas:  88% (402/455)
Resolving deltas:  89% (405/455)
Resolving deltas:  90% (411/455)
Resolving deltas:  91% (416/455)
Resolving deltas:  92% (420/455)
Resolving deltas:  93% (424/455)
Resolving deltas:  94% (428/455)
Resolving deltas:  95% (433/455)
Resolving deltas:  96% (438/455)
Resolving deltas:  97% (442/455)
Resolving deltas:  98% (447/455)
Resolving deltas:  99% (451/455)
Resolving deltas: 100% (455/455)
Resolving deltas: 100% (455/455), done.
Note: switching to '5fbe51c94aa1f5c8c8cbdca379eb00436d6491bf'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

→ initialized lamindb: testuser1/rna-seq-star-deseq2

import lamindb as ln

Show code cell output Hide code cell output

→ connected lamindb: testuser1/rna-seq-star-deseq2

Registering inputs

root_dir = "rna-seq-star-deseq2"
sample_sheet = ln.Artifact(f"{root_dir}/.test/config_basic/samples.tsv").save()
input_fastqs = ln.Artifact.from_dir(f"{root_dir}/.test/ngs-test-data/reads/")
ln.save(input_fastqs)

Show code cell output Hide code cell output

! no run & transform got linked, call `ln.track()` & re-run

! no run & transform got linked, call `ln.track()` & re-run

! no run & transform got linked, call `ln.track()` & re-run

! no run & transform got linked, call `ln.track()` & re-run

! no run & transform got linked, call `ln.track()` & re-run

! no run & transform got linked, call `ln.track()` & re-run

! no run & transform got linked, call `ln.track()` & re-run

! no run & transform got linked, call `ln.track()` & re-run

! no run & transform got linked, call `ln.track()` & re-run

! no run & transform got linked, call `ln.track()` & re-run

! no run & transform got linked, call `ln.track()` & re-run

! there are multiple artifact uids with the same hashes, dropping 2 duplicates out of 10 artifacts:
    xYLhBgxLVmnl4dMJ0000
    HMwOX54zhpdM1O4S0000

Track a Snakemake run

Track the Snakemake workflow & run:

transform = ln.Transform(
    key="snakemake-rna-seq-star-deseq2",
    version="2.0.0",
    type="pipeline",
    reference="https://github.com/snakemake-workflows/rna-seq-star-deseq2",
)
ln.track(transform)

Show code cell output Hide code cell output

→ created Transform('MfauBHJyleLz0000', key='snakemake-rna-seq-star-deseq2'), started new Run('zVsfpZFn6Xz3zu8R') at 2025-10-30 07:57:50 UTC

• recommendation: to identify the script across renames, pass the uid: ln.track("MfauBHJyleLz")

If we call cache() on the input artifacts, they’ll be downloaded into a cache and tracked as run inputs. In this test case however, no download happened because the files are already available locally.

input_sample_sheet_path = sample_sheet.cache()
input_paths = [input_fastq.cache() for input_fastq in input_fastqs]

Let’s run the pipeline.

!snakemake \
    --directory rna-seq-star-deseq2/.test \
    --snakefile rna-seq-star-deseq2/workflow/Snakefile \
    --configfile rna-seq-star-deseq2/.test/config_basic/config.yaml \
    --use-conda \
    --show-failed-logs \
    --cores 2 \
    --conda-frontend conda \
    --conda-cleanup-pkgs cache

Registering outputs

Quality control.

multiqc_file = ln.Artifact(f"{root_dir}/.test/results/qc/multiqc_report.html").save()

How would I register all QC files?

multiqc_results = ln.Artifact.from_dir(f"{root_dir}/results/qc/multiqc_report_data/")
ln.save(multiqc_results)

Count matrix.

count_matrix = ln.Artifact(f"{root_dir}/.test/results/counts/all.symbol.tsv")
count_matrix.save()

Show code cell output Hide code cell output

Artifact(uid='dO1P8TWuGdIED2zY0000', version=None, is_latest=True, key='.test/results/counts/all.symbol.tsv', description=None, suffix='.tsv', kind=None, otype=None, size=115656, hash='raT2a-pfpiOK_4XYNfdxDA', n_files=None, n_observations=None, branch_id=1, space_id=1, storage_id=1, run_id=1, schema_id=None, created_by_id=1, created_at=2025-10-30 08:04:04 UTC, is_locked=False)

Visualize

View data lineage:

count_matrix.view_lineage()

Appendix

Linking biological entities

To make the count matrix queryable by biological entities (genes, experimental metadata, etc.), we can now proceed with: Bulk RNA-seq

Linking a Snakemake run ID

Snakemake does not have an easily accessible ID that is associated with a run. Therefore, we need to extract it from the log files.

import pathlib
from datetime import datetime

PATH_TO_DOT_SNAKEMAKE_LOG = "rna-seq-star-deseq2/.test/.snakemake/log"
log_files_file_names = list(
    map(
        lambda lf: str(lf).split("/")[-1],
        list(pathlib.Path(PATH_TO_DOT_SNAKEMAKE_LOG).glob("*.snakemake.log")),
    )
)

timestamps = [
    datetime.strptime(filename.split(".")[0], "%Y-%m-%dT%H%M%S")
    for filename in log_files_file_names
]
snakemake_id = log_files_file_names[timestamps.index(max(timestamps))].split(".")[1]

Let us add the information about the session ID to our run record:

run = ln.context.run  # let's grab the global run record
run.reference = snakemake_id
run.reference_type = "snakemake_id"
run.save()

Show code cell output Hide code cell output

Run(uid='zVsfpZFn6Xz3zu8R', name=None, started_at=2025-10-30 07:57:50 UTC, finished_at=None, params=None, reference='371734', reference_type='snakemake_id', branch_id=1, space_id=1, transform_id=1, report_id=None, environment_id=None, created_by_id=1, initiated_by_run_id=None, created_at=2025-10-30 07:57:50 UTC, is_locked=False)