Nextflow¶

Nextflow is a workflow management system used for executing scientific workflows across platforms scalably, portably, and reproducibly.

Here, we’ll run a demo of the microscopy pipeline mcmicro to correct uneven illumination. Reference

Note

Typically, you run the Nextflow workflow from the command line or Seqera Platform and then register input and output data with a script. The Seqera Platform allows for post-run scripts that can automate this process.

!lamin init --storage . --name nextflow-mcmicro

import lamindb as ln

💡 connected lamindb: testuser1/nextflow-mcmicro

Run mcmicro/exemplar pipeline¶

Register the exemplar-001 data as input of the mcmicro pipeline by running the script below:

!python mcmicro_exemplar.py exemplar-001

Run mcmicro pipeline¶

Run mcmicro pipeline and track input/output data with the script below:

# mcmicro.py

"""This script runs the MCMICRO pipeline and tracks input/output data in LaminDB.

Usage: python mcmicro.py exemplar-001
"""

import subprocess
import argparse
import lamindb as ln
import yaml

# get args from command line
parser = argparse.ArgumentParser()
parser.add_argument("input", type=str, help="Input folder name.")
args = parser.parse_args()

transform = ln.Transform(
    name="MCMICRO",
    version="1.0.0",
    type="pipeline",
    reference="https://github.com/labsyspharm/mcmicro",
)
run = ln.track(transform=transform)

# get the input data from LaminDB
mcmicro_input = ln.Artifact.filter(description=args.input).one()
input_dir = mcmicro_input.cache()

# execute the nextflow pipeline to download example data
report = f"{args.input}-mcmicro-execution_report.html"
subprocess.run(
    [
        "nextflow",
        "run",
        "https://github.com/labsyspharm/mcmicro",
        "--in",
        input_dir,
        "--start-at",
        "illumination",
        "--stop-at",
        "registration",
        "-with-report",
        report,
    ]
)

# get the nextflow execution id from the log (first row)
nextflow_id = subprocess.getoutput(
    "nextflow log | tail -n 1 | awk -F '\t' '{print $6}'"
)
# optionally, tag the transform
ulabel = ln.ULabel(name="nextflow").save()
run.transform.ulabels.add(ulabel)
# track the execution report, set visibility to "hidden" to avoid cluttering the artifacts
report_artifact = ln.Artifact(
    report, description=f"nextflow report of {nextflow_id}", visibility=0, run=False
).save()
run.report = report_artifact
run.reference = nextflow_id
run.reference_type = "nextflow_id"
run.save()
# optionally, track the pipeline parameters
with open(f"{input_dir}/qc/params.yml") as params_file:
    qc_params = yaml.safe_load(params_file)
ln.Param(name="qc_params", dtype="dict").save()
run.params.add_values({"qc_params": qc_params})

# register the output artifact
output = ln.Artifact.from_dir(f"{input_dir}/registration")
ln.save(output)

ln.finish()

!python mcmicro.py exemplar-001

Data lineage¶

View data lineage:

output = ln.Artifact.filter(key__icontains="exemplar-001.ome.tif").one()

output.view_lineage()

_images/105cefeb040f137284188c18aae2c26fdaaa893714e7120d88dd3038f50ca0eb.svg

View transforms and runs in LaminHub¶

View the database content¶

ln.view()

Artifact

	uid	version	description	key	suffix	type	accessor	size	hash	hash_type	n_objects	n_observations	visibility	key_is_virtual	storage_id	transform_id	run_id	created_by_id	updated_at
id
4	2TwWA8djdHQ0rbTuWEeZ	None	None	exemplar-001/registration/exemplar-001.ome.tif	.tif	dataset	None	175490712	gHRVbQOFd4P2eO5_EMexZr	sha1-fl	NaN	None	1	False	1	2	2	1	2024-07-26 14:38:57.045982+00:00
2	a1nqhzfJQroJWsRC0g1Z	None	exemplar-001	exemplar-001		dataset	None	332630483	lpOStRajj_SuLBtzXF4KjQ	md5-d	10.0	None	1	False	1	1	1	1	2024-07-26 14:37:06.768824+00:00

Run

	uid	started_at	finished_at	is_consecutive	reference	reference_type	transform_id	report_id	environment_id	created_by_id
id
1	E7G9YabclsJILSkVERHz	2024-07-26 14:37:05.469329+00:00	2024-07-26 14:37:06.771436+00:00	None	00ec8982-8172-4915-8f9a-d7ce54abebdc	nextflow_id	1	1	None	1
2	Fc8ENFkKm2h3YdMOu6LW	2024-07-26 14:37:08.882861+00:00	2024-07-26 14:38:57.048630+00:00	None	5596733f-2bc8-4e2b-b2df-d2b6d55ba656	nextflow_id	2	3	None	1

Storage

	uid	root	description	type	region	instance_uid	run_id	created_by_id	updated_at
id
1	RwbhtZx7metu	/home/runner/work/nextflow-lamin/nextflow-lami...	None	local	None	None	None	1	2024-07-26 14:36:43.478136+00:00

Transform

	uid	version	name	key	description	type	reference	reference_type	latest_report_id	source_code_id	created_by_id	updated_at
id
2	gMaCA4s3GSkwZOyX	1.0.0	MCMICRO	None	None	pipeline	https://github.com/labsyspharm/mcmicro	None	None	None	1	2024-07-26 14:37:08.878023+00:00
1	HXJzgRZiHrZ6cSof	1.0.0	MCMICRO exemplar	None	None	pipeline	https://github.com/labsyspharm/mcmicro	None	None	None	1	2024-07-26 14:37:05.463292+00:00

ULabel

	uid	name	description	reference	reference_type	run_id	created_by_id	updated_at
id
1	alvQoyv4	nextflow	None	None	None	1	1	2024-07-26 14:38:56.906401+00:00

User

	uid	handle	name	updated_at
id
1	DzTjkKse	testuser1	Test User1	2024-07-26 14:36:43.473391+00:00