Nextflow¶

Nextflow is the most widely used workflow manager in bioinformatics.

This guide shows how to register a Nextflow run with inputs & outputs for the example of the nf-core/scrnaseq pipeline by running a Python script.

The approach could be automated by deploying the script via

a serverless environment trigger (e.g., AWS Lambda)
a post-run script on the Seqera Platform

!lamin init --storage ./test-nextflow --name test-nextflow

Run the pipeline¶

Let’s download the input data from an S3 bucket.

import lamindb as ln

input_path = ln.UPath("s3://lamindb-test/scrnaseq_input")
input_path.download_to("scrnaseq_input")

→ connected lamindb: testuser1/test-nextflow

And run the nf-core/scrnaseq pipeline.

# the test profile uses all downloaded input files as an input
!nextflow run nf-core/scrnaseq -r 2.7.1 -profile docker,test -resume --outdir scrnaseq_output

Show code cell output Hide code cell output

N E X T F L O W  ~  version 24.10.1

Pulling nf-core/scrnaseq ...

 downloaded from https://github.com/nf-core/scrnaseq.git

WARN: It appears you have never run this project before -- Option `-resume` is ignored

Launching `https://github.com/nf-core/scrnaseq` [curious_volhard] DSL2 - revision: 4171377f40 [2.7.1]
Downloading plugin [email protected]

------------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/scrnaseq v2.7.1-g4171377
------------------------------------------------------
Core Nextflow options
  revision                  : 2.7.1
  runName                   : curious_volhard
  containerEngine           : docker
  launchDir                 : /home/runner/work/nextflow-lamin/nextflow-lamin/docs
  workDir                   : /home/runner/work/nextflow-lamin/nextflow-lamin/docs/work
  projectDir                : /home/runner/.nextflow/assets/nf-core/scrnaseq
  userName                  : runner
  profile                   : docker,test
  configFiles               : 

Input/output options
  input                     : https://github.com/nf-core/test-datasets/raw/scrnaseq/samplesheet-2-0.csv
  outdir                    : scrnaseq_output

Mandatory arguments
  aligner                   : star
  protocol                  : 10XV2

Skip Tools
  skip_emptydrops           : true

Reference genome options
  fasta                     : https://github.com/nf-core/test-datasets/raw/scrnaseq/reference/GRCm38.p6.genome.chr19.fa
  gtf                       : https://github.com/nf-core/test-datasets/raw/scrnaseq/reference/gencode.vM19.annotation.chr19.gtf

Institutional config options
  config_profile_name       : Test profile
  config_profile_description: Minimal test dataset to check pipeline function

Max job request options
  max_cpus                  : 2
  max_memory                : 6.GB
  max_time                  : 6.h

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/scrnaseq for your analysis please cite:

* The pipeline
  https://doi.org/10.5281/zenodo.3568187

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
  https://github.com/nf-core/scrnaseq/blob/master/CITATIONS.md
------------------------------------------------------

[f6/eb925f] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:FASTQC_CHECK:FASTQC (Sample_Y)
[72/3cd902] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:FASTQC_CHECK:FASTQC (Sample_X)

[56/7da668] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:GTF_GENE_FILTER (GRCm38.p6.genome.chr19.fa)

[1c/efb7cd] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:STARSOLO:STAR_GENOMEGENERATE (GRCm38.p6.genome.chr19.fa)

[15/451e54] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:STARSOLO:STAR_ALIGN (Sample_X)
[3f/c77c59] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:STARSOLO:STAR_ALIGN (Sample_Y)

[99/7c112e] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_SEURAT (Sample_X)

[e5/26740e] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_H5AD (Sample_X)

[bf/1ce111] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_H5AD (Sample_X)

[ce/d52c39] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_SEURAT (Sample_X)

[08/8718cf] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_SEURAT (Sample_Y)

[2c/e6d3d4] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_SEURAT (Sample_Y)

[4c/679d1c] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_H5AD (Sample_Y)

[c6/495c9b] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_H5AD (Sample_Y)

[fd/2ade91] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MULTIQC

[6f/b7dae4] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:CONCAT_H5AD (1)

[3b/55fd17] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:CONCAT_H5AD (2)

-[nf-core/scrnaseq] Pipeline completed successfully-

Run the registration script¶

After the pipeline has completed, a Python script registers inputs & outputs in LaminDB.

nf-core/scrnaseq run registration¶

import argparse
import lamindb as ln
import json
import re
from pathlib import Path


def parse_arguments() -> argparse.Namespace:
    parser = argparse.ArgumentParser()
    parser.add_argument("--input", type=str, required=True)
    parser.add_argument("--output", type=str, required=True)
    return parser.parse_args()


def register_pipeline_io(input_dir: str, output_dir: str, run: ln.Run) -> None:
    """Register input and output artifacts for an `nf-core/scrnaseq` run."""
    input_artifacts = ln.Artifact.from_dir(input_dir, run=False)
    ln.save(input_artifacts)
    run.input_artifacts.set(input_artifacts)
    ln.Artifact(f"{output_dir}/multiqc", description="multiqc report", run=run).save()
    ln.Artifact(
        f"{output_dir}/star/mtx_conversions/combined_filtered_matrix.h5ad",
        description="filtered count matrix",
        run=run,
    ).save()


def register_pipeline_metadata(output_dir: str, run: ln.Run) -> None:
    """Register nf-core run metadata stored in the 'pipeline_info' folder."""
    ulabel = ln.ULabel(name="nextflow").save()
    run.transform.ulabels.add(ulabel)

    # nextflow run id
    content = next(Path(f"{output_dir}/pipeline_info").glob("execution_report_*.html")).read_text()
    match = re.search(r"run id \[([^\]]+)\]", content)
    nextflow_id = match.group(1) if match else ""
    run.reference = nextflow_id
    run.reference_type = "nextflow_id"

    # execution report and software versions
    for file_pattern, description, run_attr in [
        ("execution_report*", "execution report", "report"),
        ("nf_core_pipeline_software*", "software versions", "environment"),
    ]:
        artifact = ln.Artifact(
            next(Path(f"{output_dir}/pipeline_info").glob(file_pattern)),
            description=f"nextflow run {description} of {nextflow_id}",
            visibility=0,
            run=False,
        ).save()
        setattr(run, run_attr, artifact)

    # nextflow run parameters
    params_path = next(Path(f"{output_dir}/pipeline_info").glob("params*"))
    with params_path.open() as params_file:
        params = json.load(params_file)
    ln.Param(name="params", dtype="dict").save()
    run.params.add_values({"params": params})
    run.save()


args = parse_arguments()
scrnaseq_transform = ln.Transform(
    name="scrna-seq",
    version="2.7.1",
    type="pipeline",
    reference="https://github.com/nf-core/scrnaseq",
).save()
run = ln.Run(transform=scrnaseq_transform).save()
register_pipeline_io(args.input, args.output, run)
register_pipeline_metadata(args.output, run)

!python register_scrnaseq_run.py --input scrnaseq_input --output scrnaseq_output

Data lineage¶

The output data could now be accessed (in a different notebook/script) for analysis with full lineage.

matrix_af = ln.Artifact.get(description__icontains="filtered count matrix")

matrix_af.view_lineage()

_images/7fffd2a1665a298909d382c110e583a8e3ed120ac13a4d830723498b0c3a4c2f.svg

View transforms & runs on the hub¶

View the database content¶

ln.view()

Show code cell output Hide code cell output

Artifact

	uid	version	is_latest	description	key	suffix	type	size	hash	n_objects	n_observations	_hash_type	_accessor	visibility	_key_is_virtual	storage_id	transform_id	run_id	created_at	created_by_id
id
5	IgjuCHl0O2AWgntV0000	None	True	filtered count matrix	None	.h5ad	None	659819	f2cVqU6dyiqYvtKPM0XwLQ	NaN	None	md5	AnnData	1	True	1	1.0	1.0	2024-11-21 05:44:35.091754+00:00	1
4	sjscwiE64H0vIUqT0000	None	True	multiqc report	None		None	9676753	0FjHN6ty2KZzmiT5tnauVQ	59.0	None	md5-d	None	1	True	1	1.0	1.0	2024-11-21 05:44:35.065166+00:00	1
3	3OT0z6tHJcDpNqQ70000	None	True	None	scrnaseq_input/S10_L001_R2_001.fastq.gz	.fastq.gz	None	4259756	W-dGV6rDQMWXfGSAv_xL0g	NaN	None	md5	None	1	True	1	NaN	NaN	2024-11-21 05:44:35.009943+00:00	1
2	ZTT1NkrlBIGoGBBV0000	None	True	None	scrnaseq_input/S10_L001_R1_001.fastq.gz	.fastq.gz	None	1727503	UrpdRtwcAhl3QV7xfzI29w	NaN	None	md5	None	1	True	1	NaN	NaN	2024-11-21 05:44:35.009329+00:00	1
1	JWIILC5IMI923fwZ0000	None	True	None	scrnaseq_input/samplesheet.csv	.csv	None	236	QXMVrT5ZucmidIxbYJ9KHA	NaN	None	md5	None	1	True	1	NaN	NaN	2024-11-21 05:44:35.008302+00:00	1

! No records found

! No records found

! No records found

Run

	uid	started_at	finished_at	is_consecutive	reference	reference_type	transform_id	report_id	environment_id	parent_id	created_at	created_by_id
id
1	CvqlLFmjpEM9y2bloCQ4	2024-11-21 05:44:34.979255+00:00	None	None	curious_volhard	nextflow_id	1	6	7	None	2024-11-21 05:44:34.979308+00:00	1

Storage

	uid	root	description	type	region	instance_uid	run_id	created_at	created_by_id
id
1	FrTrDbU46rzE	/home/runner/work/nextflow-lamin/nextflow-lami...	None	local	None	7JUvfoPu6nFp	None	2024-11-21 05:37:18.951568+00:00	1

Transform

	uid	version	is_latest	name	key	description	type	source_code	hash	reference	reference_type	_source_code_artifact_id	created_at	created_by_id
id
1	ETS6f2KiODHO0000	2.7.1	True	scrna-seq	None	None	pipeline	None	None	https://github.com/nf-core/scrnaseq	None	None	2024-11-21 05:44:34.975276+00:00	1

ULabel

	uid	name	description	reference	reference_type	run_id	created_at	created_by_id
id
1	9SeEibaB	nextflow	None	None	None	None	2024-11-21 05:44:35.098767+00:00	1

User

	uid	handle	name	created_at
id
1	DzTjkKse	testuser1	Test User1	2024-11-21 05:37:18.946040+00:00