Nextflow¶
Nextflow is the most widely used workflow manager in bioinformatics.
This guide shows how to register a Nextflow run with inputs & outputs for the example of the nf-core/scrnaseq pipeline by running a Python script.
The approach could be automated by deploying the script via
a serverless environment trigger (e.g., AWS Lambda)
a post-run script on the Seqera Platform
What steps are executed by the nf-core/scrnaseq pipeline?
!lamin init --storage ./test-nextflow --name test-nextflow
Show code cell output
→ connected lamindb: testuser1/test-nextflow
Run the pipeline¶
Let’s download the input data from an S3 bucket.
import lamindb as ln
input_path = ln.UPath("s3://lamindb-test/scrnaseq_input")
input_path.download_to("scrnaseq_input")
→ connected lamindb: testuser1/test-nextflow
And run the nf-core/scrnaseq
pipeline.
# the test profile uses all downloaded input files as an input
!nextflow run nf-core/scrnaseq -r 2.7.1 -profile docker,test -resume --outdir scrnaseq_output
Show code cell output
N E X T F L O W ~ version 24.10.2
Pulling nf-core/scrnaseq ...
downloaded from https://github.com/nf-core/scrnaseq.git
WARN: It appears you have never run this project before -- Option `-resume` is ignored
Launching `https://github.com/nf-core/scrnaseq` [fervent_kalman] DSL2 - revision: 4171377f40 [2.7.1]
Downloading plugin [email protected]
------------------------------------------------------
,--./,-.
___ __ __ __ ___ /,-._.--~'
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
nf-core/scrnaseq v2.7.1-g4171377
------------------------------------------------------
Core Nextflow options
revision : 2.7.1
runName : fervent_kalman
containerEngine : docker
launchDir : /home/runner/work/nextflow-lamin/nextflow-lamin/docs
workDir : /home/runner/work/nextflow-lamin/nextflow-lamin/docs/work
projectDir : /home/runner/.nextflow/assets/nf-core/scrnaseq
userName : runner
profile : docker,test
configFiles :
Input/output options
input : https://github.com/nf-core/test-datasets/raw/scrnaseq/samplesheet-2-0.csv
outdir : scrnaseq_output
Mandatory arguments
aligner : star
protocol : 10XV2
Skip Tools
skip_emptydrops : true
Reference genome options
fasta : https://github.com/nf-core/test-datasets/raw/scrnaseq/reference/GRCm38.p6.genome.chr19.fa
gtf : https://github.com/nf-core/test-datasets/raw/scrnaseq/reference/gencode.vM19.annotation.chr19.gtf
Institutional config options
config_profile_name : Test profile
config_profile_description: Minimal test dataset to check pipeline function
Max job request options
max_cpus : 2
max_memory : 6.GB
max_time : 6.h
!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/scrnaseq for your analysis please cite:
* The pipeline
https://doi.org/10.5281/zenodo.3568187
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
* Software dependencies
https://github.com/nf-core/scrnaseq/blob/master/CITATIONS.md
------------------------------------------------------
[7c/a5aa1b] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:FASTQC_CHECK:FASTQC (Sample_X)
[17/952210] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:FASTQC_CHECK:FASTQC (Sample_Y)
[25/1fc9b6] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:GTF_GENE_FILTER (GRCm38.p6.genome.chr19.fa)
[c0/d1b670] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:STARSOLO:STAR_GENOMEGENERATE (GRCm38.p6.genome.chr19.fa)
[a1/d0816c] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:STARSOLO:STAR_ALIGN (Sample_X)
[3b/244516] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:STARSOLO:STAR_ALIGN (Sample_Y)
[43/94e521] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_SEURAT (Sample_X)
[ee/b09db6] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_H5AD (Sample_X)
[f8/d6155f] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_H5AD (Sample_X)
[73/567044] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_SEURAT (Sample_X)
[92/65aed9] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_SEURAT (Sample_Y)
[6e/c04cb2] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_SEURAT (Sample_Y)
[4a/ff4381] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_H5AD (Sample_Y)
[2b/a4aae7] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_H5AD (Sample_Y)
[93/25c336] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MULTIQC
[30/46ba7b] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:CONCAT_H5AD (2)
[d0/cd8e05] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:CONCAT_H5AD (1)
-[nf-core/scrnaseq] Pipeline completed successfully-
What is the full run command for the test profile?
nextflow run nf-core/scrnaseq -r 2.7.1 \
-profile docker \
-resume \
--outdir scrnaseq_output \
--input 'scrnaseq_input/samplesheet-2-0.csv' \
--skip_emptydrops \
--fasta 'https://github.com/nf-core/test-datasets/raw/scrnaseq/reference/GRCm38.p6.genome.chr19.fa' \
--gtf 'https://github.com/nf-core/test-datasets/raw/scrnaseq/reference/gencode.vM19.annotation.chr19.gtf' \
--aligner 'star' \
--protocol '10XV2' \
--max_cpus 2 \
--max_memory '6.GB' \
--max_time '6.h'
Run the registration script¶
After the pipeline has completed, a Python script registers inputs & outputs in LaminDB.
import argparse
import lamindb as ln
import json
import re
from pathlib import Path
def parse_arguments() -> argparse.Namespace:
parser = argparse.ArgumentParser()
parser.add_argument("--input", type=str, required=True)
parser.add_argument("--output", type=str, required=True)
return parser.parse_args()
def register_pipeline_io(input_dir: str, output_dir: str, run: ln.Run) -> None:
"""Register input and output artifacts for an `nf-core/scrnaseq` run."""
input_artifacts = ln.Artifact.from_dir(input_dir, run=False)
ln.save(input_artifacts)
run.input_artifacts.set(input_artifacts)
ln.Artifact(f"{output_dir}/multiqc", description="multiqc report", run=run).save()
ln.Artifact(
f"{output_dir}/star/mtx_conversions/combined_filtered_matrix.h5ad",
description="filtered count matrix",
run=run,
).save()
def register_pipeline_metadata(output_dir: str, run: ln.Run) -> None:
"""Register nf-core run metadata stored in the 'pipeline_info' folder."""
ulabel = ln.ULabel(name="nextflow").save()
run.transform.ulabels.add(ulabel)
# nextflow run id
content = next(Path(f"{output_dir}/pipeline_info").glob("execution_report_*.html")).read_text()
match = re.search(r"run id \[([^\]]+)\]", content)
nextflow_id = match.group(1) if match else ""
run.reference = nextflow_id
run.reference_type = "nextflow_id"
# execution report and software versions
for file_pattern, description, run_attr in [
("execution_report*", "execution report", "report"),
("nf_core_pipeline_software*", "software versions", "environment"),
]:
artifact = ln.Artifact(
next(Path(f"{output_dir}/pipeline_info").glob(file_pattern)),
description=f"nextflow run {description} of {nextflow_id}",
visibility=0,
run=False,
).save()
setattr(run, run_attr, artifact)
# nextflow run parameters
params_path = next(Path(f"{output_dir}/pipeline_info").glob("params*"))
with params_path.open() as params_file:
params = json.load(params_file)
ln.Param(name="params", dtype="dict").save()
run.params.add_values({"params": params})
run.save()
args = parse_arguments()
scrnaseq_transform = ln.Transform(
name="scrna-seq",
version="2.7.1",
type="pipeline",
reference="https://github.com/nf-core/scrnaseq",
).save()
run = ln.Run(transform=scrnaseq_transform).save()
register_pipeline_io(args.input, args.output, run)
register_pipeline_metadata(args.output, run)
!python register_scrnaseq_run.py --input scrnaseq_input --output scrnaseq_output
Show code cell output
→ connected lamindb: testuser1/test-nextflow
! folder is outside existing storage location, will copy files from scrnaseq_input to /home/runner/work/nextflow-lamin/nextflow-lamin/docs/test-nextflow/scrnaseq_input
Data lineage¶
The output data could now be accessed (in a different notebook/script) for analysis with full lineage.
matrix_af = ln.Artifact.get(description__icontains="filtered count matrix")
matrix_af.view_lineage()
View transforms & runs on the hub¶
View the database content¶
ln.view()
Show code cell output
Artifact
uid | version | is_latest | description | key | suffix | type | size | hash | n_objects | n_observations | _hash_type | _accessor | visibility | _key_is_virtual | storage_id | transform_id | run_id | created_at | created_by_id | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
id | ||||||||||||||||||||
5 | 0LjCSlUSP9ZEkxub0000 | None | True | filtered count matrix | None | .h5ad | None | 659819 | bOG6U2iaH8_p2nDdZ6_ulQ | NaN | None | md5 | AnnData | 1 | True | 1 | 1.0 | 1.0 | 2024-12-02 00:41:16.100269+00:00 | 1 |
4 | e96Bo8vvqesfIuYW0000 | None | True | multiqc report | None | None | 9676739 | vQoqMAtyaeZ06Vp0-a5vLg | 59.0 | None | md5-d | None | 1 | True | 1 | 1.0 | 1.0 | 2024-12-02 00:41:16.073303+00:00 | 1 | |
3 | n3qn9jKf4i4PS6Eg0000 | None | True | None | scrnaseq_input/S10_L001_R2_001.fastq.gz | .fastq.gz | None | 4259756 | W-dGV6rDQMWXfGSAv_xL0g | NaN | None | md5 | None | 1 | True | 1 | NaN | NaN | 2024-12-02 00:41:16.022692+00:00 | 1 |
2 | rJGbHAxGQh9D7fVa0000 | None | True | None | scrnaseq_input/S10_L001_R1_001.fastq.gz | .fastq.gz | None | 1727503 | UrpdRtwcAhl3QV7xfzI29w | NaN | None | md5 | None | 1 | True | 1 | NaN | NaN | 2024-12-02 00:41:16.021825+00:00 | 1 |
1 | 8SR3N7tN2KbP2QyZ0000 | None | True | None | scrnaseq_input/samplesheet.csv | .csv | None | 236 | QXMVrT5ZucmidIxbYJ9KHA | NaN | None | md5 | None | 1 | True | 1 | NaN | NaN | 2024-12-02 00:41:16.020620+00:00 | 1 |
! No records found
! No records found
! No records found
Run
uid | started_at | finished_at | is_consecutive | reference | reference_type | transform_id | report_id | environment_id | parent_id | created_at | created_by_id | |
---|---|---|---|---|---|---|---|---|---|---|---|---|
id | ||||||||||||
1 | QyAb7ZbAKHiFoQtKeMac | 2024-12-02 00:41:15.988189+00:00 | None | None | fervent_kalman | nextflow_id | 1 | 6 | 7 | None | 2024-12-02 00:41:15.988236+00:00 | 1 |
Storage
uid | root | description | type | region | instance_uid | run_id | created_at | created_by_id | |
---|---|---|---|---|---|---|---|---|---|
id | |||||||||
1 | 5U1Z56ymSRUU | /home/runner/work/nextflow-lamin/nextflow-lami... | None | local | None | 7JUvfoPu6nFp | None | 2024-12-02 00:32:43.715797+00:00 | 1 |
Transform
uid | version | is_latest | name | key | description | type | source_code | hash | reference | reference_type | _source_code_artifact_id | created_at | created_by_id | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
id | ||||||||||||||
1 | ZbMkm21oolns0000 | 2.7.1 | True | scrna-seq | None | None | pipeline | None | None | https://github.com/nf-core/scrnaseq | None | None | 2024-12-02 00:41:15.983476+00:00 | 1 |
ULabel
uid | name | description | reference | reference_type | run_id | created_at | created_by_id | |
---|---|---|---|---|---|---|---|---|
id | ||||||||
1 | 5r4bTuMF | nextflow | None | None | None | None | 2024-12-02 00:41:16.108128+00:00 | 1 |
User
uid | handle | name | created_at | |
---|---|---|---|---|
id | ||||
1 | DzTjkKse | testuser1 | Test User1 | 2024-12-02 00:32:43.711623+00:00 |
Show code cell content
# clean up the test instance:
!rm -rf test-nextflow
!lamin delete --force test-nextflow
• deleting instance testuser1/test-nextflow