facs2/4 Jupyter Notebook lamindata

Append a new dataset

We have one dataset in storage and are about to receive a new dataset.

In this notebook, we’ll see how to manage the situation.

import lamindb as ln
import bionty as bt
import readfcs

bt.settings.organism = "human"
💡 connected lamindb: testuser1/test-facs
ln.settings.transform.stem_uid = "SmQmhrhigFPL"
ln.settings.transform.version = "1"
ln.track()
💡 notebook imports: bionty==0.44.0 lamindb==0.74.0 pytometry==0.1.4 readfcs==1.1.8 scanpy==1.10.1
💡 saved: Transform(uid='SmQmhrhigFPL5zKv', version='1', name='Append a new dataset', key='facs2', type='notebook', created_by_id=1, updated_at='2024-06-19 23:19:33 UTC')
💡 saved: Run(uid='i8muPuKXIp1KtiuaG6sa', transform_id=2, created_by_id=1)
Run(uid='i8muPuKXIp1KtiuaG6sa', started_at='2024-06-19 23:19:33 UTC', is_consecutive=True, transform_id=2, created_by_id=1)

Ingest a new artifact

Access

Let us validate and register another .fcs file from Oetjen18:

filepath = readfcs.datasets.Oetjen18_t1()

adata = readfcs.read(filepath)
adata
AnnData object with n_obs × n_vars = 241552 × 20
    var: 'n', 'channel', 'marker', '$PnR', '$PnB', '$PnE', '$PnV', '$PnG'
    uns: 'meta'

Transform: normalize

import pytometry as pm
pm.pp.split_signal(adata, var_key="channel")
pm.pp.compensate(adata)
pm.tl.normalize_biExp(adata)
adata = adata[  # subset to rows that do not have nan values
    adata.to_df().isna().sum(axis=1) == 0
]
adata.to_df().describe()
CD95 CD8 CD27 CXCR4 CCR7 LIVE/DEAD CD4 CD45RA CD3 CD49B CD14/19 CD69 CD103
count 241552.000000 241552.000000 241552.000000 241552.000000 241552.000000 241552.000000 241552.000000 241552.000000 241552.000000 241552.000000 241552.000000 241552.000000 241552.000000
mean 887.579860 1302.985717 1221.257257 877.533482 977.505533 1883.358298 556.687953 929.493316 941.166747 966.012244 1210.769935 741.523184 1003.064857
std 573.549695 827.850302 672.851319 411.966073 584.217139 932.113729 480.875917 795.550133 658.984751 456.437094 694.622980 473.287558 642.728024
min 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
25% 462.757715 493.413744 605.463427 588.047798 495.437303 1063.670965 240.623098 404.087640 477.932659 592.294399 575.401173 380.247262 475.108131
50% 774.350833 1207.624048 1110.367681 782.939692 782.981430 1951.855099 484.355203 557.904360 655.909639 800.280049 1124.574275 705.802991 775.101973
75% 1327.792103 2036.849496 1721.730010 1070.479036 1453.929567 2623.975657 729.754419 1345.771633 1218.445208 1347.042403 1742.288464 1069.175380 1420.744291
max 4053.903716 4065.495666 4095.351322 4025.827267 3999.075551 4096.000000 4088.719985 3961.255364 3940.061146 4089.445928 3982.769373 3810.774988 4023.968008

Validate cell markers

Let’s see how many markers validate:

validated = bt.CellMarker.validate(adata.var.index)
9 terms (69.20%) are not validated for name: CD95, CXCR4, CCR7, LIVE/DEAD, CD4, CD49B, CD14/19, CD69, CD103

Let’s standardize and re-validate:

adata.var.index = bt.CellMarker.standardize(adata.var.index)
validated = bt.CellMarker.validate(adata.var.index)
7 terms (53.80%) are not validated for name: CD95, CXCR4, LIVE/DEAD, CD49B, CD14/19, CD69, CD103

Next, register non-validated markers from Bionty:

records = bt.CellMarker.from_values(adata.var.index[~validated])
ln.save(records)
did not create CellMarker records for 2 non-validated names: 'CD14/19', 'LIVE/DEAD'

Manually create 1 marker:

bt.CellMarker(name="CD14/19").save()

Move metadata to obs:

validated = bt.CellMarker.validate(adata.var.index)
adata.obs = adata[:, ~validated].to_df()
adata = adata[:, validated].copy()
1 term (7.70%) is not validated for name: LIVE/DEAD

Now all markers pass validation:

validated = bt.CellMarker.validate(adata.var.index)
assert all(validated)

Register

annotate = ln.Annotate.from_anndata(adata, var_index=bt.CellMarker.name, categoricals={})
annotate.validate()
1 non-validated categories are not saved in Feature.name: ['LIVE/DEAD']!
      → to lookup categories, use lookup().columns
      → to save, run add_new_from_columns
✅ var_index is validated against CellMarker.name
True
artifact = annotate.save_artifact(description="Oetjen18_t1")
💡 path content will be copied to default storage upon `save()` with key `None` ('.lamindb/CLfzXf5jEJVr9FkMKkXG.h5ad')
✅ storing artifact 'CLfzXf5jEJVr9FkMKkXG' at '/home/runner/work/lamin-usecases/lamin-usecases/docs/test-facs/.lamindb/CLfzXf5jEJVr9FkMKkXG.h5ad'
💡 parsing feature names of X stored in slot 'var'
12 terms (100.00%) are validated for name
✅    linked: FeatureSet(uid='BiyUsbxiEyG9dJkiqDH3', n=12, dtype='float', registry='bionty.CellMarker', hash='YXolP9mtiV6-oHKhY4h6', created_by_id=1, run_id=2)
💡 parsing feature names of slot 'obs'
1 term (100.00%) is not validated for name: LIVE/DEAD
❗ skip linking features to artifact in slot 'obs'
✅ saved 1 feature set for slot: 'var'

Annotate with more labels:

efs = bt.ExperimentalFactor.lookup()
organism = bt.Organism.lookup()

artifact.labels.add(efs.fluorescence_activated_cell_sorting)
artifact.labels.add(organism.human)
artifact.describe()
Artifact(uid='CLfzXf5jEJVr9FkMKkXG', description='Oetjen18_t1', suffix='.h5ad', type='dataset', accessor='AnnData', size=46506448, hash='WbPHGIMM_5GT68rC8ZydHA', hash_type='md5', n_observations=241552, visibility=1, key_is_virtual=True, updated_at='2024-06-19 23:19:39 UTC')
  Provenance
    .created_by = 'testuser1'
    .storage = '/home/runner/work/lamin-usecases/lamin-usecases/docs/test-facs'
    .transform = 'Append a new dataset'
    .run = '2024-06-19 23:19:33 UTC'
  Labels
    .organisms = 'human'
    .experimental_factors = 'fluorescence-activated cell sorting'
  Feature sets
    'var' = 'Cd4', 'CD8', 'CD95', 'CXCR4', 'CD49B', 'CD69', 'CD3', 'CD103', 'CD27', 'CD14/19', 'Ccr7', 'CD45RA'

Inspect a PCA fo QC - this collection looks much like noise:

import scanpy as sc

markers = bt.CellMarker.lookup()

sc.pp.pca(adata)
sc.pl.pca(adata, color=markers.cd8.name)

Create a new version of the collection by appending a artifact

Query the old version:

collection_v1 = ln.Collection.filter(name="My versioned cytometry collection").one()
collection_v2 = ln.Collection(
    [artifact, collection_v1.artifacts[0]], is_new_version_of=collection_v1, version="2"
)
collection_v2.describe()
💡 adding collection ids [1] as inputs for run 2, adding parent transform 1
💡 adding artifact ids [1] as inputs for run 2, adding parent transform 1
Collection(uid='1WJqterzH0uGULecJG4W', version='2', name='My versioned cytometry collection', hash='aIyjTZDm9LEyi4udLlQ-', visibility=1)
  Provenance
    .created_by = 'testuser1'
    .transform = 'Append a new dataset'
    .run = '2024-06-19 23:19:33 UTC'
  Feature sets
    'var' = 'CD57', 'Cd19', 'Cd4', 'CD8', 'Igd', 'CD85j', 'CD11c', 'CD16', 'CD3', 'CD38', 'CD27', 'CD11B', 'Cd14', 'Ccr6', 'CD94', 'CD86', 'CXCR5', 'CXCR3', 'Ccr7', 'CD45RA'
    'obs' = 'Time', 'Cell_length', 'Dead', '(Ba138)Dd', 'Bead'
collection_v2.save()
✅ saved 1 feature set for slot: 'var'