Query & search registries

This guide walks through all the ways of finding metadata records in LaminDB registries.

# !pip install lamindb
!lamin init --storage ./test-registries
Hide code cell output 
→ connected lamindb: testuser1/test-registries

We’ll need some toy data.

import lamindb as ln

# create toy data
ln.Artifact(ln.core.datasets.file_jpg_paradisi05(), description="My image").save()
ln.Artifact.from_df(ln.core.datasets.df_iris(), description="The iris collection").save()
ln.Artifact(ln.core.datasets.file_fastq(), description="My fastq").save()

# see the content of the artifact registry
ln.Artifact.df()
Hide code cell output 
→ connected lamindb: testuser1/test-registries

! no run & transform got linked, call `ln.track()` & re-run

! no run & transform got linked, call `ln.track()` & re-run

! no run & transform got linked, call `ln.track()` & re-run
uid version is_latest description key suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id run_id created_at created_by_id
id
3 5HDCARjvOUq8cyrt0000 None True My fastq None .fastq.gz None 20 hi7ZmAzz8sfMd3vIQr-57Q None None md5 None 1 True 1 None None 2024-11-21 05:39:17.108712+00:00 1
2 fXGQY0XFbbAIV77K0000 None True The iris collection None .parquet dataset 5097 K1jn6pPlqIC6ebZQfW84NQ None None md5 DataFrame 1 True 1 None None 2024-11-21 05:39:17.094165+00:00 1
1 AYFndDSI0Ut9RKhH0000 None True My image None .jpg None 29358 r4tnqmKI_SjrkdLzpuWp4g None None md5 None 1 True 1 None None 2024-11-21 05:39:16.976294+00:00 1

Look up metadata

For registries with less than 100k records, auto-completing a Lookup object is the most convenient way of finding a record.

For example, take the User registry:

# query the database for all users, optionally pass the field that creates the key
users = ln.User.lookup(field="handle")

# the lookup object is a NamedTuple
users
Hide code cell output 
Lookup(testuser1=User(uid='DzTjkKse', handle='testuser1', name='Test User1', created_at=2024-11-21 05:39:12 UTC), dict=<bound method Lookup.dict of <lamin_utils._lookup.Lookup object at 0x7f7e648a2300>>)

With auto-complete, we find a specific user record:

user = users.testuser1
user
Hide code cell output 
User(uid='DzTjkKse', handle='testuser1', name='Test User1', created_at=2024-11-21 05:39:12 UTC)

You can also get a dictionary:

users_dict = ln.User.lookup().dict()
users_dict
Hide code cell output 
{'testuser1': User(uid='DzTjkKse', handle='testuser1', name='Test User1', created_at=2024-11-21 05:39:12 UTC)}

Query exactly one record

get errors if more than one matching records are found.

# by the universal base62 uid
ln.User.get("DzTjkKse")

# by any expression involving fields
ln.User.get(handle="testuser1")
Hide code cell output 
User(uid='DzTjkKse', handle='testuser1', name='Test User1', created_at=2024-11-21 05:39:12 UTC)

Query sets of records

Filter for all artifacts created by a user:

ln.Artifact.filter(created_by=user).df()
Hide code cell output
uid version is_latest description key suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id run_id created_at created_by_id
id
1 AYFndDSI0Ut9RKhH0000 None True My image None .jpg None 29358 r4tnqmKI_SjrkdLzpuWp4g None None md5 None 1 True 1 None None 2024-11-21 05:39:16.976294+00:00 1
2 fXGQY0XFbbAIV77K0000 None True The iris collection None .parquet dataset 5097 K1jn6pPlqIC6ebZQfW84NQ None None md5 DataFrame 1 True 1 None None 2024-11-21 05:39:17.094165+00:00 1
3 5HDCARjvOUq8cyrt0000 None True My fastq None .fastq.gz None 20 hi7ZmAzz8sfMd3vIQr-57Q None None md5 None 1 True 1 None None 2024-11-21 05:39:17.108712+00:00 1

To access the results encoded in a filter statement, execute its return value with one of:

  • .df(): A pandas DataFrame with each record in a row.

  • .all(): A QuerySet.

  • .one(): Exactly one record. Will raise an error if there is none. Is equivalent to the .get() method shown above.

  • .one_or_none(): Either one record or None if there is no query result.

Note

filter() returns a QuerySet.

The ORMs in LaminDB are Django Models and any Django query works. LaminDB extends Django’s API for data scientists.

Under the hood, any .filter() call translates into a SQL select statement.

.one() and .one_or_none() are two parts of LaminDB’s API that are borrowed from SQLAlchemy.

Search for records

Search the toy data:

ln.Artifact.search("iris").df()
Hide code cell output
uid version is_latest description key suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id run_id created_at created_by_id
id
2 fXGQY0XFbbAIV77K0000 None True The iris collection None .parquet dataset 5097 K1jn6pPlqIC6ebZQfW84NQ None None md5 DataFrame 1 True 1 None None 2024-11-21 05:39:17.094165+00:00 1

Let us create 500 notebook objects with fake titles, save, and search them:

transforms = [ln.Transform(name=title, type="notebook") for title in ln.core.datasets.fake_bio_notebook_titles(n=500)]
ln.save(transforms)

# search
ln.Transform.search("intestine").df().head(5)
Hide code cell output
uid version is_latest name key description type source_code hash reference reference_type _source_code_artifact_id created_at created_by_id
id
4 0DUVYlNRh0Jk0000 None True Iga research intestine efficiency IgY. None None notebook None None None None None 2024-11-21 05:39:26.834415+00:00 1
13 3BMQY9oqxJNv0000 None True Igy IgY Subcutaneous tissue IgA IgD intestine ... None None notebook None None None None None 2024-11-21 05:39:26.835327+00:00 1
14 aJesr4LYoFrA0000 None True Enterochromaffin-Like Cell Choroid plexus IgA ... None None notebook None None None None None 2024-11-21 05:39:26.835427+00:00 1
23 LNzC0n31VOC80000 None True Igd IgA Enterochromaffin-like cell intestine O... None None notebook None None None None None 2024-11-21 05:39:26.836489+00:00 1
28 balQGFNoCi1E0000 None True Ige IgA intestine efficiency Interstitial kidn... None None notebook None None None None None 2024-11-21 05:39:26.837089+00:00 1

Note

Currently, the LaminHub UI search is more powerful than the search of the lamindb open-source package.

Leverage relations

Django has a double-under-score syntax to filter based on related tables.

This syntax enables you to traverse several layers of relations and leverage different comparators.

ln.Artifact.filter(created_by__handle__startswith="testuse").df()
Hide code cell output
uid version is_latest description key suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id run_id created_at created_by_id
id
1 AYFndDSI0Ut9RKhH0000 None True My image None .jpg None 29358 r4tnqmKI_SjrkdLzpuWp4g None None md5 None 1 True 1 None None 2024-11-21 05:39:16.976294+00:00 1
2 fXGQY0XFbbAIV77K0000 None True The iris collection None .parquet dataset 5097 K1jn6pPlqIC6ebZQfW84NQ None None md5 DataFrame 1 True 1 None None 2024-11-21 05:39:17.094165+00:00 1
3 5HDCARjvOUq8cyrt0000 None True My fastq None .fastq.gz None 20 hi7ZmAzz8sfMd3vIQr-57Q None None md5 None 1 True 1 None None 2024-11-21 05:39:17.108712+00:00 1

The filter selects all artifacts based on the users who ran the generating notebook.

Under the hood, in the SQL database, it’s joining the artifact table with the run and the user table.

Comparators

You can qualify the type of comparison in a query by using a comparator.

Below follows a list of the most import, but Django supports about two dozen field comparators field__comparator=value.

and

ln.Artifact.filter(suffix=".jpg", created_by=user).df()
Hide code cell output
uid version is_latest description key suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id run_id created_at created_by_id
id
1 AYFndDSI0Ut9RKhH0000 None True My image None .jpg None 29358 r4tnqmKI_SjrkdLzpuWp4g None None md5 None 1 True 1 None None 2024-11-21 05:39:16.976294+00:00 1

less than/ greater than

Or subset to artifacts smaller than 10kB. Here, we can’t use keyword arguments, but need an explicit where statement.

ln.Artifact.filter(created_by=user, size__lt=1e4).df()
Hide code cell output
uid version is_latest description key suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id run_id created_at created_by_id
id
2 fXGQY0XFbbAIV77K0000 None True The iris collection None .parquet dataset 5097 K1jn6pPlqIC6ebZQfW84NQ None None md5 DataFrame 1 True 1 None None 2024-11-21 05:39:17.094165+00:00 1
3 5HDCARjvOUq8cyrt0000 None True My fastq None .fastq.gz None 20 hi7ZmAzz8sfMd3vIQr-57Q None None md5 None 1 True 1 None None 2024-11-21 05:39:17.108712+00:00 1

in

ln.Artifact.filter(suffix__in=[".jpg", ".fastq.gz"]).df()
Hide code cell output
uid version is_latest description key suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id run_id created_at created_by_id
id
1 AYFndDSI0Ut9RKhH0000 None True My image None .jpg None 29358 r4tnqmKI_SjrkdLzpuWp4g None None md5 None 1 True 1 None None 2024-11-21 05:39:16.976294+00:00 1
3 5HDCARjvOUq8cyrt0000 None True My fastq None .fastq.gz None 20 hi7ZmAzz8sfMd3vIQr-57Q None None md5 None 1 True 1 None None 2024-11-21 05:39:17.108712+00:00 1

order by

ln.Artifact.filter().order_by("-updated_at").df()
Hide code cell output
uid version is_latest description key suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id run_id created_at created_by_id
id
3 5HDCARjvOUq8cyrt0000 None True My fastq None .fastq.gz None 20 hi7ZmAzz8sfMd3vIQr-57Q None None md5 None 1 True 1 None None 2024-11-21 05:39:17.108712+00:00 1
2 fXGQY0XFbbAIV77K0000 None True The iris collection None .parquet dataset 5097 K1jn6pPlqIC6ebZQfW84NQ None None md5 DataFrame 1 True 1 None None 2024-11-21 05:39:17.094165+00:00 1
1 AYFndDSI0Ut9RKhH0000 None True My image None .jpg None 29358 r4tnqmKI_SjrkdLzpuWp4g None None md5 None 1 True 1 None None 2024-11-21 05:39:16.976294+00:00 1

contains

ln.Transform.filter(name__contains="search").df().head(5)
Hide code cell output
uid version is_latest name key description type source_code hash reference reference_type _source_code_artifact_id created_at created_by_id
id
4 0DUVYlNRh0Jk0000 None True Iga research intestine efficiency IgY. None None notebook None None None None None 2024-11-21 05:39:26.834415+00:00 1
7 z3MXjxym8nNI0000 None True Ige IgY IgG4 research. None None notebook None None None None None 2024-11-21 05:39:26.834708+00:00 1
18 DnotbbIEcnQM0000 None True Igg4 IgE IgY research Clitoris visualize inves... None None notebook None None None None None 2024-11-21 05:39:26.835888+00:00 1
20 YYvHo4wdFO3s0000 None True Research rank cluster IgG IgG4 IgG3 IgD. None None notebook None None None None None 2024-11-21 05:39:26.836132+00:00 1
37 kL90tGGLwjns0000 None True Igy research Hyaline cartilage IgY. None None notebook None None None None None 2024-11-21 05:39:26.838166+00:00 1

And case-insensitive:

ln.Transform.filter(name__icontains="Search").df().head(5)
Hide code cell output
uid version is_latest name key description type source_code hash reference reference_type _source_code_artifact_id created_at created_by_id
id
4 0DUVYlNRh0Jk0000 None True Iga research intestine efficiency IgY. None None notebook None None None None None 2024-11-21 05:39:26.834415+00:00 1
7 z3MXjxym8nNI0000 None True Ige IgY IgG4 research. None None notebook None None None None None 2024-11-21 05:39:26.834708+00:00 1
18 DnotbbIEcnQM0000 None True Igg4 IgE IgY research Clitoris visualize inves... None None notebook None None None None None 2024-11-21 05:39:26.835888+00:00 1
20 YYvHo4wdFO3s0000 None True Research rank cluster IgG IgG4 IgG3 IgD. None None notebook None None None None None 2024-11-21 05:39:26.836132+00:00 1
37 kL90tGGLwjns0000 None True Igy research Hyaline cartilage IgY. None None notebook None None None None None 2024-11-21 05:39:26.838166+00:00 1

startswith

ln.Transform.filter(name__startswith="Research").df()
Hide code cell output
uid version is_latest name key description type source_code hash reference reference_type _source_code_artifact_id created_at created_by_id
id
20 YYvHo4wdFO3s0000 None True Research rank cluster IgG IgG4 IgG3 IgD. None None notebook None None None None None 2024-11-21 05:39:26.836132+00:00 1
50 OpGqAUtwlnVX0000 None True Research Enterochromaffin-like cell SA node ce... None None notebook None None None None None 2024-11-21 05:39:26.839443+00:00 1
58 Z3ljWlw5PDBV0000 None True Research visualize rank. None None notebook None None None None None 2024-11-21 05:39:26.840206+00:00 1
60 xF0XX3VQi7UM0000 None True Research Hyaline cartilage intestinal. None None notebook None None None None None 2024-11-21 05:39:26.840397+00:00 1
139 kur3ysQWIgIL0000 None True Research IgD Enterochromaffin-like cell IgG2 s... None None notebook None None None None None 2024-11-21 05:39:26.855603+00:00 1
228 pTVWRbzH1Ffs0000 None True Research intestinal Cholinergic neurons visual... None None notebook None None None None None 2024-11-21 05:39:26.867326+00:00 1
255 KMPNRurxE1Cx0000 None True Research cluster IgD. None None notebook None None None None None 2024-11-21 05:39:26.869795+00:00 1
257 BLTl6uwFoKGD0000 None True Research result IgY cluster result. None None notebook None None None None None 2024-11-21 05:39:26.869979+00:00 1
288 erEd29ZrCPnO0000 None True Research IgA Schwann cells Enterochromaffin-li... None None notebook None None None None None 2024-11-21 05:39:26.876386+00:00 1
289 lDnoXo6KKqsT0000 None True Research visualize investigate IgG1 IgG IgD. None None notebook None None None None None 2024-11-21 05:39:26.876498+00:00 1
331 VeESLE16uiRg0000 None True Research study IgY SA node cell IgA IgY IgG IgY. None None notebook None None None None None 2024-11-21 05:39:26.883976+00:00 1
337 FQGarzj9t8mb0000 None True Research Osteoprogenitor cell Osteoprogenitor ... None None notebook None None None None None 2024-11-21 05:39:26.884546+00:00 1
401 yhpMlujjtS6v0000 None True Research IgE Peptidergic neural cells IgD. None None notebook None None None None None 2024-11-21 05:39:26.894045+00:00 1
444 8Ighsp3M1BDp0000 None True Research SA node cell IgA cluster. None None notebook None None None None None 2024-11-21 05:39:26.898060+00:00 1
483 HTAD8IojxGtO0000 None True Research Clitoris Hyaline cartilage IgE intest... None None notebook None None None None None 2024-11-21 05:39:26.905272+00:00 1

or

ln.Artifact.filter(ln.Q(suffix=".jpg") | ln.Q(suffix=".fastq.gz")).df()
Hide code cell output
uid version is_latest description key suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id run_id created_at created_by_id
id
1 AYFndDSI0Ut9RKhH0000 None True My image None .jpg None 29358 r4tnqmKI_SjrkdLzpuWp4g None None md5 None 1 True 1 None None 2024-11-21 05:39:16.976294+00:00 1
3 5HDCARjvOUq8cyrt0000 None True My fastq None .fastq.gz None 20 hi7ZmAzz8sfMd3vIQr-57Q None None md5 None 1 True 1 None None 2024-11-21 05:39:17.108712+00:00 1

negate/ unequal

ln.Artifact.filter(~ln.Q(suffix=".jpg")).df()
Hide code cell output
uid version is_latest description key suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id run_id created_at created_by_id
id
2 fXGQY0XFbbAIV77K0000 None True The iris collection None .parquet dataset 5097 K1jn6pPlqIC6ebZQfW84NQ None None md5 DataFrame 1 True 1 None None 2024-11-21 05:39:17.094165+00:00 1
3 5HDCARjvOUq8cyrt0000 None True My fastq None .fastq.gz None 20 hi7ZmAzz8sfMd3vIQr-57Q None None md5 None 1 True 1 None None 2024-11-21 05:39:17.108712+00:00 1

Clean up the test instance.

!rm -r ./test-registries
!lamin delete --force test-registries
Hide code cell output 
• deleting instance testuser1/test-registries