Query & search registries

Find & access data using registries.

Setup

!lamin init --storage ./mydata
Hide code cell output
💡 connected lamindb: testuser1/mydata
import lamindb as ln

ln.settings.verbosity = "info"
💡 connected lamindb: testuser1/mydata

We’ll need some toy data:

ln.Artifact(ln.core.datasets.file_jpg_paradisi05(), description="My image").save()
ln.Artifact.from_df(ln.core.datasets.df_iris(), description="The iris collection").save()
ln.Artifact(ln.core.datasets.file_fastq(), description="My fastq").save()
Hide code cell output
❗ no run & transform get linked, consider calling ln.track()
✅ storing artifact '4srzIrbdTYxyXS7MtImd' at '/home/runner/work/lamindb/lamindb/docs/mydata/.lamindb/4srzIrbdTYxyXS7MtImd.jpg'
❗ no run & transform get linked, consider calling ln.track()
✅ storing artifact 'utOfsm3iuYMbwS2dy2vj' at '/home/runner/work/lamindb/lamindb/docs/mydata/.lamindb/utOfsm3iuYMbwS2dy2vj.parquet'
❗ no run & transform get linked, consider calling ln.track()
✅ storing artifact 'y21nvTmqKYs1ruDhwIlN' at '/home/runner/work/lamindb/lamindb/docs/mydata/.lamindb/y21nvTmqKYs1ruDhwIlN.fastq.gz'
Artifact(uid='y21nvTmqKYs1ruDhwIlN', description='My fastq', suffix='.fastq.gz', type='dataset', size=20, hash='hi7ZmAzz8sfMd3vIQr-57Q', hash_type='md5', visibility=1, key_is_virtual=True, created_by_id=1, storage_id=1, updated_at='2024-06-19 23:12:37 UTC')

Look up metadata

For entities where we don’t store more than 100k records, a look up object can be a convenient way of selecting a record.

Consider the User registry:

users = ln.User.lookup(field="handle")

With auto-complete, we find a user:

user = users.testuser1
user
User(uid='DzTjkKse', handle='testuser1', name='Test User1', updated_at='2024-06-19 23:12:35 UTC')

Note

You can also auto-complete in a dictionary:

users_dict = ln.User.lookup().dict()

Filter by metadata

Filter for all artifacts created by a user:

ln.Artifact.filter(created_by=user).df()
uid version description key suffix type accessor size hash hash_type n_objects n_observations visibility key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
1 4srzIrbdTYxyXS7MtImd None My image None .jpg dataset None 29358 r4tnqmKI_SjrkdLzpuWp4g md5 None None 1 True 1 None None 1 2024-06-19 23:12:36.998788+00:00
2 utOfsm3iuYMbwS2dy2vj None The iris collection None .parquet dataset DataFrame 5629 ah24lV9Ncc8nPL0MumEsdw md5 None None 1 True 1 None None 1 2024-06-19 23:12:37.122540+00:00
3 y21nvTmqKYs1ruDhwIlN None My fastq None .fastq.gz dataset None 20 hi7ZmAzz8sfMd3vIQr-57Q md5 None None 1 True 1 None None 1 2024-06-19 23:12:37.130086+00:00

To access the results encoded in a filter statement, execute its return value with one of:

  • .df(): A pandas DataFrame with each record stored as a row.

  • .all(): An indexable django QuerySet.

  • .list(): A list of records.

  • .one(): Exactly one record. Will raise an error if there is none.

  • .one_or_none(): Either one record or None if there is no query result.

Note

filter() returns a QuerySet.

The ORMs in LaminDB are Django Models and any Django query works. LaminDB extends Django’s API for data scientists.

Under the hood, any .filter() call translates into a SQL select statement.

.one() and .one_or_none() are two parts of LaminDB’s API that are borrowed from SQLAlchemy.

Search for metadata

ln.Artifact.search("iris").df()
uid version description key suffix type accessor size hash hash_type n_objects n_observations visibility key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
2 utOfsm3iuYMbwS2dy2vj None The iris collection None .parquet dataset DataFrame 5629 ah24lV9Ncc8nPL0MumEsdw md5 None None 1 True 1 None None 1 2024-06-19 23:12:37.122540+00:00

Let us create 500 notebook objects with fake titles and save them:

ln.save(
    [
        ln.Transform(name=title, type="notebook")
        for title in ln.core.datasets.fake_bio_notebook_titles(n=500)
    ]
)

We can now search for any combination of terms:

ln.Transform.search("intestine").df().head()
uid version name key description type reference reference_type latest_report_id source_code_id created_by_id updated_at
id
39 Rbjt59m0PqI2yBJd None Ige Proximal tubule brush border cell research... None None notebook None None None None 1 2024-06-19 23:12:42.099072+00:00
51 Pc5uFq4yjl1fcC5n None Olfactory Epithelium Supporting Cells IgG2 int... None None notebook None None None None 1 2024-06-19 23:12:42.100911+00:00
61 NXAC7LKR6Oh8ZfkI None Bone Marrow intestinal intestine Olfactory epi... None None notebook None None None None 1 2024-06-19 23:12:42.102461+00:00
67 r3Twy9cDxBjDFhwS None Intestine IgG3 visualize Ascending colon clust... None None notebook None None None None 1 2024-06-19 23:12:42.103376+00:00
70 Oyhsz4QBh9lxAu93 None Midbrain Dendritic cell cluster IgG1 IgG1 inte... None None notebook None None None None 1 2024-06-19 23:12:42.103832+00:00

Leverage relations

Django has a double-under-score syntax to filter based on related tables.

This syntax enables you to traverse several layers of relations:

ln.Artifact.filter(run__created_by__handle__startswith="testuse").df()
uid version description key suffix type accessor size hash hash_type n_objects n_observations visibility key_is_virtual storage_id transform_id run_id created_by_id updated_at
id

The filter selects all artifacts based on the users who ran the generating notebook.

(Under the hood, in the SQL database, it’s joining the artifact table with the run and the user table.)

Beyond __startswith, Django supports about two dozen field comparators field__comparator=value.

Here are some of them.

and

ln.Artifact.filter(suffix=".jpg", created_by=user).df()
uid version description key suffix type accessor size hash hash_type n_objects n_observations visibility key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
1 4srzIrbdTYxyXS7MtImd None My image None .jpg dataset None 29358 r4tnqmKI_SjrkdLzpuWp4g md5 None None 1 True 1 None None 1 2024-06-19 23:12:36.998788+00:00

less than/ greater than

Or subset to artifacts greater than 10kB. Here, we can’t use keyword arguments, but need an explicit where statement.

ln.Artifact.filter(created_by=user, size__lt=1e4).df()
uid version description key suffix type accessor size hash hash_type n_objects n_observations visibility key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
2 utOfsm3iuYMbwS2dy2vj None The iris collection None .parquet dataset DataFrame 5629 ah24lV9Ncc8nPL0MumEsdw md5 None None 1 True 1 None None 1 2024-06-19 23:12:37.122540+00:00
3 y21nvTmqKYs1ruDhwIlN None My fastq None .fastq.gz dataset None 20 hi7ZmAzz8sfMd3vIQr-57Q md5 None None 1 True 1 None None 1 2024-06-19 23:12:37.130086+00:00

or

from django.db.models import Q

ln.Artifact.filter().filter(Q(suffix=".jpg") | Q(suffix=".fastq.gz")).df()
uid version description key suffix type accessor size hash hash_type n_objects n_observations visibility key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
1 4srzIrbdTYxyXS7MtImd None My image None .jpg dataset None 29358 r4tnqmKI_SjrkdLzpuWp4g md5 None None 1 True 1 None None 1 2024-06-19 23:12:36.998788+00:00
3 y21nvTmqKYs1ruDhwIlN None My fastq None .fastq.gz dataset None 20 hi7ZmAzz8sfMd3vIQr-57Q md5 None None 1 True 1 None None 1 2024-06-19 23:12:37.130086+00:00

in

ln.Artifact.filter(suffix__in=[".jpg", ".fastq.gz"]).df()
uid version description key suffix type accessor size hash hash_type n_objects n_observations visibility key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
1 4srzIrbdTYxyXS7MtImd None My image None .jpg dataset None 29358 r4tnqmKI_SjrkdLzpuWp4g md5 None None 1 True 1 None None 1 2024-06-19 23:12:36.998788+00:00
3 y21nvTmqKYs1ruDhwIlN None My fastq None .fastq.gz dataset None 20 hi7ZmAzz8sfMd3vIQr-57Q md5 None None 1 True 1 None None 1 2024-06-19 23:12:37.130086+00:00

order by

ln.Artifact.filter().order_by("-updated_at").df()
uid version description key suffix type accessor size hash hash_type n_objects n_observations visibility key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
3 y21nvTmqKYs1ruDhwIlN None My fastq None .fastq.gz dataset None 20 hi7ZmAzz8sfMd3vIQr-57Q md5 None None 1 True 1 None None 1 2024-06-19 23:12:37.130086+00:00
2 utOfsm3iuYMbwS2dy2vj None The iris collection None .parquet dataset DataFrame 5629 ah24lV9Ncc8nPL0MumEsdw md5 None None 1 True 1 None None 1 2024-06-19 23:12:37.122540+00:00
1 4srzIrbdTYxyXS7MtImd None My image None .jpg dataset None 29358 r4tnqmKI_SjrkdLzpuWp4g md5 None None 1 True 1 None None 1 2024-06-19 23:12:36.998788+00:00

contains

ln.Transform.filter(name__contains="search").df().head(10)
uid version name key description type reference reference_type latest_report_id source_code_id created_by_id updated_at
id
2 WwuOYkNzr119YNHo None Igg Ascending colon research study candidate r... None None notebook None None None None 1 2024-06-19 23:12:42.093368+00:00
13 I2KzvOVVVka2gk63 None Igg IgG1 Basophil granulocyte IgG research can... None None notebook None None None None 1 2024-06-19 23:12:42.095071+00:00
18 JBbO5Iwnp2IwNJoS None Investigate IgG3 Dendritic cell visualize rese... None None notebook None None None None 1 2024-06-19 23:12:42.095837+00:00
21 6EzN5qvN8hJetE2F None Research IgE IgA IgE. None None notebook None None None None 1 2024-06-19 23:12:42.096295+00:00
29 mhYkocmYXPyOyYaC None Study IgM classify IgG1 research Midbrain. None None notebook None None None None 1 2024-06-19 23:12:42.097538+00:00
32 RCQ11TZ3N0eGp9Hr None Igd IgG research Intercalated duct IgM. None None notebook None None None None 1 2024-06-19 23:12:42.097999+00:00
38 zGsLRBfMltCrce8c None Igm Organ of Corti IgG1 research. None None notebook None None None None 1 2024-06-19 23:12:42.098918+00:00
39 Rbjt59m0PqI2yBJd None Ige Proximal tubule brush border cell research... None None notebook None None None None 1 2024-06-19 23:12:42.099072+00:00
52 PTGePdc2rDTv86fH None Satellite Glial Cells Mammary gland IgM Duoden... None None notebook None None None None 1 2024-06-19 23:12:42.101079+00:00
93 99YwCs2pSlsos0yC None Igm Satellite glial cells Dendritic cell IgG r... None None notebook None None None None 1 2024-06-19 23:12:42.110419+00:00

And case-insensitive:

ln.Transform.filter(name__icontains="Search").df().head(10)
uid version name key description type reference reference_type latest_report_id source_code_id created_by_id updated_at
id
2 WwuOYkNzr119YNHo None Igg Ascending colon research study candidate r... None None notebook None None None None 1 2024-06-19 23:12:42.093368+00:00
13 I2KzvOVVVka2gk63 None Igg IgG1 Basophil granulocyte IgG research can... None None notebook None None None None 1 2024-06-19 23:12:42.095071+00:00
18 JBbO5Iwnp2IwNJoS None Investigate IgG3 Dendritic cell visualize rese... None None notebook None None None None 1 2024-06-19 23:12:42.095837+00:00
21 6EzN5qvN8hJetE2F None Research IgE IgA IgE. None None notebook None None None None 1 2024-06-19 23:12:42.096295+00:00
29 mhYkocmYXPyOyYaC None Study IgM classify IgG1 research Midbrain. None None notebook None None None None 1 2024-06-19 23:12:42.097538+00:00
32 RCQ11TZ3N0eGp9Hr None Igd IgG research Intercalated duct IgM. None None notebook None None None None 1 2024-06-19 23:12:42.097999+00:00
38 zGsLRBfMltCrce8c None Igm Organ of Corti IgG1 research. None None notebook None None None None 1 2024-06-19 23:12:42.098918+00:00
39 Rbjt59m0PqI2yBJd None Ige Proximal tubule brush border cell research... None None notebook None None None None 1 2024-06-19 23:12:42.099072+00:00
52 PTGePdc2rDTv86fH None Satellite Glial Cells Mammary gland IgM Duoden... None None notebook None None None None 1 2024-06-19 23:12:42.101079+00:00
93 99YwCs2pSlsos0yC None Igm Satellite glial cells Dendritic cell IgG r... None None notebook None None None None 1 2024-06-19 23:12:42.110419+00:00

startswith

ln.Transform.filter(name__startswith="Research").df()
uid version name key description type reference reference_type latest_report_id source_code_id created_by_id updated_at
id
21 6EzN5qvN8hJetE2F None Research IgE IgA IgE. None None notebook None None None None 1 2024-06-19 23:12:42.096295+00:00
94 YptRRkqpqRfoiFYP None Research candidate Thymus study. None None notebook None None None None 1 2024-06-19 23:12:42.110566+00:00
114 FaJ3xscOlyzRtHzU None Research Intercalated duct IgA Basophil granul... None None notebook None None None None 1 2024-06-19 23:12:42.113505+00:00
221 eATuZ3zALR56MStR None Research Bone marrow Dendritic cell Olfactory ... None None notebook None None None None 1 2024-06-19 23:12:42.131898+00:00
225 jNrxYmnMGLVEs1IY None Research investigate IgG2 Ascending colon cand... None None notebook None None None None 1 2024-06-19 23:12:42.132472+00:00
231 Wk3mOlm6sosxTnho None Research Satellite glial cells IgG4 IgG4 Dendr... None None notebook None None None None 1 2024-06-19 23:12:42.202919+00:00
243 P8E8zP382vS0nOBg None Research Organ of Corti IgE IgG2 Dendritic cel... None None notebook None None None None 1 2024-06-19 23:12:42.204814+00:00
304 VAuRzRpZgrztQ102 None Research Organ of Corti Intercalated duct IgE. None None notebook None None None None 1 2024-06-19 23:12:42.213817+00:00
Hide code cell content
# clean up test instance
!lamin delete --force mydata
!rm -r mydata
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.11.9/x64/bin/lamin", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/rich_click/rich_command.py", line 367, in __call__
    return super().__call__(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/rich_click/rich_command.py", line 152, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/lamin_cli/__main__.py", line 103, in delete
    return delete(instance, force=force)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/lamindb_setup/_delete.py", line 98, in delete
    n_objects = check_storage_is_empty(
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/lamindb_setup/core/upath.py", line 779, in check_storage_is_empty
    raise InstanceNotEmpty(message)
lamindb_setup.core.upath.InstanceNotEmpty: Storage /home/runner/work/lamindb/lamindb/docs/mydata/.lamindb contains 3 objects ('_is_initialized' ignored) - delete them prior to deleting the instance
['/home/runner/work/lamindb/lamindb/docs/mydata/.lamindb/4srzIrbdTYxyXS7MtImd.jpg', '/home/runner/work/lamindb/lamindb/docs/mydata/.lamindb/_is_initialized', '/home/runner/work/lamindb/lamindb/docs/mydata/.lamindb/utOfsm3iuYMbwS2dy2vj.parquet', '/home/runner/work/lamindb/lamindb/docs/mydata/.lamindb/y21nvTmqKYs1ruDhwIlN.fastq.gz']