Query & search registries¶
!lamin init --storage ./mydata
Show code cell output
→ connected lamindb: testuser1/mydata
import lamindb as ln
# create toy data
ln.Artifact(ln.core.datasets.file_jpg_paradisi05(), description="My image").save()
ln.Artifact.from_df(ln.core.datasets.df_iris(), description="The iris collection").save()
ln.Artifact(ln.core.datasets.file_fastq(), description="My fastq").save()
Show code cell output
→ connected lamindb: testuser1/mydata
! no run & transform get linked, consider calling ln.context.track()
! no run & transform get linked, consider calling ln.context.track()
! no run & transform get linked, consider calling ln.context.track()
Artifact(uid='9kQngQRFgzS9Ypt10000', is_latest=True, description='My fastq', suffix='.fastq.gz', size=20, hash='hi7ZmAzz8sfMd3vIQr-57Q', _hash_type='md5', visibility=1, _key_is_virtual=True, created_by_id=1, storage_id=1, updated_at='2024-09-06 08:58:00 UTC')
Look up metadata¶
For entities where we don’t store more than 100k records, a look up object can be a convenient way of selecting a record.
Consider the User
registry:
users = ln.User.lookup(field="handle")
With auto-complete, we find a user:
user = users.testuser1
user
User(uid='DzTjkKse', handle='testuser1', name='Test User1', updated_at='2024-09-06 08:57:58 UTC')
You can also get a dictionary, if you prefer:
users_dict = ln.User.lookup().dict()
Query exactly one record¶
# by uid
ln.User.get("DzTjkKse")
# by any expression involving fields
ln.User.get(handle="testuser1")
User(uid='DzTjkKse', handle='testuser1', name='Test User1', updated_at='2024-09-06 08:57:58 UTC')
Query sets of records¶
Filter for all artifacts created by a user:
ln.Artifact.filter(created_by=user).df()
uid | version | is_latest | description | key | suffix | type | size | hash | n_objects | n_observations | _hash_type | _accessor | visibility | _key_is_virtual | storage_id | transform_id | run_id | created_by_id | updated_at | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
id | ||||||||||||||||||||
1 | hkeIlncMhi6Xg7RD0000 | None | True | My image | None | .jpg | None | 29358 | r4tnqmKI_SjrkdLzpuWp4g | None | None | md5 | None | 1 | True | 1 | None | None | 1 | 2024-09-06 08:57:59.994196+00:00 |
2 | WVabCyERwTAIDT770000 | None | True | The iris collection | None | .parquet | dataset | 5629 | VQGEXvcC5ZvBwMSm-3CHWg | None | None | md5 | DataFrame | 1 | True | 1 | None | None | 1 | 2024-09-06 08:58:00.102105+00:00 |
3 | 9kQngQRFgzS9Ypt10000 | None | True | My fastq | None | .fastq.gz | None | 20 | hi7ZmAzz8sfMd3vIQr-57Q | None | None | md5 | None | 1 | True | 1 | None | None | 1 | 2024-09-06 08:58:00.108506+00:00 |
To access the results encoded in a filter statement, execute its return value with one of:
.df()
: A pandasDataFrame
with each record in a row..all()
: AQuerySet
..one()
: Exactly one record. Will raise an error if there is none. Is equivalent to the.get()
method shown above..one_or_none()
: Either one record orNone
if there is no query result.
Note
The ORMs in LaminDB are Django Models and any Django query works. LaminDB extends Django’s API for data scientists.
Under the hood, any .filter()
call translates into a SQL select statement.
.one()
and .one_or_none()
are two parts of LaminDB’s API that are borrowed from SQLAlchemy.
Search for metadata¶
ln.Artifact.search("iris").df()
Show code cell output
uid | version | is_latest | description | key | suffix | type | size | hash | n_objects | n_observations | _hash_type | _accessor | visibility | _key_is_virtual | storage_id | transform_id | run_id | created_by_id | updated_at | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
id | ||||||||||||||||||||
2 | WVabCyERwTAIDT770000 | None | True | The iris collection | None | .parquet | dataset | 5629 | VQGEXvcC5ZvBwMSm-3CHWg | None | None | md5 | DataFrame | 1 | True | 1 | None | None | 1 | 2024-09-06 08:58:00.102105+00:00 |
Let us create 500 notebook objects with fake titles and save them:
transforms = [ln.Transform(name=title, type="notebook") for title in ln.core.datasets.fake_bio_notebook_titles(n=500)]
ln.save(transforms)
We can now search for any combination of terms:
ln.Transform.search("intestine").df().head()
Show code cell output
uid | version | is_latest | name | key | description | type | source_code | hash | reference | reference_type | _source_code_artifact_id | created_by_id | updated_at | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
id | ||||||||||||||
1 | plvBYoAOmxHn0000 | None | True | Igg1 Speed cells classify intestine. | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.697821+00:00 |
20 | mEPvglTZOYyV0000 | None | True | Intestine research IgG1 IgG IgG IgA. | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.699070+00:00 |
21 | U4prgJp7OsSH0000 | None | True | Intestine Olfactory ensheathing cells IgA IgG1... | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.699133+00:00 |
42 | ieHQXKmg0lQu0000 | None | True | Intestine cluster IgD IgG visualize intestine. | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.700470+00:00 |
46 | j4cpPN7UBGQ30000 | None | True | Igg1 Vestibule of the ear IgG IgG IgA intestine. | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.700725+00:00 |
Leverage relations¶
Django has a double-under-score syntax to filter based on related tables.
This syntax enables you to traverse several layers of relations:
ln.Artifact.filter(run__created_by__handle__startswith="testuse").df()
Show code cell output
uid | version | is_latest | description | key | suffix | type | size | hash | n_objects | n_observations | _hash_type | _accessor | visibility | _key_is_virtual | storage_id | transform_id | run_id | created_by_id | updated_at | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
id |
The filter selects all artifacts based on the users who ran the generating notebook.
Under the hood, in the SQL database, it’s joining the artifact table with the run and the user table.
Beyond __startswith
, Django supports about two dozen field comparators field__comparator=value
. Below follow some of them.
and¶
ln.Artifact.filter(suffix=".jpg", created_by=user).df()
Show code cell output
uid | version | is_latest | description | key | suffix | type | size | hash | n_objects | n_observations | _hash_type | _accessor | visibility | _key_is_virtual | storage_id | transform_id | run_id | created_by_id | updated_at | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
id | ||||||||||||||||||||
1 | hkeIlncMhi6Xg7RD0000 | None | True | My image | None | .jpg | None | 29358 | r4tnqmKI_SjrkdLzpuWp4g | None | None | md5 | None | 1 | True | 1 | None | None | 1 | 2024-09-06 08:57:59.994196+00:00 |
less than/ greater than¶
Or subset to artifacts smaller than 10kB. Here, we can’t use keyword arguments, but need an explicit where statement.
ln.Artifact.filter(created_by=user, size__lt=1e4).df()
Show code cell output
uid | version | is_latest | description | key | suffix | type | size | hash | n_objects | n_observations | _hash_type | _accessor | visibility | _key_is_virtual | storage_id | transform_id | run_id | created_by_id | updated_at | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
id | ||||||||||||||||||||
2 | WVabCyERwTAIDT770000 | None | True | The iris collection | None | .parquet | dataset | 5629 | VQGEXvcC5ZvBwMSm-3CHWg | None | None | md5 | DataFrame | 1 | True | 1 | None | None | 1 | 2024-09-06 08:58:00.102105+00:00 |
3 | 9kQngQRFgzS9Ypt10000 | None | True | My fastq | None | .fastq.gz | None | 20 | hi7ZmAzz8sfMd3vIQr-57Q | None | None | md5 | None | 1 | True | 1 | None | None | 1 | 2024-09-06 08:58:00.108506+00:00 |
or¶
ln.Artifact.filter().filter(ln.Q(suffix=".jpg") | ln.Q(suffix=".fastq.gz")).df()
Show code cell output
uid | version | is_latest | description | key | suffix | type | size | hash | n_objects | n_observations | _hash_type | _accessor | visibility | _key_is_virtual | storage_id | transform_id | run_id | created_by_id | updated_at | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
id | ||||||||||||||||||||
1 | hkeIlncMhi6Xg7RD0000 | None | True | My image | None | .jpg | None | 29358 | r4tnqmKI_SjrkdLzpuWp4g | None | None | md5 | None | 1 | True | 1 | None | None | 1 | 2024-09-06 08:57:59.994196+00:00 |
3 | 9kQngQRFgzS9Ypt10000 | None | True | My fastq | None | .fastq.gz | None | 20 | hi7ZmAzz8sfMd3vIQr-57Q | None | None | md5 | None | 1 | True | 1 | None | None | 1 | 2024-09-06 08:58:00.108506+00:00 |
in¶
ln.Artifact.filter(suffix__in=[".jpg", ".fastq.gz"]).df()
Show code cell output
uid | version | is_latest | description | key | suffix | type | size | hash | n_objects | n_observations | _hash_type | _accessor | visibility | _key_is_virtual | storage_id | transform_id | run_id | created_by_id | updated_at | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
id | ||||||||||||||||||||
1 | hkeIlncMhi6Xg7RD0000 | None | True | My image | None | .jpg | None | 29358 | r4tnqmKI_SjrkdLzpuWp4g | None | None | md5 | None | 1 | True | 1 | None | None | 1 | 2024-09-06 08:57:59.994196+00:00 |
3 | 9kQngQRFgzS9Ypt10000 | None | True | My fastq | None | .fastq.gz | None | 20 | hi7ZmAzz8sfMd3vIQr-57Q | None | None | md5 | None | 1 | True | 1 | None | None | 1 | 2024-09-06 08:58:00.108506+00:00 |
order by¶
ln.Artifact.filter().order_by("-updated_at").df()
Show code cell output
uid | version | is_latest | description | key | suffix | type | size | hash | n_objects | n_observations | _hash_type | _accessor | visibility | _key_is_virtual | storage_id | transform_id | run_id | created_by_id | updated_at | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
id | ||||||||||||||||||||
3 | 9kQngQRFgzS9Ypt10000 | None | True | My fastq | None | .fastq.gz | None | 20 | hi7ZmAzz8sfMd3vIQr-57Q | None | None | md5 | None | 1 | True | 1 | None | None | 1 | 2024-09-06 08:58:00.108506+00:00 |
2 | WVabCyERwTAIDT770000 | None | True | The iris collection | None | .parquet | dataset | 5629 | VQGEXvcC5ZvBwMSm-3CHWg | None | None | md5 | DataFrame | 1 | True | 1 | None | None | 1 | 2024-09-06 08:58:00.102105+00:00 |
1 | hkeIlncMhi6Xg7RD0000 | None | True | My image | None | .jpg | None | 29358 | r4tnqmKI_SjrkdLzpuWp4g | None | None | md5 | None | 1 | True | 1 | None | None | 1 | 2024-09-06 08:57:59.994196+00:00 |
contains¶
ln.Transform.filter(name__contains="search").df().head(10)
Show code cell output
uid | version | is_latest | name | key | description | type | source_code | hash | reference | reference_type | _source_code_artifact_id | created_by_id | updated_at | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
id | ||||||||||||||
6 | zrpb30rlALx10000 | None | True | Apocrine Sweat Gland research IgG1 rank. | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.698178+00:00 |
11 | pNlRzkhnwwwM0000 | None | True | Apocrine Sweat Gland IgG2 Midbrain IgG2 IgG1 r... | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.698499+00:00 |
14 | P4LNM1r5iGwh0000 | None | True | Veins result Olfactory ensheathing cells Speed... | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.698689+00:00 |
16 | HvVnIPEysmA50000 | None | True | Peripolar Cell Midbrain research Apocrine swea... | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.698816+00:00 |
20 | mEPvglTZOYyV0000 | None | True | Intestine research IgG1 IgG IgG IgA. | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.699070+00:00 |
28 | XouO3uBRIUrm0000 | None | True | Research visualize IgG2 rank. | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.699576+00:00 |
29 | JTrGMTkZABpp0000 | None | True | Igg2 Unipolar brush cells IgG3 research intest... | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.699640+00:00 |
32 | X1cAk0OinqW00000 | None | True | Olfactory Ensheathing Cells IgA research Sperm... | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.699831+00:00 |
36 | c3VAt72ouuJQ0000 | None | True | Igg3 research Spermatozoon visualize Melanotro... | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.700085+00:00 |
37 | HNKunYFPnyaw0000 | None | True | Igg3 Unipolar brush cells IgA research Olfacto... | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.700148+00:00 |
And case-insensitive:
ln.Transform.filter(name__icontains="Search").df().head(10)
Show code cell output
uid | version | is_latest | name | key | description | type | source_code | hash | reference | reference_type | _source_code_artifact_id | created_by_id | updated_at | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
id | ||||||||||||||
6 | zrpb30rlALx10000 | None | True | Apocrine Sweat Gland research IgG1 rank. | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.698178+00:00 |
11 | pNlRzkhnwwwM0000 | None | True | Apocrine Sweat Gland IgG2 Midbrain IgG2 IgG1 r... | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.698499+00:00 |
14 | P4LNM1r5iGwh0000 | None | True | Veins result Olfactory ensheathing cells Speed... | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.698689+00:00 |
16 | HvVnIPEysmA50000 | None | True | Peripolar Cell Midbrain research Apocrine swea... | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.698816+00:00 |
20 | mEPvglTZOYyV0000 | None | True | Intestine research IgG1 IgG IgG IgA. | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.699070+00:00 |
28 | XouO3uBRIUrm0000 | None | True | Research visualize IgG2 rank. | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.699576+00:00 |
29 | JTrGMTkZABpp0000 | None | True | Igg2 Unipolar brush cells IgG3 research intest... | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.699640+00:00 |
32 | X1cAk0OinqW00000 | None | True | Olfactory Ensheathing Cells IgA research Sperm... | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.699831+00:00 |
36 | c3VAt72ouuJQ0000 | None | True | Igg3 research Spermatozoon visualize Melanotro... | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.700085+00:00 |
37 | HNKunYFPnyaw0000 | None | True | Igg3 Unipolar brush cells IgA research Olfacto... | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.700148+00:00 |
startswith¶
ln.Transform.filter(name__startswith="Research").df()
Show code cell output
uid | version | is_latest | name | key | description | type | source_code | hash | reference | reference_type | _source_code_artifact_id | created_by_id | updated_at | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
id | ||||||||||||||
28 | XouO3uBRIUrm0000 | None | True | Research visualize IgG2 rank. | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.699576+00:00 |
65 | CabDJZDSjocn0000 | None | True | Research IgG2 result IgG Midbrain IgG2 intestine. | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.701950+00:00 |
71 | Q8NcGKt9ANDy0000 | None | True | Research IgG1 Apocrine sweat gland IgG result. | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.705192+00:00 |
93 | 5hABW8ua4lQj0000 | None | True | Research candidate Semicircular canals IgG. | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.706539+00:00 |
131 | FbV2ajlCEewE0000 | None | True | Research IgA intestine candidate Unipolar brus... | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.708808+00:00 |
143 | TqxLrssuauBq0000 | None | True | Research Olfactory ensheathing cells intestina... | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.711963+00:00 |
177 | z3FRhHwBc8ql0000 | None | True | Research IgA Peripolar cell rank Melanotropes ... | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.714034+00:00 |
281 | Ti1sYoEZqGqI0000 | None | True | Research Oligodendrocytes Peripolar cell. | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.725331+00:00 |
342 | N6Z131OHjspO0000 | None | True | Research IgG1 investigate classify cluster. | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.731439+00:00 |
364 | OUx78m6gHrEl0000 | None | True | Research Oligodendrocytes IgG3 IgG1 Midbrain I... | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.732753+00:00 |
379 | hBMSpYcGAx810000 | None | True | Research IgG IgA IgG1. | None | None | notebook | None | None | None | None | None | 1 | 2024-09-06 08:58:01.733674+00:00 |
Show code cell content
# clean up test instance
!lamin delete --force mydata
!rm -r mydata
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.11.9/x64/bin/lamin", line 8, in <module>
sys.exit(main())
^^^^^^
File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/rich_click/rich_command.py", line 367, in __call__
return super().__call__(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/rich_click/rich_command.py", line 152, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/lamin_cli/__main__.py", line 195, in delete
return delete(instance, force=force)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/lamindb_setup/_delete.py", line 98, in delete
n_objects = check_storage_is_empty(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/lamindb_setup/core/upath.py", line 776, in check_storage_is_empty
raise InstanceNotEmpty(message)
lamindb_setup.core.upath.InstanceNotEmpty: Storage /home/runner/work/lamindb/lamindb/docs/mydata/.lamindb contains 3 objects ('_is_initialized' ignored) - delete them prior to deleting the instance