What happens if I save the same artifacts & records twice?¶
LaminDB’s operations are idempotent in the sense defined in this document.
This allows you to re-run a notebook or script without erroring or duplicating data. Similar behavior holds for human data entry.
Summary¶
Metadata records¶
If you try to create any metadata record (Record
) and search_names
is True
(the default):
LaminDB will warn you if a record with similar
name
exists and display a table of similar existing records.You can then decide whether you’d like to save a record to the database or rather query an existing one from the table.
If a name already has an exact match in a registry, LaminDB will return it instead of creating a new record. For versioned entities, also the version must be passed.
If you set search_names
to False
, you’ll directly populate the DB.
Data: artifacts & collections¶
If you try to create a Artifact
object from the same content, depending on artifact_if_hash_exists
,
you’ll get an existing object, if
creation.artifact_if_hash_exists = "warn_return_existing"
(the default)you’ll get an error, if
creation.artifact_if_hash_exists = "error"
you’ll get a warning and a new object, if
creation.artifact_if_hash_exists = "warn_create_new"
Examples¶
# !pip install 'lamindb[jupyter]'
!lamin init --storage ./test-idempotency
→ connected lamindb: testuser1/test-idempotency
import lamindb as ln
import pytest
ln.settings.verbosity = "hint"
ln.context.uid = "ANW20Fr4eZgM0000"
ln.context.track()
→ connected lamindb: testuser1/test-idempotency
→ notebook imports: lamindb==0.76.6 pytest==8.3.3
• tracked pip freeze > /home/runner/.cache/lamindb/run_env_pip_c7vyAwlD6YuvMbQ6floT.txt
→ created Transform(uid='ANW20Fr4eZgM0000') & created Run(started_at='2024-09-10 15:14:07 UTC')
Metadata records¶
assert ln.settings.creation.search_names
Let us add a first record to the ULabel
registry:
label = ln.ULabel(name="My project 1")
label.save()
ULabel(uid='vN1xYc8l', name='My project 1', created_by_id=1, run_id=1, updated_at='2024-09-10 15:14:08 UTC')
If we create a new record, we’ll automatically get search results that give clues on whether we are prone to duplicating an entry:
label = ln.ULabel(name="My project 1a")
! record with similar name exists! did you mean to load it?
uid | name | description | reference | reference_type | run_id | created_by_id | updated_at | |
---|---|---|---|---|---|---|---|---|
id | ||||||||
1 | vN1xYc8l | My project 1 | None | None | None | 1 | 1 | 2024-09-10 15:14:08.654710+00:00 |
label.save()
ULabel(uid='uWhcQBBP', name='My project 1a', created_by_id=1, run_id=1, updated_at='2024-09-10 15:14:08 UTC')
In case we match an existing name directly, we’ll get the existing object:
label = ln.ULabel(name="My project 1")
→ returning existing ULabel record with same name: 'My project 1'
If we save it again, it will not create a new entry in the registry:
label.save()
ULabel(uid='vN1xYc8l', name='My project 1', created_by_id=1, run_id=1, updated_at='2024-09-10 15:14:08 UTC')
Now, if we create a third record, we’ll get two alternatives:
label = ln.ULabel(name="My project 1b")
! records with similar names exist! did you mean to load one of them?
uid | name | description | reference | reference_type | run_id | created_by_id | updated_at | |
---|---|---|---|---|---|---|---|---|
id | ||||||||
1 | vN1xYc8l | My project 1 | None | None | None | 1 | 1 | 2024-09-10 15:14:08.717514+00:00 |
2 | uWhcQBBP | My project 1a | None | None | None | 1 | 1 | 2024-09-10 15:14:08.694772+00:00 |
If we prefer to not perform a search, e.g. for performance reasons or too noisy logging, we can switch it off.
ln.settings.creation.search_names = False
label = ln.ULabel(name="My project 1c")
In this walkthrough, switch it back on:
ln.settings.creation.search_names = True
Data: artifacts and collections¶
Warn upon trying to re-ingest an existing artifact¶
assert ln.settings.creation.artifact_if_hash_exists == "warn_return_existing"
filepath = ln.core.datasets.file_fcs()
Create an Artifact
:
artifact = ln.Artifact(filepath, description="My fcs artifact")
artifact.save()
• path content will be copied to default storage upon `save()` with key `None` ('.lamindb/JHSrkrkekDFkmGZA0000.fcs')
✓ storing artifact 'JHSrkrkekDFkmGZA0000' at '/home/runner/work/lamindb/lamindb/docs/faq/test-idempotency/.lamindb/JHSrkrkekDFkmGZA0000.fcs'
Artifact(uid='JHSrkrkekDFkmGZA0000', is_latest=True, description='My fcs artifact', suffix='.fcs', size=6785467, hash='KCEXRahJ-Ui9Y6nksQ8z1A', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, updated_at='2024-09-10 15:14:09 UTC')
Show code cell content
assert artifact.hash == "KCEXRahJ-Ui9Y6nksQ8z1A"
Create an Artifact
from the same path:
artifact2 = ln.Artifact(filepath, description="My fcs artifact")
→ returning existing artifact with same hash: Artifact(uid='JHSrkrkekDFkmGZA0000', is_latest=True, description='My fcs artifact', suffix='.fcs', size=6785467, hash='KCEXRahJ-Ui9Y6nksQ8z1A', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, updated_at='2024-09-10 15:14:09 UTC')
It gives us the existing object:
assert artifact.id == artifact2.id
assert artifact.run == artifact2.run
If you save it again, nothing will happen (the operation is idempotent):
artifact2.save()
Artifact(uid='JHSrkrkekDFkmGZA0000', is_latest=True, description='My fcs artifact', suffix='.fcs', size=6785467, hash='KCEXRahJ-Ui9Y6nksQ8z1A', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, updated_at='2024-09-10 15:14:09 UTC')
In the hidden cell below, you’ll see how this interplays with data lineage.
Show code cell content
ln.context.track(new_run=True)
artifact3 = ln.Artifact(filepath, description="My fcs artifact")
assert artifact3.id == artifact2.id
assert artifact3.run != artifact2.run
assert artifact3._previous_runs.first() == artifact2.run
→ notebook imports: lamindb==0.76.6 pytest==8.3.3
• tracked pip freeze > /home/runner/.cache/lamindb/run_env_pip_31BXhq3U3NkNNzJ7FG7C.txt
→ loaded Transform(uid='ANW20Fr4eZgM0000') & created Run(started_at='2024-09-10 15:14:09 UTC')
→ returning existing artifact with same hash: Artifact(uid='JHSrkrkekDFkmGZA0000', is_latest=True, description='My fcs artifact', suffix='.fcs', size=6785467, hash='KCEXRahJ-Ui9Y6nksQ8z1A', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, updated_at='2024-09-10 15:14:09 UTC')
Error upon trying to re-ingest an existing artifact¶
ln.settings.creation.artifact_if_hash_exists = "error"
In this case, you’ll not be able to create an object from the same content:
with pytest.raises(FileExistsError):
artifact3 = ln.Artifact(filepath, description="My new fcs artifact")
Warn and create a new artifact¶
Lastly, let us discuss the following setting:
ln.settings.creation.artifact_if_hash_exists = "warn_create_new"
In this case, you’ll create a new object:
artifact4 = ln.Artifact(filepath, description="My new fcs artifact")
artifact4.save()
! creating new Artifact object despite existing artifact with same hash: Artifact(uid='JHSrkrkekDFkmGZA0000', is_latest=True, description='My fcs artifact', suffix='.fcs', size=6785467, hash='KCEXRahJ-Ui9Y6nksQ8z1A', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, updated_at='2024-09-10 15:14:09 UTC')
• path content will be copied to default storage upon `save()` with key `None` ('.lamindb/wwPG1wHA2T9UEKsv0000.fcs')
✓ storing artifact 'wwPG1wHA2T9UEKsv0000' at '/home/runner/work/lamindb/lamindb/docs/faq/test-idempotency/.lamindb/wwPG1wHA2T9UEKsv0000.fcs'
Artifact(uid='wwPG1wHA2T9UEKsv0000', is_latest=True, description='My new fcs artifact', suffix='.fcs', size=6785467, hash='KCEXRahJ-Ui9Y6nksQ8z1A', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=2, created_by_id=1, updated_at='2024-09-10 15:14:10 UTC')
You can verify that it’s a new entry by comparing the ids:
assert artifact4.id != artifact.id
ln.Artifact.filter(hash="KCEXRahJ-Ui9Y6nksQ8z1A").df()
uid | version | is_latest | description | key | suffix | type | size | hash | n_objects | n_observations | _hash_type | _accessor | visibility | _key_is_virtual | storage_id | transform_id | run_id | created_by_id | updated_at | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
id | ||||||||||||||||||||
1 | JHSrkrkekDFkmGZA0000 | None | True | My fcs artifact | None | .fcs | None | 6785467 | KCEXRahJ-Ui9Y6nksQ8z1A | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 1 | 2024-09-10 15:14:09.345927+00:00 |
2 | wwPG1wHA2T9UEKsv0000 | None | True | My new fcs artifact | None | .fcs | None | 6785467 | KCEXRahJ-Ui9Y6nksQ8z1A | None | None | md5 | None | 1 | True | 1 | 1 | 2 | 1 | 2024-09-10 15:14:10.053290+00:00 |
Show code cell content
assert len(ln.Artifact.filter(hash="KCEXRahJ-Ui9Y6nksQ8z1A").all()) == 2
!lamin delete --force test-idempotency
Show code cell output
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.11.9/x64/bin/lamin", line 8, in <module>
sys.exit(main())
^^^^^^
File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/rich_click/rich_command.py", line 367, in __call__
return super().__call__(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/rich_click/rich_command.py", line 152, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/lamin_cli/__main__.py", line 195, in delete
return delete(instance, force=force)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/lamindb_setup/_delete.py", line 98, in delete
n_objects = check_storage_is_empty(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/lamindb_setup/core/upath.py", line 776, in check_storage_is_empty
raise InstanceNotEmpty(message)
lamindb_setup.core.upath.InstanceNotEmpty: Storage /home/runner/work/lamindb/lamindb/docs/faq/test-idempotency/.lamindb contains 2 objects ('_is_initialized' ignored) - delete them prior to deleting the instance