Keep artifacts local in a cloud instance¶
If you want to default to keeping artifacts local in a cloud instance, enable keep_artifacts_local.
Show code cell content
!lamin login testuser1
!lamin init --storage s3://lamindb-ci/keep-artifacts-local
✓ logged in with email [email protected] (uid: DzTjkKse)
! updating cloud SQLite 's3://lamindb-ci/keep-artifacts-local/.lamindb/lamin.db' of instance 'testuser1/keep-artifacts-local'
! locked instance (to unlock and push changes to the cloud SQLite file, call: lamin disconnect)
→ initialized lamindb: testuser1/keep-artifacts-local
# !pip install 'lamindb[jupyter,aws]'
import lamindb as ln
ln.track("l9lFf83aPwRc0000")
→ connected lamindb: testuser1/keep-artifacts-local
→ created Transform('l9lFf83aPwRc0000'), started new Run('OTaHeSK8...') at 2025-06-10 20:17:00 UTC
→ notebook imports: lamindb==1.6.2
Show code cell content
# the setting should be enabled on lamin.ai
# we're temporarily setting it here only for testing purposes
ln.setup.settings.instance._keep_artifacts_local = True
You can register a managed local storage location as follows:
ln.settings.storage_local = "./my_storage_local"
→ defaulting to local storage: /home/runner/work/lamindb/lamindb/docs/faq/my_storage_local
Now, you have two storage locations: one in the S3 bucket, and the other locally.
ln.Storage.df()
Show code cell output
| uid | root | description | type | region | instance_uid | space_id | run_id | created_at | created_by_id | _aux | branch_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||
| 2 | 4y30BMVT4RjK | /home/runner/work/lamindb/lamindb/docs/faq/my_... | None | local | None | 6uGWmLpZlNoJ | 1 | None | 2025-06-10 20:17:01.128000+00:00 | 1 | None | 1 |
| 1 | jMwKVe3EGmos | s3://lamindb-ci/keep-artifacts-local | None | s3 | us-west-1 | 6uGWmLpZlNoJ | 1 | None | 2025-06-10 20:16:51.912000+00:00 | 1 | None | 1 |
Update storage description¶
You can add a description to the storage by using the description parameter:
storage_record = ln.Storage.get(root=ln.settings.storage_local.root)
storage_record.description = "Files stored locally in site X on server Y for reason ABC"
storage_record.save()
ln.Storage.df()
| uid | root | description | type | region | instance_uid | space_id | run_id | created_at | created_by_id | _aux | branch_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||
| 2 | 4y30BMVT4RjK | /home/runner/work/lamindb/lamindb/docs/faq/my_... | Files stored locally in site X on server Y for... | local | None | 6uGWmLpZlNoJ | 1 | None | 2025-06-10 20:17:01.128000+00:00 | 1 | None | 1 |
| 1 | jMwKVe3EGmos | s3://lamindb-ci/keep-artifacts-local | None | s3 | us-west-1 | 6uGWmLpZlNoJ | 1 | None | 2025-06-10 20:16:51.912000+00:00 | 1 | None | 1 |
Use local storage¶
If you save an artifact, by default, it’s stored in local storage.
original_filepath = ln.core.datasets.file_fcs()
artifact = ln.Artifact(original_filepath, description="My fcs file").save()
local_path = artifact.path
local_path
Show code cell output
PosixUPath('/home/runner/work/lamindb/lamindb/docs/faq/my_storage_local/.lamindb/JLrhJ7BsFzzJL6Do0000.fcs')
You’ll see the .fcs file named by the uid in your .lamindb/ directory under ./my_storage_local/:
ln.settings.storage_local.root.view_tree()
Show code cell output
1 sub-directory & 2 files with suffixes '.txt', '.fcs'
/home/runner/work/lamindb/lamindb/docs/faq/my_storage_local
└── .lamindb/
├── storage_uid.txt
└── JLrhJ7BsFzzJL6Do0000.fcs
Show code cell content
assert local_path.exists()
assert artifact.path.as_posix().startswith(ln.settings.storage_local.root.as_posix())
If you’d like to upload an artifact, you pass upload=True to the save() method.
artifact.save(upload=True)
Show code cell output
→ moved local artifact to cache: /home/runner/.cache/lamindb/JLrhJ7BsFzzJL6Do0000.fcs
Artifact(uid='JLrhJ7BsFzzJL6Do0000', is_latest=True, description='My fcs file', suffix='.fcs', size=19330507, hash='rCPvmZB19xs4zHZ7p_-Wrg', branch_id=1, space_id=1, storage_id=1, run_id=1, created_by_id=1, created_at=2025-06-10 20:17:01 UTC)
You now see the artifact in the S3 bucket:
ln.setup.settings.storage.root.view_tree()
Show code cell output
2 sub-directories & 3 files with suffixes '.txt', '.db', '.fcs'
s3://lamindb-ci/keep-artifacts-local
└── .lamindb/
├── JLrhJ7BsFzzJL6Do0000.fcs
├── lamin.db
├── storage_uid.txt
└── _exclusion/
And it’s no longer present in local storage:
ln.setup.settings.instance.storage_local.root.view_tree()
Show code cell output
1 sub-directory & 1 files with suffixes '.txt'
/home/runner/work/lamindb/lamindb/docs/faq/my_storage_local
└── .lamindb/
└── storage_uid.txt
Show code cell content
assert artifact.path.exists()
assert not local_path.exists()
assert artifact.path.as_posix().startswith(
ln.setup.settings.instance.storage.root.as_posix()
)
Direct upload¶
You can also directly upload a file by passing upload=True:
filepath = ln.core.datasets.file_mini_csv()
artifact2 = ln.Artifact(filepath, description="My csv file").save(upload=True)
artifact2.path
Show code cell output
S3QueryPath('s3://lamindb-ci/keep-artifacts-local/.lamindb/OkLMKkTc0n9fgrVZ0000.csv')
Now we have two files on S3:
ln.Artifact.df(include="storage__root")
Show code cell output
| uid | key | storage__root | |
|---|---|---|---|
| id | |||
| 2 | OkLMKkTc0n9fgrVZ0000 | None | s3://lamindb-ci/keep-artifacts-local |
| 1 | JLrhJ7BsFzzJL6Do0000 | None | s3://lamindb-ci/keep-artifacts-local |
Show code cell content
assert artifact2.path.exists()
Pre-existing artifacts¶
Assume we already have a file in our registered local storage location:
Show code cell source
file_in_local_storage = ln.core.datasets.file_bam()
file_in_local_storage.rename("./my_storage_local/output.bam")
ln.UPath("my_storage_local/").view_tree()
1 sub-directory & 2 files with suffixes '.txt', '.bam'
/home/runner/work/lamindb/lamindb/docs/faq/my_storage_local
├── .lamindb/
│ └── storage_uid.txt
└── output.bam
If we create an artifact from it, it remains where it is during saving:
my_existing_file = ln.Artifact(
"./my_storage_local/output.bam", description="my existing file"
).save()
ln.UPath("my_storage_local/").view_tree()
Show code cell output
1 sub-directory & 2 files with suffixes '.txt', '.bam'
/home/runner/work/lamindb/lamindb/docs/faq/my_storage_local
├── .lamindb/
│ └── storage_uid.txt
└── output.bam
The storage path of the artifact is constructed using key because key_is_virtual=False:
my_existing_file
Show code cell output
Artifact(uid='bsNBrK9EVxOH98pM0000', is_latest=True, key='output.bam', description='my existing file', suffix='.bam', size=18, hash='D2yxELM5U3VLeyvrwWUMUA', branch_id=1, space_id=1, storage_id=2, run_id=1, created_by_id=1, created_at=2025-06-10 20:17:03 UTC)
However, if we decide to upload the artifact, we’ll use the uid for constructing the storage path and switch key_is_virtual=True:
my_existing_file.save(upload=True)
Show code cell output
→ moved local artifact to cache: /home/runner/.cache/lamindb/output.bam
Artifact(uid='bsNBrK9EVxOH98pM0000', is_latest=True, key='output.bam', description='my existing file', suffix='.bam', size=18, hash='D2yxELM5U3VLeyvrwWUMUA', branch_id=1, space_id=1, storage_id=1, run_id=1, created_by_id=1, created_at=2025-06-10 20:17:03 UTC)
Here is the remote path of the artifact:
my_existing_file.path
Show code cell output
S3QueryPath('s3://lamindb-ci/keep-artifacts-local/.lamindb/bsNBrK9EVxOH98pM0000.bam')
And here are the contents of the storage locations:
# the path on S3
ln.setup.settings.storage.root.view_tree()
# the local path
ln.setup.settings.instance.storage_local.root.view_tree()
Show code cell output
2 sub-directories & 5 files with suffixes '.txt', '.csv', '.bam', '.db', '.fcs'
s3://lamindb-ci/keep-artifacts-local
└── .lamindb/
├── JLrhJ7BsFzzJL6Do0000.fcs
├── OkLMKkTc0n9fgrVZ0000.csv
├── bsNBrK9EVxOH98pM0000.bam
├── lamin.db
├── storage_uid.txt
└── _exclusion/
1 sub-directory & 1 files with suffixes '.txt'
/home/runner/work/lamindb/lamindb/docs/faq/my_storage_local
└── .lamindb/
└── storage_uid.txt
Delete the test instance¶
Delete the artifacts:
artifact.delete(permanent=True)
artifact2.delete(permanent=True)
my_existing_file.delete(permanent=True)
Delete the instance:
ln.setup.delete("keep-artifacts-local", force=True)
Show code cell output
→ deleted storage record on hub 3ca40af4cf6f42c5bc158a41de9c8d1d
→ deleted storage record on hub eb35d80e9c4d5d02aab65a03c95bb70c
→ deleted instance record on hub cc7f2489bf7251f79ff9ca8df7ac045b