Keep artifacts local in a cloud instance¶

If you want to default to keeping artifacts local in a cloud instance, enable keep_artifacts_local.

# !pip install 'lamindb[jupyter,aws]'
import lamindb as ln

ln.track("l9lFf83aPwRc0000")

→ connected lamindb: testuser1/keep-artifacts-local

→ created Transform('l9lFf83a'), started new Run('tYlY3THz') at 2024-12-20 15:04:39 UTC

→ notebook imports: lamindb==0.77.3

You can register a managed local storage location as follows:

ln.settings.storage_local = "./my_storage_local"

→ defaulting to local storage: /home/runner/work/lamindb/lamindb/docs/faq/my_storage_local

Now, you have two storage locations: one in the S3 bucket, and the other locally.

ln.Storage.df()

Show code cell output Hide code cell output

	uid	root	description	type	region	instance_uid	run_id	created_at	created_by_id
id
2	5R9kM1sjs4kL	/home/runner/work/lamindb/lamindb/docs/faq/my_...	None	local	None	6uGWmLpZlNoJ	None	2024-12-20 15:04:41.004826+00:00	1
1	nnvIf9j3GqJ1	s3://lamindb-ci/keep-artifacts-local	None	s3	us-west-1	6uGWmLpZlNoJ	None	2024-12-20 15:04:30.223670+00:00	1

Update storage description¶

You can add a description to the storage by using the description parameter:

storage_record = ln.Storage.get(root=ln.settings.storage_local.root)
storage_record.description = "Files stored locally in site X on server Y for reason ABC"
storage_record.save()
ln.Storage.df()

	uid	root	description	type	region	instance_uid	run_id	created_at	created_by_id
id
2	5R9kM1sjs4kL	/home/runner/work/lamindb/lamindb/docs/faq/my_...	Files stored locally in site X on server Y for...	local	None	6uGWmLpZlNoJ	None	2024-12-20 15:04:41.004826+00:00	1
1	nnvIf9j3GqJ1	s3://lamindb-ci/keep-artifacts-local	None	s3	us-west-1	6uGWmLpZlNoJ	None	2024-12-20 15:04:30.223670+00:00	1

Use local storage¶

If you save an artifact, by default, it’s stored in local storage.

original_filepath = ln.core.datasets.file_fcs()
artifact = ln.Artifact(original_filepath, description="My fcs file").save()
local_path = artifact.path
local_path

You’ll see the .fcs file named by the uid in your .lamindb/ directory under ./my_storage_local/:

ln.settings.storage_local.root.view_tree()

If you’d like to upload an artifact, you pass upload=True to the save() method.

artifact.save(upload=True)

You now see the artifact in the S3 bucket:

ln.setup.settings.storage.root.view_tree()

And it’s no longer present in local storage:

ln.setup.settings.instance.storage_local.root.view_tree()

Direct upload¶

You can also directly upload a file by passing upload=True:

filepath = ln.core.datasets.file_mini_csv()
artifact2 = ln.Artifact(filepath, description="My csv file").save(upload=True)
artifact2.path

Now we have two files on S3:

ln.Artifact.df(include="storage__root")

Show code cell output Hide code cell output

	uid	key	description	storage__root
id
2	hDML5vEpJaZ2i3B40000	None	My csv file	s3://lamindb-ci/keep-artifacts-local
1	uwOdTdGOYGJBx3eB0000	None	My fcs file	s3://lamindb-ci/keep-artifacts-local

Pre-existing artifacts¶

Assume we already have a file in our registered local storage location:

1 sub-directory & 2 files with suffixes '', '.bam'
/home/runner/work/lamindb/lamindb/docs/faq/my_storage_local
├── .lamindb/
│   └── _is_initialized
└── output.bam

If we create an artifact from it, it remains where it is during saving:

my_existing_file = ln.Artifact(
    "./my_storage_local/output.bam", description="my existing file"
).save()
ln.UPath("my_storage_local/").view_tree()

The storage path of the artifact is constructed using key because key_is_virtual=False:

my_existing_file

However, if we decide to upload the artifact, we’ll use the uid for constructing the storage path and switch key_is_virtual=True:

my_existing_file.save(upload=True)

Here is the remote path of the artifact:

my_existing_file.path

And here are the contents of the storage locations:

# the path on S3
ln.setup.settings.storage.root.view_tree()
# the local path
ln.setup.settings.instance.storage_local.root.view_tree()

Delete the test instance¶

Delete the artifacts:

artifact.delete(permanent=True)
artifact2.delete(permanent=True)
my_existing_file.delete(permanent=True)

Delete the instance:

ln.setup.delete("keep-artifacts-local", force=True)