## lamindb.Storage | class lamindb.Storage(root: str, *, description: str | None = None, space: Space | None = None, host: str | None = None) | class lamindb.Storage(*db_args) Bases: "SQLRecord", "TracksRun", "TracksUpdates" Storage locations of artifacts such as local directories or S3 buckets. A storage location is either a directory (local or a folder in the cloud) or an entire S3/GCP bucket. Parameters: * **root** -- "str" The root path of the storage location, e.g., ""./mydir"", ""s3://my-bucket"", ""s3://my-bucket/myfolder"", ""gs://my-bucket/myfolder"", ""/nfs/shared/datasets/genomics"", ""/weka/shared/models/"", ... | * **description** -- "str | None = None" An optional | description. | * **space** -- "Space | None = None" A space to restrict access | permissions to the storage location. | * **host** -- "str | None = None" For local storage locations, a | globally unique identifier for the physical machine/server hosting the storage. This distinguishes storage locations that may have the same local path but exist on different servers, e.g. ""my-institute-cluster-1"", ""my-server-abcd"". See also: "lamindb.core.Settings.storage" Current default storage location of your compute session for writing artifacts. "StorageSettings" Storage settings. Keep artifacts local in a cloud instance Avoid storing artifacts in the cloud, but keep them on local infrastructure. # Examples When you create a LaminDB instance, you configure its default storage location via "--storage": lamin init --storage ./mydatadir # or "s3://my-bucket/myfolder", "gs://my-bucket/myfolder", ... View the current default storage location for writing artifacts: import lamindb as ln ln.settings.storage Create a new cloud storage location: ln.Storage(root="s3://our-bucket/our-folder").save() Create a new local storage location: ln.Storage(root="/dir/our-shared-dir", host="our-server-123").save() Globally switch to another storage location: ln.settings.storage = "/dir/our-shared-dir" # or "s3://our-bucket/our-folder", "gs://our-bucket/our-folder", ... Or if you're operating in "keep-artifacts-local" mode (Keep artifacts local in a cloud instance ): ln.settings.local_storage = "/dir/our-other-shared-dir" View all storage locations used in your LaminDB instance: ln.Storage.to_dataframe() # Notes A storage location is written to by at most one LaminDB instance: the location’s *managing instance*. Some locations are not managed with LaminDB and, hence, do not have a managing instance. -[ Writable vs. read-only storage locations ]- The "instance_uid" field of "Storage" defines its *managing instance*. Only if a storage location's "instance_uid" matches your current instance's "uid" ("ln.settings.instance_uid"), you can write to it. All other storage locations are read-only in your current instance. Here is an example (source). [image] Some storage locations are not managed by any LaminDB instance, hence, their "instance_uid" is "None". -[ Managing access to storage locations across instances ]- You can manage access through LaminHub's fine-grained access management or through AWS policies that you attach to your S3 bucket. To enable access management via LaminHub, head over to "https://lamin.ai/{account}/infrastructure". By clicking the green button that says "Connect S3 bucket", your collaborators will access data based on their LaminHub permissions. Manage access permissions has more details. [image] By default, a storage location inherits the access permissions of its instance. If you want to further restrict access to a storage location, you can move it into a space: space = ln.Space.get(name="my-space") storage_loc = ln.Storage.get(root="s3://my-storage-location") storage_loc.space = space storage_loc.save() If you don't want to store data in the cloud, you can use local storage locations: Keep artifacts local in a cloud instance . -[ What is the ".lamindb/" directory inside a storage location? ]- It stores all artifacts that are ingested through "lamindb", indexed by the artifact "uid". This means you don't have to worry about renaming or moving files, as this all happens on the database level. Existing artifacts are typically stored in hierarchical structures with semantic folder names. Instead of copying such artifacts into ".lamindb/" upon calls of "Artifact("legacy_path").save()", LaminDB registers them with the semantic "key" representing the relative path within the storage location. These artifacts are marked with "artifact._key_is_virtual = False" and treated correspondingly. There is only a single ".lamindb/" directory per storage location. -[ What should I do if I want to bulk migrate all artifacts to another storage? ]- Currently, you can only achieve this manually and you should be careful with it. 1. Copy or move artifacts into the desired new storage location 2. Adapt the corresponding record in the {class}`~lamindb.Storage` registry by setting the "root" field to the new location 3. If your LaminDB storage location is connected to the hub, you also need to update the storage record on the hub | property host: str | None | Host identifier for local storage locations. Is "None" for locations with "type != "local"". A globally unique user-defined host identifier (cluster, server, laptop, etc.). property path: UPath Path. Uses the ".root" field and converts it into a "Path" or "UPath". uid: str Universal id, valid across DB instances. root: str Root path of storage (cloud or local path). | description: str | None | A description. type: StorageType Can be "local" vs. "s3" vs. "gs". Is auto-detected from the format of the "root" path. | region: str | None | Storage region for cloud storage locations. Host identifier for local storage locations. | instance_uid: str | None | The writing instance. Only the LaminDB instance with this "uid" can write to this storage location. This instance also governs the access permissions of the storage location unless the location is moved into a space. artifacts: Artifact Artifacts contained in this storage location. save(*args, **kwargs) Save the storage record. delete(permanent=None) Delete the storage location. This errors in case the storage location is not empty. Unlike other "SQLRecord"-based registries, this does *not* move the storage record into the trash. Parameters: | **permanent** ("bool" | "None", default: "None") -- "False" | raises an error, as soft delete is impossible. Return type: "None"