lamindb.UPath ¶

class lamindb.UPath(*args, protocol=None, **storage_options)¶

Bases: PathlibPathShim, Path

Path-like access to files.

Offers the typical access patterns of file systems and object stores, for example:

upath = ln.UPath("s3://my-bucket/my-folder/my-file.txt")
upath.exists()  # file exists in storage

The class is an extension of universal_pathlib.UPath.

Parameters:: pathlike – A string or Path to a local or cloud file or folder.

See also

from_auth(): If the S3 URI is in an S3 storage managed by LaminHub, use this method to request federated AWS credentials.

Examples

Create a path object from a local file:

upath = ln.UPath("./my-folder/my-file.txt")

Create a path from a S3 URI:

upath = ln.UPath("s3://my-bucket/my-folder/my-file.txt")      # create a path that detects local AWS credentials
upath = ln.UPath.from_auth("s3://managed-bucket/my-folder/")  # create a path that requests federated AWS credentials from LaminHub

Create a path object from a GS URI:

upath = ln.UPath("gs://my-bucket/my-folder/my-file.txt")

In addition to what pathlib.Path and universal_pathlib.UPath offer, ln.UPath offers the following methods:

upath.view_tree() # view a file tree
upath.upload_from("local-file.txt") # upload a local file
upath.download_to("local-file.txt") # download a file
upath.synchronize_to("local-folder/") # synchronize a folder

property anchor¶: The concatenation of the drive and root, or ‘’.

property drive¶: The drive prefix (letter or UNC path), if any.

property fs: AbstractFileSystem¶: The cached fsspec filesystem instance for the path.

property modified: datetime | None¶: Return modified time stamp.

property name: str¶

property parent¶: The logical parent of the path.

property parents¶: A sequence of this path’s logical parents.

property parts¶: An object providing sequence-like access to the components in the filesystem path.

property path: str¶: The path that a fsspec filesystem can use.

property protocol: str¶: The fsspec protocol for the path.

property root¶: The root of the path, if any.

property stem¶: The final path component, minus its last suffix.

property storage_options: Mapping[str, Any]¶: The fsspec storage options for the path.

property suffix¶

The final component’s last suffix, if any.

This includes the leading period. For example: ‘.txt’

property suffixes¶

A list of the final component’s suffixes, if any.

These include the leading periods. For example: [‘.tar’, ‘.gz’]

classmethod cwd()¶

Return type:: UPath

classmethod home()¶

Return type:: UPath

classmethod from_auth()¶

Create an authenticated path object.

This method makes a request to a LaminHub to obtain standard federated AWS credentials for the UPath object, compliant with universal_pathlib and fsspec.

Note: This only works for paths inside storage locations whose access is managed by LaminHub (Storage). For paths outside managed storage locations, local or environment credentials are used using the standard UPath search strategy, from AWS environment variables or AWS configuration files. Non-S3 paths are returned unchanged if they are already UPath objects.

Parameters:: path (lamindb.core.types.UPathStr) – A S3 URI.
Return type:: UPath

Example

Create a path object from a S3 URI with federated AWS credentials:

upath = ln.UPath.from_auth("s3://managed-bucket/my-folder/")

joinuri(uri)¶

Join with urljoin behavior for UPath instances

Return type:: UPath

with_segments(*pathsegments)¶

Return type:: Self

joinpath(*pathsegments)¶

Return type:: Self

as_uri()¶

Return type:: str

is_reserved()¶

Return type:: bool

relative_to(other, /, *_deprecated, walk_up=False)¶

Return the relative path to another path identified by the passed arguments. If the operation is not possible (because this is not related to the other path), raise ValueError.

The walk_up parameter controls whether .. may be used to resolve the path.

Return type:: Self

is_relative_to(other, /, *_deprecated)¶

Return type:: bool

stat(*, follow_symlinks=True)¶

Return the result of the stat() system call on this path, like os.stat() does.

Return type:: UPathStatResult

lstat()¶

Return type:: UPathStatResult

exists(*, follow_symlinks=True)¶

Whether this path exists.

This method normally follows symlinks; to check whether a symlink exists, add the argument follow_symlinks=False.

Return type:: bool

is_dir()¶

Whether this path is a directory.

Return type:: bool

is_file()¶

Whether this path is a regular file (also True for symlinks pointing to regular files).

Return type:: bool

is_mount()¶

Return type:: bool

is_symlink()¶

Return type:: bool

is_junction()¶

Return type:: bool

is_block_device()¶

Return type:: bool

is_char_device()¶

Return type:: bool

is_fifo()¶

Return type:: bool

is_socket()¶

Return type:: bool

samefile(other_path)¶

Return type:: bool

open(mode='r', *args, **fsspec_kwargs)¶

Open the file pointed by this path and return a file object, as the built-in open() function does.

Return type:: IO[Any]

iterdir()¶

Yield path objects of the directory contents.

The children are yielded in arbitrary order, and the special entries ‘.’ and ‘..’ are not included.

Return type:: Generator[UPath, None, None]

glob(pattern, *, case_sensitive=None)¶

Iterate over this subtree and yield all existing files (of any kind, including directories) matching the given relative pattern.

Return type:: Generator[UPath, None, None]

rglob(pattern, *, case_sensitive=None)¶

Recursively yield all existing files (of any kind, including directories) matching the given relative pattern, anywhere in this subtree.

Return type:: Generator[UPath, None, None]

absolute()¶

Return type:: Self

is_absolute()¶

Return type:: bool

resolve(strict=False)¶

Make the path absolute, resolving all symlinks on the way and also normalizing it.

Return type:: Self

owner()¶

Return type:: str

group()¶

Return type:: str

readlink()¶

Return type:: Self

touch(mode=438, exist_ok=True)¶

Return type:: None

mkdir(mode=511, parents=False, exist_ok=False)¶

Return type:: None

chmod(mode, *, follow_symlinks=True)¶

Return type:: None

lchmod(mode)¶

Return type:: None

unlink(missing_ok=False)¶

Remove this file or link. If the path is a directory, use rmdir() instead.

Return type:: None

rmdir(recursive=True)¶

Return type:: None

rename(target, *, recursive=<object object>, maxdepth=<object object>, **kwargs)¶

Move file, see fsspec.AbstractFileSystem.mv.

For example:

upath = UPath("s3://my-bucket/my-file")
upath.rename(UPath("s3://my-bucket/my-file-renamed"))
upath.rename("my-file-renamed")

Return type:: Self

replace(target)¶

Return type:: UPath

symlink_to(target, target_is_directory=False)¶

Return type:: None

hardlink_to(target)¶

Return type:: None

expanduser()¶

Return type:: Self

synchronize_to(destination, error_no_origin=True, print_progress=False, just_check=False, disable_boto3=False, **kwargs)¶

Sync to a local destination path.

Return type:: bool

upload_from(local_path, create_folder=None, print_progress=True, **kwargs)¶

Upload from the local path to self (a destination in the cloud).

If the local path is a directory, recursively upload its contents.

Parameters:

local_path (lamindb.core.types.UPathStr) – A local path of a file or directory.
create_folder (bool | None, default: None) – Only applies if local_path is a directory and then defaults to True. If True, make a new folder in the destination using the directory name of local_path. If False, upload the contents of the directory to to the root-level of the destination.
print_progress (bool, default: True) – Print progress.

Return type:

UPath

Returns:

The destination path.

to_url()¶

Public storage URL.

Generates a public URL for an object in an S3 bucket using fsspec’s UPath, considering the bucket’s region.

Parameters:: upath – A UPath object representing an S3 path.
Return type:: str
Returns:: A string containing the public URL to the S3 object.

download_to(local_path, print_progress=True, use_boto3=False, **kwargs)¶

Download from self (a destination in the cloud) to the local path.

Parameters:

local_path (lamindb.core.types.UPathStr) – A local path to download to.
print_progress (bool, default: True) – Print progress.
use_boto3 (bool, default: False) – Use boto3 instead of s3fs to download a single file from s3. Ignored if the path is not a file or not in s3
**kwargs – Additional arguments for the download.

view_tree(*, level=2, only_dirs=False, n_max_files_per_dir_and_type=100, n_max_files=1000, include_paths=None, skip_suffixes=None)¶

Print a visual tree structure of files & directories.

Parameters:

level (int, default: 2) – If 1, only iterate through one level, if 2 iterate through 2 levels, if -1 iterate through entire hierarchy.
only_dirs (bool, default: False) – Only iterate through directories.
n_max_files (int, default: 1000) – Display limit. Will only show this many files. Doesn’t affect count.
include_paths (set[Any] | None, default: None) – Restrict to these paths.
skip_suffixes (list[str] | None, default: None) – Skip directories with these suffixes.

Return type:

None

Example

View the file tree of a directory:

import lamindb as ln
dir_path = ln.examples.datasets.generate_cell_ranger_files(
    "sample_001", ln.settings.storage
)
ln.UPath(dir_path).view_tree()
#> 3 subdirectories, 15 files
#> sample_001
#> ├── web_summary.html
#> ├── metrics_summary.csv
#> ├── molecule_info.h5
#> ├── filtered_feature_bc_matrix
#> │   ├── features.tsv.gz
#> │   ├── barcodes.tsv.gz
#> │   └── matrix.mtx.gz
#> ├── analysis
#> │   └── analysis.csv
#> ├── raw_feature_bc_matrix
#> │   ├── features.tsv.gz
#> │   ├── barcodes.tsv.gz
#> │   └── matrix.mtx.gz
#> ├── possorted_genome_bam.bam.bai
#> ├── cloupe.cloupe
#> ├── possorted_genome_bam.bam
#> ├── filtered_feature_bc_matrix.h5
#> └── raw_feature_bc_matrix.h5

read_bytes()¶: Open the file in bytes mode, read it, and close the file.

read_text(encoding=None, errors=None)¶: Open the file in text mode, read it, and close the file.

write_bytes(data)¶: Open the file in bytes mode, write to it, and close the file.

write_text(data, encoding=None, errors=None, newline=None)¶: Open the file in text mode, write to it, and close the file.

walk(top_down=True, on_error=None, follow_symlinks=False)¶: Walk the directory tree from this directory, similar to os.walk().

as_posix()¶: Return the string representation of the path with forward (/) slashes.

with_name(name)¶: Return a new path with the file name changed.

with_stem(stem)¶: Return a new path with the stem changed.

with_suffix(suffix)¶: Return a new path with the file suffix changed. If the path has no suffix, add given suffix. If the given suffix is an empty string, remove the suffix from the path.

match(path_pattern, *, case_sensitive=None)¶: Return True if this path matches the given pattern.