rs_server_common.s3_storage_handler package

Submodules

rs_server_common.s3_storage_handler.s3_storage_handler module

TODO Docstring to be added.

class rs_server_common.s3_storage_handler.s3_storage_handler.GetKeysFromS3Config(s3_files: list, bucket: str, local_prefix: str, overwrite: bool = False, max_retries: int = 20)

Bases: object

S3 configuration for download

s3_files

A list with the S3 object keys to be downloaded.

Type:: list

bucket

The S3 bucket name.

Type:: str

local_prefix

The local prefix where files will be downloaded.

Type:: str

overwrite

Flag indicating whether to overwrite existing files. Default is False.

Type:: bool, optional

max_retries

The maximum number of download retries. Default is DWN_S3FILE_RETRIES.

Type:: int, optional

bucket: str

local_prefix: str

max_retries: int = 20

overwrite: bool = False

s3_files: list

class rs_server_common.s3_storage_handler.s3_storage_handler.PutFilesToS3Config(files: list, bucket: str, s3_path: str, max_retries: int = 20)

Bases: object

Configuration for uploading files to S3.

files

A list with the local file paths to be uploaded.

Type:: List

bucket

The S3 bucket name.

Type:: str

s3_path

The S3 path where files will be uploaded.

Type:: str

max_retries

The maximum number of upload retries. Default is UP_S3FILE_RETRIES.

Type:: int, optional

bucket: str

files: list

max_retries: int = 20

s3_path: str

class rs_server_common.s3_storage_handler.s3_storage_handler.S3StorageHandler(access_key_id, secret_access_key, endpoint_url, region_name)

Bases: object

Interacts with an S3 storage

S3StorageHandler for interacting with an S3 storage service.

access_key_id

The access key ID for S3 authentication.

Type:: str

secret_access_key

The secret access key for S3 authentication.

Type:: str

endpoint_url

The endpoint URL for the S3 service.

Type:: str

region_name

The region name.

Type:: str

s3_client

The s3 client to interact with the s3 storage

Type:: boto3.client

check_bucket_access(bucket)

Check the accessibility of an S3 bucket.

Parameters:: bucket (str) – The S3 bucket name.
Raises:: RuntimeError – If an error occurs during the bucket access check.

check_file_overwriting(local_file, overwrite)

Check if file exists and determine if it should be overwritten.

Parameters:

local_file (str) – Path to the local file.
overwrite (bool) – Whether to overwrite the existing file.

Returns:

True if the file should be overwritten, False otherwise.

Return type:

bool

Note: - If the file already exists and the overwrite flag is set to True, the function logs a message, deletes the existing file, and returns True. - If the file already exists and the overwrite flag is set to False, the function logs a warning message, and returns False. In this case, the existing file won’t be deleted. - If the file doesn’t exist, the function returns True.

connect_s3()

Establish a connection to the S3 service.

If the S3 client is not already instantiated, this method calls the private __get_s3_client method to create an S3 client instance using the provided credentials and configuration (see __init__).

delete_file_from_s3(bucket, s3_obj)

Delete a file from S3.

Parameters:

bucket (str) – The S3 bucket name.
s3_obj (str) – The S3 object key.

Raises:

RuntimeError – If an error occurs during the bucket access check.

disconnect_s3(): Close the connection to the S3 service.

files_to_be_downloaded(bucket, paths)

Create a list with the S3 keys to be downloaded.

The list will have the s3 keys to be downloaded from the bucket. It contains pairs (local_prefix_where_the_file_will_be_downloaded, full_s3_key_path) If a s3 key doesn’t exist, the pair will be (None, requested_s3_key_path)

Parameters:

bucket (str) – The S3 bucket name.
paths (list) – List of S3 object keys.

Returns:

List of tuples (local_prefix, full_s3_key_path).

Return type:

list

files_to_be_uploaded(paths)

Creates a list with the local files to be uploaded.

The list contains pairs (s3_path, absolute_local_file_path) If the local file doesn’t exist, the pair will be (None, requested_file_to_upload)

Parameters:: paths (list) – List of local file paths.
Returns:: List of tuples (s3_path, absolute_local_file_path).
Return type:: list

static get_basename(input_path)

Get the filename from a full path.

Parameters:: int_path (str) – The full path.
Returns:: The filename.
Return type:: str

get_keys_from_s3(config: GetKeysFromS3Config) → list

Download S3 keys specified in the configuration.

Parameters:: config (GetKeysFromS3Config) – Configuration for the S3 download.
Returns:: A list with the S3 keys that couldn’t be downloaded.
Return type:: List[str]
Raises:: Exception – Any unexpected exception raised during the download process.

The function attempts to download files from S3 according to the provided configuration. It returns a list of S3 keys that couldn’t be downloaded successfully.

static get_secrets_from_file(secrets, secret_file)

Read secrets from a specified file.

It reads the secrets from .s3cfg or aws credentials files This function should not be used in production

Parameters:

secrets (dict) – Dictionary to store retrieved secrets.
secret_file (str) – Path to the file containing secrets.
logger (Logger, optional) – Logger instance for error logging.

list_s3_files_obj(bucket, prefix)

Retrieve the content of an S3 directory.

Parameters:

bucket (str) – The S3 bucket name.
prefix (str) – The S3 object key prefix.

Returns:

List containing S3 object keys.

Return type:

list

put_files_to_s3(config: PutFilesToS3Config) → list

Upload files to S3 according to the provided configuration.

Parameters:: config (PutFilesToS3Config) – Configuration for the S3 upload.
Returns:: A list with the local file paths that couldn’t be uploaded.
Return type:: List[str]
Raises:: Exception – Any unexpected exception raised during the upload process.

The function attempts to upload files to S3 according to the provided configuration. It returns a list of local files that couldn’t be uploaded successfully.

static s3_path_parser(s3_url)

Parses S3 URL to extract bucket, prefix, and file.

Parameters:: s3_url (str) – The S3 URL.
Returns:: Tuple containing bucket, prefix, and file.
Return type:: tuple

transfer_from_s3_to_s3(config: TransferFromS3ToS3Config) → list

Copy S3 keys specified in the configuration. :param config: Configuration object containing bucket source, bucket destination,

S3 files, maximum retries.

Returns:: A list of S3 keys that failed to be copied.
Return type:: list
Raises:: Exception – Any unexpected exception raised during the upload process.

wait_timeout(timeout)

Wait for a specified timeout duration (minimum 200 ms).

This function implements a simple timeout mechanism, where it sleeps for 0.2 seconds in each iteration until the cumulative sleep time reaches the specified timeout duration.

Parameters:: timeout (float) – The total duration to wait in seconds.
Returns:: None

class rs_server_common.s3_storage_handler.s3_storage_handler.TransferFromS3ToS3Config(s3_files: list, bucket_src: str, bucket_dst: str, copy_only: bool = False, max_retries: int = 20)

Bases: object

S3 configuration for copying a list with keys between buckets

s3_files

A list with the S3 object keys to be copied.

Type:: list

bucket_src

The source S3 bucket name.

Type:: str

bucket_dst

The destination S3 bucket name.

Type:: str

max_retries

The maximum number of download retries. Default is DWN_S3FILE_RETRIES.

Type:: int, optional

bucket_dst: str

bucket_src: str

copy_only: bool = False

max_retries: int = 20

s3_files: list

Module contents

S3 storage handler package.

TODO Is it really useful to add a package here ?