swh.core.config module

swh.core.config.exists_accessible(filepath: str) → bool

Check whether a file exists, and is accessible.

Returns:

True if the file exists and is accessible False if the file does not exist

Raises:

PermissionError if the file cannot be read. –

swh.core.config.read_raw_config(base_config_path: str) → Dict[str, Any]

Read the raw config corresponding to base_config_path.

Can read yml files.

swh.core.config.config_exists(path)

Check whether the given config exists

swh.core.config.config_basepath(config_path: str) → str

Return the base path of a configuration file

swh.core.config.config_path(config_path)

Check whether the given config exists

swh.core.config.read(conf_file: str | None = None, default_conf: Dict[str, Tuple[str, Any]] | None = None) → Dict[str, Any]

Read the user’s configuration file.

Fill in the gap using default_conf. default_conf is similar to this:

DEFAULT_CONF = {
    'a': ('str', '/tmp/swh-loader-git/log'),
    'b': ('str', 'dbname=swhloadergit')
    'c': ('bool', true)
    'e': ('bool', None)
    'd': ('int', 10)
}

If conf_file is None, return the default config.

swh.core.config.priority_read(conf_filenames: List[str], default_conf: Dict[str, Tuple[str, Any]] | None = None)

Try reading the configuration files from conf_filenames, in order, and return the configuration from the first one that exists.

default_conf has the same specification as it does in read.

swh.core.config.merge_default_configs(base_config, *other_configs)

Merge several default config dictionaries, from left to right

swh.core.config.merge_configs(base: Dict[str, Any] | None, other: Dict[str, Any] | None)

Merge two config dictionaries

This does merge config dicts recursively, with the rules, for every value of the dicts (with ‘val’ not being a dict):

• None + type -> type

• type + None -> None

• dict + dict -> dict (merged)

• val + dict -> TypeError

• dict + val -> TypeError

• val + val -> val (other)

for instance:

>>> d1 = {
...   'key1': {
...     'skey1': 'value1',
...     'skey2': {'sskey1': 'value2'},
...   },
...   'key2': 'value3',
... }

with

>>> d2 = {
...   'key1': {
...     'skey1': 'value4',
...     'skey2': {'sskey2': 'value5'},
...   },
...   'key3': 'value6',
... }

will give:

>>> d3 = {
...   'key1': {
...     'skey1': 'value4',  # <-- note this
...     'skey2': {
...       'sskey1': 'value2',
...       'sskey2': 'value5',
...     },
...   },
...   'key2': 'value3',
...   'key3': 'value6',
... }
>>> assert merge_configs(d1, d2) == d3

Note that no type checking is done for anything but dicts.

swh.core.config.swh_config_paths(base_filename: str) → List[str]

Return the Software Heritage specific configuration paths for the given filename.

swh.core.config.prepare_folders(conf, *keys)

Prepare the folder mentioned in config under keys.

swh.core.config.load_global_config()

Load the global Software Heritage config

swh.core.config.load_named_config(name, default_conf=None, global_conf=True)

Load the config named name from the Software Heritage configuration paths.

If global_conf is True (default), read the global configuration too.

swh.core.config.load_from_envvar(default_config: Dict[str, Any] | None = None) → Dict[str, Any]

Load configuration yaml file from the environment variable SWH_CONFIG_FILENAME, eventually enriched with default configuration key/value from the default_config dict if provided.

Returns:

Configuration dict

Raises:

AssertionError if SWH_CONFIG_FILENAME is undefined –

swh.core.config.get_swh_backend_module(swh_package: str, cls: str) → Tuple[str, type | None]

swh.core.config.get_swh_backend_from_fullmodule(fullmodule: str) → Tuple[str | None, str | None]

swh.core.config.list_swh_backends(package: str) → List[str]

swh.core.config.list_db_config_entries(cfg) → Generator[Tuple[str, str, dict, str], None, None]

List all the db config entries in the given config structure

Generates quadruplets (module, path, cfg, cnxstr) where:

• the swh module name (aka top level config entries, eg. ‘storage’, ‘scheduler’, etc.)

• path: the path within the config structure of the (sub)config entry in which the db connection has been found,

• cfg: the config subentry from the given gcfg in which the db config has been found; it contains at least a ‘cls’ key,

• db: the db connection string