swh.loader.debian package

Submodules

swh.loader.debian.converters module

swh.loader.debian.converters.package_metadata_to_revision(package, directory, metadata)[source]

Convert a package dictionary to a revision suitable for storage.

Parameters:package

a dictionary with the following keys:

  • metadata: the metadata for the package, containing:
    package_info:
        changelog
        pgp_signature
    
  • directory
Returns:A revision suitable for persistence in swh.storage
swh.loader.debian.converters.package_to_release(package)[source]

Convert a package dictionary to a revision suitable for storage.

Parameters:package

a dictionary with the following keys:

  • metadata: the metadata for the package, containing:
    package_info
        changelog
            person
            date
    
  • revision
Returns:A revision suitable for persistence in swh.storage
swh.loader.debian.converters.uid_to_person(uid, encode=True)[source]

Convert an uid to a person suitable for insertion.

Parameters:
  • uid – an uid of the form “Name <email@ddress>”
  • encode – whether to convert the output to bytes or not
Returns:

a dictionary with the following keys:

  • name: the name associated to the uid
  • email: the mail associated to the uid

Return type:

dict

swh.loader.debian.loader module

exception swh.loader.debian.loader.DebianLoaderException[source]

Bases: Exception

__module__ = 'swh.loader.debian.loader'
__weakref__

list of weak references to the object (if defined)

exception swh.loader.debian.loader.PackageDownloadFailed[source]

Bases: swh.loader.debian.loader.DebianLoaderException

Raise this exception when a package extraction failed

__module__ = 'swh.loader.debian.loader'
exception swh.loader.debian.loader.PackageExtractionFailed[source]

Bases: swh.loader.debian.loader.DebianLoaderException

Raise this exception when a package extraction failed

__module__ = 'swh.loader.debian.loader'
swh.loader.debian.loader._debian_to_hashlib(hashname)[source]

Convert Debian hash names to hashlib-compatible names

swh.loader.debian.loader.download_package(package)[source]

Fetch a source package in a temporary directory and check the checksums for all files

swh.loader.debian.loader.extract_package(package, tempdir)[source]

Extract a Debian source package to a given directory

Note that after extraction the target directory will be the root of the extracted package, rather than containing it.

Parameters:
  • package (dict) – package information dictionary
  • tempdir (str) – directory where the package files are stored
Returns:

path to the dsc (str) and extraction directory (str)

Return type:

tuple

swh.loader.debian.loader.get_file_info(filepath)[source]

Retrieve the original file information from the file at filepath.

Parameters:filepath – the path to the original file
Returns:information about the original file, in a dictionary with the following keys
  • name: the file name
  • sha1, sha1_git, sha256: original file hashes
  • length: original file length
Return type:dict
swh.loader.debian.loader.get_package_metadata(package, dsc_path, extracted_path)[source]

Get the package metadata from the source package at dsc_path, extracted in extracted_path.

Parameters:
  • package – the package dict (with a dsc_path key)
  • dsc_path – path to the package’s dsc file
  • extracted_path – the path where the package got extracted
Returns:

a dictionary with the following keys:

  • history: list of (package_name, package_version) tuples parsed from the package changelog
  • source_files: information about all the files in the source package

Return type:

dict

swh.loader.debian.loader.process_package(package)[source]

Process a source package into its constituent components.

The source package will be decompressed in a temporary directory.

Parameters:

package (dict) –

a dict with the following keys:

  • name: source package name
  • version: source package version
  • dsc: the full path of the package’s DSC file.

Returns:

A tuple with two elements:

  • package: the original package dict augmented with the following keys:

    • metadata: the metadata from get_package_metadata
    • directory: the sha1_git of the root directory of the package
  • objects: a dictionary of the parsed directories and files, both indexed by id

Return type:

tuple

Raises:
class swh.loader.debian.loader.DebianLoader(config=None)[source]

Bases: swh.loader.core.loader.BufferedLoader

A loader for Debian packages

CONFIG_BASE_FILENAME = 'loader/debian'
ADDITIONAL_CONFIG = {'lister_db_url': ('str', 'postgresql:///lister-debian')}
__init__(config=None)[source]

Initialize self. See help(type(self)) for accurate signature.

load(*, origin, date, packages)[source]

Loading logic for the loader to follow:

    1. Call prepare_origin_visit() to prepare the origin and visit we will associate loading data to
    1. Store the actual origin_visit to storage
    1. Call prepare() to prepare any eventual state
    1. Call get_origin() to get the origin we work with and store
  • while True:
    1. Call cleanup() to clean up any eventual state put in place in prepare() method.
prepare_origin_visit(*, origin, date, packages)[source]

First step executed by the loader to prepare origin and visit references. Set/update self.origin, and optionally self.origin_url, self.visit_date.

prepare(*, origin, date, packages)[source]

Second step executed by the loader to prepare some state needed by the loader.

fetch_data()[source]
Fetch the data from the source the loader is currently loading
(ex: git/hg/svn/… repository).
Returns:a value that is interpreted as a boolean. If True, fetch_data needs to be called again to complete loading.
store_data()[source]

Store fetched data in the database.

Should call the maybe_load_xyz() methods, which handle the bundles sent to storage, rather than send directly.

update_packages()[source]
generate_and_load_snapshot()[source]

Create a SWH archive “snapshot” of the package being loaded, and send it to the archive.

load_status()[source]

Detailed loading status.

Defaults to logging an eventful load.

Returns: a dictionary that is eventually passed back as the task’s
result to the scheduler, allowing tuning of the task recurrence mechanism.
visit_status()[source]

Detailed visit status.

Defaults to logging a full visit.

cleanup()[source]

Last step executed by the loader.

__abstractmethods__ = frozenset()
__module__ = 'swh.loader.debian.loader'
_abc_cache = <_weakrefset.WeakSet object>
_abc_negative_cache = <_weakrefset.WeakSet object>
_abc_negative_cache_version = 115
_abc_registry = <_weakrefset.WeakSet object>

swh.loader.debian.tasks module

Module contents