swh.loader.core package

Submodules:

• swh.loader.core.converters module
  • prepare_contents()
  • content_for_storage()
• swh.loader.core.discovery module
  • Sample
    • Sample.contents
    • Sample.directories
    • Sample.skipped_contents
  • ArchiveDiscoveryInterface
    • ArchiveDiscoveryInterface.contents
    • ArchiveDiscoveryInterface.skipped_contents
    • ArchiveDiscoveryInterface.directories
    • ArchiveDiscoveryInterface.content_missing()
    • ArchiveDiscoveryInterface.skipped_content_missing()
    • ArchiveDiscoveryInterface.directory_missing()
  • DiscoveryStorageConnection
    • DiscoveryStorageConnection.content_missing()
    • DiscoveryStorageConnection.skipped_content_missing()
    • DiscoveryStorageConnection.directory_missing()
    • DiscoveryStorageConnection.contents
    • DiscoveryStorageConnection.skipped_contents
    • DiscoveryStorageConnection.directories
  • BaseDiscoveryGraph
    • BaseDiscoveryGraph.mark_known()
    • BaseDiscoveryGraph.mark_unknown()
    • BaseDiscoveryGraph.get_sample()
    • BaseDiscoveryGraph.do_query()
  • RandomDirSamplingDiscoveryGraph
    • RandomDirSamplingDiscoveryGraph.get_sample()
  • filter_known_objects()
• swh.loader.core.loader module
  • BaseLoader
    • BaseLoader.visit_type
    • BaseLoader.origin
    • BaseLoader.loaded_snapshot_id
    • BaseLoader.parent_origins
    • BaseLoader.from_config()
    • BaseLoader.from_configfile()
    • BaseLoader.save_data()
    • BaseLoader.get_save_data_path()
    • BaseLoader.flush()
    • BaseLoader.cleanup()
    • BaseLoader.prepare()
    • BaseLoader.get_origin()
    • BaseLoader.fetch_data()
    • BaseLoader.process_data()
    • BaseLoader.store_data()
    • BaseLoader.load_status()
    • BaseLoader.post_load()
    • BaseLoader.visit_status()
    • BaseLoader.pre_cleanup()
    • BaseLoader.build_partial_snapshot()
    • BaseLoader.load()
    • BaseLoader.load_metadata_objects()
    • BaseLoader.build_extrinsic_origin_metadata()
    • BaseLoader.statsd_timed()
    • BaseLoader.statsd_timing()
    • BaseLoader.statsd_average()
  • NodeLoader
    • NodeLoader.prepare()
    • NodeLoader.load_status()
    • NodeLoader.cleanup()
    • NodeLoader.fetch_artifact()
    • NodeLoader.process_artifact()
    • NodeLoader.fetch_data()
    • NodeLoader.store_nar_as_extids()
    • NodeLoader.visit_type
    • NodeLoader.origin
    • NodeLoader.loaded_snapshot_id
    • NodeLoader.parent_origins
  • ContentLoader
    • ContentLoader.visit_type
    • ContentLoader.fetch_artifact()
    • ContentLoader.process_artifact()
    • ContentLoader.process_data()
    • ContentLoader.store_data()
    • ContentLoader.visit_status()
    • ContentLoader.origin
    • ContentLoader.loaded_snapshot_id
    • ContentLoader.parent_origins
    • ContentLoader.snapshot
    • ContentLoader.artifact_path
    • ContentLoader.mirror_urls
  • BaseDirectoryLoader
    • BaseDirectoryLoader.visit_type
    • BaseDirectoryLoader.process_artifact()
    • BaseDirectoryLoader.process_data()
    • BaseDirectoryLoader.store_data()
    • BaseDirectoryLoader.origin
    • BaseDirectoryLoader.loaded_snapshot_id
    • BaseDirectoryLoader.parent_origins
    • BaseDirectoryLoader.snapshot
    • BaseDirectoryLoader.artifact_path
    • BaseDirectoryLoader.mirror_urls
    • BaseDirectoryLoader.visit_status()
  • TarballDirectoryLoader
    • TarballDirectoryLoader.origin
    • TarballDirectoryLoader.loaded_snapshot_id
    • TarballDirectoryLoader.parent_origins
    • TarballDirectoryLoader.snapshot
    • TarballDirectoryLoader.artifact_path
    • TarballDirectoryLoader.mirror_urls
    • TarballDirectoryLoader.directory
    • TarballDirectoryLoader.cnts
    • TarballDirectoryLoader.skipped_cnts
    • TarballDirectoryLoader.dirs
    • TarballDirectoryLoader.fetch_artifact()
• swh.loader.core.metadata_fetchers module
  • MetadataFetcherProtocol
    • MetadataFetcherProtocol.SUPPORTED_LISTERS
    • MetadataFetcherProtocol.FETCHER_NAME
    • MetadataFetcherProtocol.get_origin_metadata()
    • MetadataFetcherProtocol.get_parent_origins()
  • get_fetchers_for_lister()
• swh.loader.core.nar module
  • Nar
    • Nar.str_()
    • Nar.update()
    • Nar.serialize()
    • Nar.digest()
    • Nar.hexdigest()
    • Nar.b64digest()
    • Nar.b32digest()
• swh.loader.core.tasks module
• swh.loader.core.utils module
  • clean_dangling_folders()
  • CloneTimeout
  • CloneFailure
  • clone_with_timeout()
  • parse_visit_date()
  • compute_nar_hashes()

Module contents:

swh.loader.core.register_content() → Mapping[str, Any]

Register the current worker module’s definition

swh.loader.core.register_directory() → Mapping[str, Any]

Register the current worker module’s definition