swh.loader.core package

Submodules:

• swh.loader.core.converters module
  • prepare_contents()
  • content_for_storage()
• swh.loader.core.discovery module
  • Sample
    • Sample.contents
    • Sample.directories
    • Sample.skipped_contents
  • ArchiveDiscoveryInterface
    • ArchiveDiscoveryInterface.contents
    • ArchiveDiscoveryInterface.skipped_contents
    • ArchiveDiscoveryInterface.directories
    • ArchiveDiscoveryInterface.content_missing()
    • ArchiveDiscoveryInterface.skipped_content_missing()
    • ArchiveDiscoveryInterface.directory_missing()
  • DiscoveryStorageConnection
    • DiscoveryStorageConnection.content_missing()
    • DiscoveryStorageConnection.skipped_content_missing()
    • DiscoveryStorageConnection.directory_missing()
    • DiscoveryStorageConnection.contents
    • DiscoveryStorageConnection.skipped_contents
    • DiscoveryStorageConnection.directories
  • BaseDiscoveryGraph
    • BaseDiscoveryGraph.mark_known()
    • BaseDiscoveryGraph.mark_unknown()
    • BaseDiscoveryGraph.get_sample()
    • BaseDiscoveryGraph.do_query()
  • RandomDirSamplingDiscoveryGraph
    • RandomDirSamplingDiscoveryGraph.get_sample()
  • filter_known_objects()
• swh.loader.core.loader module
  • BaseLoader
    • BaseLoader.visit_type
    • BaseLoader.origin
    • BaseLoader.loaded_snapshot_id
    • BaseLoader.parent_origins
    • BaseLoader.from_config()
    • BaseLoader.from_configfile()
    • BaseLoader.save_data()
    • BaseLoader.get_save_data_path()
    • BaseLoader.flush()
    • BaseLoader.cleanup()
    • BaseLoader.prepare()
    • BaseLoader.get_origin()
    • BaseLoader.fetch_data()
    • BaseLoader.process_data()
    • BaseLoader.store_data()
    • BaseLoader.load_status()
    • BaseLoader.post_load()
    • BaseLoader.visit_status()
    • BaseLoader.pre_cleanup()
    • BaseLoader.load()
    • BaseLoader.load_metadata_objects()
    • BaseLoader.build_extrinsic_origin_metadata()
    • BaseLoader.statsd_timed()
    • BaseLoader.statsd_timing()
    • BaseLoader.statsd_average()
  • DVCSLoader
    • DVCSLoader.cleanup()
    • DVCSLoader.has_contents()
    • DVCSLoader.get_contents()
    • DVCSLoader.has_directories()
    • DVCSLoader.get_directories()
    • DVCSLoader.has_revisions()
    • DVCSLoader.get_revisions()
    • DVCSLoader.has_releases()
    • DVCSLoader.get_releases()
    • DVCSLoader.get_snapshot()
    • DVCSLoader.eventful()
    • DVCSLoader.store_data()
    • DVCSLoader.visit_type
    • DVCSLoader.origin
    • DVCSLoader.loaded_snapshot_id
    • DVCSLoader.parent_origins
  • NodeLoader
    • NodeLoader.prepare()
    • NodeLoader.load_status()
    • NodeLoader.cleanup()
    • NodeLoader.visit_type
    • NodeLoader.origin
    • NodeLoader.loaded_snapshot_id
    • NodeLoader.parent_origins
  • ContentLoader
    • ContentLoader.visit_type
    • ContentLoader.fetch_data()
    • ContentLoader.process_data()
    • ContentLoader.store_data()
    • ContentLoader.visit_status()
    • ContentLoader.origin
    • ContentLoader.loaded_snapshot_id
    • ContentLoader.parent_origins
    • ContentLoader.snapshot
    • ContentLoader.mirror_urls
  • DirectoryLoader
    • DirectoryLoader.visit_type
    • DirectoryLoader.fetch_data()
    • DirectoryLoader.origin
    • DirectoryLoader.loaded_snapshot_id
    • DirectoryLoader.parent_origins
    • DirectoryLoader.snapshot
    • DirectoryLoader.mirror_urls
    • DirectoryLoader.process_data()
    • DirectoryLoader.store_data()
    • DirectoryLoader.visit_status()
• swh.loader.core.metadata_fetchers module
  • MetadataFetcherProtocol
    • MetadataFetcherProtocol.SUPPORTED_LISTERS
    • MetadataFetcherProtocol.FETCHER_NAME
    • MetadataFetcherProtocol.get_origin_metadata()
    • MetadataFetcherProtocol.get_parent_origins()
  • get_fetchers_for_lister()
• swh.loader.core.tasks module
• swh.loader.core.utils module
  • clean_dangling_folders()
  • CloneTimeout
  • CloneFailure
  • clone_with_timeout()
  • parse_visit_date()
  • nix_hashes()
  • compute_nar_hashes()

Module contents:

swh.loader.core.register_content() → Mapping[str, Any]

Register the current worker module’s definition

swh.loader.core.register_directory() → Mapping[str, Any]

Register the current worker module’s definition