swh.storage package

Subpackages

• swh.storage.algos package
  • Submodules
    • swh.storage.algos.diff module
    • swh.storage.algos.dir_iterators module
    • swh.storage.algos.directory module
    • swh.storage.algos.discovery module
    • swh.storage.algos.origin module
    • swh.storage.algos.revisions_walker module
    • swh.storage.algos.snapshot module
  • Module contents
• swh.storage.api package
  • Submodules
    • swh.storage.api.client module
    • swh.storage.api.serializers module
    • swh.storage.api.server module
  • Module contents
• swh.storage.cassandra package
  • Submodules
    • swh.storage.cassandra.common module
    • swh.storage.cassandra.converters module
    • swh.storage.cassandra.cql module
    • swh.storage.cassandra.diagram module
    • swh.storage.cassandra.model module
    • swh.storage.cassandra.schema module
    • swh.storage.cassandra.storage module
  • Module contents
    • create_keyspace()
    • CassandraStorage
• swh.storage.postgresql package
  • Submodules
    • swh.storage.postgresql.converters module
    • swh.storage.postgresql.db module
    • swh.storage.postgresql.storage module
  • Module contents
• swh.storage.proxies package
  • Subpackages
    • swh.storage.proxies.blocking package
    • swh.storage.proxies.masking package
  • Submodules
    • swh.storage.proxies.buffer module
    • swh.storage.proxies.counter module
    • swh.storage.proxies.filter module
    • swh.storage.proxies.record_references module
    • swh.storage.proxies.retry module
    • swh.storage.proxies.tenacious module
    • swh.storage.proxies.validate module
  • Module contents

Submodules

• swh.storage.backfill module
  • directory_converter()
  • raw_extrinsic_metadata_converter()
  • extid_converter()
  • revision_converter()
  • release_converter()
  • snapshot_converter()
  • object_to_offset()
  • byte_ranges()
  • raw_extrinsic_metadata_target_ranges()
  • integer_ranges()
  • compute_query()
  • fetch()
  • JournalBackfiller
    • JournalBackfiller.db
    • JournalBackfiller.check_config()
    • JournalBackfiller.parse_arguments()
    • JournalBackfiller.run()
• swh.storage.cli module
  • ensure_check_config()
  • main()
• swh.storage.common module
  • origin_url_to_sha1()
• swh.storage.exc module
  • StorageDBError
  • StorageAPIError
  • NonRetryableException
  • StorageArgumentException
  • QueryTimeout
  • UnknownMetadataAuthority
  • UnknownMetadataFetcher
  • BlockedOriginException
  • MaskedObjectException
  • HashCollision
    • HashCollision.colliding_content_hashes()
• swh.storage.fixer module
  • fix_objects()
• swh.storage.in_memory module
  • Table
    • Table.partition_key()
    • Table.clustering_key()
    • Table.primary_key()
    • Table.primary_key_from_dict()
    • Table.token()
    • Table.get_partition()
    • Table.insert()
    • Table.delete()
    • Table.split_primary_key()
    • Table.get_from_partition_key()
    • Table.get_from_primary_key()
    • Table.get_from_token()
    • Table.iter_all()
    • Table.get_random()
  • InMemoryCqlRunner
    • InMemoryCqlRunner.increment_counter()
    • InMemoryCqlRunner.stat_counters()
    • InMemoryCqlRunner.content_add_prepare()
    • InMemoryCqlRunner.content_get_from_pk()
    • InMemoryCqlRunner.content_get_from_tokens()
    • InMemoryCqlRunner.content_get_random()
    • InMemoryCqlRunner.content_get_token_range()
    • InMemoryCqlRunner.content_missing_from_all_hashes()
    • InMemoryCqlRunner.content_delete()
    • InMemoryCqlRunner.content_missing_by_sha1_git()
    • InMemoryCqlRunner.content_index_add_one()
    • InMemoryCqlRunner.content_get_tokens_from_single_algo()
    • InMemoryCqlRunner.skipped_content_add_prepare()
    • InMemoryCqlRunner.skipped_content_get_from_pk()
    • InMemoryCqlRunner.skipped_content_get_from_token()
    • InMemoryCqlRunner.skipped_content_delete()
    • InMemoryCqlRunner.skipped_content_index_add_one()
    • InMemoryCqlRunner.skipped_content_get_tokens_from_single_hash()
    • InMemoryCqlRunner.directory_missing()
    • InMemoryCqlRunner.directory_add_one()
    • InMemoryCqlRunner.directory_get_random()
    • InMemoryCqlRunner.directory_get()
    • InMemoryCqlRunner.directory_get_token_range()
    • InMemoryCqlRunner.directory_delete()
    • InMemoryCqlRunner.directory_entry_add_one()
    • InMemoryCqlRunner.directory_entry_get()
    • InMemoryCqlRunner.directory_entry_get_from_name()
    • InMemoryCqlRunner.directory_entry_delete()
    • InMemoryCqlRunner.revision_missing()
    • InMemoryCqlRunner.revision_add_one()
    • InMemoryCqlRunner.revision_get_ids()
    • InMemoryCqlRunner.revision_get()
    • InMemoryCqlRunner.revision_get_token_range()
    • InMemoryCqlRunner.revision_get_random()
    • InMemoryCqlRunner.revision_delete()
    • InMemoryCqlRunner.revision_parent_add_one()
    • InMemoryCqlRunner.revision_parent_get()
    • InMemoryCqlRunner.revision_parent_delete()
    • InMemoryCqlRunner.release_missing()
    • InMemoryCqlRunner.release_add_one()
    • InMemoryCqlRunner.release_get()
    • InMemoryCqlRunner.release_get_token_range()
    • InMemoryCqlRunner.release_get_random()
    • InMemoryCqlRunner.release_delete()
    • InMemoryCqlRunner.snapshot_missing()
    • InMemoryCqlRunner.snapshot_add_one()
    • InMemoryCqlRunner.snapshot_get_token_range()
    • InMemoryCqlRunner.snapshot_get_random()
    • InMemoryCqlRunner.snapshot_branch_get_from_name()
    • InMemoryCqlRunner.snapshot_delete()
    • InMemoryCqlRunner.snapshot_branch_add_one()
    • InMemoryCqlRunner.snapshot_count_branches()
    • InMemoryCqlRunner.snapshot_branch_get()
    • InMemoryCqlRunner.snapshot_branch_delete()
    • InMemoryCqlRunner.origin_add_one()
    • InMemoryCqlRunner.origin_get_by_sha1()
    • InMemoryCqlRunner.origin_get_by_url()
    • InMemoryCqlRunner.origin_list()
    • InMemoryCqlRunner.origin_iter_all()
    • InMemoryCqlRunner.origin_bump_next_visit_id()
    • InMemoryCqlRunner.origin_generate_unique_visit_id()
    • InMemoryCqlRunner.origin_delete()
    • InMemoryCqlRunner.origin_visit_get()
    • InMemoryCqlRunner.origin_visit_add_one()
    • InMemoryCqlRunner.origin_visit_get_one()
    • InMemoryCqlRunner.origin_visit_iter_all()
    • InMemoryCqlRunner.origin_visit_iter()
    • InMemoryCqlRunner.origin_visit_delete()
    • InMemoryCqlRunner.origin_visit_status_get_range()
    • InMemoryCqlRunner.origin_visit_status_get_all_range()
    • InMemoryCqlRunner.origin_visit_status_add_one()
    • InMemoryCqlRunner.origin_visit_status_get_latest()
    • InMemoryCqlRunner.origin_visit_status_get()
    • InMemoryCqlRunner.origin_snapshot_get_all()
    • InMemoryCqlRunner.origin_visit_status_delete()
    • InMemoryCqlRunner.metadata_authority_add()
    • InMemoryCqlRunner.metadata_authority_get()
    • InMemoryCqlRunner.metadata_fetcher_add()
    • InMemoryCqlRunner.metadata_fetcher_get()
    • InMemoryCqlRunner.raw_extrinsic_metadata_by_id_add()
    • InMemoryCqlRunner.raw_extrinsic_metadata_get_by_ids()
    • InMemoryCqlRunner.raw_extrinsic_metadata_by_id_delete()
    • InMemoryCqlRunner.raw_extrinsic_metadata_add()
    • InMemoryCqlRunner.raw_extrinsic_metadata_get_after_date()
    • InMemoryCqlRunner.raw_extrinsic_metadata_get_after_date_and_id()
    • InMemoryCqlRunner.raw_extrinsic_metadata_get()
    • InMemoryCqlRunner.raw_extrinsic_metadata_get_authorities()
    • InMemoryCqlRunner.raw_extrinsic_metadata_delete()
    • InMemoryCqlRunner.extid_add_prepare()
    • InMemoryCqlRunner.extid_index_add_one()
    • InMemoryCqlRunner.extid_delete()
    • InMemoryCqlRunner.extid_delete_from_by_target_table()
    • InMemoryCqlRunner.extid_get_from_pk()
    • InMemoryCqlRunner.extid_get_from_extid()
    • InMemoryCqlRunner.extid_get_from_extid_and_version()
    • InMemoryCqlRunner.extid_get_from_target()
    • InMemoryCqlRunner.object_reference_add_concurrent()
    • InMemoryCqlRunner.object_reference_get()
    • InMemoryCqlRunner.object_references_list_tables()
    • InMemoryCqlRunner.object_references_create_table()
    • InMemoryCqlRunner.object_references_drop_table()
  • InMemoryStorage
    • InMemoryStorage.reset()
    • InMemoryStorage.check_config()
• swh.storage.interface module
  • ListOrder
    • ListOrder.ASC
    • ListOrder.DESC
  • PartialBranches
    • PartialBranches.id
    • PartialBranches.branches
    • PartialBranches.next_branch
  • SnapshotBranchByNameResponse
    • SnapshotBranchByNameResponse.branch_found
    • SnapshotBranchByNameResponse.target
    • SnapshotBranchByNameResponse.aliases_followed
  • HashDict
    • HashDict.sha1
    • HashDict.sha1_git
    • HashDict.sha256
    • HashDict.blake2s256
  • TotalHashDict
    • TotalHashDict.sha1
    • TotalHashDict.sha1_git
    • TotalHashDict.sha256
    • TotalHashDict.blake2s256
  • OriginVisitWithStatuses
  • ObjectReference
  • deprecated()
  • StorageInterface
    • StorageInterface.check_config()
    • StorageInterface.content_add()
    • StorageInterface.content_update()
    • StorageInterface.content_add_metadata()
    • StorageInterface.content_get_data()
    • StorageInterface.content_get_partition()
    • StorageInterface.content_get()
    • StorageInterface.content_missing()
    • StorageInterface.content_missing_per_sha1()
    • StorageInterface.content_missing_per_sha1_git()
    • StorageInterface.content_find()
    • StorageInterface.content_get_random()
    • StorageInterface.skipped_content_add()
    • StorageInterface.skipped_content_find()
    • StorageInterface.skipped_content_missing()
    • StorageInterface.directory_add()
    • StorageInterface.directory_missing()
    • StorageInterface.directory_ls()
    • StorageInterface.directory_entry_get_by_path()
    • StorageInterface.directory_get_entries()
    • StorageInterface.directory_get_raw_manifest()
    • StorageInterface.directory_get_random()
    • StorageInterface.directory_get_id_partition()
    • StorageInterface.revision_add()
    • StorageInterface.revision_missing()
    • StorageInterface.revision_get_partition()
    • StorageInterface.revision_get()
    • StorageInterface.revision_log()
    • StorageInterface.revision_shortlog()
    • StorageInterface.revision_get_random()
    • StorageInterface.extid_get_from_extid()
    • StorageInterface.extid_get_from_target()
    • StorageInterface.extid_add()
    • StorageInterface.release_add()
    • StorageInterface.release_missing()
    • StorageInterface.release_get()
    • StorageInterface.release_get_random()
    • StorageInterface.release_get_partition()
    • StorageInterface.snapshot_add()
    • StorageInterface.snapshot_missing()
    • StorageInterface.snapshot_get()
    • StorageInterface.snapshot_count_branches()
    • StorageInterface.snapshot_get_branches()
    • StorageInterface.snapshot_get_random()
    • StorageInterface.snapshot_branch_get_by_name()
    • StorageInterface.snapshot_get_id_partition()
    • StorageInterface.origin_visit_add()
    • StorageInterface.origin_visit_status_add()
    • StorageInterface.origin_visit_get()
    • StorageInterface.origin_visit_find_by_date()
    • StorageInterface.origin_visit_get_by()
    • StorageInterface.origin_visit_get_latest()
    • StorageInterface.origin_visit_status_get()
    • StorageInterface.origin_visit_status_get_latest()
    • StorageInterface.origin_visit_get_with_statuses()
    • StorageInterface.origin_visit_status_get_random()
    • StorageInterface.origin_get()
    • StorageInterface.origin_get_by_sha1()
    • StorageInterface.origin_list()
    • StorageInterface.origin_search()
    • StorageInterface.origin_count()
    • StorageInterface.origin_snapshot_get_all()
    • StorageInterface.origin_add()
    • StorageInterface.object_find_recent_references()
    • StorageInterface.object_references_add()
    • StorageInterface.object_find_by_sha1_git()
    • StorageInterface.stat_counters()
    • StorageInterface.refresh_stat_counters()
    • StorageInterface.raw_extrinsic_metadata_add()
    • StorageInterface.raw_extrinsic_metadata_get()
    • StorageInterface.raw_extrinsic_metadata_get_by_ids()
    • StorageInterface.raw_extrinsic_metadata_get_authorities()
    • StorageInterface.metadata_fetcher_add()
    • StorageInterface.metadata_fetcher_get()
    • StorageInterface.metadata_authority_add()
    • StorageInterface.metadata_authority_get()
    • StorageInterface.clear_buffers()
    • StorageInterface.flush()
  • ObjectDeletionInterface
    • ObjectDeletionInterface.object_delete()
    • ObjectDeletionInterface.extid_delete_for_target()
  • ObjectReferencesPartition
    • ObjectReferencesPartition.year
    • ObjectReferencesPartition.week
  • PartitionsManagementInterface
    • PartitionsManagementInterface.object_references_create_partition()
    • PartitionsManagementInterface.object_references_drop_partition()
    • PartitionsManagementInterface.object_references_list_partitions()
• swh.storage.metrics module
  • timed()
  • DifferentialTimer
    • DifferentialTimer.inner()
  • send_metric()
  • process_metrics()
• swh.storage.migrate_extrinsic_metadata module
  • pypi_project_from_filename()
  • pypi_origin_from_project_name()
  • pypi_origin_from_filename()
  • cran_package_from_url()
  • npm_package_from_source_url()
  • remove_atom_codemeta_metadata_with_xmlns()
  • remove_atom_codemeta_metadata_without_xmlns()
  • debian_origins_from_row()
  • assert_origin_exists()
  • check_origin_exists()
  • load_metadata()
  • handle_deposit_row()
  • handle_row()
  • create_fetchers()
  • iter_revision_rows()
  • main()
• swh.storage.objstorage module
  • ObjStorage
    • ObjStorage.content_get()
    • ObjStorage.content_add()
• swh.storage.pytest_plugin module
  • cassandra_auth_provider_config()
  • swh_storage_cassandra_cluster()
  • swh_storage_cassandra_keyspace()
  • swh_storage_cassandra_backend_config()
  • swh_storage_postgresql_backend_config()
  • swh_storage_backend_config()
  • swh_storage_backend()
  • swh_storage()
  • sample_data()
• swh.storage.replay module
  • ModelObjectDeserializer
    • ModelObjectDeserializer.convert()
    • ModelObjectDeserializer.report_failure()
  • process_replay_objects()
  • collision_aware_content_add()
• swh.storage.utils module
  • now()
  • map_optional()
  • get_partition_bounds_bytes()
  • extract_collision_hash()
  • content_hex_hashes()
  • content_bytes_hashes()
  • remove_keys()
  • round_to_milliseconds()
• swh.storage.writer module
  • model_object_dict_sanitizer()
  • JournalWriter
    • JournalWriter.write_addition()
    • JournalWriter.write_additions()
    • JournalWriter.content_add()
    • JournalWriter.content_update()
    • JournalWriter.content_add_metadata()
    • JournalWriter.skipped_content_add()
    • JournalWriter.directory_add()
    • JournalWriter.revision_add()
    • JournalWriter.release_add()
    • JournalWriter.snapshot_add()
    • JournalWriter.origin_visit_add()
    • JournalWriter.origin_visit_status_add()
    • JournalWriter.origin_add()
    • JournalWriter.raw_extrinsic_metadata_add()
    • JournalWriter.metadata_fetcher_add()
    • JournalWriter.metadata_authority_add()
    • JournalWriter.extid_add()

Module contents

swh.storage.get_storage(cls: str, **kwargs) → StorageInterface

Get a storage object of class cls with arguments **kwargs. cls must be one of the swh.storage.classes entry-points

Parameters:

• cls (str) –

  storage’s class, can be:

  • local to use a postgresql database

  • cassandra to use a cassandra database

  • remote to connect to a swh-storage server

  • memory for an in-memory storage, useful for fast tests

  • filter, buffer, … to use specific storage “proxies”, see their respective documentations

• args (dict) – dictionary with keys

Returns:

an instance of swh.storage.Storage or compatible class

Raises:

ValueError if passed an unknown storage class. –

swh.storage.get_storage_pipeline(steps: List[Dict[str, Any]], check_config=None) → StorageInterface

Recursively get a storage object that may use other storage objects as backends.

Parameters:

steps (List[dict]) – List of dicts that may be used as kwargs for get_storage.

Returns:

an instance of swh.storage.Storage or compatible class

Raises:

ValueError if passed an unknown storage class. –