swh.storage.proxies.blocking package

Submodules

• swh.storage.proxies.blocking.cli module
  • RequestType
    • RequestType.name
    • RequestType.convert()
  • BlockedStateType
  • parse_blocked_state()
  • format_blocked_state()
  • read_origins()
• swh.storage.proxies.blocking.db module
  • BlockingState
    • BlockingState.NON_BLOCKED
    • BlockingState.DECISION_PENDING
    • BlockingState.BLOCKED
  • BlockingStatus
  • BlockingRequest
    • BlockingRequest.id
    • BlockingRequest.slug
    • BlockingRequest.date
    • BlockingRequest.reason
  • RequestHistory
    • RequestHistory.request
    • RequestHistory.date
    • RequestHistory.message
  • BlockingLogEntry
    • BlockingLogEntry.url
    • BlockingLogEntry.url_match
    • BlockingLogEntry.request
    • BlockingLogEntry.date
    • BlockingLogEntry.state
  • BlockedOrigin
  • BlockingDb
    • BlockingDb.current_version
  • get_urls_to_check()
  • BlockingAdmin
    • BlockingAdmin.create_request()
    • BlockingAdmin.find_request()
    • BlockingAdmin.find_request_by_id()
    • BlockingAdmin.get_requests()
    • BlockingAdmin.set_origins_state()
    • BlockingAdmin.get_states_for_request()
    • BlockingAdmin.find_blocking_states()
    • BlockingAdmin.delete_blocking_states()
    • BlockingAdmin.record_history()
    • BlockingAdmin.get_history()
    • BlockingAdmin.get_log()
  • BlockingQuery
    • BlockingQuery.origins_are_blocked()
    • BlockingQuery.origin_is_blocked()

Module contents

swh.storage.proxies.blocking.get_datastore(cls, db=None, blocking_db=None, **kwargs)

swh.storage.proxies.blocking.blocking_overhead_timer(method_name: str) → DifferentialTimer

Return a properly setup DifferentialTimer for method_name of the storage

class swh.storage.proxies.blocking.BlockingProxyStorage(storage: Dict | StorageInterface, db: str | None = None, blocking_db: str | None = None, min_pool_conns: int = 1, max_pool_conns: int = 5)

Bases: object

Blocking storage proxy

This proxy prevents visits from a known list of origins to be performed at all.

It uses a specific PostgreSQL database (which for now is colocated with the swh.storage PostgreSQL database), the access to which is implemented in the db submodule.

Sample configuration

origin_visit_status_add(visit_statuses: List[OriginVisitStatus]) → Dict[str, int]

origin_visit_add(visits: List[OriginVisit]) → Iterable[OriginVisit]

origin_add(origins: List[Origin]) → Dict[str, int]