swh.objstorage.backends.http module

class swh.objstorage.backends.http.HTTPReadOnlyObjStorage(url=None, compression: Literal['bz2', 'lzma', 'gzip', 'zlib', 'none'] = 'none', **kwargs)

Bases: ObjStorage

Simple ObjStorage retrieving objects from an HTTP server.

For example, can be used to retrieve objects from S3:

objstorage:
  cls: http
  url: https://softwareheritage.s3.amazonaws.com/content/

Retry strategy can be defined via the ‘retry’ configuration, e.g.:

objstorage:
  cls: http
  url: https://softwareheritage.s3.amazonaws.com/content/
  retry:
    total: 5
    backoff_factor: 0.2
    status_forcelist:
      - 404
      - 500

See https://urllib3.readthedocs.io/en/stable/reference/urllib3.util.html#urllib3.util.Retry for more details on the possible configuration entries.

name: str = 'http'

Default objstorage name; can be overloaded at instantiation time giving a ‘name’ argument to the constructor

check_config(*, check_write)

Check the configuration for this object storage

add(content: bytes, obj_id: bytes | CompositeObjId, check_presence: bool = True) → None

delete(obj_id: bytes | CompositeObjId)

restore(content: bytes, obj_id: bytes | CompositeObjId) → None

list_content(last_obj_id: bytes | CompositeObjId | None = None, limit: int | None = 10000) → Iterator[CompositeObjId]

get(obj_id: bytes | CompositeObjId) → bytes

download_url(obj_id: bytes | CompositeObjId, content_disposition: str | None = None, expiry: timedelta | None = None) → str | None