swh.objstorage.backends.http module#

class swh.objstorage.backends.http.HTTPReadOnlyObjStorage(url=None, compression: Literal['bz2', 'lzma', 'gzip', 'zlib', 'none'] = 'none', **kwargs)[source]#

Bases: ObjStorage

Simple ObjStorage retrieving objects from an HTTP server.

For example, can be used to retrieve objects from S3:

objstorage:
  cls: http
  url: https://softwareheritage.s3.amazonaws.com/content/

Retry strategy can be defined via the ‘retry’ configuration, e.g.:

objstorage:
  cls: http
  url: https://softwareheritage.s3.amazonaws.com/content/
  retry:
    total: 5
    backoff_factor: 0.2
    status_forcelist:
      - 404
      - 500

See https://urllib3.readthedocs.io/en/stable/reference/urllib3.util.html#urllib3.util.Retry for more details on the possible configuration entries.

name: str = 'http'#

Default objstorage name; can be overloaded at instantiation time giving a ‘name’ argument to the constructor

check_config(*, check_write)[source]#

Check the configuration for this object storage

add(content: bytes, obj_id: bytes | CompositeObjId, check_presence: bool = True) None[source]#
delete(obj_id: bytes | CompositeObjId)[source]#
restore(content: bytes, obj_id: bytes | CompositeObjId) None[source]#
list_content(last_obj_id: bytes | CompositeObjId | None = None, limit: int | None = 10000) Iterator[CompositeObjId][source]#
get(obj_id: bytes | CompositeObjId) bytes[source]#
download_url(obj_id: bytes | CompositeObjId, content_disposition: str | None = None, expiry: timedelta | None = None) str | None[source]#