swh.objstorage.backends.http module

class swh.objstorage.backends.http.HTTPReadOnlyObjStorage(url=None, compression: Literal['bz2', 'lzma', 'gzip', 'zlib', 'none'] | None = None, batch_max_connections: int = 100, batch_max_connections_per_host: int = 0, **kwargs)

Bases: ObjStorage

Simple ObjStorage retrieving objects from an HTTP server.

For example, can be used to retrieve objects from S3:

objstorage:
  cls: http
  url: https://softwareheritage.s3.amazonaws.com/content/
  compression: gzip

Retry strategy can be defined via the ‘retry’ configuration, e.g.:

objstorage:
  cls: http
  url: https://softwareheritage.s3.amazonaws.com/content/
  compression: gzip
  retry:
    total: 5
    backoff_factor: 0.2
    status_forcelist:
      - 404
      - 500

See https://urllib3.readthedocs.io/en/stable/reference/urllib3.util.html#urllib3.util.Retry for more details on the possible configuration entries.

The get_batch() method is implemented with aiohttp to improve the performance of object downloads. The maximum number of simultaneous connections can be set using the batch_max_connections parameter of that class (default to 100). The maximum number of simultaneous connections to the same host can be set using the batch_max_connections_per_host parameter of that class (default to 0 for no limit).

primary_hash: Literal['sha1', 'sha256'] = 'sha1'

name: str = 'http'

Default objstorage name; can be overloaded at instantiation time giving a ‘name’ argument to the constructor

check_config(*, check_write)

Check the configuration for this object storage

add(content: bytes, obj_id: HashDict, check_presence: bool = True) → None

Add a new object to the object storage.

Parameters:

• content – object’s raw content to add in storage.

• obj_id – dict of checksums.

• check_presence (bool) – indicate if the presence of the content should be verified before adding the file.

Returns:

the id (bytes) of the object into the storage.

delete(obj_id: HashDict)

Delete an object.

Parameters:

obj_id – object identifier.

Raises:

ObjNotFoundError – if the requested object is missing.

restore(content: bytes, obj_id: HashDict) → None

Restore a content that have been corrupted.

This function is identical to add but does not check if the object id is already in the file system. The default implementation provided by the current class is suitable for most cases.

Parameters:

• content – object’s raw content to add in storage

• obj_id – dict of hashes of the content (or only the sha1, for legacy clients)

get(obj_id: HashDict) → bytes

Retrieve the content of a given object.

Parameters:

obj_id – object id.

Returns:

the content of the requested object as bytes.

Raises:

ObjNotFoundError – if the requested object is missing.

get_batch(obj_ids: Iterable[HashDict]) → Iterator[bytes | None]

Retrieve objects’ raw content in bulk from storage.

Note: This function does have a default implementation in ObjStorage that is suitable for most cases.

For object storages that needs to do the minimal number of requests possible (ex: remote object storages), that method can be overridden to perform a more efficient operation.

Parameters:

obj_ids – list of object ids.

Returns:

list of resulting contents, or None if the content could not be retrieved. Do not raise any exception as a fail for one content will not cancel the whole request.

download_url(obj_id: HashDict, content_disposition: str | None = None, expiry: timedelta | None = None) → str | None

Get a direct download link for the object if the obstorage backend supports such feature.

Some objstorage backends, typically cloud based ones like azure or s3, can provide a direct download link for a stored object.

Parameters:

• obj_id – object identifier

• content_disposition – set Content-Disposition header for the generated URL response if the objstorage backend supports it

• expiry – the duration after which the URL expires if the objstorage backend supports it, if not provided the URL expires 24 hours after its creation

Returns:

Direct download URL for the object or None if the objstorage backend does

not support such feature.