swh.objstorage.objstorage module#
- swh.objstorage.objstorage.timed(f)[source]#
A simple decorator used to add statsd probes on main ObjStorage methods (add, get and __contains__)
- swh.objstorage.objstorage.objid_to_default_hex(obj_id: ObjId, algo: Literal['sha1', 'sha256'] = 'sha1') str[source]#
Converts SHA1 hashes and multi-hashes to the hexadecimal representation of the SHA1.
- swh.objstorage.objstorage.objid_for_content(content: bytes) ObjId[source]#
Compute the content’s hashes.
- Parameters:
content – The raw content to hash
hash_names – Names of hashing algorithms (default to
swh.model.hashutil.DEFAULT_ALGORITHMS)
- Returns:
A dict mapping algo name to hash value
- class swh.objstorage.objstorage.ObjStorage(*, allow_delete: bool = False, **kwargs)[source]#
Bases:
ObjStorageInterface- name: str = 'objstorage'#
Default objstorage name; can be overloaded at instantiation time giving a ‘name’ argument to the constructor
- add_batch(contents: Iterable[Tuple[ObjId, bytes]], check_presence: bool = True) Dict[source]#
Add a batch of new objects to the object storage.
- Parameters:
contents – list of pairs of composite object ids and object contents
- Returns:
the summary of objects added to the storage (count of object, count of bytes object)
- restore(content: bytes, obj_id: ObjId) None[source]#
Restore a content that have been corrupted.
This function is identical to add but does not check if the object id is already in the file system. The default implementation provided by the current class is suitable for most cases.
- Parameters:
content – object’s raw content to add in storage
obj_id – dict of hashes of the content (or only the sha1, for legacy clients)
- get_batch(obj_ids: Iterable[ObjId]) Iterator[bytes | None][source]#
Retrieve objects’ raw content in bulk from storage.
Note: This function does have a default implementation in ObjStorage that is suitable for most cases.
For object storages that needs to do the minimal number of requests possible (ex: remote object storages), that method can be overridden to perform a more efficient operation.
- Parameters:
obj_ids – list of object ids.
- Returns:
list of resulting contents, or None if the content could not be retrieved. Do not raise any exception as a fail for one content will not cancel the whole request.
- abstract delete(obj_id: ObjId)[source]#
Delete an object.
- Parameters:
obj_id – object identifier.
- Raises:
ObjNotFoundError – if the requested object is missing.
- list_content(last_obj_id: ObjId | None = None, limit: int | None = 10000) Iterator[ObjId][source]#
Generates known object ids.
- Parameters:
last_obj_id – object id from which to iterate from (excluded).
limit (int) – max number of object ids to generate. If unset (None), generate all objects (behavior might not be guaranteed for all backends).
- Generates:
obj_id: object ids.
- download_url(obj_id: ObjId, content_disposition: str | None = None, expiry: timedelta | None = None) str | None[source]#
Get a direct download link for the object if the obstorage backend supports such feature.
Some objstorage backends, typically cloud based ones like azure or s3, can provide a direct download link for a stored object.
- Parameters:
obj_id – object identifier
content_disposition – set Content-Disposition header for the generated URL response if the objstorage backend supports it
expiry – the duration after which the URL expires if the objstorage backend supports it, if not provided the URL expires 24 hours after its creation
- Returns:
- Direct download URL for the object or
Noneif the objstorage backend does not support such feature.
- Direct download URL for the object or
- abstract get(obj_id: ObjId) bytes[source]#
Retrieve the content of a given object.
- Parameters:
obj_id – object id.
- Returns:
the content of the requested object as bytes.
- Raises:
ObjNotFoundError – if the requested object is missing.
