swh.storage.proxies.buffer module#
- swh.storage.proxies.buffer.estimate_revision_size(revision: Revision) int [source]#
Estimate the size of a revision, by summing the size of variable length fields
- swh.storage.proxies.buffer.estimate_release_size(release: Release) int [source]#
Estimate the size of a release, by summing the size of variable length fields
- class swh.storage.proxies.buffer.BufferingProxyStorage(storage: Mapping, min_batch_size: Mapping = {})[source]#
Bases:
object
- Storage implementation in charge of accumulating objects prior to
discussing with the “main” storage.
Deduplicates values based on a tuple of keys depending on the object type.
Sample configuration use case for buffering storage:
storage: cls: buffer args: storage: cls: remote args: http://storage.internal.staging.swh.network:5002/ min_batch_size: content: 10000 content_bytes: 100000000 skipped_content: 10000 directory: 5000 directory_entries: 100000 revision: 1000 revision_parents: 2000 revision_bytes: 100000000 release: 10000 release_bytes: 100000000 snapshot: 5000
- content_add(contents: Sequence[Content]) Dict[str, int] [source]#
Push contents to write to the storage in the buffer.
Following policies apply:
if the buffer’s threshold is hit, flush content to the storage.
otherwise, if the total size of buffered contents’s threshold is hit, flush content to the storage.
- object_add(objects: Sequence[BaseModel], *, object_type: typing_extensions.Literal[content, skipped_content, directory, revision, release, snapshot, extid], keys: Iterable[str]) Dict[str, int] [source]#
Push objects to write to the storage in the buffer. Flushes the buffer to the storage if the threshold is hit.
- flush(object_types: Sequence[typing_extensions.Literal[content, skipped_content, directory, revision, release, snapshot, extid]] = ('content', 'skipped_content', 'directory', 'revision', 'release', 'snapshot', 'extid')) Dict[str, int] [source]#
- clear_buffers(object_types: Sequence[typing_extensions.Literal[content, skipped_content, directory, revision, release, snapshot, extid]] = ('content', 'skipped_content', 'directory', 'revision', 'release', 'snapshot', 'extid')) None [source]#
Clear objects from current buffer.
Warning
data that has not been flushed to storage will be lost when this method is called. This should only be called when flush fails and you want to continue your processing.