swh.loader.core.converters module

Convert objects to dictionaries suitable for swh.storage

swh.loader.core.converters.prepare_contents(contents: Iterable[Dict], max_content_size: int | None = None, origin_url: str | None = None) → Tuple[List[Dict], List[Dict]]

Prepare contents for storage from a list of contents

Returns

tuple of content iterable, skipped content iterable

swh.loader.core.converters.content_for_storage(content: Dict, max_content_size: int | None = None, origin_url: str | None = None) → BaseContent

Prepare content to be ready for storage

Note: - ‘data’ is returned only if max_content_size is not reached.

Returns:

content with added data (or reason for being missing)