swh.loader.core.converters module#

Convert objects to dictionaries suitable for swh.storage

swh.loader.core.converters.prepare_contents(contents: Iterable[Dict], max_content_size: int | None = None, origin_url: str | None = None) → Tuple[List[Dict], List[Dict]][source]#

Prepare contents for storage from a list of contents

Returns: tuple of content iterable, skipped content iterable

swh.loader.core.converters.content_for_storage(content: Dict, max_content_size: int | None = None, origin_url: str | None = None) → BaseContent[source]#

Prepare content to be ready for storage

Note: - ‘data’ is returned only if max_content_size is not reached.

Returns:: content with added data (or reason for being missing)