swh.dataset.exporters.orc module

swh.dataset.exporters.orc.hash_to_hex_or_none(hash)

swh.dataset.exporters.orc.swh_date_to_tuple(obj)

swh.dataset.exporters.orc.datetime_to_tuple(obj: datetime | None) → Tuple[int, int] | None

class swh.dataset.exporters.orc.SWHTimestampConverter

Bases: object

This is an ORCConverter compatible class to convert timestamps from/to ORC files

timestamps in python are given as a couple (seconds, microseconds) and are serialized as a couple (seconds, nanoseconds) in the ORC file.

Reimplemented because we do not want the Python object to be converted as ORC timestamp to be Python datatime objects, since swh.model’s Timestamp cannot be converted without loss a Python datetime objects.

static from_orc(seconds: int, nanoseconds: int, timezone: Any) → Tuple[int, int]

static to_orc(obj: Tuple[int, int] | None, timezone: Any) → Tuple[int, int] | None

class swh.dataset.exporters.orc.ORCExporter(*args, **kwargs)

Bases: ExporterDispatch

Implementation of an exporter which writes the entire graph dataset as ORC files. Useful for large scale processing, notably on cloud instances (e.g BigQuery, Amazon Athena, Azure).

maybe_close_writer_for(table_name: str)

get_writer_for(table_name: str, unique_id=None)

process_origin(origin)

process_origin_visit(visit)

process_origin_visit_status(visit_status)

process_snapshot(snapshot)

process_release(release)

process_revision(revision)

process_directory(directory)

process_content(content)

process_skipped_content(skipped_content)