swh.storage.cassandra.model module#
Classes representing tables in the Cassandra database.
They are very close to classes found in swh.model.model, but most of them are subtly different:
Large objects are split into other classes (eg. RevisionRow has no ‘parents’ field, because parents are stored in a different table, represented by RevisionParentRow)
They have a “cols” field, which returns the list of column names of the table
They only use types that map directly to Cassandra’s schema (ie. no enums)
Therefore, this model doesn’t reuse swh.model.model, except for types that can be mapped to UDTs (Person and TimestampWithTimezone).
- swh.storage.cassandra.model.MAGIC_NULL_PK = b'<null>'#
NULLs (or all-empty blobs) are not allowed in primary keys; instead we use a special value that can’t possibly be a valid hash.
- swh.storage.cassandra.model.content_index_table_name(algo: str, skipped_content: bool) str [source]#
Given an algorithm name, returns the name of one of the ‘content_by_*’ and ‘skipped_content_by_*’ tables that serve as index for the ‘content’ and ‘skipped_content’ tables based on this algorithm’s hashes.
For now it is a simple substitution, but future versions may append a version number to it, if needed for schema updates.
- class swh.storage.cassandra.model.ContentRow(sha1: bytes, sha1_git: bytes, sha256: bytes, blake2s256: bytes, length: int, ctime: datetime.datetime, status: str)[source]#
Bases:
BaseRow
- class swh.storage.cassandra.model.SkippedContentRow(sha1: Union[bytes, NoneType], sha1_git: Union[bytes, NoneType], sha256: Union[bytes, NoneType], blake2s256: Union[bytes, NoneType], length: Union[int, NoneType], ctime: Union[datetime.datetime, NoneType], status: str, reason: str, origin: str)[source]#
Bases:
BaseRow
- class swh.storage.cassandra.model.DirectoryRow(id: bytes, raw_manifest: Union[bytes, NoneType])[source]#
Bases:
BaseRow
- class swh.storage.cassandra.model.DirectoryEntryRow(directory_id: bytes, name: bytes, target: bytes, perms: int, type: str)[source]#
Bases:
BaseRow
- class swh.storage.cassandra.model.RevisionRow(id: bytes, date: Union[swh.model.model.TimestampWithTimezone, NoneType], committer_date: Union[swh.model.model.TimestampWithTimezone, NoneType], type: str, directory: bytes, message: bytes, author: swh.model.model.Person, committer: swh.model.model.Person, synthetic: bool, metadata: str, extra_headers: dict, raw_manifest: Union[bytes, NoneType])[source]#
Bases:
BaseRow
- date: Optional[TimestampWithTimezone]#
- committer_date: Optional[TimestampWithTimezone]#
- class swh.storage.cassandra.model.RevisionParentRow(id: bytes, parent_rank: int, parent_id: bytes)[source]#
Bases:
BaseRow
- class swh.storage.cassandra.model.ReleaseRow(id: bytes, target_type: str, target: bytes, date: swh.model.model.TimestampWithTimezone, name: bytes, message: bytes, author: swh.model.model.Person, synthetic: bool, raw_manifest: Union[bytes, NoneType])[source]#
Bases:
BaseRow
- date: TimestampWithTimezone#
- class swh.storage.cassandra.model.SnapshotBranchRow(snapshot_id: bytes, name: bytes, target_type: Optional[str], target: Optional[bytes])[source]#
Bases:
BaseRow
For a given snapshot_id, branches are sorted by their name, allowing easy pagination.
- class swh.storage.cassandra.model.OriginVisitRow(origin: str, visit: int, date: datetime.datetime, type: str)[source]#
Bases:
BaseRow
- class swh.storage.cassandra.model.OriginVisitStatusRow(origin: str, visit: int, date: datetime.datetime, type: str, status: str, metadata: str, snapshot: bytes)[source]#
Bases:
BaseRow
- class swh.storage.cassandra.model.OriginRow(sha1: bytes, url: str, next_visit_id: int)[source]#
Bases:
BaseRow
- next_visit_id: int#
We need integer visit ids for compatibility with the pgsql storage, so we’re using lightweight transactions with this trick: https://stackoverflow.com/a/29391877/539465
- class swh.storage.cassandra.model.MetadataFetcherRow(name: str, version: str)[source]#
Bases:
BaseRow
- class swh.storage.cassandra.model.RawExtrinsicMetadataRow(id: bytes, type: str, target: str, authority_type: str, authority_url: str, discovery_date: datetime, fetcher_name: str, fetcher_version: str, format: str, metadata: bytes, origin: Optional[str], visit: Optional[int], snapshot: Optional[str], release: Optional[str], revision: Optional[str], path: Optional[bytes], directory: Optional[str])[source]#
Bases:
BaseRow
An explanation is in order for the primary key:
Intuitively, the primary key should only be ‘id’, because two metadata entries are the same iff the id is the same; and ‘id’ is used for deduplication.
However, we also want to query by (target, authority_type, authority_url, discovery_date) The naive solution to this would be an extra table, to use as index; but it means 1. extra code to keep them in sync 2. overhead when writing 3. overhead + random reads (instead of linear) when reading.
Therefore, we use a single table for both, by adding the column we want to query with before the id. It solves both a) the query/order issues and b) the uniqueness issue because:
adding the id at the end of the primary key does not change the rows’ order: for two different rows, id1 != id2, so (target1, …, date1) < (target2, …, date2) <=> (target1, …, date1, id1) < (target2, …, date2, id2)
the id is a hash of all the columns, so: rows are the same <=> id1 == id2 <=> (target1, …, date1, id1) == (target2, …, date2, id2)
- class swh.storage.cassandra.model.RawExtrinsicMetadataByIdRow(id: bytes, target: str, authority_type: str, authority_url: str)[source]#
Bases:
BaseRow
- class swh.storage.cassandra.model.ObjectCountRow(partition_key: int, object_type: str, count: int)[source]#
Bases:
BaseRow
- class swh.storage.cassandra.model.ExtIDRow(extid_type: str, extid: bytes, extid_version: int, target_type: str, target: bytes)[source]#
Bases:
BaseRow
- class swh.storage.cassandra.model.ExtIDByTargetRow(target_type: str, target: bytes, target_token: int)[source]#
Bases:
BaseRow