swh.model.identifiers module¶
-
class
swh.model.identifiers.
ObjectType
(value)[source]¶ Bases:
enum.Enum
Possible object types of a QualifiedSWHID or CoreSWHID.
The values of each variant is what is used in the SWHID’s string representation.
-
SNAPSHOT
= 'snp'¶
-
REVISION
= 'rev'¶
-
RELEASE
= 'rel'¶
-
DIRECTORY
= 'dir'¶
-
CONTENT
= 'cnt'¶
-
-
class
swh.model.identifiers.
ExtendedObjectType
(value)[source]¶ Bases:
enum.Enum
Possible object types of an ExtendedSWHID.
The variants are a superset of :cls:`ObjectType`’s
-
SNAPSHOT
= 'snp'¶
-
REVISION
= 'rev'¶
-
RELEASE
= 'rel'¶
-
DIRECTORY
= 'dir'¶
-
CONTENT
= 'cnt'¶
-
ORIGIN
= 'ori'¶
-
RAW_EXTRINSIC_METADATA
= 'emd'¶
-
-
swh.model.identifiers.
identifier_to_bytes
(identifier)[source]¶ Convert a text identifier to bytes.
- Parameters
identifier – an identifier, either a 40-char hexadecimal string or a bytes object of length 20
- Returns
The length 20 bytestring corresponding to the given identifier
- Raises
ValueError – if the identifier is of an unexpected type or length.
-
swh.model.identifiers.
identifier_to_str
(identifier)[source]¶ Convert an identifier to an hexadecimal string.
- Parameters
identifier – an identifier, either a 40-char hexadecimal string or a bytes object of length 20
- Returns
The length 40 string corresponding to the given identifier, hex encoded
- Raises
ValueError – if the identifier is of an unexpected type or length.
-
swh.model.identifiers.
content_identifier
(content)[source]¶ Return the intrinsic identifier for a content.
A content’s identifier is the sha1, sha1_git and sha256 checksums of its data.
- Parameters
content – a content conforming to the Software Heritage schema
- Returns
A dictionary with all the hashes for the data
- Raises
KeyError – if the content doesn’t have a data member.
-
swh.model.identifiers.
escape_newlines
(snippet)[source]¶ Escape the newlines present in snippet according to git rules.
New lines in git manifests are escaped by indenting the next line by one space.
-
swh.model.identifiers.
directory_identifier
(directory)[source]¶ Return the intrinsic identifier for a directory.
A directory’s identifier is the tree sha1 à la git of a directory listing, using the following algorithm, which is equivalent to the git algorithm for trees:
Entries of the directory are sorted using the name (or the name with ‘/’ appended for directory entries) as key, in bytes order.
For each entry of the directory, the following bytes are output:
the octal representation of the permissions for the entry (stored in the ‘perms’ member), which is a representation of the entry type:
b’100644’ (int 33188) for files
b’100755’ (int 33261) for executable files
b’120000’ (int 40960) for symbolic links
b’40000’ (int 16384) for directories
b’160000’ (int 57344) for references to revisions
an ascii space (b’ ‘)
the entry’s name (as raw bytes), stored in the ‘name’ member
a null byte (b’')
the 20 byte long identifier of the object pointed at by the entry, stored in the ‘target’ member:
for files or executable files: their blob sha1_git
for symbolic links: the blob sha1_git of a file containing the link destination
for directories: their intrinsic identifier
for revisions: their intrinsic identifier
(Note that there is no separator between entries)
-
swh.model.identifiers.
format_date
(date)[source]¶ Convert a date object into an UTC timestamp encoded as ascii bytes.
Git stores timestamps as an integer number of seconds since the UNIX epoch.
However, Software Heritage stores timestamps as an integer number of microseconds (postgres type “datetime with timezone”).
Therefore, we print timestamps with no microseconds as integers, and timestamps with microseconds as floating point values. We elide the trailing zeroes from microsecond values, to “future-proof” our representation if we ever need more precision in timestamps.
-
swh.model.identifiers.
format_offset
(offset, negative_utc=None)[source]¶ Convert an integer number of minutes into an offset representation.
The offset representation is [+-]hhmm where:
hh is the number of hours;
mm is the number of minutes.
A null offset is represented as +0000.
-
swh.model.identifiers.
normalize_timestamp
(time_representation)[source]¶ Normalize a time representation for processing by Software Heritage
This function supports a numeric timestamp (representing a number of seconds since the UNIX epoch, 1970-01-01 at 00:00 UTC), a
datetime.datetime
object (with timezone information), or a normalized Software Heritage time representation (idempotency).- Parameters
time_representation – the representation of a timestamp
- Returns
a normalized dictionary with three keys:
timestamp: a dict with two optional keys:
seconds: the integral number of seconds since the UNIX epoch
microseconds: the integral number of microseconds
offset: the timezone offset as a number of minutes relative to UTC
negative_utc: a boolean representing whether the offset is -0000 when offset = 0.
- Return type
dict
Format the specification of an author.
An author is either a byte string (passed unchanged), or a dict with three keys, fullname, name and email.
If the fullname exists, return it; if it doesn’t, we construct a fullname using the following heuristics: if the name value is None, we return the email in angle brackets, else, we return the name, a space, and the email in angle brackets.
-
swh.model.identifiers.
format_manifest
(headers: Iterable[Tuple[bytes, bytes]], message: Optional[bytes] = None) → bytes[source]¶ Format a manifest comprised of a sequence of headers and an optional message.
The manifest format, compatible with the git format for tag and commit objects, is as follows:
for each key, value in headers, emit:
the key, literally
an ascii space (
\x20
)the value, with newlines escaped using
escape_newlines()
,an ascii newline (
\x0a
)
if the message is not None, emit:
an ascii newline (
\x0a
)the message, literally
- Parameters
headers – a sequence of key/value headers stored in the manifest;
message – an optional message used to trail the manifest.
- Returns
the formatted manifest as bytes
-
swh.model.identifiers.
hash_manifest
(type: str, headers: Iterable[Tuple[bytes, bytes]], message: Optional[bytes] = None)[source]¶ Hash the manifest of an object of type type, comprised of a sequence of headers and an optional message.
Before hashing, the manifest is serialized with the
format_manifest()
function.We then use the git “salted sha1” (
swh.model.hashutil.hash_git_data()
) with the given type to hash the manifest.- Parameters
type – the type of object for which we’re computing a manifest (e.g. “tag”, “commit”, …)
headers – a sequence of key/value headers stored in the manifest;
message – an optional message used to trail the manifest.
Format authorship data according to git standards.
Git authorship data has two components:
an author specification, usually a name and email, but in practice an arbitrary bytestring
optionally, a timestamp with a UTC offset specification
The authorship data is formatted thus:
`name and email`[ `timestamp` `utc_offset`]
The timestamp is encoded as a (decimal) number of seconds since the UNIX epoch (1970-01-01 at 00:00 UTC). As an extension to the git format, we support fractional timestamps, using a dot as the separator for the decimal part.
The utc offset is a number of minutes encoded as ‘[+-]HHMM’. Note that some tools can pass a negative offset corresponding to the UTC timezone (‘-0000’), which is valid and is encoded as such.
- Parameters
author – an author specification (dict with two bytes values: name and email, or byte value)
date_offset – a normalized date/time representation as returned by
normalize_timestamp()
.
- Returns
the byte string containing the authorship data
-
swh.model.identifiers.
revision_identifier
(revision)[source]¶ Return the intrinsic identifier for a revision.
The fields used for the revision identifier computation are:
directory
parents
author
author_date
committer
committer_date
extra_headers or metadata -> extra_headers
message
A revision’s identifier is the ‘git’-checksum of a commit manifest constructed as follows (newlines are a single ASCII newline character):
tree <directory identifier> [for each parent in parents] parent <parent identifier> [end for each parents] author <author> <author_date> committer <committer> <committer_date> [for each key, value in extra_headers] <key> <encoded value> [end for each extra_headers] <message>
The directory identifier is the ascii representation of its hexadecimal encoding.
Author and committer are formatted with the
format_author()
function. Dates are formatted with theformat_offset()
function.Extra headers are an ordered list of [key, value] pairs. Keys are strings and get encoded to utf-8 for identifier computation. Values are either byte strings, unicode strings (that get encoded to utf-8), or integers (that get encoded to their utf-8 decimal representation).
Multiline extra header values are escaped by indenting the continuation lines with one ascii space.
If the message is None, the manifest ends with the last header. Else, the message is appended to the headers after an empty line.
The checksum of the full manifest is computed using the ‘commit’ git object type.
-
swh.model.identifiers.
target_type_to_git
(target_type)[source]¶ Convert a software heritage target type to a git object type
-
swh.model.identifiers.
release_identifier
(release)[source]¶ Return the intrinsic identifier for a release.
-
swh.model.identifiers.
snapshot_identifier
(snapshot, *, ignore_unresolved=False)[source]¶ Return the intrinsic identifier for a snapshot.
Snapshots are a set of named branches, which are pointers to objects at any level of the Software Heritage DAG.
As well as pointing to other objects in the Software Heritage DAG, branches can also be alias*es, in which case their target is the name of another branch in the same snapshot, or *dangling, in which case the target is unknown (and represented by the
None
value).A snapshot identifier is a salted sha1 (using the git hashing algorithm with the
snapshot
object type) of a manifest following the algorithm:Branches are sorted using the name as key, in bytes order.
For each branch, the following bytes are output:
the type of the branch target:
content
,directory
,revision
,release
orsnapshot
for the corresponding entries in the DAG;alias
for branches referencing another branch;dangling
for dangling branches
an ascii space (
\x20
)the branch name (as raw bytes)
a null byte (
\x00
)the length of the target identifier, as an ascii-encoded decimal number (
20
for current intrinsic identifiers,0
for dangling branches, the length of the target branch name for branch aliases)a colon (
:
)the identifier of the target object pointed at by the branch, stored in the ‘target’ member:
for contents: their sha1_git
for directories, revisions, releases or snapshots: their intrinsic identifier
for branch aliases, the name of the target branch (as raw bytes)
for dangling branches, the empty string
Note that, akin to directory manifests, there is no separator between entries. Because of symbolic branches, identifiers are of arbitrary length but are length-encoded to avoid ambiguity.
- Parameters
snapshot (dict) – the snapshot of which to compute the identifier. A single entry is needed,
'branches'
, which is itself adict
mapping each branch to its targetignore_unresolved (bool) – if True, ignore unresolved branch aliases.
- Returns
the intrinsic identifier for snapshot
- Return type
str
-
swh.model.identifiers.
origin_identifier
(origin)[source]¶ Return the intrinsic identifier for an origin.
An origin’s identifier is the sha1 checksum of the entire origin URL
-
swh.model.identifiers.
raw_extrinsic_metadata_identifier
(metadata: Dict[str, Any]) → str[source]¶ Return the intrinsic identifier for a RawExtrinsicMetadata object.
A raw_extrinsic_metadata identifier is a salted sha1 (using the git hashing algorithm with the
raw_extrinsic_metadata
object type) of a manifest following the format:``` target $ExtendedSwhid discovery_date $Timestamp authority $StrWithoutSpaces $IRI fetcher $Str $Version format $StrWithoutSpaces origin $IRI <- optional visit $IntInDecimal <- optional snapshot $CoreSwhid <- optional release $CoreSwhid <- optional revision $CoreSwhid <- optional path $Bytes <- optional directory $CoreSwhid <- optional
$IRI must be RFC 3987 IRIs (so they may contain newlines, that are escaped as described below)
$StrWithoutSpaces and $Version are ASCII strings, and may not contain spaces.
$Str is an UTF-8 string.
$CoreSwhid are core SWHIDs, as defined in SoftWare Heritage persistent IDentifiers (SWHIDs). $ExtendedSwhid is a core SWHID, with extra types allowed (‘ori’ for origins and ‘emd’ for raw extrinsic metadata)
$Timestamp is a decimal representation of the rounded-down integer number of seconds since the UNIX epoch (1970-01-01 00:00:00 UTC), with no leading ‘0’ (unless the timestamp value is zero) and no timezone. It may be negative by prefixing it with a ‘-‘, which must not be followed by a ‘0’.
Newlines in $Bytes, $Str, and $Iri are escaped as with other git fields, ie. by adding a space after them.
- Returns
the intrinsic identifier for metadata
- Return type
str
-
class
swh.model.identifiers.
CoreSWHID
(*, namespace: str = 'swh', scheme_version: int = 1, object_id: bytes, object_type)[source]¶ Bases:
swh.model.identifiers._BaseSWHID
[swh.model.identifiers.ObjectType
]Dataclass holding the relevant info associated to a SoftWare Heritage persistent IDentifier (SWHID).
Unlike QualifiedSWHID, it is restricted to core SWHIDs, ie. SWHIDs with no qualifiers.
- Raises
swh.model.exceptions.ValidationError – In case of invalid object type or id
To get the raw SWHID string from an instance of this class, use the
str()
function:>>> swhid = CoreSWHID( ... object_type=ObjectType.CONTENT, ... object_id=bytes.fromhex('8ff44f081d43176474b267de5451f2c2e88089d0'), ... ) >>> str(swhid) 'swh:1:cnt:8ff44f081d43176474b267de5451f2c2e88089d0'
And vice-versa with
CoreSWHID.from_string()
:>>> swhid == CoreSWHID.from_string( ... "swh:1:cnt:8ff44f081d43176474b267de5451f2c2e88089d0" ... ) True
-
object_type
: _TObjectType¶ the type of object the identifier points to
-
to_extended
() → swh.model.identifiers.ExtendedSWHID[source]¶ Converts this CoreSWHID into an ExtendedSWHID.
As ExtendedSWHID is a superset of CoreSWHID, this is lossless.
-
class
swh.model.identifiers.
QualifiedSWHID
(*, namespace: str = 'swh', scheme_version: int = 1, object_id: bytes, object_type, origin: Optional[str] = None, visit=None, anchor=None, path=None, lines=None)[source]¶ Bases:
swh.model.identifiers._BaseSWHID
[swh.model.identifiers.ObjectType
]Dataclass holding the relevant info associated to a SoftWare Heritage persistent IDentifier (SWHID)
- Raises
swh.model.exceptions.ValidationError – In case of invalid object type or id
To get the raw SWHID string from an instance of this class, use the
str()
function:>>> swhid = QualifiedSWHID( ... object_type=ObjectType.CONTENT, ... object_id=bytes.fromhex('8ff44f081d43176474b267de5451f2c2e88089d0'), ... lines=(5, 10), ... ) >>> str(swhid) 'swh:1:cnt:8ff44f081d43176474b267de5451f2c2e88089d0;lines=5-10'
And vice-versa with
QualifiedSWHID.from_string()
:>>> swhid == QualifiedSWHID.from_string( ... "swh:1:cnt:8ff44f081d43176474b267de5451f2c2e88089d0;lines=5-10" ... ) True
-
object_type
: _TObjectType¶ the type of object the identifier points to
-
origin
¶ the software origin where an object has been found or observed in the wild, as an URI
-
visit
¶ the core identifier of a snapshot corresponding to a specific visit of a repository containing the designated object
-
anchor
¶ a designated node in the Merkle DAG relative to which a path to the object is specified, as the core identifier of a directory, a revision, a release, or a snapshot
-
path
¶ the absolute file path, from the root directory associated to the anchor node, to the object; when the anchor denotes a directory or a revision, and almost always when it’s a release, the root directory is uniquely determined; when the anchor denotes a snapshot, the root directory is the one pointed to by HEAD (possibly indirectly), and undefined if such a reference is missing
-
lines
¶ line number(s) of interest, usually within a content object
- Type
lines
-
classmethod
from_string
(s: str) → swh.model.identifiers.QualifiedSWHID[source]¶
-
class
swh.model.identifiers.
ExtendedSWHID
(*, namespace: str = 'swh', scheme_version: int = 1, object_id: bytes, object_type)[source]¶ Bases:
swh.model.identifiers._BaseSWHID
[swh.model.identifiers.ExtendedObjectType
]Dataclass holding the relevant info associated to a SoftWare Heritage persistent IDentifier (SWHID).
It extends CoreSWHID, by allowing non-standard object types; and should only be used internally to Software Heritage.
- Raises
swh.model.exceptions.ValidationError – In case of invalid object type or id
To get the raw SWHID string from an instance of this class, use the
str()
function:>>> swhid = ExtendedSWHID( ... object_type=ExtendedObjectType.CONTENT, ... object_id=bytes.fromhex('8ff44f081d43176474b267de5451f2c2e88089d0'), ... ) >>> str(swhid) 'swh:1:cnt:8ff44f081d43176474b267de5451f2c2e88089d0'
And vice-versa with
CoreSWHID.from_string()
:>>> swhid == ExtendedSWHID.from_string( ... "swh:1:cnt:8ff44f081d43176474b267de5451f2c2e88089d0" ... ) True
-
object_type
: _TObjectType¶ the type of object the identifier points to