swh.graph.luigi.provenance module#

Luigi tasks to help compute the provenance of content blobs#

This module contains Luigi tasks driving the computation of a topological order, and count the number of paths to every node.

This assumes a local compressed graph (from swh.graph.luigi.compressed_graph) is present, and generates/manipulates the following files:

base_dir/
    <date>[_<flavor>]/
        provenance/
            topological_order_dfs.csv.zst

class swh.graph.luigi.provenance.SortRevrelByDate(*args, **kwargs)[source]#

Bases: Task

Creates a file that contains all revision/release author dates and their SWHIDs in date order from a graph export.

requires() → Dict[str, Task][source]#: Returns LocalGraph and LocalExport instances

output() → LocalTarget[source]#: Returns {provenance_dir}/revrel_by_author_date.csv.zst

run() → None[source]#: For each ORC revision or release file, read it with pyorc, produce a “date,swhid” CSV, and sort it with GNU sort. Then merge all outputs with GNU sort.

static orc_to_csv()[source]#

Must be called as a CLI script. Syntax: {rev,rel} path/to/dataset/file.orc

Reads an ORC file containing revisions or releases, and writes a CSV to its stdout, containing a date and a SWHID on each row.

class swh.graph.luigi.provenance.ListEarliestRevisions(*args, **kwargs)[source]#

Bases: Task

Creates a file that contains all directory/content SWHIDs, along with the first revision/release author date and SWHIDs they occur in.

requires() → Dict[str, Task][source]#: Returns LocalGraph and SortRevrelByDate instances.

output() → LocalTarget[source]#: Returns {provenance_dir}/revrel_by_author_date.csv.zst

run() → None[source]#: Runs org.softwareheritage.graph.utils.ListEarliestRevisions