swh.datasets package

Subpackages:

• swh.datasets.luigi package
  • swh.datasets.luigi.aggregate_datasets module
    • Luigi tasks for producing the aggregated derived datasets
    • ExportNodesTable
    • AggregateContentDatasets
    • UploadNodesTable
    • UploadAggregatedContentDataset
    • RunAggregatedDatasets
  • swh.datasets.luigi.blobs_datasets module
    • Luigi tasks for blob-centric datasets
    • atomic_zstd_writer()
    • atomic_csv_zstd_writer()
    • check_csv()
    • SelectBlobs
    • DownloadBlobs
    • MakeBlobTarball
    • MakeSampleBlobTarball
    • ComputeBlobFileinfo
    • BlobScancode
    • FindBlobOrigins
    • CountBlobOrigins
    • FindEarliestRevisions
    • RunBlobDataset
  • swh.datasets.luigi.file_names module
    • Luigi tasks for producing the most common names of every content and datasets based on file names
    • PopularContentNames
    • PopularContentPaths
    • PopularContentNamesOrcToS3
    • ListFilesByName
  • swh.datasets.luigi.impact module
    • Luigi tasks to measure institutional impact
    • SelectPersons
    • ComputeRawImpact
    • opt_int()
    • typeddict_parser()
    • OriginWithEmails
    • OriginWithoutEmails
    • ComputeIndexedImpact
    • DenormalizeImpactedRevrels
  • swh.datasets.luigi.origin_contributors module
    • Luigi tasks for contribution graph
    • ListOriginContributors
    • ExportDeanonymizationTable
    • DeanonymizeContributors
    • DeanonymizeOriginContributors
    • RunOriginContributors
  • swh.datasets.luigi.specific_languages_datasets module
    • ExtractFileExtension
    • JoinFilteredContentsWithNodes
  • Luigi tasks
  • UploadExportAndCompressedGraphToS3
    • UploadExportAndCompressedGraphToS3.requires()
  • RunNewGraph
    • RunNewGraph.requires()

Submodules:

• swh.datasets.cli module
  • PathlibPath
    • PathlibPath.convert()
  • get_all_subclasses()
  • main()
• swh.datasets.download module
  • DatasetDownloader
    • DatasetDownloader.filter_objects()
    • DatasetDownloader.post_downloads()
• swh.datasets.shell module
  • Rust

Module contents: