.. _roadmap-2022:
Roadmap 2022
============
(Version 1.0, last modified 2022-04-01)
This document provides an overview of the technical roadmap of the Software
Heritage initiative for the year 2022.
Live tracking of the roadmap implementation progress during the year is
available from a dedicated `Kanban board
`_.
.. contents::
:depth: 3
..
Collect
-------
Extend archive coverage (2+2 loaders/listers)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: ardumont
- Tags: coverage
- Task: `T4079 `__
- Effort: variable, depending on the chosen listers/loaders (4PM ?)
- Priority: Medium
Deploy at least 2 additional loaders (of currently unsupported VCS/package formats) and 2 additional listers (of currently unsupported hosting platforms), expanding the coverage of the Software Heritage archive. Listers and loaders can be developed in house or contributed by external partners, e.g., via dedicated grants.
KPIs:
- Number of new loaders/listers deployed
- Number of origins archived/listed
Minimize archival lag w.r.t. upstream code hosting platforms
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: olasd
- Tags: performance, coverage
- Task: `T4080 `__
- Effort: 3 PM
- Priority: High
Includes work:
- Quantify and monitor in real-time the lag, especially for major platforms (GitHub, GitLab.com, etc.)
- Improve ingestion efficiency (optimize loaders, especially the Git loader, optimize scheduling policies) - `T2207 `__
- Make lag monitoring dashboards easy to find (for decision makers)
KPIs:
- Number of out of date repos (absolute and per platform)
- Total archive lag (e.g., in days)
Add forge now
^^^^^^^^^^^^^
- Lead: ardumont
- Tags: coverage
- Task: `T1538 `__
- Effort: 3 PM
- Priority: High
Includes work:
Make it user-driven, simple, and efficient to fully and recurrently archive a new instance of an already supported code hosting platform.
- User-facing web form allowing any user to *propose* the archival of a new forge instance, and moderation web UI to validate archival requests before ingestion. `T4047 `__
- Admin tooling and UI to deal with received submissions. `T4058 `__
- Include free-from box suggestion form for forges that are not supported yet (to replace the currently poorly maintained `wiki page `__). Possibly to be integrated with the user support system elsewhere in the roadmap.
KPIs:
- Number of forges/instances added
Integrate deposit with InvenioRDM
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: moranegg
- Tags: 2021, coverage, deposit
- Task: `T2344 `__
- Effort: 1-2 PM
- Priority: Medium
Includes work:
Deploy in production support for receiving source code deposits from InvenioRDM instances, and in particular the Zenodo instance.
- Extend CodeMeta vocabulary to qualify author relationships - `T2329 `__
- Generalize usage of SWHID for referencing SWH archive objects - `T3034 `__
- Analyze deposit-client on InvenioRDM compatibility - `T3549 `__
KPIs:
- Complete on paper spec
- Number of deposits from an InvenioRDM instance (can be staging instance)
- Support deployed in InvenioRDM LTS
Admin tooling for takedown notices
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: douardda
- Tags: 2021, legal
- Task: `T3087 `__
- Effort: 3 PM
- Priority: High
Includes work:
Admin interface, private and public journal of operations.
- Low level support for blacklisting specified contents (not only URLs, also SWHIDs), with support for regexps
- Admin interface to add/remove entries from the blacklist
- A journal of these operations (what was added/removed, when and why, from the blacklist)
- A public webpage that maintains the list of accepted takedown notices
KPIs:
- Takedown tools deployed in production
- Number of processed takedown notices
Preserve
--------
Continuous data validation of all the data stores of SWH
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: vlorentz
- Tags: integrity, monitoring
- Task: `T3841 `__
- Effort: 2 PM
- Priority: Medium
Includes work:
- Set up background jobs to regularly check data validity in all SWH data stores.
- This includes both blobs (swh-objstorage) and other graph objects (swh-storage) on all the copies (in-house, kafka, azure, upcoming mirrors, etc.).
- Estimate ETA for scrubbing of the entire archive.
KPIs:
- Scrubbers deployed in production
- Monitoring tools deployed in production
- % of the archive scrubbed
Support archiving repositories containing SHA1 hash conflicts on blobs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: olasd
- Tags: crypto
- Task: `T3775 `__
- Effort: 1.5 PM
- Priority: High
Includes work:
This involves getting rid of the limitations imposed by having SHA1 as a primary key for the object storage internally.
KPIs
- Ability to archive git repos that contains sample SHAttered collisions blobs (they are currently detected and refused)
Up-to-date anonymized archive copy on Amazon S3 (except blobs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: vlorentz (originally seirl)
- Tags: 2021, archivecopy
- Task: `T3085 `__
- Effort: 3 PM
- Priority: Low
Includes work:
Periodic dumps of the (anonymized) Merkle graph on the Amazon public cloud.
- Fully automate export of the graph dataset
- Document how to export the graph edge dataset
- Define a scheduling periodicity
KPIs:
- Automatic exports scheduled
- S3 copy up to date w/ last scheduled export
Archive cold-copy at CINES via Vitam
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: douardda
- Tags: 2021, archivecopy
- Task: `T3414 `__
- Effort: 2PM
- Priority: Medium
Includes work:
Perform a first complete copy of the archive stored in Vitam @ CINES
Maintain the copy up-to-date periodically (on a period TBD)
KPIs:
- First copy stored in Vitam
- Updates calendar defined
Mirrors
^^^^^^^
- Lead: douardda
- Tags: 2021, mirror
- Task: `T3116 `__
- Effort: 2 PM
- Priority: High
Includes work:
Deploy in production at least 2 mirrors.
- Finalize ENEA Mirror deployment
- Launch Snyk mirror project
- handle takedown notice synchronization ?
- Add feature flags on web UI
KPIs:
- ENEA Mirror in production
- Snyk mirror in production
Publicly available standard for SWHID version 1
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: zack
- Tags: 2021, standard, swhid
- Task: `T3960 `__
- Effort: 1 PM
- Priority: High
Includes work:
Publish a stable version of the SWHID version 1 specification, approved by a standard organization body.
KPIs:
- Published standard for SWHID version 1
SWHID version 2
^^^^^^^^^^^^^^^
- Lead: zack
- Tags: 2021, swhid, crypto
- Task: `T3134 `__
- Effort: 4 PM
- Priority: Low
Includes work:
Complete on paper specification for SWHID version 2, including migrating to a stronger hash than SHA1.
- Complete on paper spec
- Aligned with work done on new git hashes
- Migration plan from/cohabitation with v1 (N.B.: we need to maintain SWHID v1 support forever anyway)
- Understand impact on internal microservice architecture (related to `T1805 `__, in particular use SWHIDs everywhere (core SWHIDs, without qualifiers))
- Keep correspondence with v1 (there may be multiple v2 for one v1)
- Reviewed by crypto experts
KPIs:
- Written SWHID version 2 specification
Share
-----
Show metadata on Web UI
^^^^^^^^^^^^^^^^^^^^^^^
- Lead: vlorentz
- Tags: share, present, webui
- Task: `T4081 `__
- Effort: 3 PM
- Priority: Low
Includes work:
Layer 1: show intrinsic and extrinsic metadata for artifact on web UI (design, implementation and deployment) Layer 2: add linked data capabilities (Semantic Web solutions)
- Design metadata view for Web UI
- Allow export of metadata (in multiple formats - APA/ BibTeX/ CodeMeta/ CFF)
- Assistance and contribution to CodeMeta
KPIs:
- Amount of metadata accessible on Web UI
Provide a state-of-the-art UX for web search
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: jayesh
- Tags: search
- Task: `T3952 `__
- Effort: 3 PM
- Priority: Medium
Includes work:
- Make the textual search language of archive.s.o a first-class citizen, including:
- Simplify syntax
- Conduct UX audits and user-testing of the web search UI
- Note: this does *not* include extending the type of data currently indexed and used for search (e.g., no filenames, no file content, etc.; they can come later/separately).
KPIs:
- SWH search using QL available in production
- Default user experience for archive.s.o textual searches
Self-host Software Stories software stack
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: moranegg
- Tags: communication, wikidata, docs
- Task: `T3954 `__
- Effort: 1 PM
- Priority: Medium
Includes work:
- Deploy `stories instance `__ in production on the SWH infrastructure.
KPIs:
- Software stories app deployed in production on SWH infra
- Content of current stories migrated to SWH instance
Webhook-based notification for long-running user tasks
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: anlambert
- Tags: deposit, vault, savecodenow
- Task: `T3955 `__
- Effort: 1-3 PM
- Priority: High
Includes work:
- Create a reusable webhook architecture
- Add support for webhook-based notifications of long-running user tasks, including:
- Deposit
- Vault cooking
- Save code now
- Add forge now
- Origin visit
KPIs:
- Number of services that support webhook-based notifications
Collect and index forge metadata
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: vlorentz
- Tags: 2021
- Task: `T2202 `__
- Effort: 9 PM
- Priority: High
Includes work:
- Collect extrinsic metadata from at least 1 forge (e.g., GitHub or GitLab project metadata)
- Index them into a sensible and searchable ontology/data model (could be codemeta, if suitable, or something else if needed)
- Cross-reference them to archived objects via SWHID
- Enable searches based on indexed metadata
KPIs:
- Number of forges supported
- Metadata fields collected
Prior art detection
^^^^^^^^^^^^^^^^^^^
- Lead: zack
- Tags: 2021
- Task: `T3136 `__
- Effort: 5 PM
- Priority: Medium
Includes work:
Provide a full-circle user toolchain for prior-art detection in the realm of software source code.
- `revamp swh-scanner result dashboard `__
- Integrate with swh-provenance
- Integrate with swh-graph
KPIs:
- Release and announce a beta version of swh-scanner
Documentation
-------------
docs.s.o: provide a landing page, dispatching to devel/user/sysadmin/mirrors
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: bchauvet
- Tags: docs, sys-admin
- Task: `T3867 `__
- Effort: 0.5 PM
- Priority: Medium
Includes work:
- Provide a nice landing page for all documentation at docs.s.o, dispatching by user type.
- Drop the redirection docs.s.o -> docs.s.o/devel.
- Depends on populating the /sysadm, /user and /mirrors parts.
KPIs:
- Landing page in production (https://docs.softwareheritage.org)
docs.s.o/sysadm: improve sysadmin documentation website
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: vsellier
- Tags: docs, sys-admin
- Task: `T4082 `__
- Effort: 1 PM
- Priority: Medium
Includes work:
- General goal: onboarding material + transparency about how we run the archive.
- Target user: team member, partners (e.g.mirror operators), or contributor who needs a clear view of the infrastructure architecture.
This task will be completed when it:
- Documents the configuration system of each component.
- Documents hardware architecture.
- Documents CI architecture (and other major services currently not documented).
KPIs:
- List of minimum documented items
- Number of available documented items
docs.s.o/user: bootstrap user documentation website
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: moranegg
- Tags: docs, user
- Task: `T3972 `__
- Effort: 2 PM
- Priority: Medium
Includes work:
The currently available user documentation only provides a FAQ. It should contain at least:
- An overall non-technical description of the archive and the core elements of its architecture
- A set of howto/getting started pages on main subjects (search, browse, push code in the archive, retrieve code and artifacts from the archive, metadata)
- Link to existing documentation on the main w.s.o. site as appropriate.
KPIs:
- List of minimum documented items
- Number of available documented items
High-level overview of available listers/loaders
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: anlambert
- Tags: 2021, docs, sys-admin
- Task: `T3117 `__
- Effort: 0.5 PM
- Priority: High
Includes work:
Publish a web page (under docs.s.o somewhere) providing a high-level overview of which listers/loaders are available (implemented, deployed, running, etc.) with pointers to the corresponding modules/implementations.
KPIs:
- Online web page
Technical Debt
--------------
Refactor swh-web code
^^^^^^^^^^^^^^^^^^^^^
- Lead: anlambert
- Tags: webapp, refactoring
- Task: `T3949 `__
- Effort: 3 PM
- Priority: Medium
Includes work:
Have a smaller, more modular code base
- Split the public API code from the frontend code base
- Reduce code duplication (eg. between API and frontend)
- Externalize conversion utilities towards swh-core
KPIs:
- Separate repositories for frontend and web API
New public API (GraphQL + thin layer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: jayesh
- Tags: api, refactoring
- Task: `T4083 `__
- Effort: 4 PM
- Priority: Medium
Includes work:
Provide a common unified (GraphQL based) public API
- Create a GraphQL based API
- Integrate actual API on graphQL
KPIs:
- GraphQL API in production
Organize 4+ short peer programming code-audit sprints
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: bchauvet
- Tags: refactoring
- Task: `T3956 `__
- Effort: 2.5 PM
- Priority: n/a (one 2-day sprint every 2 months)
Includes work:
- Go through the entire codebase and identify changes that should be done and dead code
- Correct identified issues or, failing that, document them with dedicated tasks
- Identify one theme per sprint
KPIs:
- Sprints done
Organize 4+ sentry-cleaning sprints
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: bchauvet
- Tags: project-management, monitoring
- Task: `T3957 `__
- Effort: 2.5 PM
- Priority: n/a (one 2-day sprint every 2 months)
Includes work:
We currently have a lot of `open Sentry issues `__, but this is very raw data that isn’t very usable or visible. They should be cleaned up so that under normal conditions, the number of reported issues stays “minimal”.
KPIs:
- Sprints done
- Number of sentry issues (before/after)
Tooling and Infrastructure
--------------------------
GitLab migration
^^^^^^^^^^^^^^^^
- Lead: olasd
- Tags: 2021
- Task: `T2225 `__
- Effort: 3 PM
- Priority: Medium
Includes work:
- Review the current workflow for the migration
- Prepare new team workflows for some “sample” projects
- Drive the migration to completion
- Sysadmin projects migration (iteration #1)
- Remaining projects migration (iteration #2)
KPIs:
- Number of migrated projects
- Phabricator switched to read-only
Polish developer-facing CI automation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: olasd
- Tags: development environment, CI
- Task: `T4084 `__
- Effort: 3 PM
- Priority: Low
Includes work:
- More automation to keep all linting / testing tools (black, flake8, tox, …) up to date and consistent
- CI support for multiple python versions (and possibly some dependency versions)
- Faster CI for diffs (e.g., consider use of `testmon `__ to only run tests affected by changes)
- Investigation of more linters or flake8 plugins
- Cypress performance (parallel testing)
KPIs:
- To be defined
Continuous Deployment
^^^^^^^^^^^^^^^^^^^^^
- Lead: vsellier
- Task: `T2231 `__
- Tags: CI, CD, packaging
- Effort: 6 PM
- Priority: Low
Includes work:
Improve bug detection Validate the future elastic infrastructure components
- Migrate away from Debian packaging for deployment
- Build a docker image per deployable service
- Build the deployment tooling
- Reset and redeploy the stack after commits
- Execute acceptance tests
KPIs:
- Operational CD platform
- CD integrated to gitlab
Continuous Integration for sysadmin tools
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: vsellier
- Tags: sysadmin, CI, tooling
- Task: `T3834 `__
- Effort: 2 PM
- Priority: Low
Includes work:
Add CI for sysadmin tasks:
- Puppet configuration
- Vagrant projects
- Terraform plans
- Container (docker) image production
Create sustainable plan for hardware provisioning/rotation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: olasd
- Tags: sysadmin, hardware
- Task: `T3959 `__
- Effort: 0.5 PM
- Priority: High
Write a policy for hardware procurement with the following in mind:
- Make sure that we properly track our current pool of hardware, and its warranty status
- Make sure we don’t get surprised by lapsing warranties
- Make sure that we don’t end up having to renew a bunch of machines *all at once*
- Allow better budget previsions
KPIs:
- Shared documented policy
Elastic loaders and listers
^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: ardumont
- Tags: sysadmin, performance, elasticity
- Task: `T3592 `__
- Effort: 3 PM
- Priority: High
Includes work:
- Deploy the listers and loaders in containers
- Deploy on a couple of bare metal servers (?)
- Easily adapt the load to the resources and the waiting tasks
KPIs:
- Running elastic infrastructure in production for loaders and listers
- Cluster / elastic workers monitoring (number of running workers, statsd, …)
Cassandra in production as primary storage
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: vsellier
- Tags: 2021, storage, sysadmin
- Task: `T2214 `__
- Effort: 3 PM
- Priority: High
Includes work:
- Have the Cassandra storage in production as primary storage
- Set up equivalent MVP in staging
KPIs:
- Cassandra primary storage in production
Scale-out objstorage in production as primary objstorage
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: olasd
- Tags: 2021, objstorage, sysadmin
- Task: `T3054 `__
- Effort: 2 PM
- Priority: High
Includes work:
- Have the Ceph-based objstorage in production as primary storage
- Set up equivalent MVP in staging (maybe use the same Ceph cluster for this)
KPIs:
- Ceph-based obj-storage in production
Provenance in production
^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: douardda
- Tags: 2021, provenance
- Task: `T3112 `__
- Effort: 3 PM
- Priority: High
Includes work:
Have the provenance index in production with less then a month of lag
Set up equivalent MVP in staging
- Produce documentation
- Finalize revisions layer processing
- Investigate/solve revisions performance issues
- Process origins layer
- Flatten directories
- Production setup (deployment / scripts)
- Implement a querying API
KPIs:
- Revisions processed per second
- % of archive covered
- Published documentation
Graph compression in production
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: vlorentz (originally seirl)
- Tags: 2021, graph compression
- Task:`T2220 `__
- Effort: 2 PM
- Priority: High
Includes work:
- Have the graph compression pipeline running in production with less then a month of lag
- Deployment, hosting and pipeline tooling
- Handle the situation for staging
KPIs:
- Graph compression pipeline in production
- Last update date / number of updates per year
Mirror tooling in production
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: douardda
- Tags: 2021, mirror
- Task: `T4085 `__
- Effort: 2 PM
- Priority: High
Includes work:
- Document the setup, the administration and the maintenance of a mirror (sprint + maintenance)
- Handle the situation for staging
- Organize the mirror operators community
KPIs:
- Mirror on staging
- Organized community
User support ticket system and process
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: bchauvet
- Tags: support, user
- Task: `T3730 `__
- Effort: 1 PM
- Priority: Medium
Includes work:
- Create a user-facing ticket system to support user requests and bug reports (e.g., a support@ address that automatically create support tasks that we can triage and follow)
- Define the process to:
- Ensure some basic quality of service (e.g., time to first answer)
- Pending tasks are not forgotten.
KPIs:
- User support feature available on web UI
Reliable user-level monitoring of services
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Lead: vsellier
- Tags: 2021, support, user
- Task: `T3129 `__
- Effort: 1 PM
- Priority: High
Includes work:
High-level view of which services are running or not, and integration with status.softwareheritage.org
KPIs:
- Services dashboard in production