swh.indexer.indexer module#

class swh.indexer.indexer.ObjectsDict(**kwargs)[source]#

Bases: dict

Typed objects whose keys are names of Kafka topics and values are list of values of messages in that topic.

content: List[Dict]#
directory: List[Dict]#
origin: List[Dict]#
origin_visit_status: List[Dict]#
raw_extrinsic_metadata: List[Dict]#
swh.indexer.indexer.write_to_temp(filename: str, data: bytes, working_directory: str) Iterator[str][source]#

Write the sha1’s content in a temporary file.

Parameters:
  • filename – one of sha1’s many filenames

  • data – the sha1’s content to write in temporary file

  • working_directory – the directory into which the file is written

Returns:

The path to the temporary file created. That file is filled in with the raw content’s data.

swh.indexer.indexer.TId#

type of the ids of index()ed objects.

alias of TypeVar(‘TId’)

swh.indexer.indexer.TData#

type of the objects passed to index().

alias of TypeVar(‘TData’)

swh.indexer.indexer.TResult#

return type of index()

alias of TypeVar(‘TResult’)

class swh.indexer.indexer.BaseIndexer(config=None, **kw)[source]#

Bases: Generic[TId, TData, TResult]

Base class for indexers to inherit from.

The main entry point is the run() function which is in charge of triggering the computations on the batch dict/ids received.

Indexers can:

  • filter out ids whose data has already been indexed.

  • retrieve ids data from storage or objstorage

  • index this data depending on the object and store the result in storage.

To implement a new object type indexer, inherit from the BaseIndexer and implement indexing:

run():

object_ids are different depending on object. For example: sha1 for content, sha1_git for revision, directory, release, and id for origin

To implement a new concrete indexer, inherit from the object level classes: ContentIndexer, DirectoryIndexer, OriginIndexer.

Then you need to implement the following functions:

filter():

filter out data already indexed (in storage).

index_object():

compute index on id with data (retrieved from the storage or the objstorage by the id key) and return the resulting index computation.

persist_index_computations():

persist the results of multiple index computations in the storage.

The new indexer implementation can also override the following functions:

prepare():

Configuration preparation for the indexer. When overriding, this must call the super().prepare() instruction.

check():

Configuration check for the indexer. When overriding, this must call the super().check() instruction.

register_tools():

This should return a dict of the tool(s) to use when indexing or filtering.

Prepare and check that the indexer is ready to run.

results: List[TResult]#
USE_TOOLS = True#
catch_exceptions = True#

Prevents exceptions in index() from raising too high. Set to False in tests to properly catch all exceptions.

scheduler: Any#
storage: StorageInterface#
objstorage: Any#
idx_storage: IndexerStorageInterface#
prepare() None[source]#

Prepare the indexer’s needed runtime configuration. Without this step, the indexer cannot possibly run.

property tool: Dict#
check() None[source]#

Check the indexer’s configuration is ok before proceeding. If ok, does nothing. If not raise error.

register_tools(tools: Union[Dict[str, Any], List[Dict[str, Any]]]) List[Dict[str, Any]][source]#

Permit to register tools to the storage.

Add a sensible default which can be overridden if not sufficient. (For now, all indexers use only one tool)

Expects the self.config[‘tools’] property to be set with one or more tools.

Parameters:

tools – Either a dict or a list of dict.

Returns:

List of dicts with additional id key.

Return type:

list

Raises:

ValueError – if not a list nor a dict.

index(id: TId, data: Optional[TData], **kwargs) List[TResult][source]#

Index computation for the id and associated raw data.

Parameters:
  • id – identifier or Dict object

  • data – id’s data from storage or objstorage depending on object type

Returns:

a dict that makes sense for the persist_index_computations() method.

Return type:

dict

filter(ids: List[TId]) Iterator[TId][source]#

Filter missing ids for that particular indexer.

Parameters:

ids – list of ids

Yields:

iterator of missing ids

abstract persist_index_computations(results: List[TResult]) Dict[str, int][source]#

Persist the computation resulting from the index.

Parameters:

results – List of results. One result is the result of the index function.

Returns:

a summary dict of what has been inserted in the storage

process_journal_objects(objects: ObjectsDict) Dict[source]#

Read swh message objects (content, origin, …) from the journal to:

  • retrieve the associated objects from the storage backend (e.g. storage, objstorage…)

  • execute the associated indexing computations

  • store the results in the indexer storage

class swh.indexer.indexer.ContentIndexer(config=None, **kw)[source]#

Bases: BaseIndexer[bytes, bytes, TResult], Generic[TResult]

A content indexer working on the journal (method process_journal_objects) or on a list of ids directly (method run).

To work on indexer partition, use the ContentPartitionIndexer instead.

Note: ContentIndexer is not an instantiable object. To use it, one should inherit from this class and override the methods mentioned in the BaseIndexer class.

Prepare and check that the indexer is ready to run.

process_journal_objects(objects: ObjectsDict) Dict[source]#

Read content objects from the journal, retrieve their raw content and compute content indexing (e.g. mimetype, fossology license, …).

Note that once this is deployed, this supersedes the main ContentIndexer.run method call and the class ContentPartitionIndexer.

run(ids: List[bytes], **kwargs) Dict[source]#

Given a list of ids:

  • retrieve the content from the storage

  • execute the indexing computations

  • store the results

Parameters:
  • ids (Iterable[Union[bytes, str]]) – sha1’s identifier list

  • **kwargs – passed to the index method

Returns:

A summary Dict of the task’s status

results: List[TResult]#
scheduler: Any#
storage: StorageInterface#
objstorage: Any#
idx_storage: IndexerStorageInterface#
class swh.indexer.indexer.ContentPartitionIndexer(config=None, **kw)[source]#

Bases: BaseIndexer[bytes, bytes, TResult], Generic[TResult]

A content partition indexer.

This expects as input a partition_id and a nb_partitions. This will then index the contents within that partition.

To work on a list of ids, use the ContentIndexer instead.

Note: ContentPartitionIndexer is not an instantiable object. To use it, one should inherit from this class and override the methods mentioned in the BaseIndexer class.

Prepare and check that the indexer is ready to run.

abstract indexed_contents_in_partition(partition_id: int, nb_partitions: int) Iterable[bytes][source]#

Retrieve indexed contents within range [start, end].

Parameters:
  • partition_id – Index of the partition to fetch

  • nb_partitions – Total number of partitions to split into

  • page_token – opaque token used for pagination

run(partition_id: int, nb_partitions: int, skip_existing: bool = True, **kwargs) Dict[source]#

Given a partition of content ids, index the contents within.

Either the indexer is incremental (filter out existing computed data) or it computes everything from scratch.

Parameters:
  • partition_id – Index of the partition to fetch

  • nb_partitions – Total number of partitions to split into

  • skip_existing – Skip existing indexed data (default) or not

  • **kwargs – passed to the index method

Returns:

dict with the indexing task status

results: List[TResult]#
scheduler: Any#
storage: StorageInterface#
objstorage: Any#
idx_storage: IndexerStorageInterface#
class swh.indexer.indexer.OriginIndexer(config=None, **kw)[source]#

Bases: BaseIndexer[str, None, TResult], Generic[TResult]

An object type indexer, inherits from the BaseIndexer and implements Origin indexing using the run method

Note: the OriginIndexer is not an instantiable object. To use it in another context one should inherit from this class and override the methods mentioned in the BaseIndexer class.

Prepare and check that the indexer is ready to run.

run(origin_urls: List[str], **kwargs) Dict[source]#

Given a list of origin urls:

  • retrieve origins from storage

  • execute the indexing computations

  • store the results

Parameters:
  • origin_urls – list of origin urls.

  • **kwargs – passed to the index method

results: List[TResult]#
scheduler: Any#
storage: StorageInterface#
objstorage: Any#
idx_storage: IndexerStorageInterface#
process_journal_objects(objects: ObjectsDict) Dict[source]#

Worker function for JournalClient.

index_list(origins: List[Origin], **kwargs) List[TResult][source]#
class swh.indexer.indexer.DirectoryIndexer(config=None, **kw)[source]#

Bases: BaseIndexer[bytes, Directory, TResult], Generic[TResult]

An object type indexer, inherits from the BaseIndexer and implements Directory indexing using the run method

Note: the DirectoryIndexer is not an instantiable object. To use it in another context one should inherit from this class and override the methods mentioned in the BaseIndexer class.

Prepare and check that the indexer is ready to run.

results: List[TResult]#
scheduler: Any#
storage: StorageInterface#
objstorage: Any#
idx_storage: IndexerStorageInterface#
run(ids: List[bytes], **kwargs) Dict[source]#

Given a list of sha1_gits:

  • retrieve directories from storage

  • execute the indexing computations

  • store the results

Parameters:

ids – sha1_git’s identifier list

process_journal_objects(objects: ObjectsDict) Dict[source]#

Worker function for JournalClient.