swh.core.api package

Submodules

• swh.core.api.asynchronous module
  • encode_msgpack()
  • render_msgpack()
  • render_json()
  • decode_data()
  • decode_request()
  • error_middleware()
  • RPCServerApp
    • RPCServerApp.client_exception_classes
    • RPCServerApp.extra_type_encoders
    • RPCServerApp.extra_type_decoders
  • SWHRemoteAPI
• swh.core.api.classes module
  • PagedResult
    • PagedResult.results
    • PagedResult.next_page_token
  • stream_results()
  • stream_results_optional()
• swh.core.api.gunicorn_config module
  • post_fork()
• swh.core.api.negotiation module
  • FormatterNotFound
  • Formatter
    • Formatter.format
    • Formatter.mimetypes
    • Formatter.configure()
    • Formatter.render()
  • Negotiator
    • Negotiator.register_formatter()
    • Negotiator.get_formatter()
    • Negotiator.accept_mimetypes
    • Negotiator.best_mimetype()
  • negotiate()
• swh.core.api.serializers module
  • encode_datetime()
  • exception_to_dict()
  • dict_to_exception()
  • encode_timedelta()
  • get_encoders()
  • get_decoders()
  • MsgpackExtTypeCodes
    • MsgpackExtTypeCodes.LONG_INT
    • MsgpackExtTypeCodes.LONG_NEG_INT
  • encode_data_client()
  • decode_response()
  • SWHJSONEncoder
    • SWHJSONEncoder.default()
  • SWHJSONDecoder
    • SWHJSONDecoder.decode_data()
    • SWHJSONDecoder.raw_decode()
  • json_dumps()
  • json_loads()
  • msgpack_dumps()
  • msgpack_loads()

Module contents

swh.core.api.get_distribution_version_for_class(cls: Type) → str

Get the version of the module shipping the given class.

When the class is in module swh.a.b.c, lookup distributions named swh-a-b-c, swh-a-b and swh-a in turn.

swh.core.api.format_user_agent(main_version: str, *extra_parameters: str) → str

Format a user-agent string from a main version and a list of extra parameters, which are then parenthesized.

class swh.core.api.Negotiator(func: Callable[[...], Any])

Bases: Negotiator

best_mimetype()

swh.core.api.negotiate(formatter_cls, *args, **kwargs)

class swh.core.api.Formatter(request_mimetype: str | None = None)

Bases: Formatter

configure(extra_encoders=None)

class swh.core.api.JSONFormatter(request_mimetype: str | None = None)

Bases: Formatter

format: str | None = 'json'

mimetypes: List[str] = ['application/json']

render(obj)

class swh.core.api.MsgpackFormatter(request_mimetype: str | None = None)

Bases: Formatter

format: str | None = 'msgpack'

mimetypes: List[str] = ['application/x-msgpack']

render(obj)

exception swh.core.api.RemoteException(payload: Any | None = None, response: Response | None = None)

Bases: Exception

raised when remote returned an out-of-band failure notification, e.g., as a HTTP status code or serialized exception

response

HTTP response corresponding to the failure

exception swh.core.api.TransientRemoteException(payload: Any | None = None, response: Response | None = None)

Bases: RemoteException

Subclass of RemoteException representing errors which are expected to be temporary.

swh.core.api.remote_api_endpoint(path: str, method: str = 'POST') → Callable[[F], F]

exception swh.core.api.APIError

Bases: Exception

API Error

class swh.core.api.MetaRPCClient(name, bases, attributes)

Bases: type

Metaclass for RPCClient, which adds a method for each endpoint of the database it is designed to access.

See for example swh.indexer.storage.api.client.RemoteStorage

class swh.core.api.RPCClient(url: str, timeout: None | Tuple[float, float] | List[float] | float = None, chunk_size: int = 4096, max_retries: int = 3, pool_connections: int = 20, pool_maxsize: int = 100, adapter_kwargs: Dict[str, Any] | None = None, api_exception: Type[Exception] | None = None, reraise_exceptions: List[Type[Exception]] | None = None, enable_requests_retry: bool | None = None, session_kwargs: Dict[str, Any] | None = None, **kwargs)

Bases: object

Proxy to an internal SWH RPC.

Parameters:

• url – base url for the RPC endpoints

• timeout – request timeout. Can be a pair of floats, used to distinguish between connection and read timeouts.

• chunk_size – used for iteration on chunked responses

• max_retries – number of automatic retries issued for requests

• pool_connections – number of connections instantiated in the default connection pool

• pool_maxsize – maximum size of the connection pool

• adapter_kwargs – extra keyword arguments to pass to the requests.adapters.HTTPAdapter

• api_exception – The exception class to raise in case of communication error with the server.

• reraise_exceptions – On server errors, if any of the exception classes in this list has the same name as the error name, then the exception will be instantiated and raised instead of a generic RemoteException.

• enable_requests_retry – If set to True, requests sent by the client will be retried when encountering specific errors. Default policy is to retry when connection errors or transient remote exceptions are raised. Subclasses can change that policy by overriding the retry_policy() method.

• session_kwargs – keyword arguments passed to the requests.Session used to make all requests. Commonly useful parameters include auth, headers and verify.

backend_class: ClassVar[type | None] = None

For each method of backend_class decorated with remote_api_endpoint(), a method with the same prototype and docstring will be added to this class. Calls to this new method will be translated into HTTP requests to a remote server.

This backend class will never be instantiated, it only serves as a template.

extra_type_encoders: List[Tuple[type, str, Callable]] = []

Value of extra_encoders passed to json_dumps or msgpack_dumps to be able to serialize more object types.

extra_type_decoders: Dict[str, Callable] = {}

Value of extra_decoders passed to json_loads or msgpack_loads to be able to deserialize more object types.

api_exception

The exception class to raise in case of communication error with the server.

alias of APIError

reraise_exceptions: List[Type[Exception]] = []

On server errors, if any of the exception classes in this list has the same name as the error name, then the exception will be instantiated and raised instead of a generic RemoteException.

enable_requests_retry: bool = False

If set to True, requests sent by the client will be retried when encountering specific errors. Default policy is to retry when connection errors or transient remote exceptions are raised. Subclasses can change that policy by overriding the retry_policy() method.

retry_policy(retry_state)

raw_verb(verb, endpoint, **opts)

post(*args, **kwargs)

post_stream(*args, **kwargs)

get(*args, **kwargs)

get_stream(*args, **kwargs)

raise_for_status(response) → None

check response HTTP status code and raise an exception if it denotes an error; do nothing otherwise

class swh.core.api.BytesRequest(environ: WSGIEnvironment, populate_request: bool = True, shallow: bool = False)

Bases: Request

Request with proper escaping of arbitrary byte sequences.

encoding = 'utf-8'

encoding_errors = 'surrogateescape'

swh.core.api.encode_data_server(data, content_type='application/x-msgpack', extra_type_encoders=None)

swh.core.api.decode_request(request, extra_decoders=None)

swh.core.api.error_handler(exception: BaseException, encoder=<function encode_data_server>, status_code: int = 500)

Error handler to be registered using flask’s error-handling decorator app.errorhandler.

This is used for exceptions that are expected in the normal execution flow of the RPC-ed API, in which case the status code should be set to a value in the 4xx range, as well as for exceptions that are unexpected (generally, a bare Exception), and for which the status code should be kept in the 5xx class.

This function only captures exceptions as sentry errors if the status code is in the 5xx range and not 502/503/504, as “expected exceptions” in the 4xx range are more, likely to be handled on the client side; and 502/503/504 are “transient” exceptions that should be resolved with client retries.

class swh.core.api.RPCServerApp(*args, backend_class=None, backend_factory=None, **kwargs)

Bases: Flask

For each endpoint of the given backend_class, tells app.route to call a function that decodes the request and sends it to the backend object provided by the factory.

Parameters:

• backend_class (Any) – The class of the backend, which will be analyzed to look for API endpoints.

• backend_factory (Optional[Callable[[], backend_class]]) – A function with no argument that returns an instance of backend_class. If unset, defaults to calling backend_class constructor directly.

For each method ‘do_x()’ of the backend_factory, subclasses may implement two methods: pre_do_x(self, kw) and post_do_x(self, ret, kw) that will be called respectively before and after do_x(**kw). kw is the dict of request parameters, and ret is the return value of do_x(**kw).

request_class

alias of BytesRequest

extra_type_encoders: List[Tuple[type, str, Callable]] = []

Value of extra_encoders passed to json_dumps or msgpack_dumps to be able to serialize more object types.

extra_type_decoders: Dict[str, Callable] = {}

Value of extra_decoders passed to json_loads or msgpack_loads to be able to deserialize more object types.

method_decorators: List[Callable[[Callable], Callable]] = []

List of decorators to all methods generated from the backend_class.

exception_status_codes: List[Tuple[Type[BaseException] | str, int]] = [(<class 'Exception'>, 500), ('psycopg.errors.OperationalError', 503), ('psycopg.errors.QueryCanceled', 500), ('swh.journal.writer.kafka.KafkaDeliveryError', 503)]

Pairs of (exception, status_code) where exception is either an exception class or a a dotted exception name to be imported (and ignored if import fails) and status_code is the HTTP code that should be returned when an instance of this exception is raised.

If a raised exception is an instance of a subclass of two classes defined here, the most specific class wins, according Flask’s MRO-based resolution.

add_backend_class(backend_class=None, backend_factory=None)

setup_psycopg_errorhandlers() → None

Deprecated method; error handlers are now setup in the constructor.