swh.deposit package

Submodules

swh.deposit.apps module

class swh.deposit.apps.DepositConfig(app_name, app_module)[source]

Bases: django.apps.config.AppConfig

name = 'swh.deposit'
ready()[source]

Override this method in subclasses to run code when Django starts.

__module__ = 'swh.deposit.apps'

swh.deposit.auth module

swh.deposit.auth.convert_response(request, content)[source]

Convert response from drf’s basic authentication mechanism to a swh-deposit one.

Args:
request (Request): Use to build the response content (bytes): The drf’s answer

Returns:

Response with the same status error as before, only the body is now an swh-deposit compliant one.
class swh.deposit.auth.WrapBasicAuthenticationResponseMiddleware(get_response)[source]

Bases: object

Middleware to capture potential authentication error and convert them to standard deposit response.

This is to be installed in django’s settings.py module.

__init__(get_response)[source]

Initialize self. See help(type(self)) for accurate signature.

__call__(request)[source]

Call self as a function.

__dict__ = mappingproxy({'__doc__': "Middleware to capture potential authentication error and convert\n them to standard deposit response.\n\n This is to be installed in django's settings.py module.\n\n ", '__call__': <function WrapBasicAuthenticationResponseMiddleware.__call__>, '__dict__': <attribute '__dict__' of 'WrapBasicAuthenticationResponseMiddleware' objects>, '__module__': 'swh.deposit.auth', '__weakref__': <attribute '__weakref__' of 'WrapBasicAuthenticationResponseMiddleware' objects>, '__init__': <function WrapBasicAuthenticationResponseMiddleware.__init__>})
__module__ = 'swh.deposit.auth'
__weakref__

list of weak references to the object (if defined)

swh.deposit.config module

swh.deposit.config.setup_django_for(platform=None, config_file=None)[source]
Setup function for command line tools (swh.deposit.create_user) to
initialize the needed db access.

Note

Do not import any django related module prior to this function call. Otherwise, this will raise an django.core.exceptions.ImproperlyConfigured error message.

Parameters:
  • platform (str) – the platform the scheduling is running
  • config_file (str) – Extra configuration file (typically for the production platform)
Raises:

ValueError in case of wrong platform inputs.
class swh.deposit.config.SWHDefaultConfig(**config)[source]

Bases: swh.core.config.SWHConfig

Mixin intended to enrich views with SWH configuration.

CONFIG_BASE_FILENAME = 'deposit/server'
DEFAULT_CONFIG = {'checks': ('bool', True), 'max_upload_size': ('int', 209715200), 'scheduler': ('dict', {'cls': 'remote', 'args': {'url': 'http://localhost:5008/'}})}
ADDITIONAL_CONFIG = {}
__init__(**config)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'swh.deposit.config'

swh.deposit.errors module

Module in charge of providing the standard sword errors

exception swh.deposit.errors.ParserError[source]

Bases: ValueError

Specific parsing error detected when parsing the xml metadata input

__module__ = 'swh.deposit.errors'
__weakref__

list of weak references to the object (if defined)

swh.deposit.errors.make_error_dict(key, summary=None, verbose_description=None)[source]

Utility function to factorize error message dictionary.

Parameters:
  • key (str) – Error status key referenced in swh.deposit.errors module
  • summary (str/None) – Error message clarifying the status
  • verbose_description (str/None) – A more verbose description or work around a potential problem.
Returns:

Dictionary with key ‘error’ detailing the ‘status’ and associated ‘message’
swh.deposit.errors.make_error_response_from_dict(req, error)[source]

Utility function to return an http response with error detail.

Parameters:
  • req (Request) – original request
  • error (dict) – Error described as dict, typically generated
  • the make_error_dict function. (from) –
Returns:

HttpResponse with detailed error.
swh.deposit.errors.make_error_response(req, key, summary=None, verbose_description=None)[source]

Utility function to create an http response with detailed error.

Parameters:
  • req (Request) – original request
  • key (str) – Error status key referenced in swh.deposit.errors module
  • summary (str) – Error message clarifying the status
  • verbose_description (str / None) – A more verbose description or work around a potential problem.
Returns:

Dictionary with key ‘error’ detailing the ‘status’ and associated ‘message’

swh.deposit.manage module

swh.deposit.models module

class swh.deposit.models.Dbversion(*args, **kwargs)[source]

Bases: django.db.models.base.Model

Db version

version

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

release

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

description

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

__str__()[source]

Return str(self).

exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

__module__ = 'swh.deposit.models'
__reduce__()
__setstate__(args)
exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

__module__ = 'swh.deposit.models'
__reduce__()
__setstate__(args)
__module__ = 'swh.deposit.models'
_meta = <Options for Dbversion>
objects = <django.db.models.manager.Manager object>
swh.deposit.models.DEPOSIT_STATUS = [('partial', 'partial'), ('expired', 'expired'), ('deposited', 'deposited'), ('verified', 'verified'), ('rejected', 'rejected'), ('loading', 'loading'), ('done', 'done'), ('failed', 'failed')]

Possible status and the detailed meaning.

class swh.deposit.models.DepositClient(*args, **kwargs)[source]

Bases: django.contrib.auth.models.User

Deposit client

collections

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

objects = <django.contrib.auth.models.UserManager object>
provider_url

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

domain

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

__str__()[source]

Return str(self).

exception DoesNotExist

Bases: django.contrib.auth.models.DoesNotExist

__module__ = 'swh.deposit.models'
__reduce__()
__setstate__(args)
exception MultipleObjectsReturned

Bases: django.contrib.auth.models.MultipleObjectsReturned

__module__ = 'swh.deposit.models'
__reduce__()
__setstate__(args)
__module__ = 'swh.deposit.models'
_meta = <Options for DepositClient>
deposit_set

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

parent.children is a ReverseManyToOneDescriptor instance.

Most of the implementation is delegated to a dynamically defined manager class built by create_forward_many_to_many_manager() defined below.

user_ptr

Accessor to the related object on the forward side of a one-to-one relation.

In the example:

class Restaurant(Model):
    place = OneToOneField(Place, related_name='restaurant')

restaurant.place is a ForwardOneToOneDescriptor instance.

user_ptr_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

class swh.deposit.models.Deposit(*args, **kwargs)[source]

Bases: django.db.models.base.Model

Deposit reception table

id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

reception_date

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

complete_date

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

collection

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

external_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

client

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

swh_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

swh_id_context

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

swh_anchor_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

swh_anchor_id_context

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

status

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

status_detail

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

parent

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

check_task_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

load_task_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

__str__()[source]

Return str(self).

exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

__module__ = 'swh.deposit.models'
__reduce__()
__setstate__(args)
exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

__module__ = 'swh.deposit.models'
__reduce__()
__setstate__(args)
__module__ = 'swh.deposit.models'
_meta = <Options for Deposit>
client_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

collection_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

deposit_set

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

parent.children is a ReverseManyToOneDescriptor instance.

Most of the implementation is delegated to a dynamically defined manager class built by create_forward_many_to_many_manager() defined below.

depositrequest_set

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

parent.children is a ReverseManyToOneDescriptor instance.

Most of the implementation is delegated to a dynamically defined manager class built by create_forward_many_to_many_manager() defined below.

get_next_by_reception_date(**morekwargs)
get_previous_by_reception_date(**morekwargs)
get_status_display(**morekwargs)
objects = <django.db.models.manager.Manager object>
parent_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

swh.deposit.models.client_directory_path(instance, filename)[source]

Callable to upload archive in MEDIA_ROOT/user_<id>/<filename>

Parameters:
  • instance (DepositRequest) – DepositRequest concerned by the upload
  • filename (str) – Filename of the uploaded file
Returns:

A path to be prefixed by the MEDIA_ROOT to access physically to the file uploaded.
class swh.deposit.models.DepositRequest(*args, **kwargs)[source]

Bases: django.db.models.base.Model

Deposit request associated to one deposit.

id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

deposit

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

child.parent is a ForwardManyToOneDescriptor instance.

date

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

metadata

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

raw_metadata

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

archive

The descriptor for the file attribute on the model instance. Returns a FieldFile when accessed so you can do stuff like:

>>> from myapp.models import MyModel
>>> instance = MyModel.objects.get(pk=1)
>>> instance.file.size

Assigns a file object on assignment so you can do:

>>> with open('/path/to/hello.world', 'r') as f:
...     instance.file = File(f)
type

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

__str__()[source]

Return str(self).

exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

__module__ = 'swh.deposit.models'
__reduce__()
__setstate__(args)
exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

__module__ = 'swh.deposit.models'
__reduce__()
__setstate__(args)
__module__ = 'swh.deposit.models'
_meta = <Options for DepositRequest>
deposit_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

get_next_by_date(**morekwargs)
get_previous_by_date(**morekwargs)
get_type_display(**morekwargs)
objects = <django.db.models.manager.Manager object>
class swh.deposit.models.DepositCollection(id, name)[source]

Bases: django.db.models.base.Model

id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

name

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

__module__ = 'swh.deposit.models'
__reduce__()
__setstate__(args)
exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

__module__ = 'swh.deposit.models'
__reduce__()
__setstate__(args)
__module__ = 'swh.deposit.models'
__str__()[source]

Return str(self).

_meta = <Options for DepositCollection>
deposit_set

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

parent.children is a ReverseManyToOneDescriptor instance.

Most of the implementation is delegated to a dynamically defined manager class built by create_forward_many_to_many_manager() defined below.

objects = <django.db.models.manager.Manager object>

swh.deposit.parsers module

Module in charge of defining parsers with SWORD 2.0 supported mediatypes.

class swh.deposit.parsers.SWHFileUploadZipParser[source]

Bases: rest_framework.parsers.FileUploadParser

File upload parser limited to zip archive.

media_type = 'application/zip'
__module__ = 'swh.deposit.parsers'
class swh.deposit.parsers.SWHFileUploadTarParser[source]

Bases: rest_framework.parsers.FileUploadParser

File upload parser limited to tarball (tar, tar.gz, tar.*) archives.

media_type = 'application/x-tar'
__module__ = 'swh.deposit.parsers'
class swh.deposit.parsers.SWHXMLParser[source]

Bases: rest_framework.parsers.BaseParser

XML parser.

media_type = 'application/xml'
parse(stream, media_type=None, parser_context=None)[source]

Parses the incoming bytestream as XML and returns the resulting data.

__module__ = 'swh.deposit.parsers'
class swh.deposit.parsers.SWHAtomEntryParser[source]

Bases: swh.deposit.parsers.SWHXMLParser

Atom entry parser limited to specific mediatype

media_type = 'application/atom+xml;type=entry'
parse(stream, media_type=None, parser_context=None)[source]

Parses the incoming bytestream as XML and returns the resulting data.

__module__ = 'swh.deposit.parsers'
class swh.deposit.parsers.SWHMultiPartParser[source]

Bases: rest_framework.parsers.MultiPartParser

Multipart parser limited to a subset of mediatypes.

media_type = 'multipart/*; *'
__module__ = 'swh.deposit.parsers'
swh.deposit.parsers.parse_xml(raw_content)[source]

Parse xml body.

Parameters:raw_content (bytes) – The content to parse
Raises:ParserError in case of a malformed xml
Returns:content parsed as dict.

swh.deposit.signals module

Module in charge of defining some uncoupled actions on deposit.

Typically, checking that the archives deposited are ok are not directly testing in the request/answer to avoid too long computations.

So this is done in the deposit_on_status_ready_for_check callback.

swh.deposit.signals.schedule_task(scheduler, task)[source]

Schedule the task and return its identifier

Parameters:task (dict) – Task to schedule
Returns:The task identifier
swh.deposit.signals.post_deposit_save(sender, instance, created, raw, using, update_fields, **kwargs)[source]
When a deposit is saved, check for the deposit’s status change and

schedule actions accordingly.

When the status passes to deposited, schedule checks. When the status pass to ready, schedule loading. Otherwise, do nothing.

Parameters:
  • sender (Deposit) – The model class
  • instance (Deposit) – The actual instance being saved
  • created (bool) – True if a new record was created
  • raw (bool) – True if the model is saved exactly as presented (i.e. when loading a fixture). One should not query/modify other records in the database as the database might not be in a consistent state yet
  • using – The database alias being used
  • update_fields – The set of fields to update as passed to Model.save(), or None if update_fields wasn’t passed to save()

swh.deposit.urls module

swhdeposit URL Configuration

swh.deposit.urls.default_view(req)[source]

swh.deposit.utils module

swh.deposit.utils.origin_url_from(deposit)[source]

Given a deposit instance, return the associated origin url.

This expects a deposit and the associated client to be correctly configured.

Parameters:

deposit (Deposit) – The deposit from which derives the origin url
Raises:
  • AssertionError if
    • the client’s provider_url field is not configured.
    • the deposit’s external_id field is not configured.
Returns
The associated origin url
swh.deposit.utils.merge(*dicts)[source]

Given an iterator of dicts, merge them losing no information.

Parameters:*dicts – arguments are all supposed to be dict to merge into one
Returns:dict merged without losing information
swh.deposit.utils.normalize_date(date)[source]

Normalize date fields as expected by swh workers.

If date is a list, elect arbitrarily the first element of that list

If date is (then) a string, parse it through dateutil.parser.parse to extract a datetime.

Then normalize it through swh.model.identifiers.normalize_timestamp.

Returns
The swh date object

Module contents