Welcome to marsha’s documentation!¶

About¶

🐠 A FUN video provider for Open edX

Hosted on Github

ADR - Architecture Decision Records¶

Purpose¶

These are the architectural decisions that are taken while developing Marsha.

The format is based on Michael Nygard’s article on the subject.

Proposed¶

Accepted¶

ADR 0001 - Actors¶

Status¶

Accepted

Context¶

There are different kinds of actors that need to interact with Marsha.

First we have the people managing a Marsha instance.

Then we have people linking their own website (this website is a “consumer site”) to a Marsha instance, to host videos.

These consumer sites can host many publishers. We call these “organizations”. And these organizations have managers that can administrate some things about authors, video sharing between authors…

Next we have the video authors, belonging to the organizations.

And finally we have the users coming to watch videos.

Decision¶

Let’s separate those 5 actors / roles:

“staff”¶

Purpose¶

To manage a Marsha instance

Implementation¶

These are simply instances of the Django User model, with the flag is_staff set to True.

“admins”¶

Purpose¶

To manage the link between a consumer site and a Marsha instance, and the organizations allowed to access this consumer site on the instance.

Implementation¶

To represent a consumer site on a Marsha instance, we have a ConsumerSite Django model. With a ManyToMany link to the User model, named admins (not a single admin, to avoid having no admin if the only one existing is not available anymore)).

“managers”¶

Purpose¶

To manage the authors in the organization (an organization could be present on many consumer sites). To allow videos to be private to authors or public for all authors. And create courses.

Implementation¶

To represent an organization on a Marsha instance, we have an Organization Django model. With a ManyToMany link to the User model, named managers.

“authors”¶

Purpose¶

To post videos on a Marsha instance to be used on a consumer site.

Implementation¶

An author is simply an instance of the User model, but has a link to an Organization via a ManyToMany link, named organizations (we can imagine an author working for many organizations).

“viewers”¶

Purpose¶

To watch videos hosted on a Marsha instance.

Implementation¶

For the viewers we don’t need to save anything in the database, so there is no instances of the User Django model for them.

Each time a user does an action to view a video, they access it via a url containing a unique token, with a limited life span. It’s this token that grant them access to the video.

This is not the scope of this document to address token generations.

To store the user preferences regarding languages, video resolution, etc, it can simply be done via a cookie.

Consequences¶

Roles and authorizations are easy to understand.

The actors hierarchy is simple.

ADR 0002 - Videos languages¶

Status¶

Accepted

Context¶

We want to think Marsha as accessible from the beginning. At least from the point of view of the videos, which are the main content available.

We can think about a video as a main content, with many auxiliary contents.

Auxiliary contents¶

Audio¶

We have a main video, with an audio track included. The author could propose many other audio tracks, as audio files, and in the player the viewer can change the one to use.

Subtitles¶

In addition to audio tracks, many subtitles tracks can be available.

Sign language¶

Some people with disabilities could want a video with the sign language transcript. For this it can be a video incorporated in the original one, or an other video displayed on the site.

As sign languages are not the same for every spoken language, there can be several sign languages videos for a single video.

Decision¶

We decided to take all these elements into account right from the beginning.

So we have a main Django model named Video, from an author, with the link to to main video file, including the default audio track.

For the other audio tracks, we have AudioTrack Django model, with a ForeignKey to the Video instance, named video, and a language field (with only one audio track for each video+language)

It’s the same for the subtitles, we have a SubtitleTrack Django model, with the same video and language fields, but with an additional cc field to indicate that this subtitle track is “closed captioning”, ie a subtitles track for deaf or hard of hearing viewers. So there can be two subtitle tracks for each video+language: one with cc on, one with cc off.

And finally, for sign-languages videos, it’s the same as for audio tracks: a Django model named SignTrack with the same video and language field.

Consequences¶

Accessibility is implemented from the start. Even if we decide to hide some things, the main concepts are here.

ADR 0003 - Content organization and accesses¶

Status¶

Accepted

Context¶

We have actors. And videos, with their auxiliary files, that we’ll call for now “content”. We want this content to be organized for actors to manage/view them.

Decision¶

Videos are grouped in playlists, which is a Django model named Playlist. A playlist belongs to an organization (Organization model defined in actors) and is created by someone who can be a manager of this organization, or an author who belongs to this organization. This link is materialized by an author field on the Playlist model, a ForeignKey to the User model.

The manager can allow many actors to manage a playlist, so there is a ManyToMany field on the Playlist model, named editors, pointing to the User model. And instead of relying on the hidden model created by Django when creating a ManyToMany, we’ll define this model and use it via the through argument, to be able to add more rights constraints later if needed.

The author of the playlist is automatically added to this list of editors. And can be removed from it by a manager, still staying marked as the author, but being the author itself doesn’t involve any rights.

A playlist can be duplicated by a manager, and if it stays in the same organization, the manager can clear or keep the list of editors. If it is to be duplicated in another organization, the list of editors will be cleared of actors not belonging to the new organization, and the manager will still be able to clear it all or keep the remaining editors.

When duplicated, a new instance of Playlist is created, with a link to the original playlist, keeping the author. We do the same for each instances of the Video linked to this playlist, but we will still point to the same files (videos/audios/subtitles…) on the hosting provider, to keep cost manageable.

And finally, there is a flag named is_public on the playlist, that can be toggled by a manager, to tell if the playlist can be viewed by anyone or only by people who were granted access to it. This kind of access is not in the scope of this document.

Consequences¶

Videos are grouped, which ease search and maintenance.

It is easy to change and check the rights for people to manage such a playlist.

ADR 0004 - Soft deletion¶

Status¶

Accepted

Context¶

We have users and objects, and everything is tied together. Deleting something may cause some problems, like deleting other things in cascade, or losing some relatioship, like not knowing who is the author of a video.

Decision¶

We don’t want things to be deleted, instead we’ll keep them in the database in a “deleted” state, so that they won’t show up anywhere.

Looking at the ` Safe/Soft/Logical deletion/trashing and restoration/undeletion <https://djangopackages.org/grids/g/deletion/>`_ page on djangopackages, we can make a choice with these constraints:

support for python 3 and django 2
simple, not over-featured
can manage relationships
supports the django admin
is maintained

Regarding this, we choose django-safedelete which proposes many options to handle deletion, and so will fit our needs.

Consequences¶

If the correct deletion options are used, depending on the relationships, we won’t lose data.

No new code to write to handle soft-deletion.

As a cons: one more dependency, but it seems maintained so it should be ok.

Deprecated¶

Superseded¶

Development¶

At the time of writing, Marsha is developed with Python 3.6 for Django 2.0.

Code quality¶

We enforce good code by using some linters and auto code formatting tools.

To run all linters at once, run the command:

make lint

You can also run each linter one by one. We have ones for the following tools:

black¶

We use black to automatically format python files to produce pep 8 compliant code.

The best is to configure your editor to automatically update the files when saved.

If you want to do this manually, run the command:

make black

And to check if all is correct without actually modifying the files:

make check-black

flake8¶

In addition to black auto-formatting, we pass the code through flake8 to check for a lot of rules. In addition to the default flake8 rules, we use these plugins:

flake8-bugbear to find likely bugs and design problems.
flake8-comprehensions to helps write better list/set/dict comprehensions.
flake8-imports to check imports order via isort.
flake8-mypy to check typing inconsistencies via mypy (see mypy).
flake8-docstrings to check docstrings (see Docstrings)

To check your code, run the command:

make flake8

pylint¶

To enforce even more rules than the ones provided by flake8, we use pylint (with the help of pylint-django).

pylint may report some violations not found by flake8, and vice-versa. More often, both will report the same ones.

To check your code, run the command:

make pylint

mypy¶

We use python typing as much as possible, and mypy (with the help of mypy-django) to check it.

We also enforce it when defining Django fields, via the use of a “Django check”. And the type of reverse related fields must always be defined.

To check if your code is valid for mypy, run the following command:

make mypy

Following is how the types of fields must be defined. To check if some fields typing is invalid, among other problems, run the following command:

make check-django

It will tell you all found errors in typing, with indication on how to correct them.

Scalar fields¶

For scalar fields (CharField, IntegerField, BooleanField, DateField…), we just add the type. For each type of Django field, there is an expected type. Theses types are defined in the fields_type_mapping dict defined in marsha.core.base_models. Add the missing ones if needed.

class Foo(models.Model):
    # we tell mypy that the attribute ``bar`` is of type ``str``
    name: str = models.CharField(...)

One-to-one fields¶

The type expected for a OneToOneField is the pointed model.

And on the pointed model we set the type of the related name to the source model.

class Foo(models.Model):
    # we tell mypy that the attribute ``bar`` is of type ``Bar``
    bar: "Bar" = models.OneToOneField(to=Bar, related_name="the_foo")

class Bar(models.Model):
    # we tell mypy that the class ``Bar`` has an attribute ``the_foo`` of type ``Foo``
    the_foo: Foo

Foreign keys¶

The type expected for a ForeignKey is the pointed model.

On the pointed model we have a many-to-one relationship. We use a type specifically defined for that, ReverseFKType, defined in marsha.stubs.

from marsha.stubs import ReverseFKType

class Foo(models.Model):
    # we tell mypy that the attribute ``bar`` is of type ``Bar``
    bar: "Bar" = models.ForeignKey(to=Bar, related_name="foos")

class Bar(models.Model):
    # we tell mypy that the class ``Bar`` has an attribute ``foos``
    # which is a reverse foreign key for the class ``Foo``
    foos: ReverseFKType[Foo]

Many-to-many fields¶

To define the type of a ManyToManyField, we use a type specifically defined for that, M2MType, defined in marsha.stubs.

On the pointed model, we use the same type, as it’s also a many-to-many fields (ie it could have been defined in one model or the other).

from marsha.stubs import M2MType

class Foo(models.Model):
    # we tell mypy that the attribute ``bar`` is a many-to-many for the class ``Bar``
    bars: M2MType["Bar"] = models.ManyToManyField(to=Bar, related_name="foos")

class Bar(models.Model):
    # we tell mypy that the class ``Bar`` has an attribute ``foos``
    # which is a many-to-many for the class ``Foo``
    foos: M2MType[Foo]

Docstrings¶

flake8 is configured to enforce docstrings rule defined in the pep 257

In addition, we document function arguments, return types, etc… using the “NumPy” style documentation, which will be validated by flake8.

Django¶

Opinionated choices¶

We made the opinionated choice of following this document, “Tips for Building High-Quality Django Apps at Scale”.

In particular:

Do not split code in many Django applications if code is tightly coupled.
Applications are inside the marsha package, not at root, so import are done like this:

from marsha.someapp.foo import bar

Database tables are specifically named: we do not rely on the Django auto-generation. And then we don’t prefix theses tables with the name of the project or the app. For example, a model named Video, will have the db_table attribute of its Meta class set to video. Enforced by a “Django check”.
Through tables for ManyToManyField relations must be defined. Enforced by a “Django check”.

In addition:

We enforce typing of fields and reverse related fields (see mypy). Enforced by a “Django check”.
We enforce defining a related name for all related field (ManyToManyField, ForeignKey, OneToOneField). Enforced by a “Django check”.

To check if theses rules are correctly applied, among other rules defined by Django itself, run:

make check-django

Note

for these checks to work, all models must inherit from BaseModel defined in marsha.core.base_models.

Specific libraries¶

Here are a list of specific Django libraries we chose to use and why.

django-configurations¶

The aim is to be more specific about inheritance in settings from doc to staging to production, instead of relying on multiple files (and changing the DJANGO_SETTINGS_MODULE environment variable accordingly), using the from .base import * pattern.

It also provides tools to get some variables from the environment and validating them.

As a consequence of this tool, some default behavior of Django don’t work anymore. It’s why the django-admin bash command is redefined in setup.cfg.

django-safedelete¶

We don’t want to lose data, so everything is kept in database, but hidden from users.

See ADR 0004 - Soft deletion for details about the reasoning behind this choice.

django-postgres-extra¶

With django-safedelete, model instances are not deleted but saved with a field deleted changing from None to the deletion date-time.

So we cannot anymore use unique_together.

django-postgres-extra provides a ConditionalUniqueIndex index, that acts like unique_together, but with a condition. We use the condition WHERE "deleted" IS NULL, to enforce the fact that only one non-deleted instance matching the fields combination can exist.

Tests¶

The whole Marsha project is tested.

Run this command to run all the tests:

make test

If you want to be more specific about the tests to run, use the Django command:

django-admin test marcha.path.to.module
django-admin test marcha.path.to.module.Class
django-admin test marcha.path.to.module.Class.method

Makefile¶

We provide a Makefile that allow to easily perform some actions.

make install: Will install the project in the current environment, with its dependencies.
make dev: Will install the project in the current environment, with its dependencies, including the ones needed in a development environment.
make dev-upgrade: Will upgrade all default+dev dependencies defined in setup.cfg.
make check: Will run all linters and checking tools.
make lint: Will run all linters (mypy, black, flake8, pylint)
make mypy: Will run the mypy tool.
make check-black: Will run the black tool in check mode only (won’t modify files)
make black: Will run the black tool and update files that need to.
make flake8: Will run the flake8 tool.
make pylint: Will run the pylint tool.
make check-django: Will run the Django check command.
make check-migrations: Will check that all needed migrations exist.
make tests: Will run django tests for the marsha project.
make doc: Will build the documentation.
make dist: Will build the package.
make clean: Will clean python build related directories and files.
make full-clean: Like make clean but will clean some other generated directories or files.

Environment variables¶

We try to follow 12 factors app and so use environment variables for configuration.

Here is a list of the ones that are needed or optional:

DJANGO_SETTINGS_MODULE¶

Description: Define the settings file to use
Type: String
Mandatory: Yes
Default: None
Choices: Must be set to marsha.settings

DJANGO_CONFIGURATION¶

Description: Define the configuration to use in settings
Type: String
Mandatory: Yes
Default: None
Choices: Actually only Dev is available

DJANGO_SECRET_KEY¶

Description: Used to provide cryptographic signing, and should be set to a unique, unpredictable value
Type: String
Mandatory: Yes
Default: None

DJANGO_DEBUG¶

Description: Turns on/off debug mode
Type: Boolean
Mandatory: No
Default: True if DJANGO_CONFIGURATION is set to Dev, False otherwise
Choices: True or False

DATABASE_URL¶

Description: URL to represent the connection string to a database
Type: String
Mandatory: No if DJANGO_CONFIGURATION is set to Dev, yes otherwise
Default: sqlite:///path/to/project/db.sqlite3 if DJANGO_CONFIGURATION is set to Dev, None otherwise
Choices: See schemas as presented by dj-database-url

marsha¶

marsha package¶

Subpackages¶

marsha.core package¶

Subpackages¶

marsha.core.migrations package¶

Submodules¶

marsha.core.migrations.0001_initial_models module¶

class marsha.core.migrations.0001_initial_models.Migration(name, app_label)[source]¶