Welcome to Amara’s documentation!¶
Contents:
Running Amara¶
Check out the Quick Start on our github page (http://github.com/pculture/unisubs/).
Reporting bugs¶
When you are reporting a bug, please look over the following suggestions. The more information you can provide, the faster the bug can be fixed. And you will life easier for developers.
- In what environment is the bug happening?
API Documentation¶
This is the documentation of v2 of Amara’s API. Please contact us if you’d like to use the Amara API for commercial purposes.
Note
The v1 of the API is deprecated, but can still be accessed through http://www.amara.org/api/1.0/documentation/ . Users should migrate to the v2 of the API. If you’re missing a feature on the API, please let us know .
Authentication¶
Before interacting with the API, you must have an API key. In order to get one, create a user on the Amara website, then go to the edit profile page. At the bottom of the page you will find a “Generate new key” button . Clicking on it will fetch your user the needed API key.
Every request must have the username and the API keys as headers. For example:
X-api-username: my_username_here
X-apikey: my_api_key_here
So a sample request would look like this:
$ curl -H 'X-api-username: my_username_here' -H 'X-apikey: my_api_key_here' \
https://staging.amara.org/api2/partners/videos/
Data Formats¶
The API accepts request data and will output the following formats: JSON, XML and YAML. Unless you have a strong reason not to, we recommend using the JSON format, as it’s the one that gets the most usage (and therefore more testing).
To specify the format, add the Accept header appropriately, for example:
Accept: application/json
You can also specify the desired format in the url, sending the request variable format=json.
API endpoint¶
The endpoint for the API is the environment base URL + /api2/partners/.
Possible environments:
- Staging: https://staging.amara.org/
- Production: https://www.amara.org/
Therefore, most clients should be making requests against: https://www.amara.org/api2/partners/
All API requests should go through https. The staging environment might need HTTP basic auth, please contact us to request credentials. When basic auth is needed on staging, you end up with a request like this:
$ curl -H 'X-api-username: my_username_here' -H 'X-apikey: my_api_key_here' \
--user basic_auth_username:basic_auth_password \
https://staging.amara.org/api2/partners/videos/
If you’re under a partnership, you might have a different base URL. Please contact us if you’re not sure.
API interaction overview¶
All resources share a common structure when it comes to the basic data operations.
- GET request is used to viewing data
- POST request is used for creating new items
- PUT request is used for updating existing items
- DELETE request is used for deleting existing items
For example, in order to request a list of teams the user is current on, you would issue the following request:
- GET /api2/partners/teams/¶
To view a detail of the test team, you could do:
- GET /api2/partners/teams/test/¶
Example response
{
"created": "2012-04-18T09:26:59",
"deleted": false,
"description": "",
"header_html_text": "",
"is_moderated": false,
"is_visible": true,
"logo": null,
"max_tasks_per_member": null,
"membership_policy": "Open",
"name": "test",
"projects_enabled": false,
"resource_uri": "/api2/partners/teams/test/",
"slug": "test",
"subtitle_policy": "Anyone",
"task_assign_policy": "Any team member",
"task_expiration": null,
"translate_policy": "Anyone",
"video_policy": "Any team member",
"workflow_enabled": false
}
Many of the available resources will allow you to filter the response by a certain field. Filters are specified as GET parameters on the request. For example, if you wanted to view all videos belong to a team called “butterfly-club”, you could do:
- GET /api2/partners/videos?team=butterfly-club¶
In addition to filters, you can request that the response is ordered in some way. To order videos by title, you would do
- GET /api2/partners/videos?order_by=title¶
Each resource section will contain a list of relevant options.
Here is an example of creating a new team via curl.
curl -i -X POST -H "Accept: application/json" \
-H 'X-api-username: my_username_here' -H 'X-apikey: my_api_key_here' \
-H "Content-Type: application/json" \
--data '{"name": "Team name", "slug": "team-name"}' \
http://host/api2/partners/teams/
You can use the same fields that you get back when requesting a team detail.
To update a team, you could issue a request like this:
curl -i -X PUT -H "Accept: application/json" \
-H 'X-api-username: my_username_here' -H 'X-apikey: my_api_key_here' \
-H "Content-Type: application/json" \
--data '{"name": "My team name"}' \
https://host/api2/partners/teams/test/
Warning
The above example only includes the name field for illustration. When sending a PUT request, always include all fields. For a list of all fields, see the response to a GET request.
Partner video ids¶
If you are a partner, you can set the id field for a video. Simply supply the usePartnerId parameter in your request and we will use your id for look ups. The parameter can be sent as a parameter to any kind of API call. This is useful if you already have a database of video ids and don’t want to maintain a mapping between those ids and Amara ids.
For example, let’s say you have an Amara video with the id of yxsSV807Dcho. Your application uses numeric id internally and you would like to tell Amara to remember that this video has an id of 12345 on your system. You can modify the video like this:
curl -i -X PUT -H "Accept: application/json" \
-H 'X-api-username: my_username_here' -H 'X-apikey: my_api_key_here' \
-H "Content-Type: application/json" \
--data '{"usePartnerId": true, "id": "12345"}' \
https://host/api2/partners/videos/yxsSV807Dcho/
And then, you can start referencing the video by the numeric id when interacting with the API. For example, the following call will retrieve the above video.
curl -i -X GET -H "Accept: application/json" \
-H 'X-api-username: my_username_here' -H 'X-apikey: my_api_key_here' \
-H "Content-Type: application/json" \
https://host/api2/partners/videos/12345/?usePartnerId=true
Available Resources¶
The following resources are available to end users:
Video Resource¶
Represents a video on Amara.
Listing videos
- GET /api2/partners/videos/¶
Query Parameters: - video_url – list only videos with the given URL, useful for finding out information about a video already on Amara.
- team – Only show videos that belong to a team identified by slug.
- project – Only show videos that belong to a project with the given slug. Passing in null will return only videos that don’t belong to a project.
- order_by –
Applies sorting to the video list. Possible values:
- title: ascending
- -title: descending
- created: older videos first
- -created : newer videos
Creating Videos:
- POST /api2/partners/videos/¶
Form Parameters: - video_url – The url for the video. Any url that Amara accepts will work here. You can send the URL for a file (e.g. http:///www.example.com/my-video.ogv) , or a link to one of our accepted providers (youtube, vimeo, dailymotion, blip.tv)
- title – The title for the video :form description: About this video
- duration – Duration in seconds
- primary_audio_language_code – The language code representing main language spoken on the video. This helps the UI to show the best title for that video, or set “Subtitle” taks in the right language from the get-go - optional.
When submitting URLs of external providers (i.e. youtube, vimeo), the metadata (title, description, duration) can be fetched from them. If you’re submitting a link to a file (mp4, flv) then you can make sure those attributes are set with these parameters. Note that these parameters do override any information from the original provider.
Information about a specific video can be retrieved from the URL:
Video Detail:
- GET /api2/partners/videos/[video-id]/¶
The video listing resource already returns a resource_uri for each video to be used when retrieving the details.
Updating a video object:
- PUT /api2/partners/videos/[video-id]/¶
With the same parameters for creation. Note that through out our system, a video cannot have it’s URLs changed. So you can change other video attributes (title, description) but the URL sent must be the same original one.
Moving videos between teams and projects¶
In order to move a video from one team to another, you can make a request to change the video where you change the team value in the Video Resource.
In order to move the video from Team A to Team B, you would make the following request.
curl -i -X PUT -H "Accept: application/json" \
-H 'X-api-username: my_username_here' -H 'X-apikey: my_api_key_here' \
-H "Content-Type: application/json" \
--data '{"team": "team_b"}' \
https://host/api2/partners/videos/video-id/
Please note that the value that is sent as the team is the team’s slug. The user making the change must have permission to remove a video from the originating team and permission to add a video to the target team.
Setting the team value to null will remove it from its current team.
A similar mechanism can be used to change what project a given video is filed under. The important difference is that when moving a video to different project, the team must be specified in the payload even if it doesn’t change.
{
"team:" "team-slug",
"project": "new-project"
}
Example response:
{
"all_urls": [
"http://vimeo.com/4951380"
],
"created": "2012-05-15T06:05:14",
"description": "Concierto Grupo NOMOI \n(Torrevieja 17/05/2009)\nProyecto TRANSMOSFERA\nAcci\u00f3n interactiva de m\u00fasica, teatro e imagen.\nJuan Pablo Zaragoza - V\u00eddeo y Guitarra sintetizada\nJos\u00e9 Mar\u00eda Pastor - Electr\u00f3nica\nRaul Ferrandez - Voz y acci\u00f3n teatral",
"duration": null,
"id": "PUuHIcJ5mq5S",
"languages": [],
"original_language": null,
"project": null,
"resource_uri": "/api2/partners/videos/PUuHIcJ5mq5S/",
"site_url": "http://unisubs.example.com:8000/videos/PUuHIcJ5mq5S/info/",
"team": null,
"thumbnail": "http://b.vimeocdn.com/ts/142/595/14259507_640.jpg",
"title": "Concierto NOMOI (Torrevieja 17/05/2009)"
}
Video Language Resource¶
Represents a language for a given video on Amara.
Listing video languages:
- GET /api2/partners/videos/[video-id]/languages/¶
Creating Video Languages:
- POST /api2/partners/videos/[video-id]/languages/¶
Form Parameters: - language_code – The language code (e.g ‘en’ or ‘pt-br’) to create.
- title – The title for the video localized to this language - optional
- description – Localized description for this language - optional.
- is_original – Boolean indicating if this is the original language for the video. - optional - defaults to false.
- is_original – If set to true, will mark this language as the primary audio language for the video ( see VideoResource) - optional, defaults to false.
See also
To list available languages, see Language Resource.
Information about a specific video language can be retrieved from the URL:
- GET /api2/partners/videos/[video-id]/languages/[lang-identifier]/¶
Parameters: - lang-identifier – language identifier can be the language code (e.g. en) or the numeric ID returned from calls to listing languages.
Example response:
{
"completion": "100%",
"created": "2012-05-17T12:25:54",
"description": "",
"id": "8",
"is_original": false,
"is_translation": false,
"language_code": "cs",
"num_versions": 1,
"original_language_code": "en",
"percent_done": 0,
"resource_uri": "/api2/partners/videos/Myn4j5OI7BxL/languages/8/",
"site_url": "http://unisubs.example.com:8000/videos/Myn4j5OI7BxL/cs/8/",
"subtitle_count": 11,
"title": "\"Postcard From 1952\" - Explosions in The Sky",
"versions": [
{
"author": "honza",
"status": "published",
"text_change": "1.0",
"time_change": "1.0",
"version_no": 0
}
]
}
Subtitles Resource¶
Represents the subtitle set for a given video language.
Fetching subtitles for a given language:
- GET /api2/partners/videos/[video-id]/languages/[lang-identifier]/subtitles/?format=srt¶
- GET /api2/partners/videos/asfssd/languages/en/subtitles/?format=dfxp¶
- GET /api2/partners/videos/asfssd/languages/111111/subtitles/?format=ssa¶
Query Parameters: - format – The format to return the subtitles in. Supports all the formats the regular website does: srt, ssa, txt, dfxp, ttml.
- version – the numeric version number to fetch. Versions are listed in the VideoLanguageResouce request.
If no version is specified, the latest public version will be returned. For videos that are not under moderation it will be the latest one. For videos under moderation only the latest published version is returned. If no version has been accepted in review, no subtitles will be returned.
Creating new subtitles for a language:
- POST /api2/partners/videos/[video-id]/languages/[lang-identifier]/subtitles/¶
- POST /api2/partners/videos/asfssd/languages/en/subtitles/¶
Query Parameters: - subtitles – The subtitles to submit
- sub_format – The format used to parse the subs. The same formats as for fetching subtitles are accepted. Optional - defaults to srt.
- title – Give a title to the new revision
- description – Give a description to the new revision
Form Parameters: - is_complete – Boolean indicating if the complete subtitling set is available for this language - optional, defaults to false.
This will create a new subtitle version with the new subtitles.
Example response:
- GET /api2/partners/videos/TRUFD3IyncAt/languages/en/subtitles/¶
{
"description": "Centipede - Knife Party www.knifeparty.com\nFireworks - Pyro Spectaculars by Souza www.pyrospectaculars.com/\n\n( Sittin' On ) The Dock of the Bay - Otis Redding\nLights - Journey\nFrisco Blues - John Lee Hooker\nSan Francisco ( Be Sure to Wear Flowers in Your Hair ) - Scott McKenzie \nI Left My Heart in San Francisco - Tony Bennett\n\nIf you didn't understand what was happening, you should probably watch it again.\nThis has been a Seventh Movement effort.",
"note": "",
"resource_uri": "",
"site_url": "http://example-host/api2/partners/videos/TRUFD3IyncAt/en/1/",
"sub_format": "srt",
"subtitles": [
{
"end": 4,
"id": 1,
"start": 3,
"start_of_paragraph": false,
"text": "This is a cool bridge"
},
{
"end": 5,
"id": 2,
"start": 4,
"start_of_paragraph": false,
"text": "Really cool"
},
{
"end": 6,
"id": 3,
"start": 5,
"start_of_paragraph": false,
"text": "I love it"
}
],
"title": "The Golden Gate Way",
"version_no": 0,
"video": "The Golden Gate Way",
"video_description": "Centipede - Knife Party www.knifeparty.com\nFireworks - Pyro Spectaculars by Souza www.pyrospectaculars.com/\n\n( Sittin' On ) The Dock of the Bay - Otis Redding\nLights - Journey\nFrisco Blues - John Lee Hooker\nSan Francisco ( Be Sure to Wear Flowers in Your Hair ) - Scott McKenzie \nI Left My Heart in San Francisco - Tony Bennett\n\nIf you didn't understand what was happening, you should probably watch it again.\nThis has been a Seventh Movement effort.",
"video_title": "The Golden Gate Way"
}
Language Resource¶
Represents a listing of all available languages on the Amara platform.
Listing available languages:
- GET /api2/partners/languages/¶
User Resource¶
One can list and create new users through the API.
Listing users:
- GET /api2/partners/users/¶
User datail:
- GET /api2/partners/users/[username]/¶
Creating Users:
- POST /api2/partners/users/¶
Form Parameters: - username – the username for later login. 30 chars or fewer alphanumeric chars, @, _ and - are accepted.
- email – A valid email address
- password – any number of chars, all chars allowed.
- first_name – Any chars, max 30 chars. Optional.
- last_name – Any chars, max 30 chars. Optional.
- create_login_token – If sent the response will also include a url that when clicked will login the recently created user. This URL expires in 2 hours
The response also includes the ‘api_key’ for that user. If clients wish to make requests on behalf of this newly created user through the api, they must hold on to this key, since it won’t be returned in the detailed view.
Example response:
{
"avatar": "http://www.gravatar.com/avatar/947b2f9a76cd39f5c7b7c8ad3a36?s=100&d=mm",
"biography": "The guy with a boring name.",
"first_name": "John",
"full_name": "John Smith",
"homepage": "http://example.com",
"last_name": "Smith",
"num_videos": 8,
"resource_uri": "/api2/partners/users/jsmith/",
"username": "jsmith"
}
Video Url Resource¶
One can list, update, delete and add new video urls to an existing video.
Listing video urls
- GET /api2/partners/videos/[video-id]/urls/¶
Video URL detail:
- GET /api2/partners/videos/[video-id]/urls/[url-id]/¶
Where the url-id can be fetched from the list of urls.
Updating video-urls:
- PUT /api2/partners/videos/[video-id]/urls/[url-id]/¶
Creating video-urls:
- POST /api2/partners/videos/[video-id]/urls/¶
Form Parameters: - url – Any URL that works for the regular site (mp4 files, youtube, vimeo, etc) can be used. Note that the url cannot be in use by another video.
- primary – A boolean. If true this is the url the will be displayed first if multiple are presents. A video must have one primary URL. If you add / change the primary status of a url, all other urls for that video will have primary set to false. If this is the only url present it will always be set to true.
- original – If this is the first url for the video.
To delete a url:
- DELETE /api2/partners/videos/[video-id]/urls/[url-id]/¶
If this is the only URL for a video, the request will fail. A video must have at least one URL.
Team Resource¶
You can list existing teams:
- GET /api2/partners/teams/¶
You can view details for an existing team:
- GET /api2/partners/teams/[team-slug]/¶
Creating a team:
- POST /api2/partners/teams/¶
Form Parameters: - name – (required) Name of the team
- slug – (required) A unique slug (used in URLs)
- description –
- is_visible – Should this team be publicly visible?
- membership_policy – See below for possible values
- video_policy – See below for possible values
- task_assign_policy – See below for possible values
- max_tasks_per_member – Maximum tasks per member
- task_expiration – Task expiration in days
Example payload:
{
"name": "Full Team",
"slug": "full-team",
"description": "One full team",
"is_visible": false,
"membership_policy": "Invitation by any team member",
"video_policy": "Admins only",
"task_assign_policy": "Managers and admins",
"max_tasks_per_member": 3,
"task_expiration": 14
}
Updating a team:
- PUT /api2/partners/teams/[team-slug]/¶
Deleting a team:
- DELETE /api2/partners/teams/[team-slug]/¶
Note
You can only create new teams if you have been granted this privilege. Contact us if you require a partner account.
Policy values¶
Membership policy:
- Open
- Application
- Invitation by any team member
- Invitation by manager
- Invitation by admin
Video policy:
- Any team member
- Managers and admins
- Admins only
Task assign policy:
- Any team member
- Managers and admins
- Admins only
Example response
{
"created": "2012-04-18T09:26:59",
"deleted": false,
"description": "",
"header_html_text": "",
"is_moderated": false,
"is_visible": true,
"logo": null,
"max_tasks_per_member": null,
"membership_policy": "Open",
"name": "test",
"projects_enabled": false,
"resource_uri": "/api2/partners/teams/test/",
"slug": "test",
"subtitle_policy": "Anyone",
"task_assign_policy": "Any team member",
"task_expiration": null,
"translate_policy": "Anyone",
"video_policy": "Any team member",
"workflow_enabled": false
}
Team Member Resource¶
This resource allows you to change team membership information without the target user’s input. This resource is only applicable to:
- Teams associated with the partner’s account
- Users who are already members of one of the partner’s teams
You can list existing members of a team:
- GET /api2/partners/teams/[team-slug]/members/¶
Adding a new member to a team:
- POST /api2/partners/teams/[team-slug]/members/¶
Updating a team member (e.g. changing their role):
- PUT /api2/partners/teams/[team-slug]/members/[username]/¶
Removing a user from a team:
- DELETE /api2/partners/teams/[team-slug]/members/[username]/¶
Example of adding a new user:
{
"username": "test-user",
"role": "manager"
}
Roles¶
- owner
- admin
- manager
- contributor
Warning
Changed behavior: the previous functionality was moved the Safe Team Member Resource documented below.
Permissions¶
If a user belongs to a partner team, any admin or above on any of the partner’s teams can move the user anywhere within the partner’s teams. Moving is done by first adding the user to the target team and then by removing the user from the originating team.
Safe Team Member Resource¶
This resource behaves the same as the normal Team Member resource with one small difference. When you add a user to a team, we will send an invitation to the user to join the team. If the user doesn’t exist, we will create it. The standard Team Member resource simply adds the user to the team and returns.
Listing:
- GET /api2/partners/teams/[team-slug]/safe-members/¶
Adding a new member to a team:
- POST /api2/partners/teams/[team-slug]/safe-members/¶
Project Resource¶
List all projects for a given team:
- GET /api2/partners/teams/[team-slug]/projects/¶
Project detail:
- GET /api2/partners/teams/[team-slug]/projects/[project-slug]/¶
Create a new project:
- POST /api2/partners/teams/[team-slug]/projects/¶
Example payload for creating a new project:
{
"name": "Project name",
"slug": "project-slug",
"description": "This is an example project.",
"guidelines": "Only post family-friendly videos."
}
Note
You can only create projects for a specific team.
Update an existing project:
- PUT /api2/partners/teams/[team-slug]/projects/[project-slug]/¶
For example, to change the project’s name:
{
"name": "Project"
}
Delete a project:
- DELETE /api2/partners/teams/[team-slug]/projects/[project-slug]/¶
Task Resource¶
List all tasks for a given team:
- GET /api2/partners/teams/[team-slug]/tasks/¶
Query Parameters: - assignee – Show only tasks assigned to a user identified by their username.
- priority – Show only tasks with a given priority
- type – Show only tasks of a given type
- video_id – Show only tasks that pertain to a given video
- order_by –
Apply sorting to the task list. Possible values:
- created Creation date
- -created Creation date (descending)
- priority Priority
- -priority Priority (descending)
- type Task type (details below)
- -type Task type (descending)
- completed – Show only complete tasks
- completed-before – Show only tasks completed before a given date (unix timestamp)
- completed-after – Show only tasks completed before a given date (unix timestamp)
- open – Show only incomplete tasks
Task detail:
- GET /api2/partners/teams/[team-slug]/tasks/[task-id]/¶
Create a new task:
- POST /api2/partners/teams/[team-slug]/tasks/¶
Update an existing task:
- PUT /api2/partners/teams/[team-slug]/tasks/[task-id]/¶
Delete an existing task:
- DELETE /api2/partners/teams/[team-slug]/tasks/[task-id]/¶
Fields¶
approved - If the team supports workflows, you can set the stage in which the task finds itself.
- In Progress
- Approved
- Rejected
assignee - The username of the user that this task will be assigned to
language
priority - An arbitrary integer denoting priority level; each team can set their own policy regarging priority of tasks
video_id - The unique identifier of the video this task relates to
type - Type of the task
- Subtitle
- Translate
- Review
- Approve
version_no - Subtitle version number (required for Approve and Review tasks)
completed - null if the task hasn’t been completed yet; a datetime string it has
An example response:
{
"approved": null,
"assignee": "johnsmith",
"language": "en",
"priority": 1,
"resource_uri": "/api2/partners/teams/all-star/tasks/3/",
"type": "Subtitle",
"video_id": "Myn4j5OI7BxL",
"completed": "2012-07-18T14:08:07"
}
Activity resource¶
This resource is read-only.
List activity items:
- GET /api2/partners/activity/¶
Query Parameters: - team – Show only items related to a given team (team slug).
- team-activity – If team is given, we normally return activity on the team’s videos. If you want to see activity for the team itself (members joining/leaving and team video deletions, then add team-activity=1)
- video – Show only items related to a given video (video id)
- type – Show only items with a given activity type (int, see below)
- language – Show only items with a given language (language code)
- before – A unix timestamp in seconds
- after – A unix timestamp in seconds
Activity types:
- Add video
- Change title
- Comment
- Add version
- Add video URL
- Add translation
- Subtitle request
- Approve version
- Member joined
- Reject version
- Member left
- Review version
- Accept version
- Decline version
- Delete video
Activity item detail:
- GET /api2/partners/activity/[activity-id]/¶
Example response:
{
"type": 1,
"comment": null,
"created": "2012-07-12T07:02:19",
"id": "1339",
"language": "en",
"new_video_title": "",
"resource_uri": "/api2/partners/activity/1339/",
"user": "test-user"
}
Message Resource¶
The message resource allows you to send messages to user and teams.
- POST /api2/partners/message/¶
Form Parameters: - subject – Subject of the message
- content – Content of the message
- user – Recipient’s username
- team – Team’s slug
You can only send the user parameter or the team parameter at once.
Application resource¶
For teams with membership by application only.
List application items:
- GET /api2/partners/teams/[team-slug]/applications¶
Query Parameters: - status – What status the application is at, possible values are ‘Denied’, ‘Approved’, ‘Pending’, ‘Member Removed’ and ‘Member Left’
- before – A unix timestamp in seconds
- after – A unix timestamp in seconds
- user – The username applying for the team
Application item detail:
- GET /api2/partners/teams/[team-slug]/applications/[application-id]/¶
Example response:
{
"created": "2012-08-09T17:48:48",
"id": "12",
"modified": null,
"note": "",
"resource_uri": "/api2/partners/teams/test-team/applications/12/",
"status": "Pending",
"user": "youtube-anonymous"
}
To delete an Application:
- DELETE /api2/partners/teams/[team-slug]/applications/[application-id]/¶
Applications can have their statuses updated:
- PUT /api2/partners/teams/[team-slug]/applications/[application-id]/¶
Query Parameters: - status – What status the application is at, possible values are ‘Denied’, ‘Approved’, ‘Pending’, ‘Member Removed’ and ‘Member Left’
Note that if an application is pending (has the status=’Pending’), the API can set it to whatever new status. Else, if the application has already been approved or denied, you won’t be able to set the new status. For cases were an approval was wrongly issues, you’d want to remove the team member. Otherwise you’d want to invite the user to the team.
Supported languages¶
Abkhazian, Afar, Afrikaans, Akan, Albanian, American, Amharic, Arabic, Aragonese, Armenian, Assamese, Asturian, Avaric, Avestan, Aymara, Azerbaijani, Bambara, Bashkir, Basque, Belarusian, Bengali, Berber, Bihari, Bislama, Bosnian, Breton, Bulgarian, Burmese, Catalan, Cebuano, Chamorro, Chechen, Chewa, Chinese, Choctaw, Church, Chuvash, Cornish, Corsican, Cree, Creole, Croatian, Czech, Danish, Divehi, Dutch, Dzongkha, Efik, English, Esperanto, Estonian, Ewe, Faroese, Fijian, Filipino, Finnish, French, Frisian, Fula, Fulah, Galician, Ganda, Georgian, German, Gikuyu, Greek, Greenlandic, Guaran, Gujarati, Haida, Hausa, Hebrew, Herero, Hindi, Hiri, Hokkien, Hungarian, Hupa, Ibibio, Icelandic, Ido, Igbo, Ilocano, Indonesian, Ingush, Interlingua, Interlingue, Inuktitut, Inupia, Irish, Iroquoian, Italian, Japanese, Javanese, Kannada, Kanuri, Karen, Kashmiri, Kazakh, Khmer, Klingon, Komi, Kongo, Korean, Kuanyama, Kurdish, Kyrgyz, Lakota, Lao, Latin, Latvian, Limburgish, Lingala, Lithuanian, Luba-Kasai, Luba-Katagana, Luhya, Luo, Luxembourgish, Macedo, Macedonian, Madurese, Malagasy, Malay, Malayalam, Maltese, Mandinka, Manipuri, Manx, Maori, Marathi, Marshallese, Metadata:, Mohawk, Moldavian, Mongolian, Mossi, Naurunan, Navajo, Ndonga, Nepali, North, Northern, Norwegian, Occitan, Ojibwe, Oriya, Oromo, Ossetian, Pali, Pashto, Persian, Polish, Portuguese, Punjabi, Quechua, Romanian, Romansh, Rundi, Russian, Rusyn, Rwandi, Samoan, Sango, Sanskrit, Sardinian, Scottish, Serbian, Serbo-Croatian, Shona, Sichuan, Sindhi, Sinhala, Slovak, Slovenian, Somali, Sotho, Southern, Spanish, Sundanese, Swahili, Swati, Swedish, Tagalog, Tahitian, Tajik, Tamil, Tartar, Telugu, Tetum, Thai, Tibetan, Tigrinya, Tonga, Tsonga, Tswana, Turkish, Turkmen, Twi, Ukrainian, Umbundu, Urdu, Uyghur, Uzbek, Venda, Vietnamese, Volapuk, Walloon, Welsh, Wolof, Xhosa, Yiddish, Yoruba, Zhuang, Zulu.
HTTP Callbacks for Teams¶
Enterprise customers can register an http callback so that any activity on their teams will fire an HTTP request.
Customers can then decide on how to act upon those notifications, for example querying through the API to fetch a new set of subtitles.
To register your Team to receive HTTP notfications get in contact with us, this process is done manually, there is no UI for this over the website at the moment.
Pick one URL where you’d like to get notified. Each team can have their own URL, or a URL can be used amongst serveral teams (for example in a public / private team setup)
Optionally you can use basic HTTP auth over that URL. We recomend the URL uses https for safer communication (even though no passwords or sensitive data will ever be sent).
A HTTP POST request will be sent to the desired URL. Bellow a list of available signals and the data passed to them.
Available Data¶
When a POST request is made to the chosen URL, the following data will be sent:
- event : A string, available events are:
- video-new : A new video has been added through the team (through the web ui)
- video-edited : Video data has been edited (video url, title, description)
- language-new : A new language has been added to the video. Either through the web UI (uploads or dialog) or through automatic transcription services.
- language-edit : An existing language has been edited (title, description). Either through the web UI (uploads or dialog) or through automatic transcription services
- subs-new : A new subtitle version has been created. Either through the web UI (uploads or dialog) or through automatic transcription services
- subs-approved : Subtitles under moderation have been approved.
- subs-rejected : Subtitles under moderation have been rejected.
- team: The slug for that team. The slug is a unique identifier that can be seen at the team’s public url page.
- project: The slug for that project. The slug is a unique identifier that can be seen at the project’s video listing page.
- video_id: The video id used in Amara to identify that video.
- api_url: The URL for the Amara API that will have the latest data for that event (subtitles, language or videos)
- language_code : The human readable language code (i.e ‘en’ or ‘es’) for the this event. If the event is a language or subtitle event the language will be sent, else if it’s a video event the parameter will be omitted.
Example¶
A notification looks like: http://<NOTIFICATION_URL>?project=_root&api_url=%2Fapi2%2Fpartners%2Fvideos%2F<VIDEO_ID>%2Flanguages%2Fen%2F&team=<TEAM_SLUG>&language_code=en&video_id=<VIDEO_ID>&event=language-edit&language_id=682965
Babelsubs¶
We’ve split the subtitle handling into it’s own separate project, Babelsubs. Anything that has do to with parsing, generating and formatting subtitles should be handled over there. The main unisubs repo should only make calls to babelsubs with the desired operations / data.
Storage¶
Internally, we’re storing subtitles as the DFXP format. DFXP is the most complex, and most capable format of all. It’s also the only one with a real spec. The advantage is that it lets us tell our users that they can input DFXP, process it throughout our system and get their data out correctly, even for features we don’t currently support (like advanced styles).
Formatting¶
Formatting we do support:
- Bold text
- Italic text
- Underline
- Like breaks
Each format handles those different. On DFXP you have attributes on the xmlnodes (span, p and div) such as fontWeigh=’bold’ and textStyle=’italic’. Line breaks are <br/> tags.
For SRT and friends, we have the ‘b’, ‘i’ and ‘u’ tags. Line breaks are displayed with the right line separator.
For HTML (which is not a download format, but it’s displayed on the website), we have ‘em’, ‘strong’ and ‘style’ tags, and ‘br’ for line breaks.
Ideally, for testing a complete set of features we need to test:
- The forementioned formats (italics, bold, underline)
- Line breaks
- Single “>” and doubles “>>” . This is used to denote speaker changes and is widely used by our customers. They must come out correctly both when displayed on the website (subtitle view, the widget, the dialogs) and when downloaded. On DFXP those should use character entities.
For anything other than these tags, let’s say you have a video on web development, and they write a ‘<script>alert();</script> ‘ tag. Here’s what should happen:
- Should be stored with the tag chars escaped
- Should show up on the website (dialog, subtitle view and the widget) as is, but escaped (javascript shouldn’t run) , but it should be editable
- Non html / xml formats (such as srt) should display them as is
In general, here’s the intended workflow:
- On intake convert what we can to dfxp (such as a line break to <br/>). Do not strip tags.
- On output (for the website only) escape anything other than the tags we expect (<script>, etc)
Syncing and Importing¶
The externalsites app handles linking Amara users/teams to accounts on externalsites. This allows for:
- Syncing subtitles to the third party site when they are edited on Amara
- Importing new videos for the third party account
We support several sites, each works slightly differently
Youtube¶
Both user and team accounts can be linked to YouTube accounts, but they are handled slightly differently. The general idea here is that the use case is different for teams and users. In general, teams want to have finer grained control over what gets imported to Amara and what gets synced back to their YouTube channel. For users, we just import everything and sync everything.
User Accounts¶
- Users can link to YouTube from account section on their profile page
- A user can only link 1 YouTube account
- A YouTube account can only be linked to 1 user
- We create a video feed and import all videos for the YouTube channel.
- All subtitles for a video in that account will be synced
Team Accounts¶
- Teams can link to YouTube from their Settings -> Integrations page
- A team can link multiple YouTube accounts
- A YouTube account can only be linked to 1 team, but there is a way to share the account with other teams.
- Subtitles are normally only synced for the team’s videos
- The linked team can add other teams to the syncing list, any of those team’s videos will also be synced.
- We don’t auto-import videos for the YouTube channel.
- A YouTube account can’t be linked to both a team and a user
Kaltura¶
- Teams can link to Kaltura from their Settings -> Integrations page
- Once a team links to Kaltura, subtitles on their team videos with their Kaltura partner id will be synced back to Kaltura.
Brightcove¶
- Teams can link to Brightcove from their Settings -> Integrations page
- Once a team links to Brightcove, subtitles on their team videos with their Brightcove publisher id will be synced back to Brightcove.
- Teams can optionally choose to import videos from their Brightcove account.
- If importing, teams can either import all videos or videos matching certain tags.
Static Media¶
Static media files are handled by the staticmedia app. This app has several goals:
- Combine multiple files into a single “media bundle”. Linking to a single JS file results in faster page loads than linking to multiple files.
- Compress JS/CSS code.
- Support preprocessors like SASS.
- Support media files served from the local server or S3
- Store media files on S3 in a unique location for each deploy. This allows us to upload media for our next deploy without affecting our current one. It also allows us to the set the expire header to the far future which is good for caching.
Settings¶
Example¶
MEDIA_BUNDLES = {
"base.css": {
"files": (
"css/v1.scss",
"css/bootstrap.css",
),
},
"site.js": {
"files": (
"js/jquery-1.4.3.js",
"js/unisubs.site.js",
),
},
}
STATIC_MEDIA_COMPRESSED = True
STATIC_MEDIA_USES_S3 = True
AWS_ACCESS_KEY_ID = 'abcdef'
AWS_SECRET_ACCESS_KEY = 'abcdef
STATIC_MEDIA_S3_BUCKET = 'bucket.name'
STATIC_MEDIA_S3_URL_BASE = '//s3.amazonaws.com/bucket.name'
MEDIA_BUNDLES¶
MEDIA_BUNDLES defines our Javascript/CSS media bundles.
The keys are the filename that we will generate. The extension of the filename controls what type of media and should either by js or css.
The values are dicts that determine how we build the bundle. They can have these properties:
- files¶
list of files to bundle together (paths are relative to the media directory)
- add_amara_conf(optional)¶
- If True, we will prepend javascript code to the source JS files. THis will create global object called _amaraConf with these properties:
- baseURL: base URL for the amara website
- staticURL: base URL to the static media
STATIC_MEDIA_COMPRESSED¶
Set to False to disable compressing/minifying Javascript and CSS
STATIC_MEDIA_USES_S3¶
If True we Will Serve media files from amazon S3. This will change the URLs that our template tags create for links to the media bundles. STATIC_MEDIA_USES_S3 is usually True for production and False for development.
If STATIC_MEDIA_USES_S3 is enabled, the following settings are available:
- AWS_ACCESS_KEY_ID: S3 access key.
- AWS_SECRET_ACCESS_KEY: S3 secret key.
- STATIC_MEDIA_S3_BUCKET: S3 bucket to store media in.
- STATIC_MEDIA_S3_URL_BASE: Base URL for S3 media.
Compilation & Minification¶
We use uglifyjs for Javascript files and SASS for CSS files. Using the SASS extensions is optional. If you just have regular CSS files that SASS will function simply as a CSS compressor.
Media Directory Structure¶
Regardless if media is uploaded to S3 or we are serving it from the local instance, we structure the files the same way:
- css/ - CSS bundles
- js/ - Javascript bundles
- images/ - Image files
- fonts/ - font files
When serving media from the local server, the root URL for media files will be /media/.
When serving media from S3, the root URL for media files will be <STATIC_MEDIA_S3_URL_BASE><git-commit-id/
Development, Media Bundles, and Caching¶
For development servers, STATIC_MEDIA_USES_S3 is usually False, which causes us to serve up the media bundles from the local server. It takes long enough to compile media bundles that we don’t want to re-do it on every page request. So we cache the result and use that for subsequent requests. Before using a cached result, we check the mtime of all source files, and if any one is later than when the cache was created, we rebuild.
This works fine for most use cases, but there are a couple ways that it will fail. For example removing a file from the sources list won’t trigger a rebuild. If you think this may be happening, just update the mtime on any source file to trigger the rebuild manually.
In Templates¶
To link to media files in templates load the media_bundle library. Then you can use these tags:
- media_bundle – include a CSS/JS media bundle (generates the entire script/link tag)
- url_for – Get the URL to a media bundle.
- static_url – Get the base URL for static media.
Developer’s Guide¶
This section contains information on the internal workings of the Amara codebase.
Development Workflow¶
This guide describes the development workflow for Amara.
Contents
Branches¶
Amara development tries to follow a “one branch per feature or bugfix” workflow (for the most part). There are two main parts.
First, the master branch is the “base” branch. It’s what gets deployed to staging and production servers. Commits should never be made directly on this base branch (except for merges).
Actual code changes should always be made on “feature” branches. Each feature branch should contain changes related to a single feature or bugfix. Each feature or bugfix should have an issue in the bug tracker (Sifter). Each feature branch should be named after its issue number (e.g. i-1234 would be a branch for issue 1234).
Workflow¶
The Amara development workflow should go something like this.
Create an Issue¶
First, a Sifter issue is created for the task. It might be a new feature, a bug fix, or some code cleanup. For this example we’ll assume the issue number is 1234.
Create a Feature Branch¶
A Git branch for the issue is created from the current head of master, and it is named i-1234.
Create an Instance¶
To test changes non-locally an instance will need to be created for the feature branch. You should do this as soon as you create the branch, so that test data will be populated (and later migrated) correctly.
Create the “demo” instance using either Launchpad (https://launchpad.amara.org) or Fabric.
Using Launchpad, login and select the “Create Demo from Branch” workflow. Select the branch from the dropdown and an optional url. You will need to enter the full url name: (i.e. mybranch.demo.amara.org). If you don’t specify a custom url, the branch name will be used.
If you use fabric, use the following:
fab demo:<username>,<branch_name> create_demo
Or to use a custom url:
fab demo:<username>,<branch_name> create_demo:url_prefix=mybranch.demo.amara.org
Make Changes on the Feature Branch¶
Changes that fulfill the issue are made on that branch. The repository now looks like this:
.
O i-1234
|
O
/
O master
|
Commit messages should start with the issue number, a colon, and a space, like this:
1234: Remove the foo from the bar
This makes it easy to grep the Git log for changes related to a specific issue.
If at all possible, the developer should add a test case that covers the feature/bug as a separate commit first.
They can then push that to the branch on GitHub, watch it fail, then add the code that fixes the problem and watch it start passing. This is a good sanity check that their code (and test) does what they think it does.
Keep the Feature Branch Up To Date¶
As the programmer works on the feature branch, other feature branches may have been merged into master by other people. The programmer should merge these changes back into their feature branch as often as possible to keep it up to date. For example:
.
master O
/|
| | O i-1234
work by another . | |
dev on a different . | |
feature branch . | |
| | O
\|/
O
|
The programmer working on i-1234 should merge these changes into their feature branch to keep it up to date:
.
O i-1234
/|
master O |
/| |
| | O
work by another . | |
dev on a different . | |
feature branch . | |
| | O
\|/
O
|
Run the Full Test Suite¶
The small set of tests should be run automatically after every commit. Once the programmer thinks they’ve solved the issue they should kick off the full suite of Selenium tests and wait for the results (by email).
TODO: Describe how to do this.
Resolve the Ticket for QA¶
Along with the automated test suite which should be run automatically, QA will need to test the changes. Once the developer has received the full tests results (and they’re passing) they should resolve the Sifter ticket. QA will then test the instance running from the i-1234 branch.
If there’s a problem, they’ll reopen the ticket and the developer can make some more changes on the feature branch. Otherwise they’ll comment on the ticket and say that it’s ready to go.
Merging Back to the Base Branch¶
Once QA has tested a feature branch, the developer should send a pull request to merge i-1234 back into master. The other developers should review all the code as a last line of defense against bugs.
If there’s a problem, the original developer should make some more changes on i-1234 that fix the problem, QA retests, and a new pull request should be made.
Otherwise, the branch can be merged into master.
Delete the Feature Branch¶
Once the feature branch (i-1234) has been merged back into the base branch (master) it can be deleted.
You can find commits made on a particular feature branch later by grepping through the commit logs for 1234:, thanks to the commit message format.
The git command to delete a branch both locally and remotely is:
git push origing --delete i-1234
Delete the Instance¶
From the launchpad, choose Delete Demo and remove it. If you use fabric, use the following:
fab demo:<username>,<branch_name> remove_demo
Deploy to Production¶
Once the feature branch has been merged back into the base branch and deleted, the base branch can be deployed to production.
TODO: Have Evan describe how to do this.
Integration Repository¶
The integration repository should function the same way as the main repository.
If you don’t need to make any changes inside of it there’s no need to create an empty i-#### feature branch in it though.
TODO: Add more details here.
“Buffer” Branches¶
Sometimes there are larger projects that span multiple Sifter issues which don’t make sense to deploy individually. When this is the case, a “buffer” branch should be used.
A “buffer” branch is a separate Git branch with a descriptive name like data-model-refactor or new-editor. Once created it takes over the role of the “base” branch for changes related to that project.
Instead of creating i-2222 as a branch off of master, it would be created as a branch off of new-editor. It would be kept up to date by merging new-editor back in, and once complete a pull request to merge it back into new-editor would be created.
Note that new-editor itself should be kept up to date with changes from master as well.
An instance can be deployed to track the buffer branch itself (in addition to instances for each feature branch off of it).
Once all the development has been completed, the buffer branch itself can be merged back into master and deployed.
Basic Example¶
Let’s walk through a full example of a workflow. First, we’ll start with a clean slate:
.
O master
|
⋯
Now someone creates a feature branch for an issue and makes some changes:
.
O i-1111
|
O
/
O master
|
⋯
At the same time, someone else creates a feature branch for a different issue:
.
i-2222 O
|
| O i-1111
| |
| O
\ /
O master
|
⋯
Now the first developer marks their ticket as resolved, QA tests, and everything is okay.
They create a pull request to merge i-1111 back into master. The other developers review it and it looks fine, so they merge it and delete the feature branch:
.
O master
i-2222 O |\
| | |
| | O
| | |
| | O
\|/
O
|
⋯
Now the second developer notices that there are new changes on master, so they merge master into their feature branch to keep the feature branch up to date:
.
i-2222 O
|\
| O master
O |\
| | |
| | O
| | |
| | O
\|/
O
|
⋯
They make a few more changes:
.
i-2222 O
|
O
|
O
|\
| O master
O |\
| | |
| | O
| | |
| | O
\|/
O
|
⋯
They mark the ticket as resolved, QA tests, they create a pull request, devs review, and their feature branch gets merged into master and deleted:
.
O master
/|
O |
| |
O |
| |
O |
|\|
| O
O |\
| | |
| | O
| | |
| | O
\|/
O
|
⋯
Buffer Branch Example¶
TODO: This.
Testing¶
The Amara project uses the Nose testing framework.
Running tests¶
You should always run your tests inside the Vagrant VM because the test suite depends on a running Solr instance.
To run all unittests:
$ dev test
To run tests for a specific Django app:
$ dev test videos
To run a specific test class:
$ dev test videos.tests:ViewsTest
To run a specific test case within a test class:
$ dev test videos.tests:ViewsTest.test_index
Caching¶
Amara uses a couple tricks for caching things.
Cache Groups¶
Cache groups are used to manage a group of related cache values. They add some extra functionality to the regular django caching system:
- Key prefixing: cache keys are prefixed with a string to avoid name collisions
- Invalidation: all values in the cache group can be invalidated together. Optionally, all values can be invalidated on server deploy
- Optimized fetching: we can remember cache usage patterns in order to use get_many() to fetch all needed keys at once (see Cache Patterns)
- Protection against race conditions: (see Race condition prevention)
Typically cache groups are associated with objects. For example we create a cache group for each user and each video. The user cache group stores things like the user menu HTML and message HTML. The video cache group stores the language list and other sections of the video/language pages.
Overview¶
- A CacheGroup is a group of cache values that can all be invalidated together
- You can automatically create a CacheGroup for each model instance
- CacheGroups can be used with a cache pattern. This makes it so we remember which cache keys are requested and fetch them all using get_many()
Let’s take the video page caching as an example. To implement caching, we create cache groups for Team, Video, and User instances. Here’s a few examples of how we use those cache groups:
- Language list: we store the rendered HTML in the video cache
- User menu: we store the rendered HTML in the user cache (and we actually use that for all pages on the site)
- Add subtitles form: we store the list of existing languages in the video cache (needed to set up the selectbox)
- Follow video button: we store a list of user ids that are following the videos in the video cache. To the user is currently following we search that list for their user ID.
- Add subtitles permissions: we store a list of member user ids in the team cache. To check if the user can view the tasks/collaboration page we search that list of the user ID
When we create the cache groups, we use the video-page cache pattern. This makes it so we can render the page with 3 cache requests. One get_many fetches the Video instance and all cache values related to the video, and similarly for the Team and User.
Cache invalidation is always tricky. We use a simple system where if a change could affect any cache value, we invalidate the entire group of values. For example if we add/remove a team member then we invalidate the cache for the team.
Cache Patterns¶
Cache patterns help optimize cache access. When a cache pattern is set for a CacheGroup we will do a couple things:
- Remember which keys were fetched from cache.
- On subsequent runs, we will try to use get_many() to fetch all cache values at once.
This speeds things up by reducing the number of round trips to memcached.
Behind the scenes¶
The main trick that CacheGroup uses is to store a “version” value in the cache, which is simply a random string. We also pack the version value together with all of our cache values. If a cache value’s version doesn’t match the version for the cache group, then it’s considered invalid. This allows us to invalidate the entire cache group by changing the version value to a different string.
Here’s some example data to show how it works.
key | value in cache | computed value |
---|---|---|
version | abc | N/A |
X | abc:foo | foo |
Y | abc:bar | bar |
Z | def:bar | invalid |
Note
We also will prefix the all cache keys with the “<prefix>:” using the prefix passed into the CacheGroup constructor.
Note
If invalidate_on_deploy is True, then we will append ”:<commit-id>” to the version key. This way the version key changes for each deploy, which will invalidate all values.
Race condition prevention¶
The typical cache usage pattern is:
Fetch from the cache
If there is a cache miss then:
- calculate the value
- store it to cache.
This pattern will often have a race condition if another process updates the DB between steps 2a and 2b. Even if the other process invalidates the cache, the step 2b will overwrite it, storing an outdated value.
This is not a problem with CacheGroup because of the way it handles the version key. When we get the value from cache, we also fetch the version value. If the version value isn’t set, we set it right then. Then when we store the value, we also store the version key that we saw when we did the get. If the version changes between the get() and set() calls, then the value stored with set() will not be valid. This works somewhat similarly to the memcached GETS and CAS operations.
Cache Groups and DB Models¶
Cache groups can save and restore django models using get_model() and set_model(). There is a pretty conservative policy around this. Only the actual row data will be stored to cache – other attributes like cached related instances are not stored. Also, restored models can’t be saved to the DB. All of this is to try to prevent overly aggressive caching from causing weird/wrong behavior.
To add caching support to your model, add ModelCacheManager as an attribute to your class definition.
- class caching.cachegroup.CacheGroup(prefix, cache_pattern=None, invalidate_on_deploy=True)¶
Manage a group of cached values
Parameters: - prefix (str) – prefix keys with this
- cache_pattern (str) – cache pattern identifier
- invalidate_on_deploy (bool) – Invalidate values when we redeploy
- get(key)¶
Get a value from the cache
This method also checks that the version of the value stored matches the version in our version key.
If there is no value set for our version key, we set it now.
- get_many(keys)¶
Get multiple keys at once
If there is no value set for our version key, we set it now.
- set(key, value, timeout=None)¶
Set a value in the cache
- set_many(values, timeout=None)¶
Set multiple values in the cache
- get_or_calc(key, work_func, *args, **kwargs)¶
Shortcut for the typical cache usage pattern
get_or_calc() is used when a cache value stores the result of a function. The steps are:
- Try self.get(key)
- If there is a cache miss then
- call work_func() to calculate the value
- store it in the cache
- get_model(ModelClass, key)¶
Get a model stored with set_model()
Note
To be catious, models fetched from the cache don’t allow saving. If the cache data is out of date, we don’t want to saave it to disk.
- set_model(key, instance, timeout=None)¶
Store a model instance in the cache
Storing a model is a tricky thing. This method works by storing a tuple containing the values of the DB row. We store it like that for 2 reasons:
- It’s space efficient
- It drops things like cached related objects. This is probably good since it makes it so we don’t also cache those objects, which can lead to unexpected behavior and bugs.
Parameters: - key – key to store the instance with
- instance – Django model instance, or None to indicate the model does not exist in the DB. This will make get_model() raise a ObjectDoesNotExist exception.
- invalidate()¶
Invalidate all values in this CacheGroup.
- class caching.cachegroup.ModelCacheManager(default_cache_pattern=None)¶
Manage CacheGroups for a django model.
ModelCacheManager is meant to be added as an attribute to a class. It does 2 things: manages CacheGroups for the model class and implements the python descriptor protocol to create a CacheGroup for each instance. If you add cache = ModelCacheManager() to your class definition, then:
- At the class level, MyModel.cache will be the ModelCacheManager instance
- At the instance level, my_model.cache will be a CacheGroup specific to that instance
- get_cache_group(pk, cache_pattern=None)¶
Create a CacheGroup for an instance of this model
Parameters: - pk – primary key value for the instance
- cache_pattern – cache pattern to use or None to use the default cache pattern for this ModelCacheManager
- invalidate_by_pk(pk)¶
Invalidate a CacheGroup for an instance
This is a shortcut for get_cache_group(pk).invalidate() and can be used to invalidate without having to load the instance from the DB.
- get_instance(pk, cache_pattern=None)¶
Get a cached instance from it’s cache group
This will create a CacheGroup, get the instance from it or load it from the DB, then reuse the CacheGroup for the instance’s cache. If a cache pattern is used this means we can load the instance and all of the needed cache values with one get_many() call.
Teams¶
Teams are a key concept in amara. A team is a group of users that work together to subtitle videos. Teams are typically made of members of a group that produces video and wants to add subtitles.
Team Workflows¶
Team workflows are ways for teams to get their subtitling work done. Team workflows compliment the Subtitle Workflows and add team-specific features.
- Team workflows are responsible for:
- Providing a SubtitleWorkflow for team videos
- Handling the workflow settings page
- Handling the dashboard page
- Creating extra tabs or the teams section
- class teams.workflows.teamworkflows.TeamWorkflow(team)¶
- type_code = NotImplemented¶
Team.workflow_type value for this workflow.
- label = NotImplemented¶
Human-friendly name for this workflow. This is what appears on the team creation form.
- dashboard_view = NotImplemented¶
view function for the dashboard page.
- workflow_settings_view = NotImplemented¶
view function for the workflow settings page.
Note
All workflows should allow the user to change membership_policy and video_policy in their workflow settings page.
- setup_team()¶
Do any additional setup for newly created teams.
- get_subtitle_workflow(team_video)¶
Get the SubtitleWorkflow for a video with this workflow.
- class teams.workflows.teamworkflows.TeamPage¶
Represents a page in the team’s section
- name¶
machine-name for this tuple. This is value to use for current in the _teams/tabs.html template
- title¶
human friendly tab title
- url¶
URL for the page
- class teams.workflows.old.workflow.OldTeamWorkflow(team)¶
Workflow for old-style teams
We have tried to tackle the issue of team workflows in several ways. The most infamous has to be the tasks sytem. This class acts the glue between the new workflow components and the old systems.
The plan is to migrate all our teams from OldTeamWorkflow to newer workflow styles. At that point we can get rid of OldTeamWorkflow and also probably a bunch of other things like the tasks code, the Workflow table, several Team model fields, etc.
The Subtitle Editor¶
The subtitle editor is one of the larger features of amara. It’s implemented using several components in a couple different areas:
- The view subtitles.views.subtitle_editor serves up the page
- The page runs javascript that lives in media/src/js/subtitle-editor
- We save subtitles using the API code (currently in a private repository, but we plan to merge it in to the main one soon)
See also
Subtitle Workflows¶
Subtitle workflows control how subtitle sets get edited and published. In particular they control:
- Work Modes – Tweak the subtitle editor behavior (for example review mode)
- Actions – User actions that can be done to subtitle sets (Publish, Approve, Send back, etc).
- Permissions – Who can edit subtitles, who can view private subtitles
- class subtitles.workflows.Workflow(video)¶
A workflow class controls the overall workflow for editing and publishing subtitles. Workflows control the work modes, actions, and permissions for a set of subtitles.
By default, we use a workflow that makes sense for public videos – Anyone can edit, the only action is Publish, etc. However, other components can create custom workflows for specific videos by:
- Creating a Workflow subclass
- Overriding get_workflow() and returning a custom workflow object
- get_work_mode(user, language_code)¶
Get the work mode to use for an editing session
Parameters: - user (User) – user who is editing
- language_code (str) – language being edited
Returns: WorkMode object to use
- get_actions(user, language_code)¶
Get available actions for a user
Parameters: - user (User) – user who is editing
- language_code (str) – language being edited
Returns: list of Action objects that are available to the user.
- action_for_add_subtitles(user, language_code, complete)¶
Get an action to use for add_subtitles()
This is used when pipeline.add_subtitles() is called, but not passed an action. This happens for a couple reasons:
- User saves a draft (in which case complete will be None)
- User is adding subtitles via the API (complete can be True, False, or None)
Subclasses can override this method if they want to use different actions to handle this case.
Parameters: - user (User) – user adding subtitles
- language_code (str) – language being edited
- complete (bool or None) – complete arg from add_subtitles()
Returns: Action object or None.
- extra_tabs(user)¶
Get extra tabs for the videos page
Returns: list of (name, title) tuples. name is used for the tab id, title is a human friendly title. For each tab name you should create a video-<name>.html and video-<name>-tab.html templates. If you need to pass variables to those templates, create a setup_tab_<name> method that inputs the same args as the methods from VideoPageContext and returns a dict of variables for the template.
- get_add_language_mode(user)¶
Control the add new language section of the video page
Parameters: user (User) – user viewing the page Returns: - None/False: Don’t display anything
- “<standard>”: Use the standard behavior – a link that opens the create subtitles dialog.
- any other string: Render this in the section. You probably want to send the string through mark_safe() to avoid escaping HTML tags.
- get_editor_notes(language_code)¶
Get notes to display in the editor
Returns: EditorNotes object
- user_can_view_private_subtitles(user, language_code)¶
Check if a user can view private subtitles
Private subtitles are subtitles with visibility or visibility_override set to “private”. A typical use is to limit viewing of the subtitles to members of a team.
Returns: True/False
- user_can_view_video(user)¶
Check if a user can view the video
Returns: True/False
- user_can_edit_subtitles(user, language_code)¶
Check if a user can edit subtitles
Returns: True/False
- subtitles.workflows.get_workflow(video)¶
Get the workflow to use for a subtitle set
This method uses the behaviors module, to allow other apps to override this and control the workflow for specific subtitles sets. A typical example is the tasks system which creates a custom workflow for videos owned by tasks teams.
Editor Notes¶
- class subtitles.workflows.EditorNotes(video, language_code)¶
Manage notes for the subtitle editor.
EditorNotes handles fetching notes for the editor and posting new ones.
- heading¶
heading for the editor section
- notes¶
list of SubtitleNotes for the editor (or any model that inherits from SubtitleNoteBase)
- post(user, body)¶
Add a new note.
Parameters: - user (CustomUser) – user adding the note
- body (unicode) – note text
Work Modes¶
- class subtitles.workflows.WorkMode¶
Work modes are used to change the workflow section of the editor and affect the overall feel of the editing session. Currently we only have 2 work modes:
Actions¶
Actions are things things that users can do to a subtitle set other than changing the actual subtitles. They correspond to the buttons in the editor at the bottom of the workflow session (publish, endorse, send back, etc). Actions can occur alongside changes to the subtitle lines or independent of them.
- class subtitles.workflows.Action¶
Base class for actions
Other components can define new actions by subclassing Action, setting the class attributes, and optionally implementing perform().
- name = NotImplemented¶
Machine-friendly name
- label = NotImplemented¶
human-friendly label. Strings should be run through ugettext_lazy()
- in_progress_text = NotImplemented¶
text to display in the editor while this action is being performed. Strings should be run through ugettext_lazy()
- visual_class = None¶
visual class to render the action with. This controls things like the icon we use in our editor button. Must be one of the CLASS_ constants
- complete = None¶
complete defines how to handle subtitles_complete. There are 3 options:
- True – this action sets subtitles_complete
- False – this action unsets subtitles_complete
- None (default) - this action doesn’t change subtitles_complete
- subtitle_visibility = 'public'¶
Visibility value for newly created SubtitleVerisons.
- CLASS_ENDORSE = 'endorse'¶
endorse/approve buttons
- CLASS_SEND_BACK = 'send-back'¶
reject/send-back buttons
- validate(user, video, subtitle_language, saved_version)¶
Check if we can perform this action.
Parameters: - user (User) – User performing the action
- video (Video) – Video being changed
- subtitle_language (SubtitleLanguage) – SubtitleLanguage being changed
- saved_version (SubtitleVersion or None) – new version that was created for subtitle changes that happened alongside this action. Will be None if no changes were made.
Raises: ActionError – this action can’t be performed –
- perform(user, video, subtitle_language, saved_version)¶
Perform this action
Parameters: - user (User) – User performing the action
- video (Video) – Video being changed
- subtitle_language (SubtitleLanguage) – SubtitleLanguage being changed
- saved_version (SubtitleVersion or None) – new version that was created for subtitle changes that happened alongside this action. Will be None if no changes were made.
- update_language(user, video, subtitle_language, saved_version)¶
Update the subtitle language after adding subtitles
Parameters: - user (User) – User performing the action
- video (Video) – Video being changed
- subtitle_language (SubtitleLanguage) – SubtitleLanguage being changed
- saved_version (SubtitleVersion or None) – new version that was created for subtitle changes that happened alongside this action. Will be None if no changes were made.
- editor_data()¶
Get a dict of data to pass to the editor for this action.
- class subtitles.workflows.Publish¶
Publish action
Publish sets the subtitles_complete flag to True
The permission system¶
The permission system in Amara subtitles is very flexible to allow for the needs of different teams. This document will give you a high level overview of what is possible. You should read this before trying to understand the source code.
Overview¶
Let’s start with some language. In the simplest case, when a user is part of a team, they can have one of the following roles:
- Contributor
- Transcribe
- Translate
- Assign tasks to themselves
- Manager
- Review subtitles
- Approve subtitles
- Assign tasks to other people
- Everything that a contributor can do
- Admin
- Assign new managers
- Delete subtitles
- Everything that a manager can do
- Owner
- Everything
Note
This is just an example to give you an idea of how this could work.
A user’s role is stored in the teams.models.TeamMember model which stores a reference to the user and team objects.
Checking for required permissions¶
When you want to check if a certain user has the required privileges to perform a task, you should use one of the functions in teams.permissions. For example, if you’d like to check if a user can approve a video, you could do something like this:
from teams.permissions import can_approve
if can_approve(video, user):
# Do something that requires the approval permission
Note
There is no middleware to attach the current user’s privileges to the request instance. Instead, you have explicitly call the necessary function whenever you want to verify the user’s privileges.
Workflows¶
A team can choose their own workflow to efficiently manage their videos, translations and volunteers. When you are setting up a workflow for your team, you can decide how certain actions will be performed. For example:
- Who can join the team?
- Who can and remove videos from the team?
- Who can assign tasks?
- How many tasks a user can have at a time?
- How many days should a user get to complete a task?
- Who can transcribe subtitles?
- Who can translate subtitles?
- Is there a review process?
- Is there an approval process?
So, why should you care? For example, you don’t trust your contributors with transcription of new videos since it’s somewhat difficult. Therefore, you can choose to only allow managers and above to transcribe videos and contributors to only translate videos to different languages. Or, the quality of the subtitles is crucial to you and you want to make sure that nothing less than that ever gets out. So, you would turn on both the review and approval process. This way three sets of eyes will look at the subtitles before it goes public.
Optional Apps¶
Amara.org uses several apps/packages that are stored in private github repositories that add extra functionality for paid partnerships. These apps are optional – the amara codebase runs fine without them.
The coding issue is how to make amara work without these repositories, but automatically pull them in if they are present. Here’s how we do it:
- For each repository we create a file inside the optional/ directory:
- The filename is the name of the repository
- The contents are the git commit ID that we want to use
- To enable a repository, it must be checked out in the amara root directory, using the same name as the git repository.
- The optionalapps module handles figuring out which repositories are present and how we should modify things at runtime
- optionalapps.get_repository_paths()¶
Get paths to optional repositories that are present
Returns: list of paths to our optional repositories. We should add these to sys.path so that we can import the apps.
- optionalapps.get_apps()¶
Get a list of optional apps
Returns: list of app names from our optional repositories to add to INSTALLED_APPS.
- optionalapps.get_urlpatterns()¶
Get Django urlpatterns for URLs from our optional apps.
This function finds urlpatterns inside the urls module for each optional app. In addition a module variable can define a variable called PREFIX to add a prefix to the urlpatterns.
Returns: url patterns containing urls for our optional apps to add to our root urlpatterns
- optionalapps.add_extra_settings(globals, locals)¶
Add extra values to the settings module.
This function looks for files named settings_extra.py in each optional repository. If that exists, then we call execfile() to run the code using the settings globals/locals. This simulates that code being inside the settings module.
Internationalization (i18n)¶
Unisubs has some complex requirements in terms of 18n. This is a rough guide of how things work.
Django’s system is gnu’s get text system. For example:
pt
The first two letters are the language code, according to ISO 639-1. In this case Portuguese.
If the locale has variation as to the country, for example Portugal’s Portuguese vs Brazilian’s portuguese then the locale name is appended an underscore + the country two digit code, which is ISO 3166. Therefore the locales for portuguese speaking countries are:
pt_BR -> Brazilian Portuguese
pt_PT -> Portugal's Portuguese
Some of the less common languages are not covered by ISO 639-1 but are by ISO 639-3.
Guidelines¶
Most of the heavy lifting is handled by our Unilangs library.
Steps for adding a new language:
Figure Out the ISO code If ISO 639-1 covers it, it’s the prefered way to handle this, as it would keep our code streamlined with Django’s. If not, then we should prefer ISO 639-3. One can depend on the list of ISO 693-1 and the list of ISO 693-3 on Wikipedia.
Also if it’s more of an uknown language it’s useful to look at the Wikipedia entry for it. Sometimes, we have a request to include a language that doesn’t make sense. For example there is no Norwegian(no), but there are dialects (nb and nn) that are used. Usually the Wikipedia page will discuss similar languages and specific dialects.
Update Unilangs INTERNAL_NAMES: Add the language code, the English name for the language, and the language in it’s own name to the unilangs.py file .
Update the unisubs standard: Add the language code to our standard .
Update other codecs: If django supports that locale (as in we can i18n the site’s UI) update the ‘dango’ standard. If other standards (such as ISO-693-1) support it, update them too.
Of course, once you’ve updated unlilangs, you’ll need to update the virtual envs on all installations of the app.
Updating Django¶
One must be careful when updating the Django’s version. As new locales are added between releases, we must check if the locale is already added on our end with a different encoding. If that happens, we’ll have duplication . This has beaten us before.
Partners¶
Different partners might have different language requirements while mapping to their own internal systems. We should update this guide once we have more specifics on how we’re implementing those mappings.
Behaviors¶
Extensible behavior functions
This module allows one app to define a “behavior function”, which other apps can then override to change the behavior. This is the Chain of responsibility pattern which helps keep modules loosely coupled.
A typical use for this is the title that we display on the video page. Normally we just print a simple video title, but if the video is part of certain teams we want to print a modified title. Putting the code that deals with this inside the videos app is bad practice because:
- It’s adding complexity to the videos app. Handling team requirements is outside of its scope.
- It requires importing from the teams app, but the teams app needs to import from the videos app. So we now have a circular dependency.
Instead, videos defines the make_video_title behavior, which can then be overriden by other apps. This allows us to change the behavior without having to add complexity/dependencies to the videos app. The code works something like this.
Example
>>> @behavior
... def make_video_title(video)
... return video.title
>>> @make_video_title.override
... def make_video_title_for_team_foo(video):
... team_video = video.get_team_video()
... if team_video and team_video.slug == 'foo'
... return 'My Team: %s' % video.title
... else:
... return DONT_OVERRIDE
Contributing¶
The source code to Amara is available on Github. Feel free to fork the project and open a pull request.
Before working on a sizeable feature, please do run it by us in our IRC channel. We don’t want you to waste your time working on something we don’t really want. We are at #amara on freenode.
License¶
Amara is freely available under the terms of the GNU Affero General Public License. You can find the full text of the license here.