Welcome to Data Hub API’s documentation!¶

The problem¶

We are migrating away from Microsoft Dynamics 2011 (CDMS) and decided to build a new CRM system (Data Hub) using a gradual incremental approach.

During a period of several months, the following constraints apply:

• data between CDMS and Data Hub needs to be kept in sync
• Data Hub needs allow re-modeling by adding/removing types/properties
• some users would continue to use CDMS whilst we transition from one system to the other

The options¶

We considered different approaches including:

• use CDMS as data store and access it directly. This has many disadvantages including hosting CDMS, not being able to easily change the schemas, architecture complexity etc.
• use two data stores with some sort of low level synchronization (via database or processes). This as well has many disadvantages including integrating with old technologies (Dynamics 2011), two separate layers (code and sync logic) depending on each other tightly and hard to manage, synchronisation conflicts etc.
• use two data stores with code-managed synchronization. This is the chosen architecture and has some disadvantages as well that we will explain later.

The chosen approach¶

Two data stores with reads and writes to CDMS happening as usual and synchronisation triggered from io actions in Data Hub.

Writes to Data Hub will:

• get the object from CDMS (if it exists)
• apply the changes and write to CDMS
• apply the changes in Data Hub

Reads from Data Hub will:

• get the object from the Data Hub data store
• get the related object from CDMS
• check if CDMS was updated after the last synchronisation
• if so, update the Data Hub object
• return the local results

Read and write operations are performed as a single transaction so that changes are rolled back in case of exceptions with CDMS.

The same object on both systems is considered in sync if the modified field value is the same. If the modified value of the CDMS version is more recent, it means that the Data Hub object has to be updated from the CDMS one. If the modified value of the Data Hub version is more recent, an exception is triggered as this should never happen. This is because writes on the Data Hub always generate writes in CDMS but the vice versa is obviously not true.

The possibility of conflicts is low as:

• objects on the two systems are kept in sync via the modified field updated after each CDMS get
• concurrent operations to a single object are low or non-existent in volume

In case two updates happen at approximately the same time, the last one wins. This should not be a problem as the system keeps a history of the changes.

Limitations¶

There are some limitations in using this approach:

• Amount of requests. This has not been measured yet but could (and should) be partially addressed by using some sort of caching strategy
• The synchronisation happens using one common CDMS user
• Some Django ORM API cannot be easily implemented. E.g. Model.objects.count(), Model.objects.filter(field1__field2='something'). This is mainly because of the old CDMS technologies
• It might not be easy to change the Django schema in many cases as the sync layer prefers a one-to-one mapping.

How it works¶

A custom django manager / queryset intercepts reads / writes and takes care of all the CDMS operations. This means that developers can ignore this extra complexity and use the django orm api as usual.

That being said, only a subset of the django orm api have been implemented and are even possible. Check Django ORM integration for the full list of the ORM calls supported.

Project setup¶

The cdms_api app contains the CDMS API library whilst the migrator app defines all the code needed for the synchronisation.

Your app and your Django model¶

1. Set up Django app/model

Create a new Django app or simply a new Django Model as needed.

2. CDMSMigrator

In a module called <your-app>/cdms_migrator.py , subclass migrator.cdms_migrator.BaseCDMSMigrator and define the mapping fields and the CDMS service.

Add the CDMSMigrator to the Django Model as per step 3.

3. Configure your model

Change your model so that it looks like the one below:

Note

for Foreign key fields, you should use core.fields.UKTIForeignKey instead of the Django one.

from reversion import revisions as reversion

from django.db import models

from core.models import CRMBaseModel
from core.managers import CRMManager

from .cdms_migrator import MyModelMigrator

@reversion.register()
class MyModel(CRMBaseModel):
    ....

    objects = CRMManager()
    cdms_migrator = MyModelMigrator()

4. Create a migration for your model as usual

./manage.py makemigrations
./manage.py migrate

CDMSMigrator¶

The mapping between your model and the CDMS one is defined in your model’s CDMSMigrator class which should be in <your-app>/cdms_migrator.py.

Extend the migrator.cdms_migrator.BaseCDMSMigrator class and define the fields and service attributes.

For example

from cdms_api import fields as cdms_fields

from migrator.cdms_migrator import BaseCDMSMigrator


class OrganisationMigrator(BaseCDMSMigrator):
    fields = {
        'name': cdms_fields.StringField('Name'),
        'uk_organisation': cdms_fields.BooleanField('optevia_ukorganisation'),
        ...
    }
    service = 'Account'  # this is the Dynamics resource name

Operations that cause synchronisation¶

.filter(...) operations make a CDMS API call to get the CDMS objects with the same translated filtering, refresh the local objects by updating or creating them and then return the standard Django results.

.get(...) operations get the object in local and in CDMS, compare the two, update the local one if needed and then return the standard Django result.

.create(...) or .save() operations create the object in local and in CDMS. In case of exceptions with CDMS the local changes are rolled back.

.save() operations update the object in local and in CDMS. In case of exceptions with CDMS the local changes are rolled back.

.delete() operations delete the object in local and in CDMS. In case of exceptions with CDMS the local changes are rolled back.

✔ Supported ✘ Not supported

Operations that skip synchronisation¶

Most of the time, you can skip CDMS operations by using the skip_cdms() method on the manager or the skip_cdms param on the save/delete methods.

✔ Supported ✘ Not supported

How django-reversion works¶

django-reversion uses revisions and versions.

Revisions are blocks of code where some changes happen. One or more objects could potentially change in the same block.

Versions are changes to an object in a given revision. Versions always have a foreign key to the related revision.

Revisions can have the following metadata:

• user: who made the changes
• comment: optional text

Metadata has to be set manually for obvious reasons.

Usually you implement django-reversion in various ways:

• via the admin integration so that every time a user uses the admin, changes are saved automatically
• via an explicit context manager with the possibility to set metadata programmatically

How django-reversion is used¶

As we wanted to create revisions/versions automatically and not lose any changes, we implemented django-reversion at a lower level.

In our system we have 2 types of changes:

• CDMS refresh changes: where we refresh a local object (update or create) from CDMS. This happens automatically by creating a version of the object with the comment CDMS refresh.
• local changes: where we make a change to the objects of our system. This happens every time the .save() method is called and it’s automatic.