Example: a task¶

Let’s go for something a little simpler. This task learns about Youtube videos; it will only retain facts that are at most 2 days old, it will reject duplicates and send notifications via Pushbullet.

{
    system: {},
    tasks: [{
        name: 'youNotify',
        chain: [{
            plugin: 'youtube',
            plan: '0 */2 * * *',  //every 2 hours
            configuration: {
                mode: 'subscriptions',
                limit: 10,
                identity: 'gglMyName'
            }
        },{
            plugin: 'filter',
            configuration: {
                'youtube:video:publishedAt': {after: [-2, 'days']}
            }
        },{
            plugin: 'deduplicate'
        },{
            plugin: 'pushbullet',
            configuration: {
                identity: 'pbMyName',
                tag: "some pushbullet tag",
                mode: "link",
                title: "{{youtube:channel:title}} - {{youtube:video:publishedAt!!MM/DD/YYYY}} - {{youtube:video:title}}",
                body: "{{youtube:video:description}}"
            }
        }
    }]
}

While all plugins are placed in a chain, only the order of mutators matters; input plugins such as youtube will just learn facts whenever they run and output plugins will always receive facts after they have gone through all of the mutators.

On the other hand, the order of mutators does matter; in this example placing the filter plugin, which forgets facts based on their own properties, before the deduplicate plugin, which hits the database once for every fact, is more sensible than the opposite (which would run more database queries, all of them useless).

On tasks¶

A task can be seen both as a container of plugins or as a container of all the facts that have been learned and processed successfully by those plugins. It can therefore be controlled - started (so that it will run on schedule), stopped (so that it won’t) or run immediately.

At this point in Egeria’s development tasks are just names given to plugin chains. They can be used to address those plugins and they’re implicitly used to isolate facts learned by a specific task’s plugins from any other fact.

On input plugins¶

All input plugins are service-alikes that can be started, stopped or run. Note: some output plugins behave like services too. Also note that starting, stopping or running an entire task means starting or stopping or running all of the service-like plugins in its chain.

Inputs, regardless of what they do and how they do it, are always configured with a plan - a cron definition that looks like this: ‘*/2 * * * *‘ (this means “every 2 minutes”). When an input plugin is in the started state, it will automatically run according to its plan. Every time it runs, it learns facts - some of them new, some of them old.

Here’s a complete guide of the Cron format.

Facts learned by input plugins are fed to the rest of the plugin chain.

On mutators¶

Mutators receive a fact from the preceding plugin (an input or another mutator), process it and return it to the chain so that it can be processed by the next plugin in line. Mutators can forget facts, effectively removing them from the chain. Forgotten facts are gone forever - unless an input plugin learns them again.

On output plugins¶

Outputs make the magic happen: a file is downloaded, a torrent is sent to a BT client, a notification is pushed, a CSV table is created.

A few outputs are service-like; meaning they are also configured with a plan. Those plugins usually need to collect a number of facts over time and process them all at specified intervals (sending an email digest, writing a CSV file etc.). Those plugins can also be started, stopped or run just like inputs, but instead of learning facts they will accumulate them over time and then consume and forget them.

On Knowledge¶

Facts are small collections of metadata. They describe something in reality or on the Internet. Facts won’t include a video, but they can tell you where it is and what its name and description are. In a few cases facts do contain all of the data (e.g. Reddit private messages).

Facts are highly expendable; they are usually learned several times by the same plugin and should your Internet connection or electricity fail they still have a high chance of being processed just a little while later, as long as Egeria is kept running at all times, restarting it as needed.

Here’s what a youtube fact looks like (printed in a somewhat legible format, omitting a few fields):

There’s a lot you could do with even this little amount of information. Notifications, keeping a log, downloading stuff... The point of Egeria is to automate the systematic consumption of data, letting you spend more time exploring new things. Egeria wants you to stop clicking on the same thing every day to see if there’s an update. Go click on entirely new stuff instead.

On Wisdom¶

Wisdom equals Knowledge plus time. You don’t need to know anything else about it.

Willful ignorance¶

Plugins of all kinds are largely ignorant of each other. pushbullet doesn’t know how the facts learned by youtube look like; in order for it to work, the user has to configure it so that it will get the right values from the right fields and concatenate them to form meaningful messages.

Similarly, transmission doesn’t know where the link to a torrent will be, nor will it search for it. rarbgLookup won’t ever try to guess how to put together a meaningful search term.

This is by design.

Templating¶

In order to generate messages, links, file paths and anything else that might be needed to make plugins work together, templates need to be written. Here’s what a template looks like:

‘The field “youtube:video:link” contains the value {{youtube:video:link}}’

If this template is applied to the fact shown in the previous chapter, the result will be this:

‘The field “youtube:video:link” contains the value https://www.youtube.com/watch?v=eWEJb_3IFBw’

The full docs of the templating system are on the egeria-temple project’s page, but here’s the part you need if all you care about is learning how to write templates.

A template is a single string that contains zero or more tokens. Text present between tokens won’t be changed. “Rendering” a template means replacing all tokens in it with a string of characters.

In a template, “||” and ”!!” are reserved sequences. “|||”, “||||” and so on are all illegal. The same goes for ”!!” and ”!!!”, ”!!!!” and so on. It’s OK if those sequences are part of the value that will be rendered, but they must not appear in the template itself.

A token is written like this: {{PRE||FIELD!!FORMAT||POST}}

The order of the various elements must be respected:

PRE is a string that will precede the value if the value is not null.

FIELD is a field of the fact being processed and resolves to wahtever value is present in that field. The value could be null.

FORMAT is the format that will be used to format the FIELD if and only if such value is a Date. It will be ignored otherwise.

POST is a string that will be appended to the value if the value is not null.

PRE, FORMAT and POST are optional. If the value is a Date, the default FORMAT is ‘YYYYMMDD’ (e.g. “20151124”).

FIELD is mandatory, but it doesn’t have to exist on data. If it’s null, undefined or empty, the entire token will be replaced by an empty string, meaning that PRE and POST will not be rendered either.

Malformed tokens will remain untouched in the final output.

The smallest token you can possibly write is {{fieldName}}. The biggest is {{On day ||happeningDate!!DD-MM-YYYY|| something happened}}.

Now let’s look again at the pushbullet configuration from the first example that appears in this guide:

{
    plugin: 'pushbullet',
    configuration: {
        identity: 'pbMyName',
        tag: "some pushbullet tag",
        mode: "link",
        title: "{{youtube:channel:title}} - {{youtube:video:publishedAt!!MM/DD/YYYY}} - {{youtube:video:title}}",
        body: "{{youtube:video:description}}"
    }
}

The parameters title and body will apply their templates to any fact sent to pushbullet. The results, if the fact is the one learned by youtube that we saw earlier, will be this:

// value generated by the template assigned to “title”:

‘Previously Recorded - 03/31/2016 - Batman V Superman Discussion (SPOILERS)’

// value generated by the template assigned to “body”:

‘From the March 30th stream, we talk about Batman V Superman at length. – Watch live at https://www.twitch.tv/previouslyrecorded_live.’

To know how to configure plugins properly you can refer to their documentation. It will tell you what you can configure, which parameter will accept templates and what the resulting value will be used for. The documentation of plugins that can learn new facts or add to them will tell you what fields will be available and what they contain.

Remember that something like 'This is a template' is also a valid template; it will always output 'This is a template' regardless of what fact it’s applied to. Think of it as a constant. Moreover, '' is also a valid template that will always output an empty string.

Private information¶

A number of services require their users to authenticate before they let them access the data. Egeria has a simple system by which it can learn and remember how to authenticate with various services on behalf of the user. This is set up through egeriactl auth <type> [credentials...].

When setting up authentication, always run egeriactl from a console within a graphical environment with a working browser. All OSes are supported. This is necessary because of how modern authentication works.

A few services such as Google may require you to create an API app. This documentation will help you do that.

How plugins can be addressed: plugin paths¶

Once the configuration is complete, a way is needed to address different plugins or different tasks. For each instance of a plugin, a path is assigned:

<task name>/<plugin name>/<instance number>

In the following example:

{   system: {},
    tasks: [{
        name: 'test',
        chain: [{
            plugin: 'youtube',
            configuration: {...}
        }, {
            plugin: 'youtube',
            configuration: {...}
        }, {
            plugin: 'filter',
            configuration: {...}
        }, {
            plugin: 'youtubeDownloader',
            configuration: {...}
        }],
    }]
}

The paths are:

• test/youtube/0
• test/youtube/1
• test/filter/0
• test/youtubeDownloader/0

The instance number always respects the order set by the user. yuotube/0 will always be the first youtube plugin that appears inside the task configuration and youtube/1 will always be the second.

Tagging facts based on the input plugins that learned them¶

All plugins accept a configuration option called tags. This option is always an Array of one or more Strings, each one a “tag”.

{   system: {},
    tasks: [{
        name: 'test',
        chain: [{
            plugin: 'youtube',
            plan: '0 0 * * *',
            tags: ['subs'],
            configuration: {
                mode: 'submissions',
                limit: 5
            }
        }, {
            plugin: 'reddit',
            plan: '0 0 * * *',
            tags: ['movies'],
            configuration: {
                subreddit: 'fullmoviesonyoutube',
                mode: 'new',
                limit: 5
            }
        }]
    }]
}

Every fact learned by test/youtube/0 will be tagged with subs, while every fact learned by test/reddit/0 will be tagged movies.

Mutators will interact with all of those facts if the mutator hasn’t been given any tag; or, if the mutator has been tagged, only with facts that share at least one tag with it.

Let’s discard any subs older than 2 days and any movies that don’t have “1080p” in the title:

{   system: {},
    tasks: [{
        name: 'test',
        chain: [{
            plugin: 'youtube',
            plan: '0 */12 * * *',
            tags: ['subs'],
            configuration: {
                mode: 'submissions',
                limit: 5
            }
        }, {
            plugin: 'reddit',
            plan: '0 0 * * *',
            tags: ['movies'],
            configuration: {
                subreddit: 'fullmoviesonyoutube',
                mode: 'new',
                limit: 5
            }
        }, {
            plugin: 'filter',
            tags: ['subs'],
            configuration: {
                'youtube:video:publishedAt': { after: [-2, 'days'] }
            }
        }, {
            plugin: 'filter',
            tags: ['movies'],
            configuration: {
                'reddit:title': { matches: /1080p/i }
            }
        }],
    }]
}

Tagging facts based on their own properties¶

Using the tagger mutator, it’s possible to virtually split facts into different groups by tagging them differently based on their own characteristics.

More in-depth documentation of this mutator is available in its specific section.

{
    plugin: 'tagger',
    configuration: {
        schemas: [{
            'youtube:video:publishedAt': {after: [-7 days], mandatory: true}
        }, {
            'youtube:video:publishedAt': {before: [-7 days], mandatory: true}
        }],
        schemaTags: [
            'recent',
            'old'
        ]
    }
}

Authentication setup¶

TODO

You can now configure tasks and plugins.

Tasks¶

A task only has a name and a chain of plugins. The chain can be empty:

{
    "system": {
        "redisHost": "localhost",
        "redisPort": 6379,
        "port": 54321,
        "logLevel": "debug"
    },
    "tasks": [{
        "name": "test",
        "chain": []
    }]
}

If you actually want the task to do something, you need to put plugins in its chain. While only the order of mutators matters, it’s best to sort the chain putting all the input plugins at the beginning, then all of the mutators in the right order, and finally all of the output plugins.