Lucidity¶
Filesystem templating and management.
Guide¶
Overview and examples of using the system in practice.
Introduction¶
Lucidity is a framework for templating filesystem structure.
It works using regular expressions, but hides much of the verbosity through the use of simple placeholders (such as you see in Python string formatting).
Consider the following paths:
/jobs/monty/assets/circus/model/high/circus_high_v001.abc
/jobs/monty/assets/circus/model/low/circus_low_v001.abc
/jobs/monty/assets/parrot/model/high/parrot_high_v002.abc
A regular expression to describe them might be:
'/jobs/(?P<job>[\w_]+?)/assets/(?P<asset_name>[\w_]+?)/model/(?P<lod>[\w_]+?)/(?P<asset_name>[\w_]+?)_(?P<lod>[\w_]+?)_v(?P<version>\d+?)\.(?P<filetype>\w+?)'
Meanwhile, the Lucidity pattern would be:
'/jobs/{job}/assets/{asset_name}/model/{lod}/{asset_name}_{lod}_v{version}.{filetype}'
With Lucidity you store this pattern as a template and can then use that template to generate paths from data as well as extract data from matching paths in a standard fashion.
Read the Tutorial to find out more.
Copyright & License¶
Copyright (c) 2013 Martin Pengelly-Phillips
Licensed under the Apache License, Version 2.0 (the “License”); you may not use this work except in compliance with the License. You may obtain a copy of the License in the LICENSE.txt file, or at:
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Installation¶
Installing Lucidity is simple with pip:
$ pip install lucidity
If the Cheeseshop (a.k.a. PyPI) is down, you can also install Lucidity from one of the mirrors:
$ pip install --use-mirrors lucidity
Alternatively, you may wish to download manually from Gitlab where Lucidity is actively developed.
You can clone the public repository:
$ git clone git@gitlab.com:4degrees/lucidity.git
Or download an appropriate zipball
Once you have a copy of the source, you can embed it in your Python package, or install it into your site-packages:
$ python setup.py install
Tutorial¶
This tutorial gives a good introduction to using Lucidity.
First make sure that you have Lucidity installed.
Patterns¶
Lucidity uses patterns to represent a path structure. A pattern is very much like the format you would use in a Python string format expression.
For example, a pattern to represent this filepath:
'/jobs/monty/assets/circus/model/high/circus_high_v001.abc'
Could be:
'/jobs/{job}/assets/{asset_name}/model/{lod}/{asset_name}_{lod}_v{version}.{filetype}'
Each {name}
in braces is a variable that can either be extracted from a
matching path, or substituted with a provided value when constructing a path.
The variable is referred to as a placeholder.
Templates¶
A Template
is a simple container for a pattern.
First, import the package:
>>> import lucidity
Now, construct a template with the pattern above:
>>> template = lucidity.Template('model', '/jobs/{job}/assets/{asset_name}/model/{lod}/{asset_name}_{lod}_v{version}.{filetype}')
Note
The template must be given a name to identify it. The name becomes useful when you have a bunch of templates to manage.
You can check the identified placeholders in a template using the
Template.keys
method:
>>> print template.keys()
set(['job', 'asset_name', 'lod', 'version', 'filetype'])
Parsing¶
With a template defined we can now parse a path and extract data from it:
>>> path = '/jobs/monty/assets/circus/model/high/circus_high_v001.abc'
>>> data = template.parse(path)
>>> print data
{
'job': 'monty',
'asset_name': 'circus',
'lod': 'high',
'version': '001',
'filetype': 'abc'
}
If a template’s pattern does not match the path then
parse()
will raise a
ParseError
:
>>> print template.parse('/other/monty/assets')
ParseError: Input '/other/monty/assets' did not match template pattern.
Handling Duplicate Placeholders¶
It is perfectly acceptable for a template to contain the same placeholder multiple times, as seen in the template constructed above. When parsing, by default, the last matching value for a placeholder is used:
>>> path = '/jobs/monty/assets/circus/model/high/spaceship_high_v001.abc'
>>> data = template.parse(path)
>>> print data['asset_name']
spaceship
This is called RELAXED
mode. If this
behaviour is not desirable then the duplicate_placeholder_mode of any
Template
can be set to
STRICT
mode instead:
>>> path = '/jobs/monty/assets/circus/model/high/spaceship_high_v001.abc'
>>> template.duplicate_placeholder_mode = template.STRICT
>>> template.parse(path)
ParseError: Different extracted values for placeholder 'asset_name' detected. Values were 'circus' and 'spaceship'.
Note
duplicate_placeholder_mode can also be passed as an argument when constructing a template.
Anchoring¶
By default, a pattern is anchored at the start, requiring that the start of a path match the pattern:
>>> job_template = lucidity.Template('job', '/job/{job}')
>>> print job_template.parse('/job/monty')
{'job': 'monty'}
>>> print job_template.parse('/job/monty/extra/path')
{'job': 'monty'}
>>> print job_template.parse('/other/job/monty')
ParseError: Input '/other/job/monty' did not match template pattern.
The anchoring can be changed when constructing a template by passing an anchor keyword in:
>>> filename_template = lucidity.Template(
... 'filename',
... '{filename}.{index}.{ext}',
... anchor=lucidity.Template.ANCHOR_END
... )
>>> print filename_template.parse('/some/path/to/file.0001.dpx')
{'filename': 'file', 'index': '0001', 'ext': 'dpx'}
The anchor can be one of:
ANCHOR_START
- Match pattern at the start of the string.ANCHOR_END
- Match pattern at the end of the string.ANCHOR_BOTH
- Match pattern exactly.None
- Match pattern once anywhere in the string.
Formatting¶
It is also possible to pass a dictionary of data to a template in order to produce a path:
>>> data = {
... 'job': 'monty',
... 'asset_name': 'circus',
... 'lod': 'high',
... 'version': '001',
... 'filetype': 'abc'
... }
>>> path = template.format(data)
>>> print path
/jobs/monty/assets/circus/model/high/circus_high_v001.abc
In the example above, we haven’t done more than could be achieved with standard
Python string formatting. In the next sections, though, you will see the need
for a dedicated format()
method.
If the supplied data does not contain enough information to fill the template
completely a FormatError
will be raised:
>>> print template.format({})
FormatError: Could not format data {} due to missing key 'job'.
Nested Data Structures¶
Often the data structure you want to use will be more complex than a single level dictionary. Therefore, Lucidity also supports nested dictionaries when both parsing or formatting a path.
To indicate a nested structure, use a dotted notation in your placeholder name:
>>> template = lucidity.Template('job', '/jobs/{job.code}')
>>> print template.parse('/jobs/monty')
{'job': {'code': 'monty'}}
>>> print template.format({'job': {'code': 'monty'}})
/jobs/monty
Note
Unlike the standard Python format syntax, the dotted notation in Lucidity always refers to a nested item structure rather than attribute access.
Custom Regular Expressions¶
Lucidity works by constructing a regular expression from a pattern. It replaces all placeholders with a default regular expression that should suit most cases.
However, if you need to customise the regular expression you can do so either at a template level or per placeholder.
At The Template Level¶
To modify the default regular expression for a template, pass it is as an additional argument:
>>> template = lucidity.Template('name', 'pattern',
default_placeholder_expression='[^/]+')
Per Placeholder¶
To alter the expression for a single placeholder, use a colon :
after the
placeholder name and follow with your custom expression:
>>> template = lucidity.Template('name', 'file_v{version:\d+}.ext')
Above, the version placeholder expression has been customised to only match one or more digits.
Note
If your custom expression requires the use of braces ({}
) you must
escape them to distinguish them from the placeholder braces. Use a
preceding backslash for the escape (\{
, \}
).
And of course, any custom expression text is omitted when formatting data:
>>> print template.format({'version': '001'})
file_v001.ext
Managing Multiple Templates¶
Representing different path structures requires the use of multiple templates.
Lucidity provides a few helper functions for dealing with multiple templates.
Template Discovery¶
Templates can be discovered by searching a list of paths for
mount points that register template
instances. By default, the list of paths is retrieved from the environment
variable LUCIDITY_TEMPLATE_PATH
.
To search and load templates in this way:
>>> import lucidity
>>> templates = lucidity.discover_templates()
To specify a specific list of paths just pass them to the function:
>>> templates = lucidity.discover_templates(paths=['/templates'])
By default each path will be recursively searched. You can disable this
behaviour by setting the recursive
keyword argument:
>>> templates = lucidity.discover_templates(recursive=False)
Template Mount Points¶
To write a template mount point, define a Python file containing a register
function. The function should return a list of instantiated
Template
instances:
# templates.py
from lucidity import Template
def register():
'''Register templates.'''
return [
Template('job', '/jobs/{job.code}'),
Template('shot', '/jobs/{job.code}/shots/{scene.code}_{shot.code}')
]
Place the file on one of the search paths for
discover_templates()
to have it take effect.
Operations Against Multiple Templates¶
Lucidity also provides two top level functions to run a
parse
or format
operation against multiple candidate templates using the first correct result
found.
Given the following templates:
>>> import lucidity
>>> templates = [
... lucidity.Template('model', '/jobs/{job}/assets/model/{lod}'),
... lucidity.Template('rig', '/jobs/{job}/assets/rig/{rig_type}')
... ]
To perform a parse:
>>> print lucidity.parse('/jobs/monty/assets/rig/anim', templates)
({'job': 'monty', 'rig_type': 'anim'},
Template(name='rig', pattern='/jobs/{job}/assets/rig/{rig_type}'))
To format data:
>>> print lucidity.format({'job': 'monty', 'rig_type': 'anim'}, templates)
('/jobs/monty/assets/rig/anim',
Template(name='rig', pattern='/jobs/{job}/assets/rig/{rig_type}'))
Note
The return value is a tuple of (result, template)
.
If no template could provide a result an appropriate error is raised (
ParseError
or
FormatError
).
Using Template References¶
When the same pattern is used repetitively in several templates, it can be useful to extract it out into a separate template that can be referenced.
To reference another template, use its name prefixed by the at symbol, @, in a placeholder:
>>> shot_path = lucidity.Template('shot_path', '{@job_path}/shots/{shot.code}')
Template references are resolved on demand when performing operations with the
template, such as calling Template.parse()
or Template.format()
.
This is why the above didn’t produce an error even though no job_path template
has been defined (or a way to lookup that template by name even). This
behaviour allows discovery of templates
without worrying about the order of template construction.
As soon as you attempt to perform an operation on the template that does require
resolving references, errors will be raised accordingly. For example, try
calling the Template.keys()
method:
>>> print shot_path.keys()
ResolveError: Failed to resolve reference 'job_path' as no template resolver set.
The error indicates that we have not provided the template with a way to
actually resolve template references. To do this we need to set the
template_resolver attribute on the template to an object that matches the
Resolver
interface. Fortunately, the resolver interface is simple so
a even a basic dictionary can act as a resolver:
>>> resolver = {}
>>> shot_path.template_resolver = resolver
Note
The template resolver can also be passed as an argument when instantiating a
new Template
.
Try getting the keys again:
>>> print shot_path.keys()
ResolveError: Failed to resolve reference 'job_path' using template resolver.
Slightly better. Now we have a resolver in place we just need to add the other template to the resolver so that it can be looked up by name:
>>> job_path = lucidity.Template('job_path', '/jobs/{job.code}')
>>> resolver[job_path.name] = job_path
Note
In a future release a dedicated template collection class will make dealing with template references even easier.
Print the keys again and it should resolve all the references and give back the full list of keys that make up the expanded template:
>>> print shot_path.keys()
set(['job.code', 'shot.code'])
There is also a method for listing the references found in a template:
>>> print shot_path.references()
set(['job_path'])
Additionally, if you would like to see the fully expanded pattern you can
manually call the Template.expanded_pattern()
method at any time:
>>> print shot_path.expanded_pattern()
/jobs/{job.code}/shots/{shot.code}
Anchor behaviour¶
A Template
has an anchor setting that
determines how the template pattern is matched when parsing. When a template is
referenced in another template its anchor setting is ignored and only the anchor
setting of the outermost template is used:
>>> template_a = lucidity.Template(
... 'a', 'path/{variable}', anchor=lucidity.Template.ANCHOR_START
... )
>>> print template_a.parse('/some/path/value')
ParseError: Path '/some/path/value' did not match template pattern.
>>> resolver = {}
>>> resolver[template_a.name] = template_a
>>> template_b = lucidity.Template(
... 'b', '{@a}', anchor=lucidity.Template.ANCHOR_END,
... template_resolver=resolver
... )
>>> print template_b.parse('/some/path/value')
{'variable': 'value'}
Reference¶
API reference providing details on the actual code.
lucidity
¶
-
lucidity.
discover_templates
(paths=None, recursive=True)[source]¶ Search paths for mount points and load templates from them.
paths should be a list of filesystem paths to search for mount points. If not specified will try to use value from environment variable
LUCIDITY_TEMPLATE_PATH
.A mount point is a Python file that defines a ‘register’ function. The function should return a list of instantiated
Template
objects.If recursive is True (the default) then all directories under a path will also be searched.
-
lucidity.
parse
(path, templates)[source]¶ Parse path against templates.
path should be a string to parse.
templates should be a list of
Template
instances in the order that they should be tried.Return
(data, template)
from first successful parse.Raise
ParseError
if path is not parseable by any of the supplied templates.
-
lucidity.
format
(data, templates)[source]¶ Format data using templates.
data should be a dictionary of data to format into a path.
templates should be a list of
Template
instances in the order that they should be tried.Return
(path, template)
from first successful format.Raise
FormatError
if data is not formattable by any of the supplied templates.
-
lucidity.
get_template
(name, templates)[source]¶ Retrieve a template from templates by name.
Raise
NotFound
if no matching template with name found in templates.
template
¶
-
class
lucidity.template.
Template
(name, pattern, anchor=1, default_placeholder_expression='[\w_.\-]+', duplicate_placeholder_mode=1, template_resolver=None)[source]¶ Bases:
object
A template.
-
ANCHOR_START
= 1¶
-
ANCHOR_END
= 2¶
-
ANCHOR_BOTH
= 3¶
-
RELAXED
= 1¶
-
STRICT
= 2¶
-
__init__
(name, pattern, anchor=1, default_placeholder_expression='[\\w_.\\-]+', duplicate_placeholder_mode=1, template_resolver=None)[source]¶ Initialise with name and pattern.
anchor determines how the pattern is anchored during a parse. A value of
ANCHOR_START
(the default) will match the pattern against the start of a path.ANCHOR_END
will match against the end of a path. To anchor at both the start and end (a full path match) useANCHOR_BOTH
. Finally,None
will try to match the pattern once anywhere in the path.duplicate_placeholder_mode determines how duplicate placeholders will be handled during parsing.
RELAXED
mode extracts the last matching value without checking the other values.STRICT
mode ensures that all duplicate placeholders extract the same value and raisesParseError
if they do not.If template_resolver is supplied, use it to resolve any template references in the pattern during operations. It should conform to the
Resolver
interface. It can be changed at any time on the instance to affect future operations.
-
name
¶ Return name of template.
-
pattern
¶ Return template pattern.
-
expanded_pattern
()[source]¶ Return pattern with all referenced templates expanded recursively.
Raise
lucidity.error.ResolveError
if pattern contains a reference that cannot be resolved by currently set template_resolver.
-
parse
(path)[source]¶ Return dictionary of data extracted from path using this template.
Raise
ParseError
if path is not parsable by this template.
-
format
(data)[source]¶ Return a path formatted by applying data to this template.
Raise
FormatError
if data does not supply enough information to fill the template fields.
-
error
¶
Custom error classes.
-
exception
lucidity.error.
ParseError
[source]¶ Bases:
exceptions.Exception
Raise when a template is unable to parse a path.
-
exception
lucidity.error.
FormatError
[source]¶ Bases:
exceptions.Exception
Raise when a template is unable to format data into a path.
Release and migration notes¶
Find out what has changed between versions and see important migration notes to be aware of when switching to a new version.
Release Notes¶
1.5.1¶
20 October 2018- fixed
Specify dependencies using case sensitive names to support systems that cannot resolve in a case insensitive manner (such as “Nexus Repository”).
1.5.0¶
6 June 2015- new
Support for referencing a template in another template.
See also
- new
Release notes in documentation using Lowdown.
1.4.1¶
26 May 2015- changed
Implemented custom formatter internally, removing dependency on Bunch and simplifying format logic.
1.4.0¶
25 May 2015- new
Added duplicate_placeholder_mode to control template behaviour when parsing templates with duplicate placeholders, including a new
STRICT
mode.See also
1.3.0¶
28 March 2014- changed
Removed dependency on Regex module to simplify installation across different platforms.
1.2.0¶
9 March 2014- new
Added
Template.keys()
for retrieving set of placeholders used in a template pattern.
1.1.0¶
8 March 2014- new
Support partial matching of template when parsing with the introduction of a new anchor setting on templates.
See also
- new
Helper function
lucidity.get_template()
for retrieving a template by name from a list of templates. - fixed
Special regex characters not escaped in pattern leading to incorrect parses.
- fixed
Template.format()
fails with unhandledAttributeError
when nested dictionary is missing a required key.
Migration notes¶
This section will show more detailed information when relevant for switching to a new version, such as when upgrading involves backwards incompatibilities.
\ Sort by:\ best rated\ newest\ oldest\
\\
Add a comment\ (markup):
\``code``
, \ code blocks:::
and an indented block after blank line