fuefit fits engine-maps on physical parameters¶

Overview¶

The Fuefit calculator was developed to apply a statistical fit on measured engine fuel consumption data (engine map). This allows the reduction of the information necessary to describe an engine fuel map from several hundred points to seven statistically calculated parameters, with limited loss of information.

More specifically this software works like that:

1. Accepts engine data as input, constituting of triplets of RPM, Power and Fuel-Consumption or equivalent quantities eg mean piston speed (CM), brake mean effective pressure (BMEP) or Torque, fuel mean effective pressure (PMF).
2. Fits the provided input to the following formula [1] [2] [3]:

\[\mathbf{BMEP} = (a + b\times{\mathbf{CM}} + c\times{\mathbf{CM^2}})\times{\mathbf{PMF}} + (a2 + b2\times{\mathbf{CM}})\times{\mathbf{PMF^2}} + loss0 + loss2\times{\mathbf{CM^2}}\]

3. Recalculates and (optionally) plots engine-maps based on the coefficients that describe the fit:

   \[a, b, c, a2, b2, loss0, loss2\]

An “execution” or a “run” of a calculation along with the most important pieces of data are depicted in the following diagram:

              .----------------------------.                    .-----------------------------.
             /        Input-Model         /                    /    Output(Fitted)-Model     /
            /----------------------------/                    /-----------------------------/
           / +--engine                  /                    / +--engine                   /
          /  |  +--...                 /                    /  |  +--fc_map_coeffs        /
         /   +--params                /  ____________      /   +--measured_eng_points    /
        /    |  +--...               /  |            |    /    |    n   p  fc  bmep ... /
       /     +--measured_eng_points /==>| Calculator |==>/     |  ... ... ...  ...     /
      /          n    p    fc      /    |____________|  /      +--fitted_eng_points   /
     /          --  ----  ---     /                    /       |    n    p   fc      /
    /            0   0.0    0    /                    /        |  ...  ...  ...     /
   /           600  42.5   25   /                    /         +--mesh_eng_points  /
  /           ...    ...  ...  /                    /               n    p   fc   /
 /                            /                    /              ...  ...  ...  /
'----------------------------'                    '-----------------------------'

Apart from various engine-characteristics under /engine the table-columns such as capacity and p_rated, the table under /measured_eng_points must contain at least one column from each of the following categories (column-headers are case-insensitive):

1. Engine-speed:

   N        [1/min]
   N_norm   [-]        : where N_norm = (N – N_idle) / (N_rated-N_idle)
   CM       [m/sec]
   

2. Load-Power-capability:

   P        [kW]
   P_norm   [-]        : where P_norm = P/P_MAX
   T        [Nm]
   BMEP     [bar]
   

3. Fuel-consumption:

   FC       [g/h]
   FC_norm  [g/KWh]    : where FC_norm = FC[g/h] / P_MAX [kW]
   PMF      [bar]
   

The Input & fitted data-model described above are trees of strings and numbers, assembled with:

• sequences,
• dictionaries,
• pandas.DataFrame,
• pandas.Series.

Quick-start¶

The program runs on Python-3.3+ and requires numpy/scipy, pandas and win32 libraries along with their native backends to be installed.

On Windows/OS X, it is recommended to use one of the following “scientific” python-distributions, as they already include the native libraries and can install without administrative priviledges:

• WinPython (Windows only),
• Anaconda,
• Canopy,

Assuming you have a working python-environment, open a command-shell (in Windows use cmd.exe BUT ensure python.exe is in its PATH) and try the following console-commands:

$ pip install fuefit
$ fuefit --winmenus                         ## Adds StartMenu-items, Windows only.

$ fuefit --version
0.0.7-alpha.1

$ fuefit --help
...

## Change-directory into the `fuefit/test/` folder in the  *sources*.
$ fuefit -I FuelFit_real.csv header+=0 \
    -I ./FuelFit.xlsx sheetname+=0 header@=None names:='["p","n","fc"]' \
    -I ./engine.csv file_frmt=SERIES model_path=/engine header@=None \
    -m /engine/fuel=petrol \
    -m /params/plot_maps@=True \
    -O full_results_model.json \
    -O fit_coeffs.csv model_path=/engine/fc_map_coeffs   index?=false \
    -O t1.csv model_path=/measured_eng_points   index?=false \
    -O t2.csv model_path=/mesh_eng_points       index?=false \

$ fuefit --excelrun                                             ## Windows & OS X only

>>> import pandas as pd
>>> from fuefit import datamodel, processor, test

>>> inp_model = datamodel.base_model()
>>> inp_model.update({...})                                     ## See "Python Usage" below.        
>>> inp_model['engine_points'] = pd.read_csv('measured.csv')    ## Pandas can read Excel, matlab, ... 
>>> datamodel.set_jsonpointer(inp_model, '/params/plot_maps', True)

>>> datamodel.validade_model(inp_model, additional_properties=False)            

>>> out_model = processor.run(inp_model)                                        

>>> print(datamodel.resolve_jsonpointer(out_model, '/engine/fc_map_coeffs'))    
a            164.110667
b           7051.867419
c          63015.519469
a2             0.121139
b2          -493.301306
loss0      -1637.894603
loss2   -1047463.140758
dtype: float64

* git, git-completion
* make, zip, unzip, bzip2
* openssh, curl, wget

Discussion¶

Installing from sources (for advanced users familiar with git)¶

If you download the sources you have more options for installation. There are various methods to get hold of them:

• Download and extract a release-snapshot from github.

• Download and extract a sdist source distribution from |pypi|_.

• Clone the git-repository at github. Assuming you have a working installation of git you can fetch and install the latest version of the project with the following series of commands:

  $ git clone "https://github.com/ankostis/fuefit.git" fuefit.git
  $ cd fuefit.git
  $ python setup.py install                                 ## Use `python3` if both python-2 & 3 installed.
  

When working with sources, you need to have installed all libraries that the project depends on. Particularly for the latest WinPython environments (Windows / OS X) you can install the necessary dependencies with:

$ pip install -r requirements/execution.txt .

The previous command installs a “snapshot” of the project as it is found in the sources. If you wish to link the project’s sources with your python environment, install the project in development mode:

$ python setup.py develop

Anaconda install¶

The installation to Anaconda (ie OS X) works without any differences from the pip procedure described so far.

To install it on miniconda environment, you need to install first the project’s native dependencies (numpy/scipy), so you need to download the sources (see above). Then open a bash-shell inside them and type the following commands:

$ coda install `cat requirements/miniconda.txt`
$ pip install lmfit             ## Workaround lmfit-py#149
$ python setup.py install
$ fuefit --version
0.0.7-alpha.1

Discussion¶

Excel usage¶

In Windows and OS X you may utilize the xlwings library to use Excel files for providing input and output to the program.

To create the necessary template-files in your current-directory, type this console-command:

$ fuefit --excel

Type fuefit --excel file_path if you want to specify a different destination path.

In windows/OS X you can type fuefit --excelrun and the files will be created in your home-directory and the Excel will immediately open them.

What the above commands do is to create 2 files:

FuefitExcelRunner#.xlsm

The python-enabled excel-file where input and output data are written, as seen in the screenshot below:

After opening it the first tie, enable the macros on the workbook, select the python-code at the left and click the Run Selection as Pyhon button; one sheet per vehicle should be created.

The excel-file contains additionally appropriate VBA modules allowing you to invoke Python code present in selected cells with a click of a button, and python-functions declared in the python-script, below, using the mypy namespace.

To add more input-columns, you need to set as column Headers the json-pointers path of the desired model item (see Python usage below,).

FuefitExcelRunner#.py

Python functions used by the above xls-file for running a batch of experiments.

The particular functions included reads multiple vehicles from the input table with various vehicle characteristics and/or experiment coefficients, and then it adds a new worksheet containing the cycle-run of each vehicle . Of course you can edit it to further fit your needs.

$ python FuefitExcelRunner.py

Some general notes regarding the python-code from excel-cells:

• An elaborate syntax to reference excel cells, rows, columns or tables from python code, and to read them as pandas.DataFrame is utilized by the Excel . Read its syntax at resolve_excel_ref().
• On each invocation, the predefined VBA module pandalon executes a dynamically generated python-script file in the same folder where the excel-file resides, which, among others, imports the “sister” python-script file. You can read & modify the sister python-script to import libraries such as ‘numpy’ and ‘pandas’, or pre-define utility python functions.
• The name of the sister python-script is automatically calculated from the name of the Excel-file, and it must be valid as a python module-name. Therefore: * Do not use non-alphanumeric characters such as spaces(` ), dashes(-) and dots(.`) on the Excel-file. * If you rename the excel-file, rename also the python-file, or add this python import <old_py_file> as mypy`
• On errors, a log-file is written in the same folder where the excel-file resides, for as long as the message-box is visible, and it is deleted automatically after you click ‘ok’!
• Read http://docs.xlwings.org/quickstart.html

Cmd-line usage¶

Example command:

fuefit -v\
  -I fuefit/test/FuelFit.xlsx sheetname+=0 header@=None names:='["p","rpm","fc"]' \
  -I fuefit/test/engine.csv file_frmt=SERIES model_path=/engine header@=None \
  -m /engine/fuel=petrol \
  -O ~t2.csv model_path=/fitted_eng_points    index?=false \
  -O ~t2.csv model_path=/mesh_eng_points      index?=false \
  -O ~t.csv model_path= -m /params/plot_maps@=True

Python usage¶

The most powerful way to interact with the project is through a python REPL. So fire-up a python or ipython shell and first try to import the project just to check its version:

>>> import fuefit

>>> fuefit.__version__                ## Check version once more.
'0.0.7-alpha.1'

>>> fuefit.__file__                   ## To check where it was installed.         
/usr/local/lib/site-package/fuefit-...

If the version was as expected, take the base-model and extend it with your engine-data (strings and numbers):

>>> from fuefit import datamodel, processor

>>> inp_model = datamodel.base_model()
>>> inp_model.update({
...     "engine": {
...         "fuel":     "diesel",
...         "p_max":    95,
...         "n_idle":   850,
...         "n_rated":  6500,
...         "stroke":   94.2,
...         "capacity": 2000,
...         "bore":     None,       ##You do not have to include these,
...         "cylinders": None,      ##  they are just for displaying some more engine properties.
...     }
... })

>>> import pandas as pd
>>> df = pd.read_excel('fuefit/test/FuelFit.xlsx', 0, header=None, names=["n","p","fc"])
>>> inp_model['measured_eng_points'] = df

For information on the accepted model-data, check both its JSON-schema at model_schema(), and the base_model():

Next you have to validate it against its JSON-schema:

>>> datamodel.validate_model(inp_model, additional_properties=False)

If validation is successful, you may then feed this model-tree to the fuefit.processor, to get back the results:

>>> out_model = processor.run(inp_model)

>>> print(datamodel.resolve_jsonpointer(out_model, '/engine/fc_map_coeffs'))
a            164.110667
b           7051.867419
c          63015.519469
a2             0.121139
b2          -493.301306
loss0      -1637.894603
loss2   -1047463.140758
dtype: float64

>>> print(out_model['fitted_eng_points'].shape)
(262, 11)

Discussion¶

Sources & Dependencies¶

To get involved with development, you need a POSIX environment to fully build it (Linux, OSX, or Cygwin on Windows).

Development team¶

• Kostis Anagnostopoulos (software design & implementation)
• Georgios Fontaras (methodology inception, engineering support & validation)

Discussion¶

General¶

Technical¶

Discussion¶

Core¶

ExcelRunner¶

Tests¶

Module: fuefit.datamodel¶

fuefit.datamodel.base_model()[source]¶

The base model for running a WLTC experiment.

It contains some default values for the experiment (ie the default ‘full-load-curve’ for the vehicles). But note that it this model is not valid - you need to override its attributes.

:return :json_tree: with the default values for the experiment.

fuefit.datamodel.jsonpointer_parts(jsonpointer)[source]¶

Iterates over the jsonpointer parts.

Parameters:	jsonpointer (str) – a jsonpointer to resolve within document
Returns:	a generator over the parts of the json-pointer
Author:	Julian Berman, ankostis

fuefit.datamodel.merge(a, b, path=, []list_merge_mode=<MergeMode.REPLACE: 1>, raise_struct_mismatches=False)[source]¶

‘Merges b into a.

List merge modes: REPLACE, APPEND_HEAD, APPEND_TAIL, OVERLAP_HEAD, OVERLAP_TAIL

fuefit.datamodel.model_schema(additional_properties=False)[source]¶

The json-schema for input/output of the fuefit experiment.

:return :dict:

fuefit.datamodel.resolve_jsonpointer(doc, jsonpointer, default=<object object at 0x7fefef9a75e0>)[source]¶

Resolve a jsonpointer within the referenced doc.

Parameters:	doc – the referrant document jsonpointer (str) – a jsonpointer to resolve within document
Returns:	the resolved doc-item or raises JsonPointerException
Author:	Julian Berman, ankostis

fuefit.datamodel.set_jsonpointer(doc, jsonpointer, value, object_factory=<class 'dict'>)[source]¶

Resolve a jsonpointer within the referenced doc.

Parameters:	doc – the referrant document jsonpointer (str) – a jsonpointer to the node to modify
Raises:	JsonPointerException (if jsonpointer empty, missing, invalid-contet)

Module: fuefit.processor¶

The core calculations required for transforming the Input-datamodel to the Output one.

Uses pandalon‘s automatic dependency extraction from calculation functions.

fuefit.processor._robust_residualfunc(coeffs, modelfunc, X, YData, is_robust=False, robust_prcntile=None)[source]¶

A non-linear iteratively-reweighted least-squares (IRLS) residual function (objective-function) that robustly fits YData = modelfunc(X).

This method applies weights on each iteration so as to downscale any outliers and high-leverage data-points based on the ‘bisquare’ standardized adjusted residuals:[1]

rac{r}{K imes hat{sigma} imes sqrt{1 - h}}

where:

\(r\) (vector)

the residuals \(\hat{y} - y\)

\(K\) (scalar)

the robust percentile tuning constant used on each iteration to filter-out adjusted-standardized-weights above 1, expressed as the Bisquare M-estimator efficiency under Gaussian model.

\(\hat{\sigma}\) (scalar)

the robust estimate of the standard deviation of the residuals based on MAD[#]_ like this: \(\hat{\sigma}=1.4826 imes\operatorname{MAD}\)

\(h\) : (vector)

the hat vector, the diagonal of the hat matrix,[#]_ which is used to reduce the weight of high-leverage data points that are having a large effect on the least-squares fit.

param modelfunc:
	The modeling function that accepts the dict of coeffs
param nparray X:
param nparray YData:
	measured-data points
param boolean is_robust:
	Whether to deleverage outlier YData.
param float robust_prcntile:
	The K percentile of the MAD, [default: 4.68, filters-out approximately 5% of the residuals as outliers]

See also

curve_fit, leastsq

[1]	http://www.mathworks.com/help/stats/robustfit.html

[2]	https://en.wikipedia.org/wiki/Median_absolute_deviation

[3]	https://en.wikipedia.org/wiki/Hat_matrix

fuefit.processor.eng_points_2_std_map(params, engine, eng_points)[source]¶

A factory of the calculation functions for reaching to the data necessary for the Fitting.

The order of the functions below not important, and the actual order of execution is calculated from their dependencies, based on the data-frame’s column access.

fuefit.processor.engine_map_modelfunc(coeff_values, X)[source]¶

The function that models the engine-map.

fuefit.processor.run(mdl, opts=None)[source]¶

Parameters:	mdl – the datamodel to process, all params and data opts (map) – flags controlling non-functional aspects of the process (ie error-handling and logging, gui, etc)

Module: fuefit.pdcalc¶

A best-effort attempt to build computation dependency-graphs from method with dict-like objects (such as pandas), inspired by XForms: http://lib.tkk.fi/Diss/2007/isbn9789512285662/article3.pdf

class fuefit.pdcalc.Dependencies[source]¶

Bases: builtins.object

Discovers and stores the rough functions-relationships needed to produce ExecutionPlanner

The relation-tuples are “rough” in the sense that they may contain duplicates etc requiring cleanup by _consolidate_relations().

Usage:

Use the harvest_{XXX}() methods or the add_func_rel() method to gather dependencies from functions and function-facts and then use the build_plan() method to freeze them into a plan.

add_func_rel(item, deps=None, func=None)[source]¶

Parameters:	item (str) – a dotted.var deps – a string or a sequence of dotted.vars func – a standalone func, or a funcs_factory as a 2-tuple (func, index)

build_plan(sources, dests)[source]¶

Builds an execution-plan that when executed with execute_plan() it will produce dests from sources.

Turns any stored-relations into a the full dependencies-graph then trims it only to those ‘’dotted.varname’’ nodes reaching from sources to dests.

Parameters:	dests (sequence) – a list of ‘’dotted.varname’’s that will be the outcome of the calculations sources – a list of ‘’dotted.varname’’s (existent or not) that are assumed to exist when the execution wil start
Returns:	the new plan that can be fed to execute_plan()
Return type:	pd.Series

classmethod from_funcs_map(funcs_map, deps=None)[source]¶

Factory method for building Dependencies by harvesting multiple functions and func_factories, at once

Parameters:

funcs_map – a mapping or a sequence of pairs (what --> bool_or_null) with values: True when what is a funcs_factory, a function returning a sequence of functions processing the data False when what is a standalone_function, or None when what is an explicit relation 3-tuple (item, deps, func) to be fed directly to Dependencies.add_func_rel(), where: item a ‘’dotted.varname’’ string, deps a string or a sequence of ‘’dotted.varname’’ strings, and func a standalone func or a funcs_factory as a 2-tuple (func, index) Both items and deps are dot-separated sequence of varnames, such as ‘foo.bar’ deps – if none, a new Dependencies instance is created to harvest relations

Returns:

a new (even inherited) or an updated Dependencies instance

Important

All functions must accept exactly the same args, or else the results will be undetermined and it will most-likely scream on execute_funcs_map.

class fuefit.pdcalc._DepFunc(func, is_funcs_factory=False, _child_index=None)[source]¶

Bases: builtins.object

A wrapper for functions explored for relations, optionally allowing them to form a hierarchy of factories and produced functions.

It can be in 3 types:

• 0, standalone function: args given to function invocation are used immediatelt,
• 10, functions-factory: args are stored and will be used by the child-funcs returned,
• 20, child-func: created internally, and no args given when invoked, will use parent-factory’s args.

Use factory methods to create one of the first 2 types:

• pdcalc._wrap_standalone_func()
• pdcalc._wrap_funcs_factory()

fuefit.pdcalc._consolidate_relations(relations)[source]¶

(item1, deps, func), (item1, ...) –> {item1, (set(deps), set(funcs))}

fuefit.pdcalc._filter_common_prefixes(deps)[source]¶

deps: not-empty set

example::

deps = [‘a’, ‘a.b’, ‘b.cc’, ‘a.d’, ‘b’, ‘ac’, ‘a.c’] res = _filter_common_prefixes(deps) assert res == [‘a.b’, ‘a.c’, ‘a.d’, ‘ac’, ‘b.cc’]

fuefit.pdcalc._find_missing_input(calc_inp_nodes, graph)[source]¶

Search for tentatively missing data.

fuefit.pdcalc._harvest_indexing(index)[source]¶

Harvest any strings, slices, etc, assuming to be DF’s indices.

fuefit.pdcalc._harvest_mock_call(mock_call, func, deps_set, func_rels)[source]¶

Adds a 2-tuple (indep, [deps]) into indeps with all deps collected so far when it visits a __setartr__.

fuefit.pdcalc._research_calculation_routes(graph, sources, dests)[source]¶

Find nodes reaching ‘dests’ but not ‘sources’.

sources: a list of nodes (existent or not) to search for all paths originating from them dests: a list of nodes to search for all paths leading to them them return: a 2-tuple with the graph and its nodes topologically-ordered

fuefit.pdcalc._strip_magic_tail(path)[source]¶

some.path___with_.__magics__ –> some.path

fuefit.pdcalc.calculation(deps=None)[source]¶

A decorator that extracts dependencies from a function.

Parameters:	deps – if None, any 2 extracted deps are added in the calculation_deps module-variable.

Example:

@calculation
def foo(a, r):
    a['bc'] = a.aa
    r['r1'] = a.bb

@calculation
def bar(a, r):
    r['r2'] = a.bc

plan    = calculation_deps.build_plan(['a.ab', 'a.bc', 'r'], ['r.r2'])
planner.execute_funcs_map(plan, 1, 2)

fuefit.pdcalc.default_arg_paths_extractor(arg_name, arg, paths)[source]¶

Dig recursively objects for indices to build the sources/dests sequences for Dependencies.build_plan()

It extracts indices recursively from maps and pandas The inner-ones paths are not added, ie df.some.key, but not df or df.some.

Note

for pandas-series their index (their infos-axis) gets appended only the 1st time (not if invoked in recursion, from DataFrame columns).

Parameters:	paths (list) – where to add the extracted paths into

fuefit.pdcalc.execute_funcs_factory(funcs_fact, dests, *args, **kwargs)[source]¶

A one-off way to run calculations from a funcions-factory (see execute_funcs_map())

fuefit.pdcalc.execute_funcs_map(funcs_map, dests, *args, **kwargs)[source]¶

A one-off way to run calculations from a funcs_map as defined in Dependencies:from_funcs_map().

Parameters:

funcs_map – a dictionary similar to the one used by Dependencies.from_funcs_map() dests (seq) – what is required as the final outcome of the execution, ie: for the func-functory: def some_funcs(foo, bar): def func_1(): foo['a1'] = ... ... def func_n(): ... foo['a2'] = ... bar['b1'] = ... return [func_1, ..., func_n] the dests may be: ['foo.a1', 'foo.a2', 'bar.b1'] args – the actual args to use for invoking the functions in the map, and the names of these args would come rom the first function to be invoked (which ever that might be!).

Note

Do not use it to run the calculations repeatedly. Preffer to cache the function-relations into an intermediate Dependencies instance, instead.

fuefit.pdcalc.tell_paths_from_named_args(named_args, arg_paths_extractor_func=<function default_arg_paths_extractor at 0x7fefea4bb510>, paths=None)[source]¶

Builds the sources or the dests sequences params of Dependencies.build_plan() from a map of function-arguments

Parameters:	named_args – an ordered map {param_name --> param_value} similar to that returned by inspect.signature(func).bind(*args).arguments: BoundArguments. Use the utility name_all_func_args() to generate such a map for some function.
Returns:	the paths updated with all ‘’dotted.vars’’ found

Module: fuefit.excel.FuefitExcelRunner¶

Module: fuefit.test.cmdline_test¶

Check cmdline parsing and model building.

Releases¶

Index¶