MasterCard file utilities¶
Contents:
MasterCard file utilities¶
Set of command line utilities to work with various MasterCard files.
- Free software: BSD license
- Documentation: https://mciutil.readthedocs.org.
Features¶
Provides the following command line utilities:
- paramconv: Utility for working with MasterCard MPE parameter extract files
- mideu: Utility for working with MasterCard IPM files
Installation¶
At the command line:
$ easy_install mciutil
Or, if you have virtualenvwrapper installed:
$ mkvirtualenv mciutil
$ pip install mciutil
Usage¶
The mciutil package provides command line tools and python functions you can use when working with MasterCard files.
Note
Make sure set binary transfers when moving MasterCard files. For
example, if using FTP, use the binary
command before starting the
transfer
paramconv - Convert MPE parameter extract files¶
Use this tool if you are moving between a mainframe and PC based clearing application and need the parameter files available on both platforms. The app currently works exclusively with 1014 blocked file format which is used for communications between MasterCard and the member.
Most simple usage, just provide a MasterCard MPE file to convert:
paramconv <inputfile>
This runs with the following assumptions
- Input file format is EBCDIC
- Output file is the input file plus a ‘.out’ extension
If you have a ASCII file and want to convert it to EBCDIC, you need to provide the source format type:
paramconv -s ascii <inputfile>
You can change the output file location:
paramconv -o <outputfile> <inputfile>
To get all the usage details:
paramconv --help
mideu - MasterCard IPM file tool¶
This utility is for working with MasterCard IPM clearing files. Currently the utility provides the following functions:
- extract - provide IPM transaction data in usable format
- convert - convert IPM files between ASCII and EBCDIC encoding
Extract command¶
Use this command to extract transactions from a MasterCard IPM format file into usable formats like csv. The app currently works exclusively with 1014 blocked file format which is used for communications between MasterCard and the member.
Get a csv file from an IPM file:
mideu extract <filename>
This runs with the following assumptions
- Input file format is EBCDIC
- Output file is the input file plus a ‘.csv’ extension
Attention
Currently python 2.6 does not print a header row as the csv library only added this support in python 2.7. This function may be added in a future release.
If you need to process an ASCII encoded file:
mideu extract -s ascii <filename>
You can change the output file location for the CSV file:
mideu extract <inputfile> --csvoutputfile <outputfile>
You can also load the transactions into a MongoDB collection:
mideu extract <inputfile> --mongohost localhost --mongodb testdb
The transactions will be added to a collection called mastercardtransactions
Currently the existing collection is deleted prior to the load.
You should consider this functionality to be beta and subject to change in the
future. Feel free to suggest changes.
To get all the usage details:
mideu extract --help
Convert command¶
Use this command to convert a MasterCard IPM format file into another encoding scheme (ASCII or EBCDIC). The app currently works exclusively with 1014 blocked file format which is used for communications between MasterCard and the member.
Most simple usage, just provide a MasterCard IPM file to convert:
mideu convert <inputfile>
This runs with the following assumptions
- Input file format is EBCDIC
- Output file is the input file plus a ‘.out’ extension
If you have a ASCII file and want to convert it to EBCDIC, you need to provide the source format type:
mideu convert -s ascii <inputfile>
To get all the usage details:
mideu convert --help
mideu.yml configuration¶
The package provides a common configuration, but you can provide your own if you wish to change it. You have 2 options for providing your own configuration.
- mideu.yml file in the current directory
- .mideu.yml file in the users home directory
The file is a basic yaml file with the following sections
- output_data_elements
- Specify fields to output. Set the order and fields to be output.
Structure:
output_data_elements:
- MTI
- DE1
- DE10
- PDS0023
- DE43_NAME
- bit_config
- Define the bitmap fields. You should not need to change this but if you think you need to, have a look at the source to see what the options mean. Will document this in more details at some point.
Structure:
bit_config:
1:
field_name: Bitmap secondary
field_type: FIXED
field_length: 8
2:
field_name: PAN
field_type: LLVAR
field_length: 0
field_processor: PAN
- mongo_config
- Specify mongo host and port details. Command line options will override options provided in a config file
Structure:
mongo_config:
host: 192.168.99.100:27017
db: test
1014 blocked file format¶
This is the format used to send and receive files from MasterCard.
All file consists of records. Each record is prefixed with a 4 byte binary length. There are no carriage returns or line feeds in the file. Before sending, the contents is blocked into lengths of 1012, and an additional 2 x‘40’ characters are appended at each block. Finally, the total file length is made a multiple of 1014 with the final incomplete record being filled with the x‘40’ character
Say you had a file with the following 2 records:
"This is first record 1234567" <- length 28
"This is second record AAAABBBBB123" <- length 34
Add binary length to the start of each record. (x‘1C’ = 28, x‘22’ = 34)
00000000: 00 00 00 1C 54 68 69 73 20 69 73 20 66 69 72 73 ....This is firs
00000010: 74 20 72 65 63 6F 72 64 20 31 32 33 34 35 36 37 t record 1234567
00000020: 00 00 00 22 54 68 69 73 20 69 73 20 73 65 63 6F ..."This is seco
00000030: 6E 64 20 72 65 63 6F 72 64 20 41 41 41 41 42 42 nd record AAAABB
00000040: 42 42 42 31 32 33 BBB123
Block to 1014 by adding 2 * x‘40’ characters every 1012 characters in the data. Finally fill with x‘40’ characters to next 1014 increment. In this case, there is only one increment:
00000000: 00 00 00 1C 54 68 69 73 20 69 73 20 66 69 72 73 ....This is firs
00000010: 74 20 72 65 63 6F 72 64 20 31 32 33 34 35 36 37 t record 1234567
00000020: 00 00 00 22 54 68 69 73 20 69 73 20 73 65 63 6F ..."This is seco
00000030: 6E 64 20 72 65 63 6F 72 64 20 41 41 41 41 42 42 nd record AAAABB
00000040: 42 42 42 31 32 33 40 40 40 40 40 40 40 40 40 40 BBB123@@@@@@@@@@
00000050: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 @@@@@@@@@@@@@@@@
... all X'40' characters
000003E0: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 @@@@@@@@@@@@@@@@
000003F0: 40 40 40 40 40 40 @@@@@@
mciutil package¶
To use Mastercard file utilities in a project:
import mciutil
There are some useful functions for working with bitmap, variable length files. Will document in a future version.
Contributing¶
Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.
You can contribute in many ways:
Types of Contributions¶
Report Bugs¶
Report bugs at https://github.com/adelosa/mciutil/issues.
If you are reporting a bug, please include:
- Your operating system name and version.
- Any details about your local setup that might be helpful in troubleshooting.
- Detailed steps to reproduce the bug.
Fix Bugs¶
Look through the GitHub issues for bugs. Anything tagged with “bug” is open to whoever wants to implement it.
Implement Features¶
Look through the GitHub issues for features. Anything tagged with “feature” is open to whoever wants to implement it.
Write Documentation¶
Mastercard file utilities could always use more documentation, whether as part of the official Mastercard file utilities docs, in docstrings, or even on the web in blog posts, articles, and such.
Submit Feedback¶
The best way to send feedback is to file an issue at https://github.com/adelosa/mciutil/issues.
If you are proposing a feature:
- Explain in detail how it would work.
- Keep the scope as narrow as possible, to make it easier to implement.
- Remember that this is a volunteer-driven project, and that contributions are welcome :)
Get Started!¶
Ready to contribute? Here’s how to set up mciutil for local development.
Fork the mciutil repo on GitHub.
Clone your fork locally:
$ git clone git@github.com:your_name_here/mciutil.git
Install your local copy into a virtualenv. Assuming you have virtualenvwrapper installed, this is how you set up your fork for local development:
$ mkvirtualenv mciutil $ cd mciutil/ $ python setup.py develop
Create a branch for local development:
$ git checkout -b name-of-your-bugfix-or-feature
Now you can make your changes locally.
When you’re done making changes, check that your changes pass flake8 and the tests, including testing other Python versions with tox:
$ flake8 mciutil tests $ python setup.py test $ tox
To get flake8 and tox, just pip install them into your virtualenv.
Commit your changes and push your branch to GitHub:
$ git add . $ git commit -m "Your detailed description of your changes." $ git push origin name-of-your-bugfix-or-feature
Submit a pull request through the GitHub website.
Pull Request Guidelines¶
Before you submit a pull request, check that it meets these guidelines:
- The pull request should include tests.
- If the pull request adds functionality, the docs should be updated. Put your new functionality into a function with a docstring, and add the feature to the list in README.rst.
- The pull request should work for Python 2.7, 3.4, and 3.5, and for PyPy. Check https://travis-ci.org/adelosa/mciutil/pull_requests and make sure that the tests pass for all supported Python versions.
Credits¶
Development Lead¶
- Anthony Delosa <adelosa@gmail.com>
Contributors¶
None yet. Why not be the first?
History¶
0.4.2 (2016-03-13)¶
- Complete data elements added to default config.
- Added versioneer support for easier package versions
0.4.1 (2015-12-16)¶
- Additional data elements added to default config file.
0.4.0 (2015-10-05)¶
- Now supporting python 2.6 (for all those still using RHEL 6)
- Headers rows in mideu csv extracts don’t work in 2.6
0.3.0 (2015-10-03)¶
- added sub commands for mideu
- mideu now supports IPM encoding conversion between ascii and ebcdic
- Now faster using list comps instead of slow loops
0.2.0 (2015-08-28)¶
- Support for config override for mideu - see usage doco
- Progress bar while using mideu.. it takes a while
- Now supports python 3.4, 3.5 and 2.7. Upgrade if you are using 2.6
- New usage documentation
0.1.0 (2015-08-20)¶
- First release.