Kemangi Documentation¶

Overview¶

Kemangi is Indonesian Language (Bahasa Indonesia) pre-processor tool.

If you are going to conduct text mining related research with Indonesian Language, there is a high probability that pre-processing is needed. The purpose of Kemangi is to clean up your text file so that it is easier to be processed.

Kemangi provides basic task pre-processing task:

# Remove non-ASCII characters # Remove non-alphanumeric characters # Case folding # Remove stop words (meaningless word) # Remove words according to pattern provided by you, e.g. hashtags, URLs, Twitter mentions, etc # Word stemming

Example¶

Raw input:

Mempermainkan peranan 12 domba di pementasan
ALAyISME iTu TETAP ada di Jakarta
Saya tidur... kemarin
Pin  BB saya   B12A3FC
bbm koq naik, warga sedih #edisicurhat
dia memblokir website http://www.lucu.com
éà ada karakter ga jelas, non ASCII appeared! #wow
Pak kepala desa tidak tahu bahwa 3 pencuri di rumah itu adalah teman lamanya!

After case folding, non-alphanumeric removal, stop words removal, and stemming:

main peran 12 domba pentas
alayisme tetap jakarta
tidur kemarin
pin bb b12a3fc
bbm naik warga sedih
blokir website
karakter ga jelas non ascii appeared
pak kepala desa tahu 3 curi rumah teman

Issues and Discussion¶

When you encounter issues in using Kemangi, please report it to Kemangi’s repository. You need GitHub account to do that. Don’t worry if you don’t have it, just create one. GitHub is a community full of generous programmers.

Support Kemangi¶

Visit http://bit.ly/kemangi-feedback for feedback and comments. Kemangi developers would love to hear your story. If you like Kemangi, don’t forget to star it at upper right Kemangi’s repository (need GitHub login).

Overview¶

Kemangi takes a plain text file, and process each line of the text as your command.

Plain Text File

A text file that can be opened using basic text editor in your system (Windows: notepad, Linux: gedit, vi, etc). If it shows up cleanly, then it is a plain text file.

Typical plain text file has .txt extension.

To do text preprocessing, you need to supply input file, list of tasks, and output target.

Input File¶

Simply by clicking “Browse” button. Remember that the file must be a plain text. Kemangi can’t process more complex extensions like .doc or .odt.

List of Tasks¶

You can add several tasks, and Kemangi will run the task in the given order. To add a task, click “Add task”. A window will appear and you can choose what kind of task to be added.

Some tasks may need additional inputs. For example, own stop words removal demands text file containing list of stop words provided by you.

Some tasks may also includes another tasks. For example, stop words removal already includes case folding.

Output Target¶

Simply by clicking “Browse” button. Then proceed as if you are going to save a file as usual.

Start Processing¶

Click the “Start” button and text processing will start.

When unexpected error occurs (required internet, but your connection is lost), Kemangi will print the latest preprocessed text “intermediateResult.bak” in Kemangi’s directory. It is a plain text file, and you can open it with any basic text editor.