Versions

Description

Ucto tokenizes text files: it separates words from punctuation, and splits sentences. This is one of the first tasks for almost any Natural Language Processing application. Ucto offers several other basic preprocessing steps such as changing case that you can all use to make your text suited for further processing such as indexing, part-of-speech tagging, or machine translation.

Ucto comes with tokenisation rules for several languages (packaged separately) and can be easily extended to suit other languages. It has been incorporated for tokenizing Dutch text in Frog (https://languagemachines.github.io/frog), our Dutch morpho-syntactic processor.

The software is intended to be used from the command-line by researchers in Natural Language Processing or related areas, as well as software developers. An Ucto python binding is also available separately.

Repository

https://github.com/LanguageMachines/ucto.git

Project Slug

ucto

Last Built

1 week, 4 days ago passed

Maintainers

Home Page

https://languagemachines.github.io/ucto

Badge

Tags

nlp, tokenizer, tokeniser

Project Privacy Level

Public

Short URLs

ucto.readthedocs.io
ucto.rtfd.io

Default Version

latest

'latest' Version

master