org.languagetool.tokenizers.eo

Class EsperantoWordTokenizer

Object

WordTokenizer

EsperantoWordTokenizer

All Implemented Interfaces:

Tokenizer

public class EsperantoWordTokenizer
extends WordTokenizer

Author:

Dominique Pelle

Constructor Summary

Constructors

Constructor and Description
EsperantoWordTokenizer()

Method Summary

Methods

Modifier and Type	Method and Description
List<String>	tokenize(String text) Tokenizes just like WordTokenizer with the exception that words such as "dank'" contain an apostrophe within it.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

EsperantoWordTokenizer

public EsperantoWordTokenizer()

Method Detail

tokenize

public List<String> tokenize(String text)

Tokenizes just like WordTokenizer with the exception that words such as "dank'" contain an apostrophe within it.

Specified by:

tokenize in interface Tokenizer

Overrides:

tokenize in class WordTokenizer

Parameters:

text - - Text to tokenize

Returns:

List of tokens. Note: a special string ##EO_APOS## is used to replace apostrophe during tokenizing.