de.danielnaber.languagetool.tokenizers.nl
Class DutchWordTokenizer

Object
  WordTokenizer
      DutchWordTokenizer

All Implemented Interfaces:

Tokenizer

public class DutchWordTokenizer
extends WordTokenizer

Constructor Summary

DutchWordTokenizer()

Method Summary

List<String>

tokenize(String text) Tokenizes just like WordTokenizer with the exception for words such as "oma's" that contains an apostrophe in their middle.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DutchWordTokenizer

public DutchWordTokenizer()

Method Detail

tokenize

public List<String> tokenize(String text)

Tokenizes just like WordTokenizer with the exception for words such as "oma's" that contains an apostrophe in their middle.

Specified by:

tokenize in interface Tokenizer

Overrides:

tokenize in class WordTokenizer

Parameters:

text - - Text to tokenize

Returns:

List of tokens. Note: a special string ##NL_APOS## is used to replace apostrophe during tokenizing.