Package de.danielnaber.languagetool.tokenizers

Interface Summary

Tokenizer

Interface for classes that tokenize text into smaller units.

Class Summary

SentenceTokenizer	Tokenizes text into sentences by looking for typical end-of-sentence markers, but considering exceptions (e.g. abbreviations).
SRXSentenceTokenizer	Class to tokenize sentences using an SRX file.
WordTokenizer	Tokenizes a sentence into words.