de.danielnaber.languagetool.tokenizers
Interface Tokenizer

All Known Implementing Classes:

CzechSentenceTokenizer, DutchSentenceTokenizer, EnglishWordTokenizer, GermanCompoundTokenizer, GermanSentenceTokenizer, PolishSentenceTokenizer, SentenceTokenizer, WordTokenizer

public interface Tokenizer

Interface for classes that tokenize text into smaller units.

Author:

Daniel Naber

Method Summary

List<String>

tokenize(String text)

Method Detail

tokenize

List<String> tokenize(String text)