org.languagetool.tokenizers

Interface Tokenizer

All Known Implementing Classes:

BretonWordTokenizer, ChineseSentenceTokenizer, ChineseWordTokenizer, CzechSentenceTokenizer, DanishSentenceTokenizer, DutchWordTokenizer, EnglishWordTokenizer, EsperantoWordTokenizer, GalicianWordTokenizer, GermanCompoundTokenizer, KhmerWordTokenizer, MalayalamWordTokenizer, RomanianWordTokenizer, SentenceTokenizer, SpanishWordTokenizer, SRXSentenceTokenizer, WordTokenizer

public interface Tokenizer

Interface for classes that tokenize text into smaller units.

Author:

Daniel Naber

Method Summary

Methods

Modifier and Type	Method and Description
List<String>	tokenize(String text)

Method Detail

tokenize

List<String> tokenize(String text)