org.languagetool.tokenizers

Class SentenceTokenizer

Object

SentenceTokenizer

All Implemented Interfaces:

Tokenizer

Direct Known Subclasses:

ChineseSentenceTokenizer, CzechSentenceTokenizer, DanishSentenceTokenizer, SRXSentenceTokenizer

public class SentenceTokenizer
extends Object
implements Tokenizer

Tokenizes text into sentences by looking for typical end-of-sentence markers, but considering exceptions (e.g. abbreviations).

Author:

Daniel Naber

Constructor Summary

Constructors

Constructor and Description
SentenceTokenizer() Create a sentence tokenizer that uses the built-in abbreviations.
SentenceTokenizer(String[] abbrevList) Create a sentence tokenizer with the given list of abbreviations, additionally to the built-in ones.

Method Summary

Methods

Modifier and Type	Method and Description
void	setSingleLineBreaksMarksParagraph(boolean lineBreakParagraphs)
boolean	singleLineBreaksMarksPara()
List<String>	tokenize(String s) Tokenize the given string to sentences.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SentenceTokenizer

public SentenceTokenizer()

Create a sentence tokenizer that uses the built-in abbreviations.

SentenceTokenizer

public SentenceTokenizer(String[] abbrevList)

Create a sentence tokenizer with the given list of abbreviations, additionally to the built-in ones.

Method Detail

setSingleLineBreaksMarksParagraph

public void setSingleLineBreaksMarksParagraph(boolean lineBreakParagraphs)

Parameters:

lineBreakParagraphs - if true, single lines breaks are assumed to end a paragraph, with false, only two ore more consecutive line breaks end a paragraph

singleLineBreaksMarksPara

public boolean singleLineBreaksMarksPara()

tokenize

public List<String> tokenize(String s)

Tokenize the given string to sentences.

Specified by:

tokenize in interface Tokenizer