This is a collection of the developer documentation available for LanguageTool. It's intended for people who want to understand LanguageTool so they can write their own rules or even add support for a new language. Software developers might also be interested in LanguageTool's API.

• Help wanted!
• Source code Checkout (Java developers only)
• Installation and usage
• Language checking process
• Adding new XML rules
• Adding new Java rules
• Translating the user interface
• Adding support for a new language
• Background information

Help wanted!

the list of supported languages

How can you help?

1. Read this page (some features described here are quite advanced, so you won't need everything)
2. Start writing rules for the error you'd like LanguageTool to detect
3. See our wiki for more tips and tricks
4. Post your rules to our mailing list so we can include them in LanguageTool

Source code Checkout (Java developers only)

If you are a Java developer and you want to extend LanguageTool or if you want to use the latest development version, check out LanguageTool from subversion:

You can then run the test with ant test or build the code with ant.

Installation and usage

README

Usage page

Language checking process

1. The text to be checked is split into sentences
2. Each sentence is split into words, so called tokens
3. Each word is assigned its part-of-speech tag(s) (e.g. cars = plural noun, talked = simple past verb)
4. The analyzed text is then matched against the built-in rules and against the rules loaded from the grammar.xml file

Adding new XML rules

• <token>think</token>
  matches the word think
• <token>think</token>
  <token>about</token>
  Matches the phrase think about - as the text is split into words, you need to list each word separately as a token. This will not work:
  <token>think about</token>
• <token regexp="yes">think|say</token>
  matches the regular expression think|say, i.e. the word think or say. You can write simple rules without knowing regular expressions, but if you want to learn more about them you can try this tutorial.
• <token postag="VB" />
  <token>house</token>
  matches a base form verb followed by the word house. See resource/en/tagset.txt for a list of possible English part-of-speech tags.
• <token>cause</token>
  <token regexp="yes" negate="yes">and|to</token>
  matches the word cause followed by any word that is not and or to
• <token postag="SENT_START" />
  <token>foobar</token>
  matches the word foobar only at the beginning of a sentence. The corresponding postag for the end of a sentence is SENT_END.

A pattern's terms are matched case-insensitively by default. This can be changed by setting the pattern's case_sensitive attribute to yes.

Here's an example of a complete rule that marks "bed English", "bat attitude" etc as an error:

A short description of the elements and their attributes:

• element rule, attribute id: An internal identifier used to address this rule. This must be unique.
• element rule, attribute name: The text displayed in the configuration.
• element pattern, attributes mark_from and mark_to: What part of the original text should be marked. The default, mark_from="0" and mark_to="0", means to mark the complete matching token. For example, if the pattern contains three token elements that match the input text, those three matching words will be marked in the text. mark_to="-1" in the example above means that the last token of the match will not be marked.
• element token, attribute regexp: interpret the given token as a regular expression
• element message: The text displayed to the user if this rule matches. Use sub-element suggestion to suggest a possible replacement that corrects the error.
• element url (optional, since LanguageTool 1.7): An URL to a page that explains the rule leading to the error in more detail.
• element example: At least two examples with one correct and one incorrect sentence. The incorrect sentence is supposed to be matched by this rule. The position of the error must be marked up with the sub-element marker. This is used by the automatic test cases that can be run using ant test.

There are more features not used in the example above:

• element token, attribute inflected is used to match not only the given form but also all of its inflected forms. For example <token inflected="yes">bicycle</token> will match bicycle, bicycles, bicycling etc.
• element token, attribute skip is used in two situations:
  
  

  1. Simulate a simple chunker for languages with flexible word order, e.g., for matching errors of rection; we could for example skip possible adverbs in some rule. skip="1" works exactly as two rules, i.e.

  <token skip="1">A</token>
  <token>B</token>

  is equivalent to the pair of rules:

  <token>A</token>
  <token/>
  <token>B</token>
  
  <token>A</token>
  <token>B</token>

  Using negative value, we can match until the B is found, no matter how many tokens are skipped. This cannot be easily encoded using empty tokens as above because the sentence could be of any length.

  
  

  2. Match coordinated words, for example to match "both... as well" we could write:

  <token skip="-1">both<exception scope="next">and</exception></token>
  <token>as</token>
  <token>well</token>

  Here the exception is applied only to the skipped tokens.

  The scope attribute of the exception is used to make exception valid only for the token the exception is specified (scope="current") or for skipped tokens (scope="next"). Default behavior is scope="current". Using scopes is useful where several different exceptions should be applied to avoid false alarms. In some cases, it's useful to use scope="previous" in rules that already have skip="-1". This way, you can set an exception against a single token that immediately precedes the matched token. For example, we want to match "tak" after "jak" which is not preceded by a comma:

  <token>tak</token>
  <token skip="-1">jak</token>
  <token>tak<exception scope="previous">,</exception></token>

  In this case, the rule excludes all sentences, where there is a comma before "tak". Note that it's very hard to make such an exclusion otherwise.

  3. Using variables in rules

  In XML rules, you can refer to previously matched tokens in the pattern. For example:

  <pattern mark_from="2">
   <token regexp="yes" skip="-1">ani|ni|i|lub|albo|czy|oraz<exception scope="next">,</exception></token>
   <token><match no="0"/></token>
  </pattern>

  This rule matches sequences like ani... ani, ni... ni, i... i but you don't have to write all these cases explicitly. The first match (matches are numbered from zero, so it's <match no="0"/>) is automatically inserted into the second token. Note that this rule will match sentences like:

  Nie kupiłem ani gruszek ani jabłek. Kupię to lub to lub tamto.

  A similar mechanism could be used in suggestions, however there are more features, and tokens are numbered from 1 (for compatibility with the older notation \1 for the first matched token). For example:

  <suggestion><match no="1"/></suggestion>

  A more complicated example:

  <pattern>
  <token regexp="yes">^(\p{Lu}{2}+[i]*\p{Lu}+[\p{L}&amp;
  &amp;[^\p{Lu}]]{1,4}+)</token>
  </pattern>
  <message>Prawdopodobny błąd zapisu odmiany;
    skrótowce odmieniamy z dywizem:
    <suggestion><match no="1" regexp_match="^(\p{Lu}{2}+[i]*\p{Lu}+)([\p{L}&amp;
  &amp;[^\p{Lu}]]{1,4}+)" regexp_replace="$1-$2"/></suggestion></message>

  This rule matches Polish inflected acronyms such as "SMSem" that should be written with a hyphen: "SMS-em". So the acronym is matched with a complicated regular expression, and the match replaces the match using Java regular expression notation. Basically, the regular expression only shows two parts and inserts a hyphen between them.

  For some languages (currently Polish, English, Catalan, Spanish, Galician, Dutch, Romanian, Slovak and Russian), element <match/> can be used to insert an inflected matched token (or another word with a specified part of speech tag). For example:

  <pattern mark_from="1" mark_to="-1">
   <token regexp="yes">has|have</token>
   <token postag="VBD|VBP|VB" postag_regexp="yes"><exception postag="VBN|NN:U.*|JJ.*|RB" postag_regexp="yes"/></token>
   <token><exception postag="VBG"/></token>
  </pattern>
  <message>Possible agreement error -- use past participle here: <suggestion><match no="2" postag="VBN"/></suggestion>.</message>

  The above rule takes the second verb with a POS tag "VBN", "VBP" or "VB" and displays its form with a POS tag "VBN" in the suggestion. You can also specify POS tags using regular expressions (postag_regexp="yes") and replace POS tags – just like in the above example with acronyms. This is useful for large and complicated tagsets (for many examples, see Polish rule file: rules/pl/grammar.xml).

  Sometimes the rule should change the case of the matched word. For this purpose, you can use case_conversion attribute values: startlower, startupper, allupper and alllower.

  Another useful thing is that <match> can refer to a token, but apply its POS to another word. This is useful for suggesting another word with the same part of speech. There is a special abbreviated syntax used for this purpose:

  <match no="1" postag="verb:.*perf">kierować</match>

  This syntax means: take the POS tag of the first matched token that matches the regular expression specified in the postag attribute, and then apply this POS tag to the verb "kierować". This way the verb will be inflected just the way the matched verb was originally inflected. The reason why you need to specify the POS tag is that the matched token can have several POS tags (several readings).

  Note that by default <match> element inside the <token> element inserts only a string – so it matches a string, and not part of speech tags. So even if it refers to a token with a POS tag, it copies the matched token, and not its POS token. However, you can use all above attributes to change the form of the token.

  You can however use the <match> element to copy POS tags alone but to do so, you must use the attribute setpos="yes". All other attributes can be applied so that the POS could be converted appropriately. This can be useful for creating rules specifying grammatical agreement. Currently, such rules must be quite wordy, somewhat more terse syntax is in development.

  4. Turning the rule off

  Some rules can be optional, useful only in specific registers, or very sensitive. You can turn them off by default by using an attribute default="off". The user can turn the rule in the Options dialog box, and this setting is being saved in the configuration file.

Adding new Java rules

Translating the user interface

Adding support for a new language

• Language.java contains the information about supported languages. You can add a new language by creating a new Language object in this class and providing a part-of-speech tagger for it, similar to de/danielnaber/languagetool/tagging/en/EnglishTagger.java. The tagger must implement the Tagger interface, any implementation details (i.e. how to actually assign tags to words) are up to you -- the easiest thing is probably to just copy the English tagger.

  A trivial tagger that only assigns null tags to words is DemoTagger. This is enough for rules that refer to words but not to part-of-speech tags. You can add those rules to a file rules/xy/grammar.xml, whereas xy is the short name for your language. You will also need to add the short name of your language to rules.dtd.

  The test cases run by "ant test" will automatically include your new language and its rules, based on the "example" elements of each rule.

  To add part-of-speech tags, please have a look at resource/en/make-dict-en.sh (note: this file is only in subversion, not in the released OXT). First try to make it work for English. You need the fsa package. Install it and add its installation directory to your PATH. Once it works for English, create your own version of manually_added.txt and use that to create a .dict file, then adapt your tagger to use it (e.g. copy EnglishTagger.java and change the getFileName() implementation). More details about building dictionaries are in the Wiki.

• Adapt openoffice/Addons.xcu and openoffice/description.xml to translate the user interface of LanguageTool into your language when used in OpenOffice.org.
• Adapt build.xml. Just search for "/en/" in that file and copy those lines, adapting them to your language.
• Add the two-letter code of your language to i18n_update.sh and create a translation on Transifex.

Background information

PDF, 650 KB

Postscript (.ps.gz), 630 KB

Page last modified: 2012-05-11