Package org.languagetool.tokenizers.eo
Class EsperantoWordTokenizer
- java.lang.Object
-
- org.languagetool.tokenizers.WordTokenizer
-
- org.languagetool.tokenizers.eo.EsperantoWordTokenizer
-
- All Implemented Interfaces:
org.languagetool.tokenizers.Tokenizer
public class EsperantoWordTokenizer extends org.languagetool.tokenizers.WordTokenizer
-
-
Constructor Summary
Constructors Constructor Description EsperantoWordTokenizer()
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description java.util.List<java.lang.String>
tokenize(java.lang.String text)
Tokenizes just like WordTokenizer with the exception that words such as "dank'" contain an apostrophe within it.
-
-
-
Method Detail
-
tokenize
public java.util.List<java.lang.String> tokenize(java.lang.String text)
Tokenizes just like WordTokenizer with the exception that words such as "dank'" contain an apostrophe within it.- Specified by:
tokenize
in interfaceorg.languagetool.tokenizers.Tokenizer
- Overrides:
tokenize
in classorg.languagetool.tokenizers.WordTokenizer
- Parameters:
text
- - Text to tokenize- Returns:
- List of tokens. Note: a special string EO@APOS is used to replace apostrophe during tokenizing.
-
-