Package org.languagetool.tokenizers.ro
Class RomanianWordTokenizer
- java.lang.Object
-
- org.languagetool.tokenizers.WordTokenizer
-
- org.languagetool.tokenizers.ro.RomanianWordTokenizer
-
- All Implemented Interfaces:
org.languagetool.tokenizers.Tokenizer
public class RomanianWordTokenizer extends org.languagetool.tokenizers.WordTokenizer
Tokenizes a sentence into words. Punctuation and whitespace gets its own token. Like EnglishWordTokenizer except for some characters: eg: "-'- Since:
- 20.02.2009 19:53:50
-
-
Constructor Summary
Constructors Constructor Description RomanianWordTokenizer()
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description java.util.List<java.lang.String>
tokenize(java.lang.String text)
-