Package org.languagetool.tokenizers
Interface Tokenizer
-
- All Known Subinterfaces:
CompoundWordTokenizer
,SentenceTokenizer
- All Known Implementing Classes:
SimpleSentenceTokenizer
,SRXSentenceTokenizer
,WordTokenizer
public interface Tokenizer
Interface for classes that tokenize text into smaller units.
-
-
Method Summary
All Methods Instance Methods Abstract Methods Modifier and Type Method Description java.util.List<java.lang.String>
tokenize(java.lang.String text)
-