Uses of Interface
org.languagetool.tokenizers.Tokenizer
-
Packages that use Tokenizer Package Description org.languagetool org.languagetool.language org.languagetool.noop org.languagetool.rules.ngrams org.languagetool.tokenizers -
-
Uses of Tokenizer in org.languagetool
Methods in org.languagetool that return Tokenizer Modifier and Type Method Description Tokenizer
Language. getWordTokenizer()
Get this language's word tokenizer implementation. -
Uses of Tokenizer in org.languagetool.language
Methods in org.languagetool.language that return Tokenizer Modifier and Type Method Description Tokenizer
LanguageBuilder.ExtendedLanguage. getWordTokenizer()
-
Uses of Tokenizer in org.languagetool.noop
Methods in org.languagetool.noop that return Tokenizer Modifier and Type Method Description Tokenizer
NoopLanguage. getWordTokenizer()
-
Uses of Tokenizer in org.languagetool.rules.ngrams
Methods in org.languagetool.rules.ngrams that return Tokenizer Modifier and Type Method Description (package private) static Tokenizer
LanguageModelUtils. getGoogleStyleWordTokenizer(Language language)
Return a tokenizer that works more like Google does for its ngram index (which doesn't seem to be properly documented).protected Tokenizer
NgramProbabilityRule. getGoogleStyleWordTokenizer()
Methods in org.languagetool.rules.ngrams with parameters of type Tokenizer Modifier and Type Method Description (package private) static java.util.List<GoogleToken>
GoogleToken. getGoogleTokens(java.lang.String sentence, boolean addStartToken, Tokenizer wordTokenizer)
(package private) static java.util.List<GoogleToken>
GoogleToken. getGoogleTokens(AnalyzedSentence sentence, boolean addStartToken, Tokenizer wordTokenizer)
-
Uses of Tokenizer in org.languagetool.tokenizers
Subinterfaces of Tokenizer in org.languagetool.tokenizers Modifier and Type Interface Description interface
CompoundWordTokenizer
Interface for components that take compound words and split them into their parts.interface
SentenceTokenizer
Tokenizes text into sentences.Classes in org.languagetool.tokenizers that implement Tokenizer Modifier and Type Class Description class
SimpleSentenceTokenizer
A very simple sentence tokenizer that splits on[.!?…]
followed by whitespace or an uppercase letter.class
SRXSentenceTokenizer
Class to tokenize sentences using rules from an SRX file.class
WordTokenizer
Tokenizes a sentence into words.
-