Package org.languagetool.rules.ngrams
Class GoogleToken
- java.lang.Object
-
- org.languagetool.rules.ngrams.GoogleToken
-
class GoogleToken extends java.lang.Object
A token as tokenized in the Google ngram index.- Since:
- 3.2
-
-
Field Summary
Fields Modifier and Type Field Description (package private) int
endPos
(package private) java.util.Set<AnalyzedToken>
posTags
(package private) int
startPos
(package private) java.lang.String
token
-
Constructor Summary
Constructors Constructor Description GoogleToken(java.lang.String token, int startPos, int endPos)
GoogleToken(java.lang.String token, int startPos, int endPos, java.util.Set<AnalyzedToken> posTags)
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description private static java.util.Set<AnalyzedToken>
findOriginalAnalyzedTokens(AnalyzedSentence sentence, int startPos, int endPos)
(package private) static java.util.List<GoogleToken>
getGoogleTokens(java.lang.String sentence, boolean addStartToken, Tokenizer wordTokenizer)
(package private) static java.util.List<GoogleToken>
getGoogleTokens(AnalyzedSentence sentence, boolean addStartToken, Tokenizer wordTokenizer)
(package private) java.util.Set<AnalyzedToken>
getPosTags()
(package private) boolean
isWhitespace()
java.lang.String
toString()
-
-
-
Field Detail
-
token
final java.lang.String token
-
startPos
final int startPos
-
endPos
final int endPos
-
posTags
final java.util.Set<AnalyzedToken> posTags
-
-
Constructor Detail
-
GoogleToken
GoogleToken(java.lang.String token, int startPos, int endPos)
-
GoogleToken
GoogleToken(java.lang.String token, int startPos, int endPos, java.util.Set<AnalyzedToken> posTags)
-
-
Method Detail
-
getPosTags
java.util.Set<AnalyzedToken> getPosTags()
-
isWhitespace
boolean isWhitespace()
-
toString
public java.lang.String toString()
- Overrides:
toString
in classjava.lang.Object
-
getGoogleTokens
static java.util.List<GoogleToken> getGoogleTokens(java.lang.String sentence, boolean addStartToken, Tokenizer wordTokenizer)
-
getGoogleTokens
static java.util.List<GoogleToken> getGoogleTokens(AnalyzedSentence sentence, boolean addStartToken, Tokenizer wordTokenizer)
-
findOriginalAnalyzedTokens
private static java.util.Set<AnalyzedToken> findOriginalAnalyzedTokens(AnalyzedSentence sentence, int startPos, int endPos)
-
-