Class TextFormat.Tokenizer
- java.lang.Object
-
- org.fusesource.hawtbuf.proto.compiler.TextFormat.Tokenizer
-
- Enclosing class:
- TextFormat
private static final class TextFormat.Tokenizer extends java.lang.Object
Represents a stream of tokens parsed from aString
.The Java standard library provides many classes that you might think would be useful for implementing this, but aren't. For example:
java.io.StreamTokenizer
: This almost does what we want -- or, at least, something that would get us close to what we want -- except for one fatal flaw: It automatically un-escapes strings using Java escape sequences, which do not include all the escape sequences we need to support (e.g. '\x').java.util.Scanner
: This seems like a great way at least to parse regular expressions out of a stream (so we wouldn't have to load the entire input into a single string before parsing). Sadly,Scanner
requires that tokens be delimited with some delimiter. Thus, although the text "foo:" should parse to two tokens ("foo" and ":"),Scanner
would recognize it only as a single token. Furthermore,Scanner
provides no way to inspect the contents of delimiters, making it impossible to keep track of line and column numbers.
Luckily, Java's regular expression support does manage to be useful to us. (Barely: We need
Matcher.usePattern()
, which is new in Java 1.5.) So, we can use that, at least. Unfortunately, this implies that we need to have the entire input in one contiguous string.
-
-
Field Summary
Fields Modifier and Type Field Description private int
column
private java.lang.String
currentToken
private static java.util.regex.Pattern
DOUBLE_INFINITY
private static java.util.regex.Pattern
FLOAT_INFINITY
private static java.util.regex.Pattern
FLOAT_NAN
private int
line
private java.util.regex.Matcher
matcher
private int
pos
private int
previousColumn
private int
previousLine
private java.lang.CharSequence
text
private static java.util.regex.Pattern
TOKEN
private static java.util.regex.Pattern
WHITESPACE
-
Constructor Summary
Constructors Constructor Description Tokenizer(java.lang.CharSequence text)
Construct a tokenizer that parses tokens from the given text.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description boolean
atEnd()
Are we at the end of the input?void
consume(java.lang.String token)
If the next token exactly matchestoken
, consume it.boolean
consumeBoolean()
If the next token is a boolean, consume it and return its value.Buffer
consumeBuffer()
If the next token is a string, consume it, unescape it as aBuffer
, and return it.double
consumeDouble()
If the next token is a double, consume it and return its value.float
consumeFloat()
If the next token is a float, consume it and return its value.java.lang.String
consumeIdentifier()
If the next token is an identifier, consume it and return its value.int
consumeInt32()
If the next token is a 32-bit signed integer, consume it and return its value.long
consumeInt64()
If the next token is a 64-bit signed integer, consume it and return its value.java.lang.String
consumeString()
If the next token is a string, consume it and return its (unescaped) value.int
consumeUInt32()
If the next token is a 32-bit unsigned integer, consume it and return its value.long
consumeUInt64()
If the next token is a 64-bit unsigned integer, consume it and return its value.private TextFormat.ParseException
floatParseException(java.lang.NumberFormatException e)
Constructs an appropriateTextFormat.ParseException
for the givenNumberFormatException
when trying to parse a float or double.private TextFormat.ParseException
integerParseException(java.lang.NumberFormatException e)
Constructs an appropriateTextFormat.ParseException
for the givenNumberFormatException
when trying to parse an integer.boolean
lookingAtInteger()
Returnstrue
if the next token is an integer, but does not consume it.void
nextToken()
Advance to the next token.TextFormat.ParseException
parseException(java.lang.String description)
Returns aTextFormat.ParseException
with the current line and column numbers in the description, suitable for throwing.TextFormat.ParseException
parseExceptionPreviousToken(java.lang.String description)
Returns aTextFormat.ParseException
with the line and column numbers of the previous token in the description, suitable for throwing.private void
skipWhitespace()
Skip over any whitespace so that the matcher region starts at the next token.boolean
tryConsume(java.lang.String token)
If the next token exactly matchestoken
, consume it and returntrue
.
-
-
-
Field Detail
-
text
private final java.lang.CharSequence text
-
matcher
private final java.util.regex.Matcher matcher
-
currentToken
private java.lang.String currentToken
-
pos
private int pos
-
line
private int line
-
column
private int column
-
previousLine
private int previousLine
-
previousColumn
private int previousColumn
-
WHITESPACE
private static java.util.regex.Pattern WHITESPACE
-
TOKEN
private static java.util.regex.Pattern TOKEN
-
DOUBLE_INFINITY
private static java.util.regex.Pattern DOUBLE_INFINITY
-
FLOAT_INFINITY
private static java.util.regex.Pattern FLOAT_INFINITY
-
FLOAT_NAN
private static java.util.regex.Pattern FLOAT_NAN
-
-
Method Detail
-
atEnd
public boolean atEnd()
Are we at the end of the input?
-
nextToken
public void nextToken()
Advance to the next token.
-
skipWhitespace
private void skipWhitespace()
Skip over any whitespace so that the matcher region starts at the next token.
-
tryConsume
public boolean tryConsume(java.lang.String token)
If the next token exactly matchestoken
, consume it and returntrue
. Otherwise, returnfalse
without doing anything.
-
consume
public void consume(java.lang.String token) throws TextFormat.ParseException
If the next token exactly matchestoken
, consume it. Otherwise, throw aTextFormat.ParseException
.- Throws:
TextFormat.ParseException
-
lookingAtInteger
public boolean lookingAtInteger()
Returnstrue
if the next token is an integer, but does not consume it.
-
consumeIdentifier
public java.lang.String consumeIdentifier() throws TextFormat.ParseException
If the next token is an identifier, consume it and return its value. Otherwise, throw aTextFormat.ParseException
.- Throws:
TextFormat.ParseException
-
consumeInt32
public int consumeInt32() throws TextFormat.ParseException
If the next token is a 32-bit signed integer, consume it and return its value. Otherwise, throw aTextFormat.ParseException
.- Throws:
TextFormat.ParseException
-
consumeUInt32
public int consumeUInt32() throws TextFormat.ParseException
If the next token is a 32-bit unsigned integer, consume it and return its value. Otherwise, throw aTextFormat.ParseException
.- Throws:
TextFormat.ParseException
-
consumeInt64
public long consumeInt64() throws TextFormat.ParseException
If the next token is a 64-bit signed integer, consume it and return its value. Otherwise, throw aTextFormat.ParseException
.- Throws:
TextFormat.ParseException
-
consumeUInt64
public long consumeUInt64() throws TextFormat.ParseException
If the next token is a 64-bit unsigned integer, consume it and return its value. Otherwise, throw aTextFormat.ParseException
.- Throws:
TextFormat.ParseException
-
consumeDouble
public double consumeDouble() throws TextFormat.ParseException
If the next token is a double, consume it and return its value. Otherwise, throw aTextFormat.ParseException
.- Throws:
TextFormat.ParseException
-
consumeFloat
public float consumeFloat() throws TextFormat.ParseException
If the next token is a float, consume it and return its value. Otherwise, throw aTextFormat.ParseException
.- Throws:
TextFormat.ParseException
-
consumeBoolean
public boolean consumeBoolean() throws TextFormat.ParseException
If the next token is a boolean, consume it and return its value. Otherwise, throw aTextFormat.ParseException
.- Throws:
TextFormat.ParseException
-
consumeString
public java.lang.String consumeString() throws TextFormat.ParseException
If the next token is a string, consume it and return its (unescaped) value. Otherwise, throw aTextFormat.ParseException
.- Throws:
TextFormat.ParseException
-
consumeBuffer
public Buffer consumeBuffer() throws TextFormat.ParseException
If the next token is a string, consume it, unescape it as aBuffer
, and return it. Otherwise, throw aTextFormat.ParseException
.- Throws:
TextFormat.ParseException
-
parseException
public TextFormat.ParseException parseException(java.lang.String description)
Returns aTextFormat.ParseException
with the current line and column numbers in the description, suitable for throwing.
-
parseExceptionPreviousToken
public TextFormat.ParseException parseExceptionPreviousToken(java.lang.String description)
Returns aTextFormat.ParseException
with the line and column numbers of the previous token in the description, suitable for throwing.
-
integerParseException
private TextFormat.ParseException integerParseException(java.lang.NumberFormatException e)
Constructs an appropriateTextFormat.ParseException
for the givenNumberFormatException
when trying to parse an integer.
-
floatParseException
private TextFormat.ParseException floatParseException(java.lang.NumberFormatException e)
Constructs an appropriateTextFormat.ParseException
for the givenNumberFormatException
when trying to parse a float or double.
-
-