Package io.pebbletemplates.pebble.lexer
Class LexerImpl
java.lang.Object
io.pebbletemplates.pebble.lexer.LexerImpl
- All Implemented Interfaces:
Lexer
This class reads the template input and builds single items out of it.
This class is not thread safe.
-
Nested Class Summary
Nested Classes -
Field Summary
FieldsModifier and TypeFieldDescriptionprivate final Collection
<BinaryOperator> Binary operatorsprivate LinkedList
<Pair<String, Integer>> Represents the brackets we are currently inside ordered by how recently we encountered them.private Deque
<LexerImpl.State> The state of the lexer is important so that we know what to expect next and to help discover errors in the template (ex.private final org.slf4j.Logger
private static final String
private static final Pattern
Matches a double quoteprivate static final Pattern
Static regular expressions for identifiers.private static final Pattern
private static final Pattern
private static final Pattern
Matches everything up to the first interpolation in a double quoted stringprivate static final Pattern
Matches single quoted strings and double quoted strings without interpolation.private Pattern
Regular expression to find operatorsprivate TemplateSource
As we progress through the source we maintain a string which is the text that has yet to be tokenized.private final Syntax
SyntaxThe list of tokens that we find and use to create a TokenStreamprivate boolean
If we encountered an END delimiter that was preceded with a whitespace trim character (ex.private final Collection
<UnaryOperator> Unary operators -
Constructor Summary
ConstructorsConstructorDescriptionLexerImpl
(Syntax syntax, Collection<UnaryOperator> unaryOperators, Collection<BinaryOperator> binaryOperators) Constructor -
Method Summary
Modifier and TypeMethodDescriptionprivate void
Retrieves the operators (both unary and binary) from the PebbleEngine and then dynamically creates one giant regular expression to detect for the existence of one of these operators.private void
checkForLeadingWhitespaceTrim
(Token leadingToken) private void
private void
lexVerbatimData
(Matcher verbatimStartMatcher) Implementation of the "verbatim" tagprivate void
popState()
Pop state from the stackprivate Token
pushToken
(Token.Type type) Create a Token with a Token Type but without no value onto the list of tokens that we are maintaining.private Token
pushToken
(Token.Type type, String value) Create a Token of a certain type and value and push it into the list of tokens that we are maintaining.This is the main method used to tokenize the raw contents of a template.private void
Tokenizes between execute delimiters.private void
Tokenizes between print delimiters.private void
Tokenizes between comment delimiters.private void
The DATA state assumes that we are current NOT in between any pair of meaningful delimiters.private void
Tokenizing an expression which can be found within both execute and print regions.private void
private void
private String
unquoteAndUnescape
(String str) This method assumes the providedstr
starts with a single or double quote.
-
Field Details
-
logger
private final org.slf4j.Logger logger -
syntax
Syntax -
unaryOperators
Unary operators -
binaryOperators
Binary operators -
source
As we progress through the source we maintain a string which is the text that has yet to be tokenized. -
tokens
The list of tokens that we find and use to create a TokenStream -
brackets
Represents the brackets we are currently inside ordered by how recently we encountered them. (i.e. peek() will return the most innermost bracket, getLast() will return the outermost). Brackets in this case includes double quotes. The String value of the pair is the bracket representation, and the Integer is the line number. -
lexerStateStack
The state of the lexer is important so that we know what to expect next and to help discover errors in the template (ex. unclosed comments). -
trimLeadingWhitespaceFromNextData
private boolean trimLeadingWhitespaceFromNextDataIf we encountered an END delimiter that was preceded with a whitespace trim character (ex. {{ foo -}}) then this boolean is toggled to "true" which tells the lexData() method to trim leading whitespace from the next text token. -
REGEX_IDENTIFIER
Static regular expressions for identifiers. -
REGEX_LONG
-
REGEX_NUMBER
-
REGEX_DOUBLEQUOTE
Matches a double quote -
REGEX_STRING_NON_INTERPOLATED_PART
Matches everything up to the first interpolation in a double quoted string -
REGEX_STRING_PLAIN
Matches single quoted strings and double quoted strings without interpolation. Extra complexity is due to ignoring escaped quotation marks. -
PUNCTUATION
- See Also:
-
regexOperators
Regular expression to find operators
-
-
Constructor Details
-
LexerImpl
public LexerImpl(Syntax syntax, Collection<UnaryOperator> unaryOperators, Collection<BinaryOperator> binaryOperators) Constructor- Parameters:
syntax
- The primary syntaxunaryOperators
- The available unary operatorsbinaryOperators
- The available binary operators
-
-
Method Details
-
tokenize
This is the main method used to tokenize the raw contents of a template. -
tokenizeStringInterpolation
private void tokenizeStringInterpolation() -
tokenizeString
private void tokenizeString() -
tokenizeData
private void tokenizeData()The DATA state assumes that we are current NOT in between any pair of meaningful delimiters. We are currently looking for the next "open" or "start" delimiter, ex. the opening comment delimiter, or the opening variable delimiter. -
tokenizeBetweenExecuteDelimiters
private void tokenizeBetweenExecuteDelimiters()Tokenizes between execute delimiters. -
tokenizeBetweenPrintDelimiters
private void tokenizeBetweenPrintDelimiters()Tokenizes between print delimiters. -
tokenizeComment
private void tokenizeComment()Tokenizes between comment delimiters.Simply find the closing delimiter for the comment and move the cursor to that point.
-
tokenizeExpression
private void tokenizeExpression()Tokenizing an expression which can be found within both execute and print regions. -
unquoteAndUnescape
This method assumes the providedstr
starts with a single or double quote. It removes the wrapping quotes, and un-escapes any quotes within the string. -
checkForLeadingWhitespaceTrim
-
checkForTrailingWhitespaceTrim
private void checkForTrailingWhitespaceTrim() -
lexVerbatimData
Implementation of the "verbatim" tag -
pushToken
Create a Token with a Token Type but without no value onto the list of tokens that we are maintaining.- Parameters:
type
- The type of Token we are creating
-
pushToken
Create a Token of a certain type and value and push it into the list of tokens that we are maintaining. `- Parameters:
type
- The type of token we are creatingvalue
- The value of the new token
-
popState
private void popState()Pop state from the stack -
buildOperatorRegex
private void buildOperatorRegex()Retrieves the operators (both unary and binary) from the PebbleEngine and then dynamically creates one giant regular expression to detect for the existence of one of these operators.
-