Class TokenStream


  • class TokenStream
    extends java.lang.Object

    Return a steady stream of tokens.

    The returned tokens are one of: LITERAL, '%', FORMAT_MODIFIER, SIMPLE_KEYWORD, COMPOSITE_KEYWORD OPTION, LEFT_PARENTHESIS, and RIGHT_PARENTHESIS.

    The '\' character is used as escape. It can be used to escape '_', '%', '(' and '('.

    Note that there is no EOS token returned.

    • Constructor Detail

      • TokenStream

        TokenStream​(java.lang.String pattern)
      • TokenStream

        TokenStream​(java.lang.String pattern,
                    IEscapeUtil escapeUtil)
    • Method Detail

      • handleRightParenthesisState

        private void handleRightParenthesisState​(char c,
                                                 java.util.List<Token> tokenList,
                                                 java.lang.StringBuffer buf)
      • processOption

        private void processOption​(char c,
                                   java.util.List<Token> tokenList,
                                   java.lang.StringBuffer buf)
                            throws ScanException
        Throws:
        ScanException
      • handleFormatModifierState

        private void handleFormatModifierState​(char c,
                                               java.util.List<Token> tokenList,
                                               java.lang.StringBuffer buf)
      • handleLiteralState

        private void handleLiteralState​(char c,
                                        java.util.List<Token> tokenList,
                                        java.lang.StringBuffer buf)
      • handleKeywordState

        private void handleKeywordState​(char c,
                                        java.util.List<Token> tokenList,
                                        java.lang.StringBuffer buf)
      • escape

        void escape​(java.lang.String escapeChars,
                    java.lang.StringBuffer buf)
      • optionEscape

        void optionEscape​(java.lang.String escapeChars,
                          java.lang.StringBuffer buf)
      • addValuedToken

        private void addValuedToken​(int type,
                                    java.lang.StringBuffer buf,
                                    java.util.List<Token> tokenList)