Class Tokenizer

java.lang.Object
org.sonatype.maven.polyglot.atom.parsing.Tokenizer

public class Tokenizer extends Object
Taken from the Loop programming language compiler pipeline.
  • Field Details

    • input

      private final String input
    • NON

      private static final int NON
      See Also:
    • SINGLE_TOKEN

      private static final int SINGLE_TOKEN
      See Also:
    • SEQUENCE_TOKEN

      private static final int SEQUENCE_TOKEN
      See Also:
    • DELIMITERS

      private static final int[] DELIMITERS
    • STRING_TERMINATORS

      private static final boolean[] STRING_TERMINATORS
  • Constructor Details

    • Tokenizer

      public Tokenizer(String input)
  • Method Details

    • tokenize

      public List<Token> tokenize()
    • isWhitespace

      private static boolean isWhitespace(char c)
    • isSingleTokenDelimiter

      static boolean isSingleTokenDelimiter(char c)
    • detokenize

      public static String detokenize(List<Token> tokens)
    • isDelimiter

      private static boolean isDelimiter(char c)
    • bakeToken

      private static void bakeToken(List<Token> tokens, char[] input, int i, int start)