Class Tokenizer


  • public class Tokenizer
    extends java.lang.Object
    A simple tokenizer for Java source text. This is not intended to be a compliant lexer; instead, it is for quick and dirty scanning.
    See Also:
    Token
    • Field Summary

      Fields 
      Modifier and Type Field Description
      private java.io.PushbackReader reader  
      private static java.util.BitSet single  
      private static java.util.BitSet whiteSpace  
    • Constructor Summary

      Constructors 
      Constructor Description
      Tokenizer​(java.io.Reader reader)
      Constructor.
    • Field Detail

      • whiteSpace

        private static final java.util.BitSet whiteSpace
      • single

        private static final java.util.BitSet single
      • reader

        private final java.io.PushbackReader reader
    • Constructor Detail

      • Tokenizer

        public Tokenizer​(java.io.Reader reader)
        Constructor.
        Parameters:
        reader - the Reader for the Java source text
    • Method Detail

      • next

        public Token next()
                   throws java.io.IOException
        Get the next Token in the stream.
        Returns:
        the Token
        Throws:
        java.io.IOException
      • skipWhitespace

        private void skipWhitespace()
                             throws java.io.IOException
        Throws:
        java.io.IOException
      • munchString

        private Token munchString​(int delimiter)
                           throws java.io.IOException
        Throws:
        java.io.IOException
      • maybeComment

        private Token maybeComment()
                            throws java.io.IOException
        Throws:
        java.io.IOException
      • parseWord

        private Token parseWord()
                         throws java.io.IOException
        Throws:
        java.io.IOException