Class Tokenizer

java.lang.Object
org.apache.hc.core5.util.Tokenizer
Direct Known Subclasses:
TokenParser

@Contract(threading=IMMUTABLE) public class Tokenizer extends Object
Tokenizer that can be used as a foundation for more complex parsing routines. Methods of this class are designed to produce near zero intermediate garbage and make no intermediate copies of input data.

This class is immutable and thread safe.

Since:
5.1
  • Field Details

  • Constructor Details

    • Tokenizer

      public Tokenizer()
  • Method Details

    • INIT_BITSET

      public static BitSet INIT_BITSET(int... b)
    • isWhitespace

      public static boolean isWhitespace(char ch)
    • parseContent

      public String parseContent(CharSequence buf, Tokenizer.Cursor cursor, BitSet delimiters)
      Extracts from the sequence of chars a token terminated with any of the given delimiters or a whitespace characters.
      Parameters:
      buf - buffer with the sequence of chars to be parsed
      cursor - defines the bounds and current position of the buffer
      delimiters - set of delimiting characters. Can be null if the token is not delimited by any character.
    • parseToken

      public String parseToken(CharSequence buf, Tokenizer.Cursor cursor, BitSet delimiters)
      Extracts from the sequence of chars a token terminated with any of the given delimiters discarding semantically insignificant whitespace characters.
      Parameters:
      buf - buffer with the sequence of chars to be parsed
      cursor - defines the bounds and current position of the buffer
      delimiters - set of delimiting characters. Can be null if the token is not delimited by any character.
    • parseValue

      public String parseValue(CharSequence buf, Tokenizer.Cursor cursor, BitSet delimiters)
      Extracts from the sequence of chars a value which can be enclosed in quote marks and terminated with any of the given delimiters discarding semantically insignificant whitespace characters.
      Parameters:
      buf - buffer with the sequence of chars to be parsed
      cursor - defines the bounds and current position of the buffer
      delimiters - set of delimiting characters. Can be null if the value is not delimited by any character.
    • skipWhiteSpace

      public void skipWhiteSpace(CharSequence buf, Tokenizer.Cursor cursor)
      Skips semantically insignificant whitespace characters and moves the cursor to the closest non-whitespace character.
      Parameters:
      buf - buffer with the sequence of chars to be parsed
      cursor - defines the bounds and current position of the buffer
    • copyContent

      public void copyContent(CharSequence buf, Tokenizer.Cursor cursor, BitSet delimiters, StringBuilder dst)
      Transfers content into the destination buffer until a whitespace character or any of the given delimiters is encountered.
      Parameters:
      buf - buffer with the sequence of chars to be parsed
      cursor - defines the bounds and current position of the buffer
      delimiters - set of delimiting characters. Can be null if the value is delimited by a whitespace only.
      dst - destination buffer
    • copyUnquotedContent

      public void copyUnquotedContent(CharSequence buf, Tokenizer.Cursor cursor, BitSet delimiters, StringBuilder dst)
      Transfers content into the destination buffer until a whitespace character, a quote, or any of the given delimiters is encountered.
      Parameters:
      buf - buffer with the sequence of chars to be parsed
      cursor - defines the bounds and current position of the buffer
      delimiters - set of delimiting characters. Can be null if the value is delimited by a whitespace or a quote only.
      dst - destination buffer
    • copyQuotedContent

      public void copyQuotedContent(CharSequence buf, Tokenizer.Cursor cursor, StringBuilder dst)
      Transfers content enclosed with quote marks into the destination buffer.
      Parameters:
      buf - buffer with the sequence of chars to be parsed
      cursor - defines the bounds and current position of the buffer
      dst - destination buffer