Package org.gjt.xpp.impl.tokenizer
Class Tokenizer
java.lang.Object
org.gjt.xpp.impl.tokenizer.Tokenizer
Simpe XML Tokenizer (SXT) performs input stream tokenizing.
Advantages:
- utility class to simplify creation of XML parsers, especially suited for pull event model but can support also push (SAX2)
- small footprint: whole tokenizer is in one file
- minimal memory utilization: does not use memory except for input and content buffer (that can grow in size)
- fast: all parsing done in one function (simple automata)
- supports most of XML 1.0 (except validation and external entities)
- low level: supports on demand parsing of Characters, CDSect, Comments, PIs etc.)
- parsed content: supports providing on demand parsed content to application (standard entities expanded all CDATA sections inserted, Comments and PIs removed) not for attribute values and element content
- mixed content: allow to dynamically disable mixed content
- small - total compiled size around 15K
- it is just a tokenizer - does not enforce grammar
- readName() is using Java identifier rules not XML
- does not parse DOCTYPE declaration (skips everyting in [...])
- Author:
- Aleksander Slominski
-
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final byte
static final byte
static final byte
char[]
static final byte
static final byte
static final byte
static final byte
static final byte
static final byte
static final byte
static final byte
static final byte
static final byte
protected static final int
protected static final char
protected static boolean[]
protected static boolean[]
int
boolean
boolean
boolean
boolean
boolean
boolean
boolean
boolean
boolean
This falg decides which buffer will be used to retrieve content for current token.char[]
This is buffer for parsed content such as actual valuue of entity ('<' in buf but in pc it is 'invalid input: '<'')int
int
Range [pcStart, pcEnd) defines part of pc that is content of current token iff parsedContent == falsestatic final byte
int
position of next char that will be read from bufferint
int
int
Range [posStart, posEnd) defines part of buf that is content of current token iff parsedContent == falseboolean
static final byte
static final byte
-
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionint
int
int
int
Return string describing current position of parsers as text 'at line %d (row) and column %d (colum) [seen %s...]'.int
boolean
boolean
protected boolean
isNameChar
(char ch) protected boolean
isNameStartChar
(char ch) protected boolean
isS
(char ch) Determine if ch is whitespace ([3] S)byte
next()
Return next recognized toke or END_DOCUMENT if no more input.void
reset()
void
setAllowedMixedContent
(boolean enable) Set support for mixed conetent.void
setBufferShrinkable
(boolean shrinkable) void
setHardLimit
(int value) Set hard limit on internal buffer size.void
setInput
(char[] data) Reset tokenizer state and set new input sourcevoid
setInput
(char[] data, int off, int len) void
Reset tokenizer state and set new input sourcevoid
setNotifyAll
(boolean enable) Set notification of all XML content tokens: Characters, Comment, CDSect, Doctype, PI, EntityRef, CharRef and AttValue (tokens for STag, ETag and Attribute are always sent).void
setParseContent
(boolean enable) Allow reporting parsed content for element content and attribute content (no need to deal with low level tokens such as in setNotifyAll).void
setSoftLimit
(int value) Set soft limit on internal buffer size.
-
Field Details
-
END_DOCUMENT
public static final byte END_DOCUMENT- See Also:
-
CONTENT
public static final byte CONTENT- See Also:
-
CHARACTERS
public static final byte CHARACTERS- See Also:
-
CDSECT
public static final byte CDSECT- See Also:
-
COMMENT
public static final byte COMMENT- See Also:
-
DOCTYPE
public static final byte DOCTYPE- See Also:
-
PI
public static final byte PI- See Also:
-
ENTITY_REF
public static final byte ENTITY_REF- See Also:
-
CHAR_REF
public static final byte CHAR_REF- See Also:
-
ETAG_NAME
public static final byte ETAG_NAME- See Also:
-
EMPTY_ELEMENT
public static final byte EMPTY_ELEMENT- See Also:
-
STAG_END
public static final byte STAG_END- See Also:
-
STAG_NAME
public static final byte STAG_NAME- See Also:
-
ATTR_NAME
public static final byte ATTR_NAME- See Also:
-
ATTR_CHARACTERS
public static final byte ATTR_CHARACTERS- See Also:
-
ATTR_CONTENT
public static final byte ATTR_CONTENT- See Also:
-
paramNotifyCharacters
public boolean paramNotifyCharacters -
paramNotifyComment
public boolean paramNotifyComment -
paramNotifyCDSect
public boolean paramNotifyCDSect -
paramNotifyDoctype
public boolean paramNotifyDoctype -
paramNotifyPI
public boolean paramNotifyPI -
paramNotifyCharRef
public boolean paramNotifyCharRef -
paramNotifyEntityRef
public boolean paramNotifyEntityRef -
paramNotifyAttValue
public boolean paramNotifyAttValue -
buf
public char[] buf -
pos
public int posposition of next char that will be read from buffer -
posStart
public int posStartRange [posStart, posEnd) defines part of buf that is content of current token iff parsedContent == false -
posEnd
public int posEnd -
posNsColon
public int posNsColon -
nsColonCount
public int nsColonCount -
seenContent
public boolean seenContent -
parsedContent
public boolean parsedContentThis falg decides which buffer will be used to retrieve content for current token. If true use pc and [pcStart, pcEnd) and if false use buf and [posStart, posEnd) -
pc
public char[] pcThis is buffer for parsed content such as actual valuue of entity ('<' in buf but in pc it is 'invalid input: '<'') -
pcStart
public int pcStartRange [pcStart, pcEnd) defines part of pc that is content of current token iff parsedContent == false -
pcEnd
public int pcEnd -
LOOKUP_MAX
protected static final int LOOKUP_MAX- See Also:
-
LOOKUP_MAX_CHAR
protected static final char LOOKUP_MAX_CHAR- See Also:
-
lookupNameStartChar
protected static boolean[] lookupNameStartChar -
lookupNameChar
protected static boolean[] lookupNameChar
-
-
Constructor Details
-
Tokenizer
public Tokenizer()
-
-
Method Details
-
reset
public void reset() -
setInput
Reset tokenizer state and set new input source -
setInput
public void setInput(char[] data) Reset tokenizer state and set new input source -
setInput
public void setInput(char[] data, int off, int len) -
setNotifyAll
public void setNotifyAll(boolean enable) Set notification of all XML content tokens: Characters, Comment, CDSect, Doctype, PI, EntityRef, CharRef and AttValue (tokens for STag, ETag and Attribute are always sent). -
setParseContent
public void setParseContent(boolean enable) Allow reporting parsed content for element content and attribute content (no need to deal with low level tokens such as in setNotifyAll). -
isAllowedMixedContent
public boolean isAllowedMixedContent() -
setAllowedMixedContent
public void setAllowedMixedContent(boolean enable) Set support for mixed conetent. If mixed content is disabled tokenizer will do its best to ensure that no element has mixed content model also ignorable whitespaces will not be reported as element content. -
getSoftLimit
public int getSoftLimit() -
setSoftLimit
Set soft limit on internal buffer size. That means suggested size that tokznzier will try to keep.- Throws:
TokenizerException
-
getHardLimit
public int getHardLimit() -
setHardLimit
Set hard limit on internal buffer size. That means that if input (such as element content) is bigger than hard limit size tokenizer will throw XmlTokenizerBufferOverflowException.- Throws:
TokenizerException
-
getBufferShrinkOffset
public int getBufferShrinkOffset() -
setBufferShrinkable
- Throws:
TokenizerException
-
isBufferShrinkable
public boolean isBufferShrinkable() -
getPosDesc
Return string describing current position of parsers as text 'at line %d (row) and column %d (colum) [seen %s...]'. -
getLineNumber
public int getLineNumber() -
getColumnNumber
public int getColumnNumber() -
isNameStartChar
protected boolean isNameStartChar(char ch) -
isNameChar
protected boolean isNameChar(char ch) -
isS
protected boolean isS(char ch) Determine if ch is whitespace ([3] S) -
next
Return next recognized toke or END_DOCUMENT if no more input.This is simple automata (in pseudo-code):
byte next() { while(state != END_DOCUMENT) { ch = more(); // read character from input state = func(ch, state); // do transition if(state is accepting) return state; // return token to caller } }
For speed (and simplicity?) it is using few procedures such as readName() or isS().
- Throws:
TokenizerException
IOException
-