Package net.loomchild.segment.srx
Class SrxTextIterator
- java.lang.Object
-
- net.loomchild.segment.AbstractTextIterator
-
- net.loomchild.segment.srx.SrxTextIterator
-
- All Implemented Interfaces:
java.util.Iterator<java.lang.String>
,TextIterator
public class SrxTextIterator extends AbstractTextIterator
Represents text iterator splitting text according to rules in SRX file. The algorithm idea is as follows:1. Rule matcher list is created based on SRX file and language. Each rule matcher is responsible for matching before break and after break regular expressions of one break rule. 2. Each rule matcher is matched to the text. If the rule was not found the rule matcher is removed from the list. 3. First rule matcher in terms of its break position in text is selected. 4. List of exception rules corresponding to break rule is retrieved. 5. If none of exception rules is matching in break position then the text is marked as split and new segment is created. In addition all rule matchers are moved so they start after the end of new segment (which is the same as break position of the matched rule). 6. All the rules that have break position behind last matched rule break position are moved until they pass it. 7. If segment was not found the whole process is repeated.
In streaming version of this algorithm character buffer is searched. When the end of it is reached or break position is in the margin (break position > buffer size - margin) and there is more text, the buffer is moved in the text until it starts after last found segment. If this happens rule matchers are reinitialized and the text is searched again. Streaming version has a limitation that read buffer must be at least as long as any segment in the text. As this algorithm uses lookbehind extensively but Java does not permit infinite regular expressions in lookbehind, so some patterns are finitized. For example a* pattern will be changed to something like a{0,100}.
-
-
Field Summary
Fields Modifier and Type Field Description static java.lang.String
BUFFER_LENGTH_PARAMETER
Reader buffer size.static int
DEFAULT_BUFFER_LENGTH
Default size of read buffer when using streaming version of this class.static int
DEFAULT_MARGIN
Default margin size.static int
DEFAULT_MAX_LOOKBEHIND_CONSTRUCT_LENGTH
Default max lookbehind construct length parameter.private SrxDocument
document
private int
end
private int
margin
static java.lang.String
MARGIN_PARAMETER
Margin size.static java.lang.String
MAX_LOOKBEHIND_CONSTRUCT_LENGTH_PARAMETER
Maximum length of a regular expression construct that occurs in lookbehind.private RuleManager
ruleManager
private java.util.List<RuleMatcher>
ruleMatcherList
private java.lang.String
segment
private int
start
private TextManager
textManager
-
Constructor Summary
Constructors Constructor Description SrxTextIterator(SrxDocument document, java.lang.String languageCode, java.io.Reader reader)
Creates streaming text iterator with no additional parameters.SrxTextIterator(SrxDocument document, java.lang.String languageCode, java.io.Reader reader, java.util.Map<java.lang.String,java.lang.Object> parameterMap)
Creates text iterator that obtains language rules from given document using given language code.SrxTextIterator(SrxDocument document, java.lang.String languageCode, java.lang.CharSequence text)
Creates text iterator with no additional parameters.SrxTextIterator(SrxDocument document, java.lang.String languageCode, java.lang.CharSequence text, java.util.Map<java.lang.String,java.lang.Object> parameterMap)
Creates text iterator that obtains language rules form given document using given language code.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description private void
cutMatchers()
Move matchers that start before previous segment end.private RuleMatcher
getMinMatcher()
boolean
hasNext()
private void
init(SrxDocument document, java.lang.String languageCode, TextManager textManager, java.util.Map<java.lang.String,java.lang.Object> parameterMap)
Initializes splitter.private void
initMatchers()
Initializes matcher list according to rules from ruleManager and text from textManager.private boolean
isException(RuleMatcher ruleMatcher)
Returns true if there are no exception rules preventing given rule matcher from breaking the text.private void
moveMatchers()
Moves all matchers to the next position if their break position is smaller than last segment end position.java.lang.String
next()
Finds the next segment in the text and returns it.-
Methods inherited from class net.loomchild.segment.AbstractTextIterator
remove, toString
-
-
-
-
Field Detail
-
MARGIN_PARAMETER
public static final java.lang.String MARGIN_PARAMETER
Margin size. Used in streaming splitter. If rule is matched but its position is in the margin (position > bufferLength - margin) then the matching is ignored, and more text is read and rule is matched again.- See Also:
- Constant Field Values
-
BUFFER_LENGTH_PARAMETER
public static final java.lang.String BUFFER_LENGTH_PARAMETER
Reader buffer size. Segments cannot be longer than this value.- See Also:
- Constant Field Values
-
MAX_LOOKBEHIND_CONSTRUCT_LENGTH_PARAMETER
public static final java.lang.String MAX_LOOKBEHIND_CONSTRUCT_LENGTH_PARAMETER
Maximum length of a regular expression construct that occurs in lookbehind.- See Also:
- Constant Field Values
-
DEFAULT_MARGIN
public static final int DEFAULT_MARGIN
Default margin size.- See Also:
- Constant Field Values
-
DEFAULT_BUFFER_LENGTH
public static final int DEFAULT_BUFFER_LENGTH
Default size of read buffer when using streaming version of this class. Any segment cannot be longer than buffer size.- See Also:
- Constant Field Values
-
DEFAULT_MAX_LOOKBEHIND_CONSTRUCT_LENGTH
public static final int DEFAULT_MAX_LOOKBEHIND_CONSTRUCT_LENGTH
Default max lookbehind construct length parameter.- See Also:
- Constant Field Values
-
document
private SrxDocument document
-
segment
private java.lang.String segment
-
start
private int start
-
end
private int end
-
textManager
private TextManager textManager
-
ruleManager
private RuleManager ruleManager
-
ruleMatcherList
private java.util.List<RuleMatcher> ruleMatcherList
-
margin
private int margin
-
-
Constructor Detail
-
SrxTextIterator
public SrxTextIterator(SrxDocument document, java.lang.String languageCode, java.lang.CharSequence text, java.util.Map<java.lang.String,java.lang.Object> parameterMap)
Creates text iterator that obtains language rules form given document using given language code. This constructor version is not streaming because it receives whole text as a string. Supported parameters:MAX_LOOKBEHIND_CONSTRUCT_LENGTH_PARAMETER
.- Parameters:
document
- SRX documentlanguageCode
- text language code of text used to retrieve the rulestext
-parameterMap
- additional segmentation parameters
-
SrxTextIterator
public SrxTextIterator(SrxDocument document, java.lang.String languageCode, java.lang.CharSequence text)
Creates text iterator with no additional parameters.- Parameters:
document
- SRX documentlanguageCode
- text language code of text used to retrieve the rulestext
-- See Also:
SrxTextIterator(SrxDocument, String, CharSequence, Map)
-
SrxTextIterator
public SrxTextIterator(SrxDocument document, java.lang.String languageCode, java.io.Reader reader, java.util.Map<java.lang.String,java.lang.Object> parameterMap)
Creates text iterator that obtains language rules from given document using given language code. This is streaming constructor - it reads text from reader using buffer with given size and margin. Single segment cannot be longer than buffer size. If rule is matched but its position is in the margin (position > bufferLength - margin) then the matching is ignored, and more text is read and rule is matched again. This is needed because incomplete rule can be located at the end of the buffer and never matched. Supported parameters:BUFFER_LENGTH_PARAMETER
,MARGIN_PARAMETER
,MAX_LOOKBEHIND_CONSTRUCT_LENGTH_PARAMETER
.- Parameters:
document
- SRX documentlanguageCode
- text language code of text used to retrieve the rulesreader
- reader from which read the textparameterMap
- additional segmentation parameters
-
SrxTextIterator
public SrxTextIterator(SrxDocument document, java.lang.String languageCode, java.io.Reader reader)
Creates streaming text iterator with no additional parameters.- Parameters:
document
- SRX documentlanguageCode
- text language code of text used to retrieve the rulesreader
- reader from which read the text- See Also:
SrxTextIterator(SrxDocument, String, Reader, Map)
-
-
Method Detail
-
next
public java.lang.String next()
Finds the next segment in the text and returns it.- Returns:
- next segment or null if it doesn't exist
- Throws:
java.lang.IllegalStateException
- if buffer is too small to hold the segmentIORuntimeException
- if IO error occurs when reading the text
-
hasNext
public boolean hasNext()
- Returns:
- true if there are more segments
-
init
private void init(SrxDocument document, java.lang.String languageCode, TextManager textManager, java.util.Map<java.lang.String,java.lang.Object> parameterMap)
Initializes splitter.- Parameters:
document
- SRX documentlanguageCode
- text language codetextManager
- text manager containing the textparameterMap
- additional segmentation parameters
-
initMatchers
private void initMatchers()
Initializes matcher list according to rules from ruleManager and text from textManager.
-
moveMatchers
private void moveMatchers()
Moves all matchers to the next position if their break position is smaller than last segment end position.
-
cutMatchers
private void cutMatchers()
Move matchers that start before previous segment end.
-
getMinMatcher
private RuleMatcher getMinMatcher()
- Returns:
- first matcher in the text or null if there are no matchers
-
isException
private boolean isException(RuleMatcher ruleMatcher)
Returns true if there are no exception rules preventing given rule matcher from breaking the text.- Parameters:
ruleMatcher
- rule matcher- Returns:
- true if rule matcher breaks the text
-
-