Class LogFilePatternReceiver
- java.lang.Object
-
- org.apache.log4j.component.spi.ComponentBase
-
- org.apache.log4j.component.plugins.PluginSkeleton
-
- org.apache.log4j.component.plugins.Receiver
-
- org.apache.log4j.receivers.varia.LogFilePatternReceiver
-
- All Implemented Interfaces:
Plugin
,Component
,Thresholdable
,org.apache.log4j.spi.OptionHandler
public class LogFilePatternReceiver extends Receiver
LogFilePatternReceiver can parse and tail log files, converting entries into LoggingEvents. If the file doesn't exist when the receiver is initialized, the receiver will look for the file once every 10 seconds.This receiver relies on java.util.regex features to perform the parsing of text in the log file, however the only regular expression field explicitly supported is a glob-style wildcard used to ignore fields in the log file if needed. All other fields are parsed by using the supplied keywords.
Features:
- specify the URL of the log file to be processed
- specify the timestamp format in the file (if one exists, using patterns fromSimpleDateFormat
)
- specify the pattern (logFormat) used in the log file using keywords, a wildcard character (*) and fixed text
- 'tail' the file (allows the contents of the file to be continually read and new events processed)
- supports the parsing of multi-line messages and exceptions - 'hostname' property set to URL host (or 'file' if not available) - 'application' property set to URL path (or value of fileURL if not available)Keywords:
TIMESTAMP
LOGGER
LEVEL
THREAD
CLASS
FILE
LINE
METHOD
RELATIVETIME
MESSAGE
NDC
PROP(key)
Use a * to ignore portions of the log format that should be ignored
Example:
If your file's patternlayout is this:
%d %-5p [%t] %C{2} (%F:%L) - %m%nspecify this as the log format:
TIMESTAMP LEVEL [THREAD] CLASS (FILE:LINE) - MESSAGETo define a PROPERTY field, use PROP(key)
Example:
If you used the RELATIVETIME pattern layout character in the file, you can use PROP(RELATIVETIME) in the logFormat definition to assign the RELATIVETIME field as a property on the event.If your file's patternlayout is this:
%r [%t] %-5p %c %x - %m%nspecify this as the log format:
PROP(RELATIVETIME) [THREAD] LEVEL LOGGER * - MESSAGENote the * - it can be used to ignore a single word or sequence of words in the log file (in order for the wildcard to ignore a sequence of words, the text being ignored must be followed by some delimiter, like '-' or '[') - ndc is being ignored in the following example.
Assign a filterExpression in order to only process events which match a filter. If a filterExpression is not assigned, all events are processed.
Limitations:
- no support for the single-line version of throwable supported by patternlayout
(this version of throwable will be included as the last line of the message)
- the relativetime patternLayout character must be set as a property: PROP(RELATIVETIME)
- messages should appear as the last field of the logFormat because the variability in message content
- exceptions are converted if the exception stack trace (other than the first line of the exception)
is stored in the log file with a tab followed by the word 'at' as the first characters in the line
- tailing may fail if the file rolls over.Example receiver configuration settings (add these as params, specifying a LogFilePatternReceiver 'plugin'):
param: "timestampFormat" value="yyyy-MM-d HH:mm:ss,SSS"
param: "logFormat" value="PROP(RELATIVETIME) [THREAD] LEVEL LOGGER * - MESSAGE"
param: "fileURL" value="file:///c:/events.log"
param: "tailing" value="true"This configuration will be able to process these sample events:
710 [ Thread-0] DEBUG first.logger first -something here something else
880 [ Thread-2] DEBUG first.logger third -something here something else
880 [ Thread-0] INFO first.logger first - infomsg-0
java.lang.Exception: someexception-first
at Generator2.run(Generator2.java:102)
-
-
Field Summary
Fields Modifier and Type Field Description private java.util.List
additionalLines
private boolean
appendNonMatches
private static java.lang.String
CLASS
private java.util.Map
currentMap
private java.util.Map
customLevelDefinitionMap
private java.lang.String
customLevelDefinitions
private java.text.SimpleDateFormat
dateFormat
private static java.lang.String
DEFAULT_GROUP
private static java.lang.String
DEFAULT_HOST
private java.lang.String[]
emptyException
private static java.lang.String
EXCEPTION_PATTERN
private java.util.regex.Pattern
exceptionPattern
private Rule
expressionRule
private static java.lang.String
FILE
private java.lang.String
fileURL
private java.lang.String
filterExpression
private static java.lang.String
GREEDY_GROUP
private java.lang.String
host
private java.util.List
keywords
private static java.lang.String
LEVEL
private static java.lang.String
LINE
private java.lang.String
logFormat
private static java.lang.String
LOGGER
private java.util.List
matchingKeywords
private static java.lang.String
MESSAGE
private static java.lang.String
METHOD
static int
MISSING_FILE_RETRY_MILLIS
private static java.lang.String
MULTIPLE_SPACES_REGEXP
private static java.lang.String
NDC
private java.lang.String
newLine
private static java.lang.String
NOSPACE_GROUP
private java.lang.String
path
private static java.lang.String
PATTERN_WILDCARD
private static java.lang.String
PROP_END
private static java.lang.String
PROP_START
private java.io.Reader
reader
private java.lang.String
regexp
private static java.lang.String
REGEXP_DEFAULT_WILDCARD
private static java.lang.String
REGEXP_GREEDY_WILDCARD
private java.util.regex.Pattern
regexpPattern
private boolean
tailing
private static java.lang.String
THREAD
private static java.lang.String
TIMESTAMP
private java.lang.String
timestampFormat
private java.lang.String
timestampPatternText
private boolean
useCurrentThread
private static java.lang.String
VALID_DATEFORMAT_CHAR_PATTERN
private static java.lang.String
VALID_DATEFORMAT_CHARS
private long
waitMillis
-
Fields inherited from class org.apache.log4j.component.plugins.Receiver
thresholdLevel
-
Fields inherited from class org.apache.log4j.component.plugins.PluginSkeleton
active, name
-
Fields inherited from class org.apache.log4j.component.spi.ComponentBase
repository
-
-
Constructor Summary
Constructors Constructor Description LogFilePatternReceiver()
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
activateOptions()
Read and process the log file.private org.apache.log4j.spi.LoggingEvent
buildEvent()
Construct a logging event from currentMap and additionalLines (additionalLines contains multiple message lines and any exception lines)private java.lang.String[]
buildException(int exceptionLine)
Combine all exception lines occuring in the additionalLines list into a String arrayprivate java.lang.String
buildMessage(java.lang.String firstMessageLine, int exceptionLine)
Combine all message lines occuring in the additionalLines list, adding a newline character between each lineprivate java.lang.String
convertTimestamp()
Helper method that will convert timestamp format to a patternprivate org.apache.log4j.spi.LoggingEvent
convertToEvent(java.util.Map fieldMap, java.lang.String[] exception)
Convert a keyword-to-values map to a LoggingEventprotected void
createPattern()
java.lang.String
getCustomLevelDefinitions()
private int
getExceptionLine()
Walk the additionalLines list, looking for the EXCEPTION_PATTERN.java.lang.String
getFileURL()
Accessorjava.lang.String
getFilterExpression()
Accessorjava.lang.String
getLogFormat()
Accessorjava.lang.String
getPath()
private java.lang.String
getShortPropertyName(java.lang.String longPropertyName)
java.lang.String
getTimestampFormat()
Accessorlong
getWaitMillis()
Accessorprotected void
initialize()
Build the regular expression needed to parse log entriesboolean
isAppendNonMatches()
Accessorprivate boolean
isInteger(java.lang.String value)
boolean
isTailing()
Accessorboolean
isUseCurrentThread()
When true, this property uses the current Thread to perform the import, otherwise when false (the default), a new Thread is created and started to manage the import.private boolean
passesExpression(org.apache.log4j.spi.LoggingEvent event)
Helper method that supports the evaluation of the expressionprotected void
process(java.io.BufferedReader bufferedReader)
Read, parse and optionally tail the log file, converting entries into logging events.private java.util.Map
processEvent(java.util.regex.MatchResult result)
Convert the match into a map.private java.lang.String
quoteTimeStampChars(java.lang.String input)
private java.lang.String
replaceMetaChars(java.lang.String input)
Some perl5 characters may occur in the log file format.void
setAppendNonMatches(boolean appendNonMatches)
Mutatorvoid
setCustomLevelDefinitions(java.lang.String customLevelDefinitions)
If the log file contains non-log4j level strings, they can be mapped to log4j levels using the format (android example): V=TRACE,D=DEBUG,I=INFO,W=WARN,E=ERROR,F=FATAL,S=OFFvoid
setFileURL(java.lang.String fileURL)
Mutatorvoid
setFilterExpression(java.lang.String filterExpression)
Mutatorprotected void
setHost(java.lang.String host)
void
setLogFormat(java.lang.String logFormat)
Mutatorprotected void
setPath(java.lang.String path)
void
setTailing(boolean tailing)
Mutatorvoid
setTimestampFormat(java.lang.String timestampFormat)
Mutator.void
setUseCurrentThread(boolean useCurrentThread)
Sets whether the current Thread or a new Thread is created to perform the import, the default being false (new Thread created).void
setWaitMillis(long waitMillis)
Mutatorvoid
shutdown()
Close the reader.private java.lang.String
singleReplace(java.lang.String inputString, java.lang.String oldString, java.lang.String newString)
private void
updateCustomLevelDefinitionMap()
-
Methods inherited from class org.apache.log4j.component.plugins.Receiver
doPost, getThreshold, isAsSevereAsThreshold, setThreshold
-
Methods inherited from class org.apache.log4j.component.plugins.PluginSkeleton
addPropertyChangeListener, addPropertyChangeListener, firePropertyChange, firePropertyChange, firePropertyChange, firePropertyChange, getLoggerRepository, getName, isActive, isEquivalent, removePropertyChangeListener, removePropertyChangeListener, setLoggerRepository, setName
-
Methods inherited from class org.apache.log4j.component.spi.ComponentBase
getLogger, getNonFloodingLogger, resetErrorCount
-
-
-
-
Field Detail
-
keywords
private final java.util.List keywords
-
PROP_START
private static final java.lang.String PROP_START
- See Also:
- Constant Field Values
-
PROP_END
private static final java.lang.String PROP_END
- See Also:
- Constant Field Values
-
LOGGER
private static final java.lang.String LOGGER
- See Also:
- Constant Field Values
-
MESSAGE
private static final java.lang.String MESSAGE
- See Also:
- Constant Field Values
-
TIMESTAMP
private static final java.lang.String TIMESTAMP
- See Also:
- Constant Field Values
-
NDC
private static final java.lang.String NDC
- See Also:
- Constant Field Values
-
LEVEL
private static final java.lang.String LEVEL
- See Also:
- Constant Field Values
-
THREAD
private static final java.lang.String THREAD
- See Also:
- Constant Field Values
-
CLASS
private static final java.lang.String CLASS
- See Also:
- Constant Field Values
-
FILE
private static final java.lang.String FILE
- See Also:
- Constant Field Values
-
LINE
private static final java.lang.String LINE
- See Also:
- Constant Field Values
-
METHOD
private static final java.lang.String METHOD
- See Also:
- Constant Field Values
-
DEFAULT_HOST
private static final java.lang.String DEFAULT_HOST
- See Also:
- Constant Field Values
-
EXCEPTION_PATTERN
private static final java.lang.String EXCEPTION_PATTERN
- See Also:
- Constant Field Values
-
REGEXP_DEFAULT_WILDCARD
private static final java.lang.String REGEXP_DEFAULT_WILDCARD
- See Also:
- Constant Field Values
-
REGEXP_GREEDY_WILDCARD
private static final java.lang.String REGEXP_GREEDY_WILDCARD
- See Also:
- Constant Field Values
-
PATTERN_WILDCARD
private static final java.lang.String PATTERN_WILDCARD
- See Also:
- Constant Field Values
-
NOSPACE_GROUP
private static final java.lang.String NOSPACE_GROUP
- See Also:
- Constant Field Values
-
DEFAULT_GROUP
private static final java.lang.String DEFAULT_GROUP
- See Also:
- Constant Field Values
-
GREEDY_GROUP
private static final java.lang.String GREEDY_GROUP
- See Also:
- Constant Field Values
-
MULTIPLE_SPACES_REGEXP
private static final java.lang.String MULTIPLE_SPACES_REGEXP
- See Also:
- Constant Field Values
-
newLine
private final java.lang.String newLine
-
emptyException
private final java.lang.String[] emptyException
-
dateFormat
private java.text.SimpleDateFormat dateFormat
-
timestampFormat
private java.lang.String timestampFormat
-
logFormat
private java.lang.String logFormat
-
customLevelDefinitions
private java.lang.String customLevelDefinitions
-
fileURL
private java.lang.String fileURL
-
host
private java.lang.String host
-
path
private java.lang.String path
-
tailing
private boolean tailing
-
filterExpression
private java.lang.String filterExpression
-
waitMillis
private long waitMillis
-
VALID_DATEFORMAT_CHARS
private static final java.lang.String VALID_DATEFORMAT_CHARS
- See Also:
- Constant Field Values
-
VALID_DATEFORMAT_CHAR_PATTERN
private static final java.lang.String VALID_DATEFORMAT_CHAR_PATTERN
- See Also:
- Constant Field Values
-
expressionRule
private Rule expressionRule
-
currentMap
private java.util.Map currentMap
-
additionalLines
private java.util.List additionalLines
-
matchingKeywords
private java.util.List matchingKeywords
-
regexp
private java.lang.String regexp
-
reader
private java.io.Reader reader
-
regexpPattern
private java.util.regex.Pattern regexpPattern
-
exceptionPattern
private java.util.regex.Pattern exceptionPattern
-
timestampPatternText
private java.lang.String timestampPatternText
-
useCurrentThread
private boolean useCurrentThread
-
MISSING_FILE_RETRY_MILLIS
public static final int MISSING_FILE_RETRY_MILLIS
- See Also:
- Constant Field Values
-
appendNonMatches
private boolean appendNonMatches
-
customLevelDefinitionMap
private final java.util.Map customLevelDefinitionMap
-
-
Method Detail
-
getFileURL
public java.lang.String getFileURL()
Accessor- Returns:
- file URL
-
setFileURL
public void setFileURL(java.lang.String fileURL)
Mutator- Parameters:
fileURL
-
-
setCustomLevelDefinitions
public void setCustomLevelDefinitions(java.lang.String customLevelDefinitions)
If the log file contains non-log4j level strings, they can be mapped to log4j levels using the format (android example): V=TRACE,D=DEBUG,I=INFO,W=WARN,E=ERROR,F=FATAL,S=OFF- Parameters:
customLevelDefinitions
- the level definition string
-
getCustomLevelDefinitions
public java.lang.String getCustomLevelDefinitions()
-
isAppendNonMatches
public boolean isAppendNonMatches()
Accessor- Returns:
- append non matches
-
setAppendNonMatches
public void setAppendNonMatches(boolean appendNonMatches)
Mutator- Parameters:
appendNonMatches
-
-
getFilterExpression
public java.lang.String getFilterExpression()
Accessor- Returns:
- filter expression
-
setFilterExpression
public void setFilterExpression(java.lang.String filterExpression)
Mutator- Parameters:
filterExpression
-
-
isTailing
public boolean isTailing()
Accessor- Returns:
- tailing
-
setTailing
public void setTailing(boolean tailing)
Mutator- Parameters:
tailing
-
-
isUseCurrentThread
public final boolean isUseCurrentThread()
When true, this property uses the current Thread to perform the import, otherwise when false (the default), a new Thread is created and started to manage the import.- Returns:
- true, if the current thread is used
-
setUseCurrentThread
public final void setUseCurrentThread(boolean useCurrentThread)
Sets whether the current Thread or a new Thread is created to perform the import, the default being false (new Thread created).- Parameters:
useCurrentThread
-
-
getLogFormat
public java.lang.String getLogFormat()
Accessor- Returns:
- log format
-
setLogFormat
public void setLogFormat(java.lang.String logFormat)
Mutator- Parameters:
logFormat
- the format
-
setTimestampFormat
public void setTimestampFormat(java.lang.String timestampFormat)
Mutator. Specify a pattern fromSimpleDateFormat
- Parameters:
timestampFormat
-
-
getTimestampFormat
public java.lang.String getTimestampFormat()
Accessor- Returns:
- timestamp format
-
getWaitMillis
public long getWaitMillis()
Accessor- Returns:
- millis between retrieves of content
-
setWaitMillis
public void setWaitMillis(long waitMillis)
Mutator- Parameters:
waitMillis
-
-
getExceptionLine
private int getExceptionLine()
Walk the additionalLines list, looking for the EXCEPTION_PATTERN.Return the index of the first matched line (the match may be the 1st line of an exception)
Assumptions:
- the additionalLines list may contain both message and exception lines
- message lines are added to the additionalLines list and then exception lines (all message lines occur in the list prior to all exception lines)- Returns:
- -1 if no exception line exists, line number otherwise
-
buildMessage
private java.lang.String buildMessage(java.lang.String firstMessageLine, int exceptionLine)
Combine all message lines occuring in the additionalLines list, adding a newline character between each linethe event will already have a message - combine this message with the message lines in the additionalLines list (all entries prior to the exceptionLine index)
- Parameters:
firstMessageLine
- primary message lineexceptionLine
- index of first exception line- Returns:
- message
-
buildException
private java.lang.String[] buildException(int exceptionLine)
Combine all exception lines occuring in the additionalLines list into a String array(all entries equal to or greater than the exceptionLine index)
- Parameters:
exceptionLine
- index of first exception line- Returns:
- exception
-
buildEvent
private org.apache.log4j.spi.LoggingEvent buildEvent()
Construct a logging event from currentMap and additionalLines (additionalLines contains multiple message lines and any exception lines)CurrentMap and additionalLines are cleared in the process
- Returns:
- event
-
process
protected void process(java.io.BufferedReader bufferedReader) throws java.io.IOException
Read, parse and optionally tail the log file, converting entries into logging events. A runtimeException is thrown if the logFormat pattern is malformed.- Parameters:
bufferedReader
-- Throws:
java.io.IOException
-
createPattern
protected void createPattern()
-
passesExpression
private boolean passesExpression(org.apache.log4j.spi.LoggingEvent event)
Helper method that supports the evaluation of the expression- Parameters:
event
-- Returns:
- true if expression isn't set, or the result of the evaluation otherwise
-
processEvent
private java.util.Map processEvent(java.util.regex.MatchResult result)
Convert the match into a map.Relies on the fact that the matchingKeywords list is in the same order as the groups in the regular expression
- Parameters:
result
-- Returns:
- map
-
convertTimestamp
private java.lang.String convertTimestamp()
Helper method that will convert timestamp format to a pattern- Returns:
- string
-
setHost
protected void setHost(java.lang.String host)
-
setPath
protected void setPath(java.lang.String path)
-
getPath
public java.lang.String getPath()
-
initialize
protected void initialize()
Build the regular expression needed to parse log entries
-
updateCustomLevelDefinitionMap
private void updateCustomLevelDefinitionMap()
-
isInteger
private boolean isInteger(java.lang.String value)
-
quoteTimeStampChars
private java.lang.String quoteTimeStampChars(java.lang.String input)
-
singleReplace
private java.lang.String singleReplace(java.lang.String inputString, java.lang.String oldString, java.lang.String newString)
-
getShortPropertyName
private java.lang.String getShortPropertyName(java.lang.String longPropertyName)
-
replaceMetaChars
private java.lang.String replaceMetaChars(java.lang.String input)
Some perl5 characters may occur in the log file format. Escape these characters to prevent parsing errors.- Parameters:
input
-- Returns:
- string
-
convertToEvent
private org.apache.log4j.spi.LoggingEvent convertToEvent(java.util.Map fieldMap, java.lang.String[] exception)
Convert a keyword-to-values map to a LoggingEvent- Parameters:
fieldMap
-exception
-- Returns:
- logging event
-
shutdown
public void shutdown()
Close the reader.
-
activateOptions
public void activateOptions()
Read and process the log file.
-
-