Class IndexStatisticsDaemonImpl
- All Implemented Interfaces:
Runnable
,IndexStatisticsDaemon
The need for updated statistics is currently determined when compiling a SELECT query. The unit of work is then scheduled with this daemon, and the work itself will be carried out in a separate thread. If the worker thread doesn't exist it is created, if it is idle the unit of work will be processed immediately, and if it is busy the unit of work has to wait in the queue.
The daemon code has a notion of a background task. If the update is run as a background task, it will try to affect other activity in the Derby database as little as possible. As far as possible, it will not set locks on the conglomerates it scans, and if it needs to take locks it will give up immediately if the locks cannot be obtained. In some cases it will also roll back to release locks already taken, ad then retry. Since we are accessing shared structures the background work may still interfere with the user activity in the database due to locking, but all such operations carried out by the daemon are of short duration.
The high level flow of an update to index statistics is:
- schedule update (the only action carried out by the user thread)
- for each index:
- scan index
- invalidate statements dependent on current statistics
- drop existing statistics
- add new statistics
List of possible improvements:
- Reduce potential impact of multiple invalidations (per table), probably by finding a way to invalidate only once after all indexes for a table have had their statistics updated. So far invalidation has proven to be the most difficult piece of the puzzle due to the interaction with the data dictionary and sensitivity to concurrent activity for the table.
Implementation notes: List of potential cleanups before going into a release:
- Consider removing all tracing code. May involve improving logging if parts of the trace output is valuable enough.
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionprivate static class
Support class used to compare keys when scanning indexes. -
Field Summary
FieldsModifier and TypeFieldDescriptionprivate static final boolean
private static final boolean
private final ContextManager
The context manager for the worker thread.private boolean
Tells if the daemon has been disabled.private LanguageConnectionContext
The language connection context for the worker thread.private final String
private final Database
The database object for the database we are handling automatic index statistics update for.private final String
The name of the database owner.private final boolean
Tells if logging is enabled.private final boolean
Tells if tracing is enabled.private int
Number of consecutive errors, used as a metric to decide if the damoen should be automatically shut down.private long
private long
private final HeaderPrintWriter
private static final int
Maximum number of work units allowed in the queue.private final ArrayList
<TableDescriptor> A list of tables that shall have their index statistics updated.private Thread
The thread in which the index statistics refresh operation is being executed, if any.private long
The period of time (ms) for which the daemon has been doing active work.final boolean
Tells if disposable stats should be generated, which will happen in soft-upgrade mode or when the user asks us to revert to the old behavior.private final long
Specifies when the daemon was created.private final boolean
Tells if traces are written to the Derby log file.private final boolean
Tells if traces are written to standard out.private final StringBuffer
private long
private long
private long
private long
private long
-
Constructor Summary
ConstructorsConstructorDescriptionIndexStatisticsDaemonImpl
(HeaderPrintWriter log, boolean doLog, String traceLevel, Database db, String userName, String databaseName) Creates a new daemon. -
Method Summary
Modifier and TypeMethodDescriptionprivate boolean
Determines if the given work can be accepted.private void
Appends runtime statistics to the given string buffer.private static String
cardToStr
(long[] cardinality) Produces a textual representation of the cardinality numbers.private boolean
Tells if the database is 10.9 or newer.private static String
Purely for debugging, to avoid printing too much info.private static String
fmtScanTimes
(long[][] timings) Format array of scan durations as a string.private void
Generates index statistics for all indexes associated with the given table descriptor.private static ContextService
Privileged lookup of the ContextService.private boolean
Handles expected errors.private boolean
Handles fatal errors that will cause the daemon to be shut down.private boolean
Handles unexpected errors.private void
invalidateStatements
(LanguageConnectionContext lcc, TableDescriptor td, boolean asBackgroundTask) Performs an invalidation action for the given table (the event being statistics update).private boolean
Return true if we are being shutdownprivate void
log
(boolean asBackgroundTask, TableDescriptor td, String msg) private void
log
(boolean asBackgroundTask, TableDescriptor td, Throwable t, String msg) Logs the information given.private void
logAlways
(TableDescriptor td, Throwable t, String msg) Logs the information given.private void
Main processing loop which will compute statistics until the queue of scheduled work units has been drained.void
run()
Drives the statistics generation.void
runExplicitly
(LanguageConnectionContext lcc, TableDescriptor td, ConglomerateDescriptor[] cds, String runContext) Runs the statistics update sequence explicitly as requested by the user.void
Schedules an update of the index statistics for the specified table.private void
setHeapRowEstimate
(TransactionController tc, long tableId, long rowEstimate) Sets the row estimate for the heap conglomerate.private static void
sleep
(long ms) Puts the current thread to sleep for maximumms
milliseconds.void
stop()
Stops the daemon.private void
private void
updateIndexStatsMinion
(LanguageConnectionContext lcc, TableDescriptor td, ConglomerateDescriptor[] cds, boolean asBackgroundTask) Updates the index statistics for the given table and the specified indexes.private void
writeUpdatedStats
(LanguageConnectionContext lcc, TableDescriptor td, UUID index, long numRows, long[] cardinality, boolean asBackgroundTask) Writes updated statistics for the specified index to the data dictionary.
-
Field Details
-
AS_BACKGROUND_TASK
private static final boolean AS_BACKGROUND_TASK- See Also:
-
AS_EXPLICIT_TASK
private static final boolean AS_EXPLICIT_TASK- See Also:
-
MAX_QUEUE_LENGTH
private static final int MAX_QUEUE_LENGTHMaximum number of work units allowed in the queue. -
logStream
-
doLog
private final boolean doLogTells if logging is enabled. -
doTrace
private final boolean doTraceTells if tracing is enabled. -
traceToDerbyLog
private final boolean traceToDerbyLogTells if traces are written to the Derby log file. -
traceToStdOut
private final boolean traceToStdOutTells if traces are written to standard out. -
daemonDisabled
private boolean daemonDisabledTells if the daemon has been disabled. -
ctxMgr
The context manager for the worker thread. -
skipDisposableStats
public final boolean skipDisposableStatsTells if disposable stats should be generated, which will happen in soft-upgrade mode or when the user asks us to revert to the old behavior.Made public to allow access for CreateIndexConstantAction and FromBaseTable, but this is no longer necessary when the debug property to keep disposable statistics is removed.
-
daemonLCC
The language connection context for the worker thread. -
db
The database object for the database we are handling automatic index statistics update for. -
dbOwner
The name of the database owner. -
databaseName
-
queue
A list of tables that shall have their index statistics updated. Note that the descriptor isn't removed before the work has been completed. -
runningThread
The thread in which the index statistics refresh operation is being executed, if any. Created as needed, but there will only be one thread doing the work. The thread is allowed to die since it is assumed that index statistics regeneration is rather infrequent. -
errorsConsecutive
private int errorsConsecutiveNumber of consecutive errors, used as a metric to decide if the damoen should be automatically shut down. -
errorsUnknown
private long errorsUnknown -
errorsKnown
private long errorsKnown -
wuProcessed
private long wuProcessed -
wuScheduled
private long wuScheduled -
wuRejectedDup
private long wuRejectedDup -
wuRejectedFQ
private long wuRejectedFQ -
wuRejectedOther
private long wuRejectedOther -
timeOfCreation
private final long timeOfCreationSpecifies when the daemon was created. -
runTime
private long runTimeThe period of time (ms) for which the daemon has been doing active work. -
tsb
-
-
Constructor Details
-
IndexStatisticsDaemonImpl
public IndexStatisticsDaemonImpl(HeaderPrintWriter log, boolean doLog, String traceLevel, Database db, String userName, String databaseName) Creates a new daemon.- Parameters:
log
- the log to write todoLog
- whether to log activity informationtraceLevel
- whether, and to where, trace information should be written ("off|log|stdout|both")db
- the database ("off|log|stdout|both")userName
- the name of the database ownerdatabaseName
- the name of the database (not stored in the db obj)
-
-
Method Details
-
dbAtLeast10_9
Tells if the database is 10.9 or newer. -
schedule
Schedules an update of the index statistics for the specified table.Assume the descriptor will be valid until we get around to generate the statistics. If it turns out to be invalid, it will be discarded.
- Specified by:
schedule
in interfaceIndexStatisticsDaemon
- Parameters:
td
- base table descriptor to update index statistics for
-
acceptWork
Determines if the given work can be accepted.- Parameters:
td
- the table descriptor to check- Returns:
true
if work can be accepted,false
if not.
-
generateStatistics
private void generateStatistics(LanguageConnectionContext lcc, TableDescriptor td) throws StandardException Generates index statistics for all indexes associated with the given table descriptor.This method is run as a background task.
- Parameters:
lcc
- connection context to use to perform the worktd
- target base table descriptor- Throws:
StandardException
- if accessing the conglomerates fail
-
isShuttingDown
private boolean isShuttingDown()Return true if we are being shutdown -
updateIndexStatsMinion
private void updateIndexStatsMinion(LanguageConnectionContext lcc, TableDescriptor td, ConglomerateDescriptor[] cds, boolean asBackgroundTask) throws StandardException Updates the index statistics for the given table and the specified indexes.API note: Using
null
to update the statistics for all conglomerates is preferred over explicitly passing an array with all the conglomerates for the table. Doing so allows for some optimizations, and will cause a disposable statistics check to be performed.- Parameters:
lcc
- language connection context used to perform the worktd
- the table to update index stats forcds
- the conglomerates to update statistics for (non-index conglomerates will be ignored),null
means all indexesasBackgroundTask
- whether the updates are done automatically as part of a background task or if explicitly invoked by the user- Throws:
StandardException
- if something goes wrong
-
writeUpdatedStats
private void writeUpdatedStats(LanguageConnectionContext lcc, TableDescriptor td, UUID index, long numRows, long[] cardinality, boolean asBackgroundTask) throws StandardException Writes updated statistics for the specified index to the data dictionary.- Parameters:
lcc
- connection context to use to perform the worktd
- the base tableindex
- the index of the base tablenumRows
- number of rows in the base tablecardinality
- the number of unique values in the index (per number of leading columns)asBackgroundTask
- whether the update is done automatically as part of a background task or if explicitly invoked by the user- Throws:
StandardException
- if updating the data dictionary fails
-
invalidateStatements
private void invalidateStatements(LanguageConnectionContext lcc, TableDescriptor td, boolean asBackgroundTask) throws StandardException Performs an invalidation action for the given table (the event being statistics update).- Parameters:
lcc
- connection context to use to perform the worktd
- the table to invalidate forasBackgroundTask
- whether the update is done automatically as part of a background task or if explicitly invoked by the user- Throws:
StandardException
- if the invalidation request fails
-
setHeapRowEstimate
private void setHeapRowEstimate(TransactionController tc, long tableId, long rowEstimate) throws StandardException Sets the row estimate for the heap conglomerate.- Parameters:
tc
- transaction to usetableId
- the heap tablerowEstimate
- estimate of number of rows in the table- Throws:
StandardException
- if accessing the table fails
-
run
public void run()Drives the statistics generation.This method will be run in a separate thread, and it will keep working as long as there is work to do. When the queue is exhausted, the method will exit (the thread dies).
-
processingLoop
private void processingLoop()Main processing loop which will compute statistics until the queue of scheduled work units has been drained. -
runExplicitly
public void runExplicitly(LanguageConnectionContext lcc, TableDescriptor td, ConglomerateDescriptor[] cds, String runContext) throws StandardException Runs the statistics update sequence explicitly as requested by the user.- Specified by:
runExplicitly
in interfaceIndexStatisticsDaemon
- Parameters:
lcc
- connection context to use to perform the worktd
- the base tablecds
- the indexes to update (non-index conglomerates are ignored)runContext
- the context in which the operation is run (i.e. 'ALTER TABLE', may benull
)- Throws:
StandardException
- if updating the index statistics fails
-
stop
public void stop()Stops the daemon.Will also clear the queue and print runtime statistics to the log the first time the method is invoked.
- Specified by:
stop
in interfaceIndexStatisticsDaemon
-
handleFatalErrors
Handles fatal errors that will cause the daemon to be shut down.- Parameters:
cm
- context managerse
- the exception to handle- Returns:
true
if the error was handled,false
otherwise
-
handleExpectedErrors
Handles expected errors.The logging of expected errors is for observability purposes only. The daemon is capable of dealing with these errors, and no interaction from the user is expected.
- Parameters:
se
- the exception to handle- Returns:
true
if the error was handled,false
otherwise
-
handleUnexpectedErrors
Handles unexpected errors.Unexpected errors are error conditions the daemon isn't set up to handle specifically. For this reason the stack trace will be logged to allow for later investigation.
In general it is expected that the daemon will be able to recover by dropping the current unit of work and move on to the next one (if any).
- Parameters:
se
- the exception to handle- Returns:
true
if the error was handled,false
otherwise
-
sleep
private static void sleep(long ms) Puts the current thread to sleep for maximumms
milliseconds.No guarantee is provided for the minimum amount of time slept. If interrupted, the interrupt flag will be set again.
- Parameters:
ms
- target sleep time
-
fmtScanTimes
Format array of scan durations as a string. -
log
- See Also:
-
log
Logs the information given.Note that if
asBackgroundTask
is false, nothing will be logged currently.- Parameters:
asBackgroundTask
-true
if logging for the background daemon automatically updating stats,false
if nottd
- current table descriptor being worked on, may benull
t
- raised error, may benull
msg
- the message to log
-
logAlways
Logs the information given.- Parameters:
td
- current table descriptor being worked on, may benull
t
- raised error, may benull
msg
- the message to log
-
trace
-
appendRunStats
Appends runtime statistics to the given string buffer.- Parameters:
sb
- the string buffer to append to
-
cardToStr
Produces a textual representation of the cardinality numbers.- Parameters:
cardinality
- index cardinality- Returns:
- A string.
-
extractIstatInfo
Purely for debugging, to avoid printing too much info. -
getContextService
Privileged lookup of the ContextService. Must be private so that user code can't call this entry point.
-