Class MVStore
- All Implemented Interfaces:
AutoCloseable
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionprivate static class
A background writer thread to automatically store changes from time to time.static final class
A builder for an MVStore.private static class
static final class
Class TxCounter is a simple data structure to hold version of the store along with the counter of open transactions, which are still operating on this version. -
Field Summary
FieldsModifier and TypeFieldDescriptionprivate int
The delay in milliseconds to automatically commit and write changes.private final int
private final int
private long
private final AtomicReference
<MVStore.BackgroundWriterThread> Reference to a background thread, which is expected to be running, if any.(package private) static final int
The block size (physical sector size) of the disk.private ThreadPoolExecutor
Single-threaded executor for saving ByteBuffer as a new Chunkprivate final CacheLongKeyLIRS
<Page<?, ?>> The page cache.private final ConcurrentHashMap
<Integer, Chunk> The map of chunks.private final CacheLongKeyLIRS
<long[]> Cache for chunks "Table of Content" used to translate page's sequential number within containing chunk into byte position within chunk's image.private final int
The compression level for new pages (0 for disabled, 1 for fast, 2 for high).private Compressor
private Compressor
private long
The time the store was created, in milliseconds since 1970.private long
The version of the current store operation (if any).private MVStore.TxCounter
Counter of open transactions for the latest (current) store versionprivate long
private final FileStore
private final boolean
private static final int
private static final int
private static final int
private static final int
private static final String
private static final String
private static final String
private static final String
private static final String
private static final String
private static final String
private static final String
private static final String
private static final String
private final int
private Chunk
The newest chunk.private int
private long
private final AtomicInteger
private long
The layout map.private long
private final ConcurrentHashMap
<Integer, MVMap<?, ?>> The metadata map.static final String
The key for the entry within "layout" map, which contains id of "meta" map.private boolean
private long
private final AtomicLong
Oldest store version in use.private final int
private MVStoreException
private static final int
private final boolean
private final Queue
<MVStore.RemovedPageInfo> private int
How long to retain old, persisted chunks, in milliseconds.private boolean
private final ReentrantLock
private boolean
private ThreadPoolExecutor
Single-threaded executor for serialization of the store snapshot into ByteBufferprivate final ReentrantLock
private int
private static final int
Store is closed.private static final int
Store is closing now, and any operation on it may fail.private static final int
Store is open.private static final int
Store is about to close now, but is still operational.private final ReentrantLock
Lock which governs access to major store operations: store(), close(), ...private int
The estimated memory used by unsaved pages.private long
private long
private final Deque
<MVStore.TxCounter> Ordered collection of all version usage counters for all versions starting from oldestVersionToKeep and up to current.private int
private final Queue
<WriteBuffer> -
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionprivate void
acceptChunkOccupancyChanges
(long time, long version) Apply the freed space to the chunk metadata.(package private) void
accountForRemovedPage
(long pos, long version, boolean pinned, int pageNo) Remove a page.(package private) void
beforeWrite
(MVMap<?, ?> map) This method is called before writing to a map.(package private) void
Put the page in the cache.private int
calculatePageNo
(long pos) private static boolean
canOverwriteChunk
(Chunk c, long oldestVersionToKeep) private void
private void
void
close()
Close the file and the store.void
close
(int allowedCompactionTime) Close the file and the store.void
Close the file and the store, without writing anything.private void
closeStore
(boolean normalShutdown, int allowedCompactionTime) collectChangedMapRoots
(long version) long
commit()
Commit the changes.private long
boolean
compact
(int targetFillRate, int write) Try to increase the fill rate by re-writing partially full chunks.void
compactFile
(int maxCompactTime) Compact store file, that is, compact blocks that have a low fill rate, and move chunks next to each other.void
Compact by moving all chunks next to each other.(package private) boolean
compactMoveChunks
(int targetFillRate, long moveSize) Compact the store by moving all chunks next to each other, if there is free space between chunks.private boolean
compactMoveChunks
(long moveSize) private void
compactMoveChunks
(Iterable<Chunk> move) private int
compactRewrite
(Set<Integer> set) private Chunk
createChunk
(long time, long version) createIdSet
(Iterable<Chunk> toCompact) private static ThreadPoolExecutor
createSingleThreadExecutor
(String threadName) (package private) void
deregisterMapRoot
(int mapId) Performs final stage of map removal - delete root location info from the layout table.void
deregisterVersionUsage
(MVStore.TxCounter txCounter) De-register (close) completed operation (transaction).private Chunk
discoverChunk
(long block) Discover a valid chunk, searching file backwards from the given blockprivate void
doMaintenance
(int targetFillRate) private int
private void
findChunksToMove
(long startBlock, long moveSize) private boolean
findLastChunkWithCompleteValidChunkSet
(Chunk[] lastChunkCandidates, Map<Long, Chunk> validChunksByLocation, Map<Integer, Chunk> validChunksById, boolean afterFullScan) findOldChunks
(int writeLimit, int targetFillRate) private void
freeChunkSpace
(Chunk chunk) private void
freeFileSpace
(long start, int length) private long
Get the index of the first block after last occupied one.int
Get the auto-commit delay.int
Get the maximum memory (in bytes) used for unsaved pages.CacheLongKeyLIRS
<Page<?, ?>> getCache()
Get the cache.int
private static int
getCacheHitRatio
(CacheLongKeyLIRS<?> cache) int
Get the maximum cache size, in MB.int
Get the amount of memory used for caching, in MB.private Chunk
getChunk
(long pos) Get the chunk for the given position.int
Get data chunks count.private Chunk
getChunkForVersion
(long version) int
Get the current fill rate (percentage of used space in the file).private int
getChunksFillRate
(boolean all) (package private) int
(package private) Compressor
(package private) Compressor
long
Get the current version of the data.private long
Get the position right after the last used byte.Get the file store.int
int
Get this store's layout map.getLayoutMap
(long version) int
int
Get live data pages count.<K,
V> MVMap <K, V> getMap
(int id) Get map by id.private int
getMapName
(int id) Get the name of the given map.Get the set of all map names.long
Get the metadata map.private int
getMovePriority
(Chunk chunk) (package private) long
Get the oldest version to retain.int
Get data pages count.int
private int
getProjectedFillRate
(int thresholdChunkFillRate) int
boolean
int
private long
getRootPos
(int mapId) Get the store header.int
Get the store version.private int
private long
private long
private long[]
int
int
Get the estimated memory (in bytes) of unsaved data.private MVStoreException
getUnsupportedWriteFormatException
(long format, int expectedFormat, String s) double
long
Get the oldest version to retain in memory (for in-memory stores).private WriteBuffer
Get a buffer for writing.private void
boolean
Check whether a given map exists and has data.boolean
Check whether a given map exists.boolean
Check whether there are any unsaved changes.boolean
boolean
isClosed()
Determine that store is open, or wait for it to be closed (by other thread)private boolean
isIdle()
private boolean
isKnownVersion
(long version) Check whether all data can be read from this version.private boolean
isOpen()
private boolean
boolean
Whether the store is read-only.private boolean
isRewritable
(Chunk chunk, long time) (package private) boolean
private boolean
isSeasonedChunk
(Chunk chunk, long time) private boolean
isWriteStoreHeader
(Chunk c, boolean storeAtEndOfFile) private long
private void
private long
private boolean
Move specified chunk into free area of the file.private boolean
moveChunkInside
(Chunk chunkToMove, long boundary) private boolean
private void
onVersionChange
(long version) static MVStore
Open a store in exclusive mode.<M extends MVMap<K,
V>, K, V>
MopenMap
(int id, MVMap.MapBuilder<M, K, V> builder) Open an existing map with the given builder.<K,
V> MVMap <K, V> Open a map with the default settings.<M extends MVMap<K,
V>, K, V>
MopenMap
(String name, MVMap.MapBuilder<M, K, V> builder) Open a map with the given builder.private void
private Chunk
readChunkFooter
(long block) Try to read a chunk footer.private Chunk
readChunkHeader
(long block) private Chunk
readChunkHeaderAndFooter
(long block, int expectedId) Read a chunk header and footer, and verify the stored data is consistent.private Chunk
readChunkHeaderOptionally
(long block) private Chunk
readChunkHeaderOptionally
(long block, int expectedId) (package private) <K,
V> Page <K, V> Read a page.private <K,
V> Page <K, V> readPageFromCache
(long pos) private void
void
registerUnsavedMemory
(int memory) Adjust amount of "unsaved memory" meaning amount of RAM occupied by pages not saved yet to the file.Register opened operation (transaction).private void
Release a buffer for writing.void
Remove map by name.void
Remove a map from the current version of the store.void
Rename a map.private boolean
private boolean
rewriteChunks
(int writeLimit, int targetFillRate) private int
rewriteChunks
(Set<Integer> set, boolean secondPass) void
rollback()
Revert to the beginning of the current version, reverting all uncommitted changes.void
rollbackTo
(long version) Revert to the beginning of the given version.private void
private void
private void
serializeAndStore
(boolean syncRun, long reservedLow, Supplier<Long> reservedHighSupplier, ArrayList<Page<?, ?>> changed, long time, long version) private void
serializeToBuffer
(WriteBuffer buff, ArrayList<Page<?, ?>> changed, Chunk c, long reservedLow, Supplier<Long> reservedHighSupplier) void
setAutoCommitDelay
(int millis) Set the maximum delay in milliseconds to auto-commit changes.void
setCacheSize
(int mb) Set the read cache size in MB.private void
setLastChunk
(Chunk last) private void
setOldestVersionToKeep
(long oldestVersionToKeep) void
setRetentionTime
(int ms) How long to retain old, persisted chunks, in milliseconds.void
setReuseSpace
(boolean reuseSpace) Whether empty space in the file should be re-used.void
setStoreVersion
(int version) Update the store version.void
setVersionsToKeep
(int count) How many versions to retain for in-memory stores.private void
setWriteVersion
(long version) private void
shrinkFileIfPossible
(int minPercent) Shrink the file if possible, and if at least a given percentage can be saved.private static void
shutdownExecutor
(ThreadPoolExecutor executor) private void
stopBackgroundThread
(boolean waitForIt) private void
store
(boolean syncWrite) private void
store
(long reservedLow, long reservedHigh) private void
storeBuffer
(Chunk c, WriteBuffer buff, ArrayList<Page<?, ?>> changed) private void
private static void
submitOrRun
(ThreadPoolExecutor executor, Runnable action, boolean syncRun) void
sync()
Force all stored changes to be written to the storage.long
Unlike regular commit this method returns immediately if there is commit in progress on another thread, otherwise it acts as regular commit.private long
private void
private boolean
validateFileLength
(String msg) private void
write
(long pos, ByteBuffer buffer) (package private) void
Commit and save all changes, if there are any, and compact the store if needed.private void
-
Field Details
-
HDR_H
- See Also:
-
HDR_BLOCK_SIZE
- See Also:
-
HDR_FORMAT
- See Also:
-
HDR_CREATED
- See Also:
-
HDR_FORMAT_READ
- See Also:
-
HDR_CHUNK
- See Also:
-
HDR_BLOCK
- See Also:
-
HDR_VERSION
- See Also:
-
HDR_CLEAN
- See Also:
-
HDR_FLETCHER
- See Also:
-
META_ID_KEY
The key for the entry within "layout" map, which contains id of "meta" map. Entry value (hex encoded) is usually equal to 1, unless it's a legacy (upgraded) database and id 1 has been taken already by another map.- See Also:
-
BLOCK_SIZE
static final int BLOCK_SIZEThe block size (physical sector size) of the disk. The store header is written twice, one copy in each block, to ensure it survives a crash.- See Also:
-
FORMAT_WRITE_MIN
private static final int FORMAT_WRITE_MIN- See Also:
-
FORMAT_WRITE_MAX
private static final int FORMAT_WRITE_MAX- See Also:
-
FORMAT_READ_MIN
private static final int FORMAT_READ_MIN- See Also:
-
FORMAT_READ_MAX
private static final int FORMAT_READ_MAX- See Also:
-
STATE_OPEN
private static final int STATE_OPENStore is open.- See Also:
-
STATE_STOPPING
private static final int STATE_STOPPINGStore is about to close now, but is still operational. Outstanding store operation by background writer or other thread may be in progress. New updates must not be initiated, unless they are part of a closing procedure itself.- See Also:
-
STATE_CLOSING
private static final int STATE_CLOSINGStore is closing now, and any operation on it may fail.- See Also:
-
STATE_CLOSED
private static final int STATE_CLOSEDStore is closed.- See Also:
-
PIPE_LENGTH
private static final int PIPE_LENGTH- See Also:
-
storeLock
Lock which governs access to major store operations: store(), close(), ... It serves as a replacement for synchronized(this), except it allows for non-blocking lock attempts. -
serializationLock
-
saveChunkLock
-
backgroundWriterThread
Reference to a background thread, which is expected to be running, if any. -
serializationExecutor
Single-threaded executor for serialization of the store snapshot into ByteBuffer -
bufferSaveExecutor
Single-threaded executor for saving ByteBuffer as a new Chunk -
reuseSpace
private volatile boolean reuseSpace -
state
private volatile int state -
fileStore
-
fileStoreIsProvided
private final boolean fileStoreIsProvided -
pageSplitSize
private final int pageSplitSize -
keysPerPage
private final int keysPerPage -
cache
The page cache. The default size is 16 MB, and the average size is 2 KB. It is split in 16 segments. The stack move distance is 2% of the expected number of entries. -
chunksToC
Cache for chunks "Table of Content" used to translate page's sequential number within containing chunk into byte position within chunk's image. Cache keyed by chunk id. -
lastChunk
The newest chunk. If nothing was stored yet, this field is not set. -
chunks
The map of chunks. -
removedPages
-
deadChunks
-
updateCounter
private long updateCounter -
updateAttemptCounter
private long updateAttemptCounter -
layout
The layout map. Contains chunks metadata and root locations for all maps. This is relatively fast changing part of metadata -
meta
The metadata map. Holds name -> id and id -> name and id -> metadata mapping for all maps. This is relatively slow changing part of metadata -
maps
-
storeHeader
-
writeBufferPool
-
lastMapId
-
lastChunkId
private int lastChunkId -
versionsToKeep
private int versionsToKeep -
compressionLevel
private final int compressionLevelThe compression level for new pages (0 for disabled, 1 for fast, 2 for high). Even if disabled, the store may contain (old) compressed pages. -
compressorFast
-
compressorHigh
-
recoveryMode
private final boolean recoveryMode -
backgroundExceptionHandler
-
currentVersion
private volatile long currentVersion -
oldestVersionToKeep
Oldest store version in use. All version beyond this can be safely dropped -
versions
Ordered collection of all version usage counters for all versions starting from oldestVersionToKeep and up to current. -
currentTxCounter
Counter of open transactions for the latest (current) store version -
unsavedMemory
private int unsavedMemoryThe estimated memory used by unsaved pages. This number is not accurate, also because it may be changed concurrently, and because temporary pages are counted. -
autoCommitMemory
private final int autoCommitMemory -
saveNeeded
private volatile boolean saveNeeded -
creationTime
private long creationTimeThe time the store was created, in milliseconds since 1970. -
retentionTime
private int retentionTimeHow long to retain old, persisted chunks, in milliseconds. For larger or equal to zero, a chunk is never directly overwritten if unused, but instead, the unused field is set. If smaller zero, chunks are directly overwritten if unused. -
lastCommitTime
private long lastCommitTime -
currentStoreVersion
private volatile long currentStoreVersionThe version of the current store operation (if any). -
metaChanged
private volatile boolean metaChanged -
autoCommitDelay
private int autoCommitDelayThe delay in milliseconds to automatically commit and write changes. -
autoCompactFillRate
private final int autoCompactFillRate -
autoCompactLastFileOpCount
private long autoCompactLastFileOpCount -
panicException
-
lastTimeAbsolute
private long lastTimeAbsolute -
leafCount
private long leafCount -
nonLeafCount
private long nonLeafCount
-
-
Constructor Details
-
MVStore
Create and open the store.- Parameters:
config
- the configuration to use- Throws:
MVStoreException
- if the file is corrupt, or an exception occurred while openingIllegalArgumentException
- if the directory does not exist
-
-
Method Details
-
openMetaMap
-
scrubLayoutMap
private void scrubLayoutMap() -
scrubMetaMap
private void scrubMetaMap() -
unlockAndCheckPanicCondition
private void unlockAndCheckPanicCondition() -
panic
-
getPanicException
-
open
Open a store in exclusive mode. For a file-based store, the parent directory must already exist.- Parameters:
fileName
- the file name (null for in-memory)- Returns:
- the store
-
openMap
Open a map with the default settings. The map is automatically create if it does not yet exist. If a map with this name is already open, this map is returned.- Type Parameters:
K
- the key typeV
- the value type- Parameters:
name
- the name of the map- Returns:
- the map
-
openMap
Open a map with the given builder. The map is automatically create if it does not yet exist. If a map with this name is already open, this map is returned.- Type Parameters:
M
- the map typeK
- the key typeV
- the value type- Parameters:
name
- the name of the mapbuilder
- the map builder- Returns:
- the map
-
openMap
Open an existing map with the given builder.- Type Parameters:
M
- the map typeK
- the key typeV
- the value type- Parameters:
id
- the map idbuilder
- the map builder- Returns:
- the map
-
getMap
Get map by id.- Type Parameters:
K
- the key typeV
- the value type- Parameters:
id
- map id- Returns:
- Map
-
getMapNames
Get the set of all map names.- Returns:
- the set of names
-
getLayoutMap
Get this store's layout map. This data is for informational purposes only. The data is subject to change in future versions.The data in this map should not be modified (changing system data may corrupt the store).
The layout map contains the following entries:
chunk.{chunkId} = {chunk metadata} root.{mapId} = {root position}
- Returns:
- the metadata map
-
getMetaMap
Get the metadata map. This data is for informational purposes only. The data is subject to change in future versions.The data in this map should not be modified (changing system data may corrupt the store).
The metadata map contains the following entries:
name.{name} = {mapId} map.{mapId} = {map metadata} setting.storeVersion = {version}
- Returns:
- the metadata map
-
getLayoutMap
-
getChunkForVersion
-
hasMap
Check whether a given map exists.- Parameters:
name
- the map name- Returns:
- true if it exists
-
hasData
Check whether a given map exists and has data.- Parameters:
name
- the map name- Returns:
- true if it exists and has data.
-
markMetaChanged
private void markMetaChanged() -
readStoreHeader
private void readStoreHeader() -
getUnsupportedWriteFormatException
private MVStoreException getUnsupportedWriteFormatException(long format, int expectedFormat, String s) -
findLastChunkWithCompleteValidChunkSet
-
setLastChunk
-
discoverChunk
Discover a valid chunk, searching file backwards from the given block- Parameters:
block
- to start search from (found chunk footer should be no further than block-1)- Returns:
- valid chunk or null if none found
-
writeStoreHeader
private void writeStoreHeader() -
write
-
close
public void close()Close the file and the store. Unsaved changes are written to disk first.- Specified by:
close
in interfaceAutoCloseable
-
close
public void close(int allowedCompactionTime) Close the file and the store. Unsaved changes are written to disk first, and compaction (up to a specified number of milliseconds) is attempted.- Parameters:
allowedCompactionTime
- the allowed time for compaction (in milliseconds)
-
closeImmediately
public void closeImmediately()Close the file and the store, without writing anything. This will try to stop the background thread (without waiting for it). This method ignores all errors. -
closeStore
private void closeStore(boolean normalShutdown, int allowedCompactionTime) -
shutdownExecutor
-
getChunk
Get the chunk for the given position.- Parameters:
pos
- the position- Returns:
- the chunk
-
setWriteVersion
private void setWriteVersion(long version) -
tryCommit
public long tryCommit()Unlike regular commit this method returns immediately if there is commit in progress on another thread, otherwise it acts as regular commit. This method may return BEFORE this thread changes are actually persisted!- Returns:
- the new version (incremented if there were changes)
-
tryCommit
-
commit
public long commit()Commit the changes.This method does nothing if there are no unsaved changes, otherwise it increments the current version and stores the data (for file based stores).
It is not necessary to call this method when auto-commit is enabled (the default setting), as in this case it is automatically called from time to time or when enough changes have accumulated. However, it may still be called to flush all changes to disk.
At most one store operation may run at any time.
- Returns:
- the new version (incremented if there were changes)
-
commit
-
store
private void store(boolean syncWrite) -
storeNow
-
submitOrRun
private static void submitOrRun(ThreadPoolExecutor executor, Runnable action, boolean syncRun) throws ExecutionException - Throws:
ExecutionException
-
collectChangedMapRoots
-
serializeAndStore
-
createChunk
-
serializeToBuffer
-
storeBuffer
-
isWriteStoreHeader
-
getWriteBuffer
Get a buffer for writing. This caller must synchronize on the store before calling the method and until after using the buffer.- Returns:
- the buffer
-
releaseWriteBuffer
Release a buffer for writing. This caller must synchronize on the store before calling the method and until after using the buffer.- Parameters:
buff
- the buffer than can be re-used
-
canOverwriteChunk
-
isSeasonedChunk
-
getTimeSinceCreation
private long getTimeSinceCreation() -
getTimeAbsolute
private long getTimeAbsolute() -
acceptChunkOccupancyChanges
private void acceptChunkOccupancyChanges(long time, long version) Apply the freed space to the chunk metadata. The metadata is updated, but completely free chunks are not removed from the set of chunks, and the disk space is not yet marked as free. They are queued instead and wait until their usage is over. -
shrinkFileIfPossible
private void shrinkFileIfPossible(int minPercent) Shrink the file if possible, and if at least a given percentage can be saved.- Parameters:
minPercent
- the minimum percentage to save
-
getFileLengthInUse
private long getFileLengthInUse()Get the position right after the last used byte.- Returns:
- the position
-
getAfterLastBlock
private long getAfterLastBlock()Get the index of the first block after last occupied one. It marks the beginning of the last (infinite) free space.- Returns:
- block index
-
measureFileLengthInUse
private long measureFileLengthInUse() -
hasUnsavedChanges
public boolean hasUnsavedChanges()Check whether there are any unsaved changes.- Returns:
- if there are any changes
-
readChunkHeader
-
readChunkHeaderOptionally
-
readChunkHeaderOptionally
-
compactMoveChunks
public void compactMoveChunks()Compact by moving all chunks next to each other. -
compactMoveChunks
boolean compactMoveChunks(int targetFillRate, long moveSize) Compact the store by moving all chunks next to each other, if there is free space between chunks. This might temporarily increase the file size. Chunks are overwritten irrespective of the current retention time. Before overwriting chunks and before resizing the file, syncFile() is called.- Parameters:
targetFillRate
- do nothing if the file store fill rate is higher than thismoveSize
- the number of bytes to move- Returns:
- true if any chunks were moved as result of this operation, false otherwise
-
compactMoveChunks
private boolean compactMoveChunks(long moveSize) -
findChunksToMove
-
getMovePriority
-
compactMoveChunks
-
store
private void store(long reservedLow, long reservedHigh) -
moveChunkInside
-
moveChunk
Move specified chunk into free area of the file. "Reserved" area specifies file interval to be avoided, when un-allocated space will be chosen for a new chunk's location.- Parameters:
chunk
- to movereservedAreaLow
- low boundary of reserved area, inclusivereservedAreaHigh
- high boundary of reserved area, exclusive- Returns:
- true if block was moved, false otherwise
-
sync
public void sync()Force all stored changes to be written to the storage. The default implementation calls FileChannel.force(true). -
compactFile
public void compactFile(int maxCompactTime) Compact store file, that is, compact blocks that have a low fill rate, and move chunks next to each other. This will typically shrink the file. Changes are flushed to the file, and old chunks are overwritten.- Parameters:
maxCompactTime
- the maximum time in milliseconds to compact
-
compact
public boolean compact(int targetFillRate, int write) Try to increase the fill rate by re-writing partially full chunks. Chunks with a low number of live items are re-written.If the current fill rate is higher than the target fill rate, nothing is done.
Please note this method will not necessarily reduce the file size, as empty chunks are not overwritten.
Only data of open maps can be moved. For maps that are not open, the old chunk is still referenced. Therefore, it is recommended to open all maps before calling this method.
- Parameters:
targetFillRate
- the minimum percentage of live entrieswrite
- the minimum number of bytes to write- Returns:
- if a chunk was re-written
-
rewriteChunks
private boolean rewriteChunks(int writeLimit, int targetFillRate) -
getChunksFillRate
public int getChunksFillRate()Get the current fill rate (percentage of used space in the file). Unlike the fill rate of the store, here we only account for chunk data; the fill rate here is how much of the chunk data is live (still referenced). Young chunks are considered live.- Returns:
- the fill rate, in percent (100 is completely full)
-
getRewritableChunksFillRate
public int getRewritableChunksFillRate() -
getChunksFillRate
private int getChunksFillRate(boolean all) -
getChunkCount
public int getChunkCount()Get data chunks count.- Returns:
- number of existing chunks in store.
-
getPageCount
public int getPageCount()Get data pages count.- Returns:
- number of existing pages in store.
-
getLivePageCount
public int getLivePageCount()Get live data pages count.- Returns:
- number of existing live pages in store.
-
getProjectedFillRate
private int getProjectedFillRate(int thresholdChunkFillRate) -
getFillRate
public int getFillRate() -
findOldChunks
-
isRewritable
-
compactRewrite
-
rewriteChunks
-
createIdSet
-
readPage
Read a page.- Type Parameters:
K
- key typeV
- value type- Parameters:
map
- the mappos
- the page position- Returns:
- the page
-
getToC
-
readPageFromCache
-
accountForRemovedPage
void accountForRemovedPage(long pos, long version, boolean pinned, int pageNo) Remove a page.- Parameters:
pos
- the position of the pageversion
- at which page was removedpinned
- whether page is considered pinnedpageNo
- sequential page number within chunk
-
calculatePageNo
private int calculatePageNo(long pos) -
getCompressorFast
Compressor getCompressorFast() -
getCompressorHigh
Compressor getCompressorHigh() -
getCompressionLevel
int getCompressionLevel() -
getPageSplitSize
public int getPageSplitSize() -
getKeysPerPage
public int getKeysPerPage() -
getMaxPageSize
public long getMaxPageSize() -
getReuseSpace
public boolean getReuseSpace() -
setReuseSpace
public void setReuseSpace(boolean reuseSpace) Whether empty space in the file should be re-used. If enabled, old data is overwritten (default). If disabled, writes are appended at the end of the file.This setting is specially useful for online backup. To create an online backup, disable this setting, then copy the file (starting at the beginning of the file). In this case, concurrent backup and write operations are possible (obviously the backup process needs to be faster than the write operations).
- Parameters:
reuseSpace
- the new value
-
getRetentionTime
public int getRetentionTime() -
setRetentionTime
public void setRetentionTime(int ms) How long to retain old, persisted chunks, in milliseconds. Chunks that are older may be overwritten once they contain no live data.The default value is 45000 (45 seconds) when using the default file store. It is assumed that a file system and hard disk will flush all write buffers within this time. Using a lower value might be dangerous, unless the file system and hard disk flush the buffers earlier. To manually flush the buffers, use
MVStore.getFile().force(true)
, however please note that according to various tests this does not always work as expected depending on the operating system and hardware.The retention time needs to be long enough to allow reading old chunks while traversing over the entries of a map.
This setting is not persisted.
- Parameters:
ms
- how many milliseconds to retain old chunks (0 to overwrite them as early as possible)
-
setVersionsToKeep
public void setVersionsToKeep(int count) How many versions to retain for in-memory stores. If not set, 5 old versions are retained.- Parameters:
count
- the number of versions to keep
-
getVersionsToKeep
public long getVersionsToKeep()Get the oldest version to retain in memory (for in-memory stores).- Returns:
- the version
-
getOldestVersionToKeep
long getOldestVersionToKeep()Get the oldest version to retain. We keep at least number of previous versions specified by "versionsToKeep" configuration parameter (default 5). Previously it was used only in case of non-persistent MVStore. Now it's honored in all cases (although H2 always sets it to zero). Oldest version determination also takes into account calls (de)registerVersionUsage(), an will not release the version, while version is still in use.- Returns:
- the version
-
setOldestVersionToKeep
private void setOldestVersionToKeep(long oldestVersionToKeep) -
lastChunkVersion
private long lastChunkVersion() -
isKnownVersion
private boolean isKnownVersion(long version) Check whether all data can be read from this version. This requires that all chunks referenced by this version are still available (not overwritten).- Parameters:
version
- the version- Returns:
- true if all data can be read
-
registerUnsavedMemory
public void registerUnsavedMemory(int memory) Adjust amount of "unsaved memory" meaning amount of RAM occupied by pages not saved yet to the file. This is the amount which triggers auto-commit.- Parameters:
memory
- adjustment
-
isSaveNeeded
boolean isSaveNeeded() -
beforeWrite
This method is called before writing to a map.- Parameters:
map
- the map
-
requireStore
private boolean requireStore() -
needStore
private boolean needStore() -
getStoreVersion
public int getStoreVersion()Get the store version. The store version is usually used to upgrade the structure of the store after upgrading the application. Initially the store version is 0, until it is changed.- Returns:
- the store version
-
setStoreVersion
public void setStoreVersion(int version) Update the store version.- Parameters:
version
- the new store version
-
rollback
public void rollback()Revert to the beginning of the current version, reverting all uncommitted changes. -
rollbackTo
public void rollbackTo(long version) Revert to the beginning of the given version. All later changes (stored or not) are forgotten. All maps that were created later are closed. A rollback to a version before the last stored version is immediately persisted. Rollback to version 0 means all data is removed.- Parameters:
version
- the version to revert to
-
clearCaches
private void clearCaches() -
getRootPos
private long getRootPos(int mapId) -
getCurrentVersion
public long getCurrentVersion()Get the current version of the data. When a new store is created, the version is 0.- Returns:
- the version
-
getFileStore
Get the file store.- Returns:
- the file store
-
getStoreHeader
Get the store header. This data is for informational purposes only. The data is subject to change in future versions. The data should not be modified (doing so may corrupt the store).- Returns:
- the store header
-
checkOpen
private void checkOpen() -
renameMap
Rename a map.- Parameters:
map
- the mapnewName
- the new name
-
removeMap
Remove a map from the current version of the store.- Parameters:
map
- the map to remove
-
deregisterMapRoot
void deregisterMapRoot(int mapId) Performs final stage of map removal - delete root location info from the layout table. Map is supposedly closed and anonymous and has no outstanding usage by now.- Parameters:
mapId
- to deregister
-
removeMap
Remove map by name.- Parameters:
name
- the map name
-
getMapName
Get the name of the given map.- Parameters:
id
- the map id- Returns:
- the name, or null if not found
-
getMapId
-
writeInBackground
void writeInBackground()Commit and save all changes, if there are any, and compact the store if needed. -
doMaintenance
private void doMaintenance(int targetFillRate) -
getTargetFillRate
private int getTargetFillRate() -
isIdle
private boolean isIdle() -
handleException
-
setCacheSize
public void setCacheSize(int mb) Set the read cache size in MB.- Parameters:
mb
- the cache size in MB.
-
isOpen
private boolean isOpen() -
isClosed
public boolean isClosed()Determine that store is open, or wait for it to be closed (by other thread)- Returns:
- true if store is open, false otherwise
-
isOpenOrStopping
private boolean isOpenOrStopping() -
stopBackgroundThread
private void stopBackgroundThread(boolean waitForIt) -
setAutoCommitDelay
public void setAutoCommitDelay(int millis) Set the maximum delay in milliseconds to auto-commit changes.To disable auto-commit, set the value to 0. In this case, changes are only committed when explicitly calling commit.
The default is 1000, meaning all changes are committed after at most one second.
- Parameters:
millis
- the maximum delay
-
createSingleThreadExecutor
-
isBackgroundThread
public boolean isBackgroundThread() -
getAutoCommitDelay
public int getAutoCommitDelay()Get the auto-commit delay.- Returns:
- the delay in milliseconds, or 0 if auto-commit is disabled.
-
getAutoCommitMemory
public int getAutoCommitMemory()Get the maximum memory (in bytes) used for unsaved pages. If this number is exceeded, unsaved changes are stored to disk.- Returns:
- the memory in bytes
-
getUnsavedMemory
public int getUnsavedMemory()Get the estimated memory (in bytes) of unsaved data. If the value exceeds the auto-commit memory, the changes are committed.The returned value is an estimation only.
- Returns:
- the memory in bytes
-
cachePage
Put the page in the cache.- Parameters:
page
- the page
-
getCacheSizeUsed
public int getCacheSizeUsed()Get the amount of memory used for caching, in MB. Note that this does not include the page chunk references cache, which is 25% of the size of the page cache.- Returns:
- the amount of memory used for caching
-
getCacheSize
public int getCacheSize()Get the maximum cache size, in MB. Note that this does not include the page chunk references cache, which is 25% of the size of the page cache.- Returns:
- the cache size
-
getCache
Get the cache.- Returns:
- the cache
-
isReadOnly
public boolean isReadOnly()Whether the store is read-only.- Returns:
- true if it is
-
getCacheHitRatio
public int getCacheHitRatio() -
getTocCacheHitRatio
public int getTocCacheHitRatio() -
getCacheHitRatio
-
getLeafRatio
public int getLeafRatio() -
getUpdateFailureRatio
public double getUpdateFailureRatio() -
registerVersionUsage
Register opened operation (transaction). This would increment usage counter for the current version. This version (and all after it) should not be dropped until all transactions involved are closed and usage counter goes to zero.- Returns:
- TxCounter to be decremented when operation finishes (transaction closed).
-
deregisterVersionUsage
De-register (close) completed operation (transaction). This will decrement usage counter for the corresponding version. If counter reaches zero, that version (and all unused after it) can be dropped immediately.- Parameters:
txCounter
- to be decremented, obtained from registerVersionUsage()
-
onVersionChange
private void onVersionChange(long version) -
dropUnusedVersions
private void dropUnusedVersions() -
dropUnusedChunks
private int dropUnusedChunks() -
freeChunkSpace
-
freeFileSpace
private void freeFileSpace(long start, int length) -
validateFileLength
-