Package org.ojalgo.data.batch
Class BatchNode.Builder<T>
- java.lang.Object
-
- org.ojalgo.data.batch.BatchNode.Builder<T>
-
-
Field Summary
Fields Modifier and Type Field Description private java.io.File
myDirectory
private java.util.function.ToIntFunction<T>
myDistributor
private java.util.concurrent.ExecutorService
myExecutor
private int
myFragmentation
private DataInterpreter<T>
myInterpreter
private int
myParallelism
private int
myQueueCapacity
-
Constructor Summary
Constructors Constructor Description Builder(java.io.File directory, DataInterpreter<T> interpreter)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description BatchNode<T>
build()
BatchNode.Builder<T>
distributor(java.util.function.ToIntFunction<T> distributor)
The default is to distribute randomly.BatchNode.Builder<T>
executor(java.util.concurrent.ExecutorService executor)
BatchNode.Builder<T>
fragmentation(int fragmentation)
The number of underlying files/shards.(package private) java.util.function.ToIntFunction<T>
getDistributor()
(package private) int
getFragmentation()
The total number of files/shards.(package private) DataInterpreter<T>
getInterpreter()
(package private) java.lang.String
getName()
(package private) java.util.function.IntSupplier
getParallelism()
Will always be power of 2(package private) ProcessingService
getProcessor()
(package private) int
getQueueCapacity()
(package private) ShardedFile
getShardedFile()
BatchNode.Builder<T>
parallelism(int parallelism)
BatchNode.Builder<T>
parallelism(java.util.function.IntSupplier parallelism)
How many worker threads should process data in parallel?BatchNode.Builder<T>
queue(int capacity)
When reading and/or writing data from/to disk data is temporarily queued.
-
-
-
Field Detail
-
myDirectory
private final java.io.File myDirectory
-
myDistributor
private java.util.function.ToIntFunction<T> myDistributor
-
myExecutor
private java.util.concurrent.ExecutorService myExecutor
-
myFragmentation
private int myFragmentation
-
myInterpreter
private final DataInterpreter<T> myInterpreter
-
myParallelism
private int myParallelism
-
myQueueCapacity
private int myQueueCapacity
-
-
Constructor Detail
-
Builder
Builder(java.io.File directory, DataInterpreter<T> interpreter)
-
-
Method Detail
-
distributor
public BatchNode.Builder<T> distributor(java.util.function.ToIntFunction<T> distributor)
The default is to distribute randomly. Most likely you want to distribute based on some property of the item/type – extract that property and get its hash code. That causes all items with same value on that property to end up in the same shard, and that you can exploit when processing the data.
-
executor
public BatchNode.Builder<T> executor(java.util.concurrent.ExecutorService executor)
-
fragmentation
public BatchNode.Builder<T> fragmentation(int fragmentation)
The number of underlying files/shards. Increasing the fragmentation (the number of shards) typically reduces memory requirements when processong. The value set here is only an indication of the desired order of magnitude. The exact number of shards actually used is a derived property.
-
parallelism
public BatchNode.Builder<T> parallelism(int parallelism)
- See Also:
parallelism(IntSupplier)
-
parallelism
public BatchNode.Builder<T> parallelism(java.util.function.IntSupplier parallelism)
How many worker threads should process data in parallel?
-
queue
public BatchNode.Builder<T> queue(int capacity)
When reading and/or writing data from/to disk data is temporarily queued. This specifies the total maximum number of items kept in the queues.
-
getDistributor
java.util.function.ToIntFunction<T> getDistributor()
-
getFragmentation
int getFragmentation()
The total number of files/shards. Will always be power of 2 as well as a multiple ofgetParallelism()
.
-
getInterpreter
DataInterpreter<T> getInterpreter()
-
getName
java.lang.String getName()
-
getParallelism
java.util.function.IntSupplier getParallelism()
Will always be power of 2
-
getProcessor
ProcessingService getProcessor()
-
getQueueCapacity
int getQueueCapacity()
-
getShardedFile
ShardedFile getShardedFile()
-
-