Package org.ojalgo.netio
Class SegmentedFile
- java.lang.Object
-
- org.ojalgo.netio.SegmentedFile
-
- All Implemented Interfaces:
java.lang.AutoCloseable
public class SegmentedFile extends java.lang.Object implements java.lang.AutoCloseable
Divides a large file in segments and then reads those (in parallel) using memory mapped files (memory mapped file segments). Basic usage:- Call
newBuilder(File)
to get aSegmentedFile.Builder
orof(File)
to get aSegmentedFile
directly with the default settings. - Then for each of the segments obtained from
segments()
: CallnewTextLineReader(Segment)
and using that read all the lines.
TextLineReader
for each segment.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
SegmentedFile.Builder
static class
SegmentedFile.Segment
-
Field Summary
Fields Modifier and Type Field Description private java.nio.channels.FileChannel
myFileChannel
private java.io.RandomAccessFile
myRandomAccessFile
private SegmentedFile.Segment[]
mySegments
-
Constructor Summary
Constructors Constructor Description SegmentedFile(java.io.RandomAccessFile file, SegmentedFile.Segment[] segments)
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description void
close()
(package private) SegmentedFile.Segment[]
getSegments()
static SegmentedFile.Builder
newBuilder(java.io.File file)
<T> FromFileReader<T>
newDataReader(DataReader.Deserializer<T> deserializer)
<T> DataReader<T>
newDataReader(SegmentedFile.Segment segment, DataReader.Deserializer<T> deserializer)
<T> java.util.function.Supplier<FromFileReader<T>>
newSequencedFactory(java.util.function.Function<SegmentedFile.Segment,FromFileReader<T>> factory)
Each reader instantiated by this factory will read from the segments in sequence, until all of them are done.FromFileReader<java.lang.String>
newTextLineReader()
TextLineReader
newTextLineReader(SegmentedFile.Segment segment)
Call this once for each file segment, and use the returnedTextLineReader
to read the file segment.<T> FromFileReader<T>
newTextLineReader(TextLineReader.Parser<T> parser)
static SegmentedFile
of(java.io.File file)
java.util.List<SegmentedFile.Segment>
segments()
-
-
-
Field Detail
-
myFileChannel
private final java.nio.channels.FileChannel myFileChannel
-
myRandomAccessFile
private final java.io.RandomAccessFile myRandomAccessFile
-
mySegments
private final SegmentedFile.Segment[] mySegments
-
-
Constructor Detail
-
SegmentedFile
SegmentedFile(java.io.RandomAccessFile file, SegmentedFile.Segment[] segments)
-
-
Method Detail
-
newBuilder
public static SegmentedFile.Builder newBuilder(java.io.File file)
-
of
public static SegmentedFile of(java.io.File file)
-
close
public void close()
- Specified by:
close
in interfacejava.lang.AutoCloseable
-
newDataReader
public <T> FromFileReader<T> newDataReader(DataReader.Deserializer<T> deserializer)
-
newDataReader
public <T> DataReader<T> newDataReader(SegmentedFile.Segment segment, DataReader.Deserializer<T> deserializer)
-
newSequencedFactory
public <T> java.util.function.Supplier<FromFileReader<T>> newSequencedFactory(java.util.function.Function<SegmentedFile.Segment,FromFileReader<T>> factory)
Each reader instantiated by this factory will read from the segments in sequence, until all of them are done. The idea is that you can create multiple readers and have them work in parallel (each segment will only be read once by one of the readers). If you only instantiate 1 reader, then maybe you shouldn't have created file segments in the first place.
-
newTextLineReader
public FromFileReader<java.lang.String> newTextLineReader()
-
newTextLineReader
public TextLineReader newTextLineReader(SegmentedFile.Segment segment)
Call this once for each file segment, and use the returnedTextLineReader
to read the file segment. TheTextLineReader
is not thread safe, and should only be used by a single thread.The segment is a range of bytes in the file. The
TextLineReader
will read the bytes in the range [offset, offset + size). The segments are obtained fromsegments()
. The segments are sorted in ascending order by offset.
-
newTextLineReader
public <T> FromFileReader<T> newTextLineReader(TextLineReader.Parser<T> parser)
-
segments
public java.util.List<SegmentedFile.Segment> segments()
-
getSegments
SegmentedFile.Segment[] getSegments()
-
-