Package org.xerial.snappy
Class SnappyOutputStream
- java.lang.Object
-
- java.io.OutputStream
-
- org.xerial.snappy.SnappyOutputStream
-
- All Implemented Interfaces:
java.io.Closeable
,java.io.Flushable
,java.lang.AutoCloseable
- Direct Known Subclasses:
SnappyHadoopCompatibleOutputStream
public class SnappyOutputStream extends java.io.OutputStream
This class implements a stream filter for writing compressed data using Snappy.The input data is blocked into 32kb size (in default), and each block is compressed and then passed to the given
The output data format is:OutputStream
.- snappy codec header defined in
SnappyCodec
(8 bytes) - codec version (4 bytes integer, at least 1) and compatible version (4 bytes integer)
- compressed block 1 : a pair of (compressed data size [4 byte integer. Big-endian], compressed data...)
- compressed block 2
- ...
SnappyOutputStream
cannot be uncompressed bySnappy.uncompress(byte[])
since the output formats ofSnappy.compress(byte[])
andSnappyOutputStream
are different. UseSnappyInputStream
for uncompress the data generated bySnappyOutputStream
.
-
-
Field Summary
Fields Modifier and Type Field Description private int
blockSize
private boolean
closed
(package private) static int
DEFAULT_BLOCK_SIZE
private boolean
headerWritten
protected byte[]
inputBuffer
private BufferAllocator
inputBufferAllocator
private int
inputCursor
static int
MAX_BLOCK_SIZE
(package private) static int
MIN_BLOCK_SIZE
protected java.io.OutputStream
out
protected byte[]
outputBuffer
private BufferAllocator
outputBufferAllocator
private int
outputCursor
-
Constructor Summary
Constructors Constructor Description SnappyOutputStream(java.io.OutputStream out)
SnappyOutputStream(java.io.OutputStream out, int blockSize)
SnappyOutputStream(java.io.OutputStream out, int blockSize, BufferAllocatorFactory bufferAllocatorFactory)
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description void
close()
close the streamprotected void
compressInput()
protected void
dumpOutput()
void
flush()
private boolean
hasSufficientOutputBufferFor(int inputSize)
void
rawWrite(java.lang.Object array, int byteOffset, int byteLength)
Compress the raw byte array data.(package private) static int
readInt(byte[] buffer, int pos)
void
write(byte[] b, int byteOffset, int byteLength)
void
write(double[] f)
Compress the input array datavoid
write(double[] f, int off, int len)
Compress the input double array datavoid
write(float[] f)
Compress the input array datavoid
write(float[] f, int off, int len)
Compress the input float array datavoid
write(int b)
Writes the specified byte to this output stream.void
write(int[] f)
Compress the input array datavoid
write(int[] f, int off, int len)
Compress the input int array datavoid
write(long[] d)
Compress the input array datavoid
write(long[] d, int off, int len)
Compress the input long array datavoid
write(short[] f)
Compress the input array datavoid
write(short[] f, int off, int len)
Compress the input short array dataprotected void
writeBlockPreemble()
Optionally write a preemble before a block.protected void
writeCurrentDataSize()
protected int
writeHeader()
(package private) static void
writeInt(byte[] dst, int offset, int v)
-
-
-
Field Detail
-
MAX_BLOCK_SIZE
public static final int MAX_BLOCK_SIZE
- See Also:
- Constant Field Values
-
MIN_BLOCK_SIZE
static final int MIN_BLOCK_SIZE
- See Also:
- Constant Field Values
-
DEFAULT_BLOCK_SIZE
static final int DEFAULT_BLOCK_SIZE
- See Also:
- Constant Field Values
-
out
protected final java.io.OutputStream out
-
blockSize
private final int blockSize
-
inputBufferAllocator
private final BufferAllocator inputBufferAllocator
-
outputBufferAllocator
private final BufferAllocator outputBufferAllocator
-
inputBuffer
protected byte[] inputBuffer
-
outputBuffer
protected byte[] outputBuffer
-
inputCursor
private int inputCursor
-
outputCursor
private int outputCursor
-
headerWritten
private boolean headerWritten
-
closed
private boolean closed
-
-
Constructor Detail
-
SnappyOutputStream
public SnappyOutputStream(java.io.OutputStream out)
-
SnappyOutputStream
public SnappyOutputStream(java.io.OutputStream out, int blockSize)
- Parameters:
out
-blockSize
- byte size of the internal buffer size- Throws:
java.lang.IllegalArgumentException
- when blockSize is larger than 512 MiB
-
SnappyOutputStream
public SnappyOutputStream(java.io.OutputStream out, int blockSize, BufferAllocatorFactory bufferAllocatorFactory)
-
-
Method Detail
-
write
public void write(byte[] b, int byteOffset, int byteLength) throws java.io.IOException
- Overrides:
write
in classjava.io.OutputStream
- Throws:
java.io.IOException
-
write
public void write(long[] d, int off, int len) throws java.io.IOException
Compress the input long array data- Parameters:
d
- input arrayoff
- offset in the arraylen
- the number of elements in the array to copy- Throws:
java.io.IOException
-
write
public void write(double[] f, int off, int len) throws java.io.IOException
Compress the input double array data- Parameters:
f
- input arrayoff
- offset in the arraylen
- the number of elements in the array to copy- Throws:
java.io.IOException
-
write
public void write(float[] f, int off, int len) throws java.io.IOException
Compress the input float array data- Parameters:
f
- input arrayoff
- offset in the arraylen
- the number of elements in the array to copy- Throws:
java.io.IOException
-
write
public void write(int[] f, int off, int len) throws java.io.IOException
Compress the input int array data- Parameters:
f
- input arrayoff
- offset in the arraylen
- the number of elements in the array to copy- Throws:
java.io.IOException
-
write
public void write(short[] f, int off, int len) throws java.io.IOException
Compress the input short array data- Parameters:
f
- input arrayoff
- offset in the arraylen
- the number of elements in the array to copy- Throws:
java.io.IOException
-
write
public void write(long[] d) throws java.io.IOException
Compress the input array data- Parameters:
d
-- Throws:
java.io.IOException
-
write
public void write(double[] f) throws java.io.IOException
Compress the input array data- Parameters:
f
-- Throws:
java.io.IOException
-
write
public void write(float[] f) throws java.io.IOException
Compress the input array data- Parameters:
f
-- Throws:
java.io.IOException
-
write
public void write(int[] f) throws java.io.IOException
Compress the input array data- Parameters:
f
-- Throws:
java.io.IOException
-
write
public void write(short[] f) throws java.io.IOException
Compress the input array data- Parameters:
f
-- Throws:
java.io.IOException
-
hasSufficientOutputBufferFor
private boolean hasSufficientOutputBufferFor(int inputSize)
-
rawWrite
public void rawWrite(java.lang.Object array, int byteOffset, int byteLength) throws java.io.IOException
Compress the raw byte array data.- Parameters:
array
- array data of any type (e.g., byte[], float[], long[], ...)byteOffset
-byteLength
-- Throws:
java.io.IOException
-
write
public void write(int b) throws java.io.IOException
Writes the specified byte to this output stream. The general contract for write is that one byte is written to the output stream. The byte to be written is the eight low-order bits of the argument b. The 24 high-order bits of b are ignored.- Specified by:
write
in classjava.io.OutputStream
- Throws:
java.io.IOException
-
flush
public void flush() throws java.io.IOException
- Specified by:
flush
in interfacejava.io.Flushable
- Overrides:
flush
in classjava.io.OutputStream
- Throws:
java.io.IOException
-
writeInt
static void writeInt(byte[] dst, int offset, int v)
-
readInt
static int readInt(byte[] buffer, int pos)
-
dumpOutput
protected void dumpOutput() throws java.io.IOException
- Throws:
java.io.IOException
-
compressInput
protected void compressInput() throws java.io.IOException
- Throws:
java.io.IOException
-
writeHeader
protected int writeHeader()
-
writeBlockPreemble
protected void writeBlockPreemble()
Optionally write a preemble before a block. Hadoop requires the actual block data size being written. This base implementation does nothing. Derive implementation can callwriteCurrentDataSize()
.
-
writeCurrentDataSize
protected void writeCurrentDataSize()
-
close
public void close() throws java.io.IOException
close the stream- Specified by:
close
in interfacejava.lang.AutoCloseable
- Specified by:
close
in interfacejava.io.Closeable
- Overrides:
close
in classjava.io.OutputStream
- Throws:
java.io.IOException
-
-