Class SnappyOutputStream

  • All Implemented Interfaces:
    java.io.Closeable, java.io.Flushable, java.lang.AutoCloseable
    Direct Known Subclasses:
    SnappyHadoopCompatibleOutputStream

    public class SnappyOutputStream
    extends java.io.OutputStream
    This class implements a stream filter for writing compressed data using Snappy.

    The input data is blocked into 32kb size (in default), and each block is compressed and then passed to the given OutputStream.

    The output data format is:
    1. snappy codec header defined in SnappyCodec (8 bytes)
    2. codec version (4 bytes integer, at least 1) and compatible version (4 bytes integer)
    3. compressed block 1 : a pair of (compressed data size [4 byte integer. Big-endian], compressed data...)
    4. compressed block 2
    5. ...

    Note that the compressed data created by SnappyOutputStream cannot be uncompressed by Snappy.uncompress(byte[]) since the output formats of Snappy.compress(byte[]) and SnappyOutputStream are different. Use SnappyInputStream for uncompress the data generated by SnappyOutputStream.

    • Field Detail

      • out

        protected final java.io.OutputStream out
      • blockSize

        private final int blockSize
      • outputBufferAllocator

        private final BufferAllocator outputBufferAllocator
      • inputBuffer

        protected byte[] inputBuffer
      • outputBuffer

        protected byte[] outputBuffer
      • inputCursor

        private int inputCursor
      • outputCursor

        private int outputCursor
      • headerWritten

        private boolean headerWritten
      • closed

        private boolean closed
    • Constructor Detail

      • SnappyOutputStream

        public SnappyOutputStream​(java.io.OutputStream out)
      • SnappyOutputStream

        public SnappyOutputStream​(java.io.OutputStream out,
                                  int blockSize)
        Parameters:
        out -
        blockSize - byte size of the internal buffer size
        Throws:
        java.lang.IllegalArgumentException - when blockSize is larger than 512 MiB
      • SnappyOutputStream

        public SnappyOutputStream​(java.io.OutputStream out,
                                  int blockSize,
                                  BufferAllocatorFactory bufferAllocatorFactory)
    • Method Detail

      • write

        public void write​(byte[] b,
                          int byteOffset,
                          int byteLength)
                   throws java.io.IOException
        Overrides:
        write in class java.io.OutputStream
        Throws:
        java.io.IOException
      • write

        public void write​(long[] d,
                          int off,
                          int len)
                   throws java.io.IOException
        Compress the input long array data
        Parameters:
        d - input array
        off - offset in the array
        len - the number of elements in the array to copy
        Throws:
        java.io.IOException
      • write

        public void write​(double[] f,
                          int off,
                          int len)
                   throws java.io.IOException
        Compress the input double array data
        Parameters:
        f - input array
        off - offset in the array
        len - the number of elements in the array to copy
        Throws:
        java.io.IOException
      • write

        public void write​(float[] f,
                          int off,
                          int len)
                   throws java.io.IOException
        Compress the input float array data
        Parameters:
        f - input array
        off - offset in the array
        len - the number of elements in the array to copy
        Throws:
        java.io.IOException
      • write

        public void write​(int[] f,
                          int off,
                          int len)
                   throws java.io.IOException
        Compress the input int array data
        Parameters:
        f - input array
        off - offset in the array
        len - the number of elements in the array to copy
        Throws:
        java.io.IOException
      • write

        public void write​(short[] f,
                          int off,
                          int len)
                   throws java.io.IOException
        Compress the input short array data
        Parameters:
        f - input array
        off - offset in the array
        len - the number of elements in the array to copy
        Throws:
        java.io.IOException
      • write

        public void write​(long[] d)
                   throws java.io.IOException
        Compress the input array data
        Parameters:
        d -
        Throws:
        java.io.IOException
      • write

        public void write​(double[] f)
                   throws java.io.IOException
        Compress the input array data
        Parameters:
        f -
        Throws:
        java.io.IOException
      • write

        public void write​(float[] f)
                   throws java.io.IOException
        Compress the input array data
        Parameters:
        f -
        Throws:
        java.io.IOException
      • write

        public void write​(int[] f)
                   throws java.io.IOException
        Compress the input array data
        Parameters:
        f -
        Throws:
        java.io.IOException
      • write

        public void write​(short[] f)
                   throws java.io.IOException
        Compress the input array data
        Parameters:
        f -
        Throws:
        java.io.IOException
      • hasSufficientOutputBufferFor

        private boolean hasSufficientOutputBufferFor​(int inputSize)
      • rawWrite

        public void rawWrite​(java.lang.Object array,
                             int byteOffset,
                             int byteLength)
                      throws java.io.IOException
        Compress the raw byte array data.
        Parameters:
        array - array data of any type (e.g., byte[], float[], long[], ...)
        byteOffset -
        byteLength -
        Throws:
        java.io.IOException
      • write

        public void write​(int b)
                   throws java.io.IOException
        Writes the specified byte to this output stream. The general contract for write is that one byte is written to the output stream. The byte to be written is the eight low-order bits of the argument b. The 24 high-order bits of b are ignored.
        Specified by:
        write in class java.io.OutputStream
        Throws:
        java.io.IOException
      • flush

        public void flush()
                   throws java.io.IOException
        Specified by:
        flush in interface java.io.Flushable
        Overrides:
        flush in class java.io.OutputStream
        Throws:
        java.io.IOException
      • writeInt

        static void writeInt​(byte[] dst,
                             int offset,
                             int v)
      • readInt

        static int readInt​(byte[] buffer,
                           int pos)
      • dumpOutput

        protected void dumpOutput()
                           throws java.io.IOException
        Throws:
        java.io.IOException
      • compressInput

        protected void compressInput()
                              throws java.io.IOException
        Throws:
        java.io.IOException
      • writeHeader

        protected int writeHeader()
      • writeBlockPreemble

        protected void writeBlockPreemble()
        Optionally write a preemble before a block. Hadoop requires the actual block data size being written. This base implementation does nothing. Derive implementation can call writeCurrentDataSize().
      • writeCurrentDataSize

        protected void writeCurrentDataSize()
      • close

        public void close()
                   throws java.io.IOException
        close the stream
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable
        Overrides:
        close in class java.io.OutputStream
        Throws:
        java.io.IOException