Class Snappy


  • public final class Snappy
    extends java.lang.Object
    Uncompresses an input ByteBuf encoded with Snappy compression into an output ByteBuf. See snappy format.
    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      private static class  Snappy.State  
    • Constructor Summary

      Constructors 
      Constructor Description
      Snappy()  
      Snappy​(boolean reuseHashtable)  
    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      private static int bitsToEncode​(int value)
      Calculates the minimum number of bits required to encode a value.
      (package private) static int calculateChecksum​(ByteBuf data)
      Computes the CRC32C checksum of the supplied data and performs the "mask" operation on the computed checksum
      (package private) static int calculateChecksum​(ByteBuf data, int offset, int length)
      Computes the CRC32C checksum of the supplied data and performs the "mask" operation on the computed checksum
      void decode​(ByteBuf in, ByteBuf out)  
      private static int decodeCopyWith1ByteOffset​(byte tag, ByteBuf in, ByteBuf out, int writtenSoFar)
      Reads a compressed reference offset and length from the supplied input buffer, seeks back to the appropriate place in the input buffer and writes the found data to the supplied output stream.
      private static int decodeCopyWith2ByteOffset​(byte tag, ByteBuf in, ByteBuf out, int writtenSoFar)
      Reads a compressed reference offset and length from the supplied input buffer, seeks back to the appropriate place in the input buffer and writes the found data to the supplied output stream.
      private static int decodeCopyWith4ByteOffset​(byte tag, ByteBuf in, ByteBuf out, int writtenSoFar)
      Reads a compressed reference offset and length from the supplied input buffer, seeks back to the appropriate place in the input buffer and writes the found data to the supplied output stream.
      (package private) static int decodeLiteral​(byte tag, ByteBuf in, ByteBuf out)
      Reads a literal from the input buffer directly to the output buffer.
      void encode​(ByteBuf in, ByteBuf out, int length)  
      private static void encodeCopy​(ByteBuf out, int offset, int length)
      Encodes a series of copies, each at most 64 bytes in length.
      private static void encodeCopyWithOffset​(ByteBuf out, int offset, int length)  
      (package private) static void encodeLiteral​(ByteBuf in, ByteBuf out, int length)
      Writes a literal to the supplied output buffer by directly copying from the input buffer.
      private static int findMatchingLength​(ByteBuf in, int minIndex, int inIndex, int maxIndex)
      Iterates over the supplied input buffer between the supplied minIndex and maxIndex to find how long our matched copy overlaps with an already-written literal value.
      private short[] getHashTable​(int hashTableSize)
      Returns a short[] to be used as a hashtable
      static short[] getHashTableFastThreadLocalArrayFill​(int hashTableSize)
      Returns a short[] from a FastThreadLocal, zeroing for correctness creating a new one and resizing it if necessary
      (package private) int getPreamble​(ByteBuf in)
      Get the length varint (a series of bytes, where the lower 7 bits are data and the upper bit is a flag to indicate more bytes to be read).
      private static int hash​(ByteBuf in, int index, int shift)
      Hashes the 4 bytes located at index, shifting the resulting hash into the appropriate range for our hash table.
      (package private) static int maskChecksum​(long checksum)
      From the spec: "Checksums are not stored directly, but masked, as checksumming data and then its own checksum can be problematic.
      private static int readPreamble​(ByteBuf in)
      Reads the length varint (a series of bytes, where the lower 7 bits are data and the upper bit is a flag to indicate more bytes to be read).
      void reset()  
      (package private) static void validateChecksum​(int expectedChecksum, ByteBuf data)
      Computes the CRC32C checksum of the supplied data, performs the "mask" operation on the computed checksum, and then compares the resulting masked checksum to the supplied checksum.
      (package private) static void validateChecksum​(int expectedChecksum, ByteBuf data, int offset, int length)
      Computes the CRC32C checksum of the supplied data, performs the "mask" operation on the computed checksum, and then compares the resulting masked checksum to the supplied checksum.
      private static void validateOffset​(int offset, int chunkSizeSoFar)
      Validates that the offset extracted from a compressed reference is within the permissible bounds of an offset (0 < offset < Integer.MAX_VALUE), and does not exceed the length of the chunk currently read so far.
      static Snappy withHashTableReuse()  
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Constructor Detail

      • Snappy

        public Snappy()
      • Snappy

        Snappy​(boolean reuseHashtable)
    • Method Detail

      • withHashTableReuse

        public static Snappy withHashTableReuse()
      • reset

        public void reset()
      • encode

        public void encode​(ByteBuf in,
                           ByteBuf out,
                           int length)
      • hash

        private static int hash​(ByteBuf in,
                                int index,
                                int shift)
        Hashes the 4 bytes located at index, shifting the resulting hash into the appropriate range for our hash table.
        Parameters:
        in - The input buffer to read 4 bytes from
        index - The index to read at
        shift - The shift value, for ensuring that the resulting value is within the range of our hash table size
        Returns:
        A 32-bit hash of 4 bytes located at index
      • getHashTable

        private short[] getHashTable​(int hashTableSize)
        Returns a short[] to be used as a hashtable
        Parameters:
        hashTableSize - the size for the hashtable
        Returns:
        An appropriately sized empty hashtable
      • getHashTableFastThreadLocalArrayFill

        public static short[] getHashTableFastThreadLocalArrayFill​(int hashTableSize)
        Returns a short[] from a FastThreadLocal, zeroing for correctness creating a new one and resizing it if necessary
        Parameters:
        hashTableSize -
        Returns:
        An appropriately sized empty hashtable
      • findMatchingLength

        private static int findMatchingLength​(ByteBuf in,
                                              int minIndex,
                                              int inIndex,
                                              int maxIndex)
        Iterates over the supplied input buffer between the supplied minIndex and maxIndex to find how long our matched copy overlaps with an already-written literal value.
        Parameters:
        in - The input buffer to scan over
        minIndex - The index in the input buffer to start scanning from
        inIndex - The index of the start of our copy
        maxIndex - The length of our input buffer
        Returns:
        The number of bytes for which our candidate copy is a repeat of
      • bitsToEncode

        private static int bitsToEncode​(int value)
        Calculates the minimum number of bits required to encode a value. This can then in turn be used to calculate the number of septets or octets (as appropriate) to use to encode a length parameter.
        Parameters:
        value - The value to calculate the minimum number of bits required to encode
        Returns:
        The minimum number of bits required to encode the supplied value
      • encodeLiteral

        static void encodeLiteral​(ByteBuf in,
                                  ByteBuf out,
                                  int length)
        Writes a literal to the supplied output buffer by directly copying from the input buffer. The literal is taken from the current readerIndex up to the supplied length.
        Parameters:
        in - The input buffer to copy from
        out - The output buffer to copy to
        length - The length of the literal to copy
      • encodeCopyWithOffset

        private static void encodeCopyWithOffset​(ByteBuf out,
                                                 int offset,
                                                 int length)
      • encodeCopy

        private static void encodeCopy​(ByteBuf out,
                                       int offset,
                                       int length)
        Encodes a series of copies, each at most 64 bytes in length.
        Parameters:
        out - The output buffer to write the copy pointer to
        offset - The offset at which the original instance lies
        length - The length of the original instance
      • readPreamble

        private static int readPreamble​(ByteBuf in)
        Reads the length varint (a series of bytes, where the lower 7 bits are data and the upper bit is a flag to indicate more bytes to be read).
        Parameters:
        in - The input buffer to read the preamble from
        Returns:
        The calculated length based on the input buffer, or 0 if no preamble is able to be calculated
      • getPreamble

        int getPreamble​(ByteBuf in)
        Get the length varint (a series of bytes, where the lower 7 bits are data and the upper bit is a flag to indicate more bytes to be read).
        Parameters:
        in - The input buffer to get the preamble from
        Returns:
        The calculated length based on the input buffer, or 0 if no preamble is able to be calculated
      • decodeLiteral

        static int decodeLiteral​(byte tag,
                                 ByteBuf in,
                                 ByteBuf out)
        Reads a literal from the input buffer directly to the output buffer. A "literal" is an uncompressed segment of data stored directly in the byte stream.
        Parameters:
        tag - The tag that identified this segment as a literal is also used to encode part of the length of the data
        in - The input buffer to read the literal from
        out - The output buffer to write the literal to
        Returns:
        The number of bytes appended to the output buffer, or -1 to indicate "try again later"
      • decodeCopyWith1ByteOffset

        private static int decodeCopyWith1ByteOffset​(byte tag,
                                                     ByteBuf in,
                                                     ByteBuf out,
                                                     int writtenSoFar)
        Reads a compressed reference offset and length from the supplied input buffer, seeks back to the appropriate place in the input buffer and writes the found data to the supplied output stream.
        Parameters:
        tag - The tag used to identify this as a copy is also used to encode the length and part of the offset
        in - The input buffer to read from
        out - The output buffer to write to
        Returns:
        The number of bytes appended to the output buffer, or -1 to indicate "try again later"
        Throws:
        DecompressionException - If the read offset is invalid
      • decodeCopyWith2ByteOffset

        private static int decodeCopyWith2ByteOffset​(byte tag,
                                                     ByteBuf in,
                                                     ByteBuf out,
                                                     int writtenSoFar)
        Reads a compressed reference offset and length from the supplied input buffer, seeks back to the appropriate place in the input buffer and writes the found data to the supplied output stream.
        Parameters:
        tag - The tag used to identify this as a copy is also used to encode the length and part of the offset
        in - The input buffer to read from
        out - The output buffer to write to
        Returns:
        The number of bytes appended to the output buffer, or -1 to indicate "try again later"
        Throws:
        DecompressionException - If the read offset is invalid
      • decodeCopyWith4ByteOffset

        private static int decodeCopyWith4ByteOffset​(byte tag,
                                                     ByteBuf in,
                                                     ByteBuf out,
                                                     int writtenSoFar)
        Reads a compressed reference offset and length from the supplied input buffer, seeks back to the appropriate place in the input buffer and writes the found data to the supplied output stream.
        Parameters:
        tag - The tag used to identify this as a copy is also used to encode the length and part of the offset
        in - The input buffer to read from
        out - The output buffer to write to
        Returns:
        The number of bytes appended to the output buffer, or -1 to indicate "try again later"
        Throws:
        DecompressionException - If the read offset is invalid
      • validateOffset

        private static void validateOffset​(int offset,
                                           int chunkSizeSoFar)
        Validates that the offset extracted from a compressed reference is within the permissible bounds of an offset (0 < offset < Integer.MAX_VALUE), and does not exceed the length of the chunk currently read so far.
        Parameters:
        offset - The offset extracted from the compressed reference
        chunkSizeSoFar - The number of bytes read so far from this chunk
        Throws:
        DecompressionException - if the offset is invalid
      • calculateChecksum

        static int calculateChecksum​(ByteBuf data)
        Computes the CRC32C checksum of the supplied data and performs the "mask" operation on the computed checksum
        Parameters:
        data - The input data to calculate the CRC32C checksum of
      • calculateChecksum

        static int calculateChecksum​(ByteBuf data,
                                     int offset,
                                     int length)
        Computes the CRC32C checksum of the supplied data and performs the "mask" operation on the computed checksum
        Parameters:
        data - The input data to calculate the CRC32C checksum of
      • validateChecksum

        static void validateChecksum​(int expectedChecksum,
                                     ByteBuf data)
        Computes the CRC32C checksum of the supplied data, performs the "mask" operation on the computed checksum, and then compares the resulting masked checksum to the supplied checksum.
        Parameters:
        expectedChecksum - The checksum decoded from the stream to compare against
        data - The input data to calculate the CRC32C checksum of
        Throws:
        DecompressionException - If the calculated and supplied checksums do not match
      • validateChecksum

        static void validateChecksum​(int expectedChecksum,
                                     ByteBuf data,
                                     int offset,
                                     int length)
        Computes the CRC32C checksum of the supplied data, performs the "mask" operation on the computed checksum, and then compares the resulting masked checksum to the supplied checksum.
        Parameters:
        expectedChecksum - The checksum decoded from the stream to compare against
        data - The input data to calculate the CRC32C checksum of
        Throws:
        DecompressionException - If the calculated and supplied checksums do not match
      • maskChecksum

        static int maskChecksum​(long checksum)
        From the spec: "Checksums are not stored directly, but masked, as checksumming data and then its own checksum can be problematic. The masking is the same as used in Apache Hadoop: Rotate the checksum by 15 bits, then add the constant 0xa282ead8 (using wraparound as normal for unsigned integers)."
        Parameters:
        checksum - The actual checksum of the data
        Returns:
        The masked checksum