Package edu.berkeley.nlp.lm.values
Class CompressibleProbBackoffValueContainer
- java.lang.Object
-
- edu.berkeley.nlp.lm.values.CompressibleProbBackoffValueContainer
-
- All Implemented Interfaces:
CompressibleValueContainer<ProbBackoffPair>
,ProbBackoffValueContainer
,ValueContainer<ProbBackoffPair>
,java.io.Serializable
public final class CompressibleProbBackoffValueContainer extends java.lang.Object implements ProbBackoffValueContainer
- See Also:
- Serialized Form
-
-
Field Summary
Fields Modifier and Type Field Description protected int
defaultValRank
protected long[]
numNgramsForEachOrder
protected boolean
storeSuffixIndexes
protected int[]
suffixBitsForOrder
protected boolean
useMapValueArray
protected VariableLengthBitCompressor
valueCoder
protected int
valueRadix
protected CustomWidthArray[]
valueRanks
protected int
valueWidth
-
Constructor Summary
Constructors Constructor Description CompressibleProbBackoffValueContainer(int valueRadix, boolean storePrefixIndexes, long[] numNgramsForEachOrder, float[] probsForRank, float[] backoffsForRank, Indexer<java.lang.Float> probIndexer, int wordWidth, Indexer<java.lang.Float> backoffIndexer, int backoffWidth)
CompressibleProbBackoffValueContainer(LongToIntHashMap countCounter, int valueRadix, boolean storePrefixes, long[] numNgramsForEachOrder)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description boolean
add(int[] ngram, int startPos, int endPos, int ngramOrder, long offset, long prefixOffset, int word, V val_, long suffixOffset, boolean ngramIsNew)
Adds a new value at the specified offset.void
clearStorageAfterCompression(int ngramOrder)
void
clearStorageForOrder(int ngramOrder)
CompressibleProbBackoffValueContainer
createFreshValues(long[] numNgramsForEachOrder_)
Creates a fresh value container for copying purposes.void
decompress(BitStream bits, int ngramOrder, boolean justConsume, ProbBackoffPair outputVal)
Reads and decompresses from the bit stream bits.float
getBackoff(int ngramOrder, long index)
float
getBackoff(CustomWidthArray valueRanksForNgramOrder, long index)
BitList
getCompressed(long offset, int ngramOrder)
Compresses the value at the given offset into a list of bits.protected long
getCountRank(long val)
protected ProbBackoffPair
getDefaultVal()
void
getFromOffset(long index, int ngramOrder, ProbBackoffPair outputVal)
Gets the value living at a particular offset.protected void
getFromRank(long rank, ProbBackoffPair outputVal)
long
getInternalVal(int ngramOrder, long index)
float
getProb(int ngramOrder, long index)
float
getProb(CustomWidthArray valueRanksForOrder, long index)
protected long
getRank(int ngramOrder, long offset)
ProbBackoffPair
getScratchValue()
Creates a fresh value of object (useful for passing as an output parameter)long
getSuffixOffset(long index, int ngramOrder)
protected int
getSuffixOffsetFromInternalVal(long internalVal)
int
numValueBits(int ngramOrder)
void
setFromOtherValues(ValueContainer<ProbBackoffPair> o)
Destructively sets internal storage from another object.void
setMap(NgramMap<V> map)
Initializes a value container with the map that contains itvoid
setSizeAtLeast(long size, int ngramOrder)
Sets internal storage for size for a particular n-gram orderboolean
storeSuffixoffsets()
void
swap(long a, long b, int ngramOrder)
Swaps values at offsets a and b.void
trim()
Final clean up of storage.void
trimAfterNgram(int ngramOrder, long size)
Clear storage after an n-gram order is completeprotected boolean
useValueStoringArray()
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface edu.berkeley.nlp.lm.values.ProbBackoffValueContainer
getSuffixOffset
-
Methods inherited from interface edu.berkeley.nlp.lm.values.ValueContainer
add, clearStorageForOrder, numValueBits, setMap, setSizeAtLeast, storeSuffixoffsets, trimAfterNgram
-
-
-
-
Field Detail
-
valueRanks
protected final CustomWidthArray[] valueRanks
-
storeSuffixIndexes
protected final boolean storeSuffixIndexes
-
valueCoder
protected final VariableLengthBitCompressor valueCoder
-
valueRadix
protected final int valueRadix
-
valueWidth
protected int valueWidth
-
defaultValRank
protected final int defaultValRank
- See Also:
- Constant Field Values
-
numNgramsForEachOrder
protected final long[] numNgramsForEachOrder
-
suffixBitsForOrder
protected final int[] suffixBitsForOrder
-
useMapValueArray
protected boolean useMapValueArray
-
-
Constructor Detail
-
CompressibleProbBackoffValueContainer
public CompressibleProbBackoffValueContainer(LongToIntHashMap countCounter, int valueRadix, boolean storePrefixes, long[] numNgramsForEachOrder)
-
CompressibleProbBackoffValueContainer
public CompressibleProbBackoffValueContainer(int valueRadix, boolean storePrefixIndexes, long[] numNgramsForEachOrder, float[] probsForRank, float[] backoffsForRank, Indexer<java.lang.Float> probIndexer, int wordWidth, Indexer<java.lang.Float> backoffIndexer, int backoffWidth)
- Parameters:
valueRadix
-storePrefixIndexes
-maxNgramOrder
-hasBackoffValIndexer
-noBackoffValIndexer
-probsAndBackoffsForRank
-probsForRank
-hasBackoffValIndexer
-
-
-
Method Detail
-
createFreshValues
public CompressibleProbBackoffValueContainer createFreshValues(long[] numNgramsForEachOrder_)
Description copied from interface:ValueContainer
Creates a fresh value container for copying purposes.- Specified by:
createFreshValues
in interfaceValueContainer<ProbBackoffPair>
- Returns:
-
getProb
public final float getProb(int ngramOrder, long index)
- Specified by:
getProb
in interfaceProbBackoffValueContainer
-
getInternalVal
public final long getInternalVal(int ngramOrder, long index)
-
getProb
public final float getProb(CustomWidthArray valueRanksForOrder, long index)
-
getFromOffset
public void getFromOffset(long index, int ngramOrder, ProbBackoffPair outputVal)
Description copied from interface:ValueContainer
Gets the value living at a particular offset.- Specified by:
getFromOffset
in interfaceValueContainer<ProbBackoffPair>
-
getBackoff
public final float getBackoff(int ngramOrder, long index)
- Specified by:
getBackoff
in interfaceProbBackoffValueContainer
-
getBackoff
public final float getBackoff(CustomWidthArray valueRanksForNgramOrder, long index)
-
getDefaultVal
protected ProbBackoffPair getDefaultVal()
-
getFromRank
protected void getFromRank(long rank, ProbBackoffPair outputVal)
-
getScratchValue
public ProbBackoffPair getScratchValue()
Description copied from interface:ValueContainer
Creates a fresh value of object (useful for passing as an output parameter)- Specified by:
getScratchValue
in interfaceProbBackoffValueContainer
- Specified by:
getScratchValue
in interfaceValueContainer<ProbBackoffPair>
- Returns:
-
setFromOtherValues
public void setFromOtherValues(ValueContainer<ProbBackoffPair> o)
Description copied from interface:ValueContainer
Destructively sets internal storage from another object.- Specified by:
setFromOtherValues
in interfaceValueContainer<ProbBackoffPair>
-
trim
public void trim()
Description copied from interface:ValueContainer
Final clean up of storage.- Specified by:
trim
in interfaceValueContainer<ProbBackoffPair>
-
getCountRank
protected long getCountRank(long val)
-
getCompressed
public BitList getCompressed(long offset, int ngramOrder)
Description copied from interface:CompressibleValueContainer
Compresses the value at the given offset into a list of bits.- Specified by:
getCompressed
in interfaceCompressibleValueContainer<ProbBackoffPair>
- Returns:
-
decompress
public final void decompress(BitStream bits, int ngramOrder, boolean justConsume, ProbBackoffPair outputVal)
Description copied from interface:CompressibleValueContainer
Reads and decompresses from the bit stream bits.- Specified by:
decompress
in interfaceCompressibleValueContainer<ProbBackoffPair>
justConsume
- If true, nothing is returned, and the function simply consumes the appropriate number of bits from the BitStream.
-
setMap
public void setMap(NgramMap<V> map)
Description copied from interface:ValueContainer
Initializes a value container with the map that contains it- Specified by:
setMap
in interfaceValueContainer<V extends LongRepresentable<V>>
-
useValueStoringArray
protected boolean useValueStoringArray()
-
swap
public void swap(long a, long b, int ngramOrder)
Description copied from interface:CompressibleValueContainer
Swaps values at offsets a and b.- Specified by:
swap
in interfaceCompressibleValueContainer<V extends LongRepresentable<V>>
-
add
public boolean add(int[] ngram, int startPos, int endPos, int ngramOrder, long offset, long prefixOffset, int word, V val_, long suffixOffset, boolean ngramIsNew)
Description copied from interface:ValueContainer
Adds a new value at the specified offset.- Specified by:
add
in interfaceValueContainer<V extends LongRepresentable<V>>
ngramOrder
- As always, ngramOrder is 0-based (0=unigram)- Returns:
- Whether or not the add was successful
-
setSizeAtLeast
public void setSizeAtLeast(long size, int ngramOrder)
Description copied from interface:ValueContainer
Sets internal storage for size for a particular n-gram order- Specified by:
setSizeAtLeast
in interfaceValueContainer<V extends LongRepresentable<V>>
-
getSuffixOffset
public long getSuffixOffset(long index, int ngramOrder)
-
getSuffixOffsetFromInternalVal
protected int getSuffixOffsetFromInternalVal(long internalVal)
- Parameters:
internalVal
-- Returns:
-
getRank
protected long getRank(int ngramOrder, long offset)
-
clearStorageAfterCompression
public void clearStorageAfterCompression(int ngramOrder)
- Specified by:
clearStorageAfterCompression
in interfaceCompressibleValueContainer<V extends LongRepresentable<V>>
-
trimAfterNgram
public void trimAfterNgram(int ngramOrder, long size)
Description copied from interface:ValueContainer
Clear storage after an n-gram order is complete- Specified by:
trimAfterNgram
in interfaceValueContainer<V extends LongRepresentable<V>>
-
clearStorageForOrder
public void clearStorageForOrder(int ngramOrder)
- Specified by:
clearStorageForOrder
in interfaceValueContainer<V extends LongRepresentable<V>>
-
storeSuffixoffsets
public boolean storeSuffixoffsets()
- Specified by:
storeSuffixoffsets
in interfaceValueContainer<V extends LongRepresentable<V>>
-
numValueBits
public int numValueBits(int ngramOrder)
- Specified by:
numValueBits
in interfaceValueContainer<V extends LongRepresentable<V>>
-
-