Package edu.berkeley.nlp.lm.values
Class CountValueContainer
- java.lang.Object
-
- edu.berkeley.nlp.lm.values.CountValueContainer
-
- All Implemented Interfaces:
CompressibleValueContainer<LongRef>
,ValueContainer<LongRef>
,java.io.Serializable
public final class CountValueContainer extends java.lang.Object
- See Also:
- Serialized Form
-
-
Field Summary
Fields Modifier and Type Field Description protected int
defaultValRank
protected long[]
numNgramsForEachOrder
protected boolean
storeSuffixIndexes
protected int[]
suffixBitsForOrder
protected boolean
useMapValueArray
protected VariableLengthBitCompressor
valueCoder
protected int
valueRadix
protected CustomWidthArray[]
valueRanks
protected int
valueWidth
-
Constructor Summary
Constructors Constructor Description CountValueContainer(LongToIntHashMap countCounter, int valueRadix, boolean storePrefixes, long[] numNgramsForEachOrder)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description boolean
add(int[] ngram, int startPos, int endPos, int ngramOrder, long offset, long prefixOffset, int word, V val_, long suffixOffset, boolean ngramIsNew)
Adds a new value at the specified offset.void
clearStorageAfterCompression(int ngramOrder)
void
clearStorageForOrder(int ngramOrder)
CountValueContainer
createFreshValues(long[] numNgramsForEachOrder_)
Creates a fresh value container for copying purposes.void
decompress(BitStream bits, int ngramOrder, boolean justConsume, V outputVal)
Reads and decompresses from the bit stream bits.BitList
getCompressed(long offset, int ngramOrder)
Compresses the value at the given offset into a list of bits.long
getCount(int ngramOrder, long index)
protected long
getCountRank(long val)
protected LongRef
getDefaultVal()
void
getFromOffset(long index, int ngramOrder, LongRef outputVal)
Gets the value living at a particular offset.protected void
getFromRank(long rank, LongRef outputVal)
protected long
getRank(int ngramOrder, long offset)
LongRef
getScratchValue()
Creates a fresh value of object (useful for passing as an output parameter)long
getSuffixOffset(long index, int ngramOrder)
protected int
getSuffixOffsetFromInternalVal(long internalVal)
long
getUnigramSum()
int
numValueBits(int ngramOrder)
void
setFromOtherValues(ValueContainer<LongRef> o)
Destructively sets internal storage from another object.void
setMap(NgramMap<V> map)
Initializes a value container with the map that contains itvoid
setSizeAtLeast(long size, int ngramOrder)
Sets internal storage for size for a particular n-gram orderboolean
storeSuffixoffsets()
void
swap(long a, long b, int ngramOrder)
Swaps values at offsets a and b.void
trim()
Final clean up of storage.void
trimAfterNgram(int ngramOrder, long size)
Clear storage after an n-gram order is completeprotected boolean
useValueStoringArray()
-
-
-
Field Detail
-
valueRanks
protected final CustomWidthArray[] valueRanks
-
storeSuffixIndexes
protected final boolean storeSuffixIndexes
-
valueCoder
protected final VariableLengthBitCompressor valueCoder
-
valueRadix
protected final int valueRadix
-
valueWidth
protected int valueWidth
-
defaultValRank
protected final int defaultValRank
- See Also:
- Constant Field Values
-
numNgramsForEachOrder
protected final long[] numNgramsForEachOrder
-
suffixBitsForOrder
protected final int[] suffixBitsForOrder
-
useMapValueArray
protected boolean useMapValueArray
-
-
Constructor Detail
-
CountValueContainer
public CountValueContainer(LongToIntHashMap countCounter, int valueRadix, boolean storePrefixes, long[] numNgramsForEachOrder)
-
-
Method Detail
-
createFreshValues
public CountValueContainer createFreshValues(long[] numNgramsForEachOrder_)
Description copied from interface:ValueContainer
Creates a fresh value container for copying purposes.- Returns:
-
getFromOffset
public void getFromOffset(long index, int ngramOrder, LongRef outputVal)
Description copied from interface:ValueContainer
Gets the value living at a particular offset.
-
getFromRank
protected void getFromRank(long rank, LongRef outputVal)
-
getCount
public final long getCount(int ngramOrder, long index)
-
getDefaultVal
protected LongRef getDefaultVal()
-
trimAfterNgram
public void trimAfterNgram(int ngramOrder, long size)
Description copied from interface:ValueContainer
Clear storage after an n-gram order is complete- Specified by:
trimAfterNgram
in interfaceValueContainer<LongRef>
-
getUnigramSum
public long getUnigramSum()
-
getScratchValue
public LongRef getScratchValue()
Description copied from interface:ValueContainer
Creates a fresh value of object (useful for passing as an output parameter)- Returns:
-
setFromOtherValues
public void setFromOtherValues(ValueContainer<LongRef> o)
Description copied from interface:ValueContainer
Destructively sets internal storage from another object.- Specified by:
setFromOtherValues
in interfaceValueContainer<LongRef>
-
trim
public void trim()
Description copied from interface:ValueContainer
Final clean up of storage.- Specified by:
trim
in interfaceValueContainer<LongRef>
-
getCountRank
protected long getCountRank(long val)
-
setMap
public void setMap(NgramMap<V> map)
Description copied from interface:ValueContainer
Initializes a value container with the map that contains it- Specified by:
setMap
in interfaceValueContainer<V extends LongRepresentable<V>>
-
useValueStoringArray
protected boolean useValueStoringArray()
-
swap
public void swap(long a, long b, int ngramOrder)
Description copied from interface:CompressibleValueContainer
Swaps values at offsets a and b.- Specified by:
swap
in interfaceCompressibleValueContainer<V extends LongRepresentable<V>>
-
add
public boolean add(int[] ngram, int startPos, int endPos, int ngramOrder, long offset, long prefixOffset, int word, V val_, long suffixOffset, boolean ngramIsNew)
Description copied from interface:ValueContainer
Adds a new value at the specified offset.- Specified by:
add
in interfaceValueContainer<V extends LongRepresentable<V>>
ngramOrder
- As always, ngramOrder is 0-based (0=unigram)- Returns:
- Whether or not the add was successful
-
getCompressed
public BitList getCompressed(long offset, int ngramOrder)
Description copied from interface:CompressibleValueContainer
Compresses the value at the given offset into a list of bits.- Specified by:
getCompressed
in interfaceCompressibleValueContainer<V extends LongRepresentable<V>>
- Returns:
-
decompress
public void decompress(BitStream bits, int ngramOrder, boolean justConsume, V outputVal)
Description copied from interface:CompressibleValueContainer
Reads and decompresses from the bit stream bits.- Specified by:
decompress
in interfaceCompressibleValueContainer<V extends LongRepresentable<V>>
justConsume
- If true, nothing is returned, and the function simply consumes the appropriate number of bits from the BitStream.
-
setSizeAtLeast
public void setSizeAtLeast(long size, int ngramOrder)
Description copied from interface:ValueContainer
Sets internal storage for size for a particular n-gram order- Specified by:
setSizeAtLeast
in interfaceValueContainer<V extends LongRepresentable<V>>
-
getSuffixOffset
public long getSuffixOffset(long index, int ngramOrder)
-
getSuffixOffsetFromInternalVal
protected int getSuffixOffsetFromInternalVal(long internalVal)
- Parameters:
internalVal
-- Returns:
-
getRank
protected long getRank(int ngramOrder, long offset)
-
clearStorageAfterCompression
public void clearStorageAfterCompression(int ngramOrder)
- Specified by:
clearStorageAfterCompression
in interfaceCompressibleValueContainer<V extends LongRepresentable<V>>
-
clearStorageForOrder
public void clearStorageForOrder(int ngramOrder)
- Specified by:
clearStorageForOrder
in interfaceValueContainer<V extends LongRepresentable<V>>
-
storeSuffixoffsets
public boolean storeSuffixoffsets()
- Specified by:
storeSuffixoffsets
in interfaceValueContainer<V extends LongRepresentable<V>>
-
numValueBits
public int numValueBits(int ngramOrder)
- Specified by:
numValueBits
in interfaceValueContainer<V extends LongRepresentable<V>>
-
-