Package edu.berkeley.nlp.lm.io
Class KneserNeyFileWritingLmReaderCallback<W>
java.lang.Object
edu.berkeley.nlp.lm.io.KneserNeyFileWritingLmReaderCallback<W>
- Type Parameters:
W
-
- All Implemented Interfaces:
ArpaLmReaderCallback<ProbBackoffPair>
,LmReaderCallback<ProbBackoffPair>
,NgramOrderedLmReaderCallback<ProbBackoffPair>
public class KneserNeyFileWritingLmReaderCallback<W>
extends Object
implements ArpaLmReaderCallback<ProbBackoffPair>
Class for producing a Kneser-Ney language model in ARPA format from raw text.
- Author:
- adampauls
-
Constructor Summary
ConstructorsConstructorDescriptionKneserNeyFileWritingLmReaderCallback
(File outputFile, WordIndexer<W> wordIndexer) KneserNeyFileWritingLmReaderCallback
(PrintWriter out, WordIndexer<W> wordIndexer) -
Method Summary
Modifier and TypeMethodDescriptionvoid
call
(int[] ngram, int startPos, int endPos, ProbBackoffPair value, String words) Called for each n-gramvoid
cleanup()
Called once all reading is done.void
handleNgramOrderFinished
(int order) Called when all n-grams of a given order are finishedvoid
handleNgramOrderStarted
(int order) Called when n-grams of a given order are startedvoid
initWithLengths
(List<Long> numNGrams) Called initially with a list of how many n-grams will appear for each order.
-
Constructor Details
-
KneserNeyFileWritingLmReaderCallback
-
KneserNeyFileWritingLmReaderCallback
-
-
Method Details
-
handleNgramOrderFinished
public void handleNgramOrderFinished(int order) Description copied from interface:NgramOrderedLmReaderCallback
Called when all n-grams of a given order are finished- Specified by:
handleNgramOrderFinished
in interfaceNgramOrderedLmReaderCallback<W>
- Parameters:
order
-
-
handleNgramOrderStarted
public void handleNgramOrderStarted(int order) Description copied from interface:NgramOrderedLmReaderCallback
Called when n-grams of a given order are started- Specified by:
handleNgramOrderStarted
in interfaceNgramOrderedLmReaderCallback<W>
- Parameters:
order
-
-
call
Description copied from interface:LmReaderCallback
Called for each n-gram- Specified by:
call
in interfaceLmReaderCallback<W>
- Parameters:
ngram
- The integer representation of the words as given by the provided WordIndexervalue
- The value of the n-gramwords
- The string representation of the n-gram (space separated)
-
cleanup
public void cleanup()Description copied from interface:LmReaderCallback
Called once all reading is done.- Specified by:
cleanup
in interfaceLmReaderCallback<W>
-
initWithLengths
Description copied from interface:ArpaLmReaderCallback
Called initially with a list of how many n-grams will appear for each order.- Specified by:
initWithLengths
in interfaceArpaLmReaderCallback<W>
- Parameters:
numNGrams
- maps n-gram orders to number of n-grams (i.e. numNGrams.get(0) is the number of unigrams)
-