Class PackWriter
- java.lang.Object
-
- org.eclipse.jgit.internal.storage.pack.PackWriter
-
- All Implemented Interfaces:
java.lang.AutoCloseable
public class PackWriter extends java.lang.Object implements java.lang.AutoCloseable
PackWriter class is responsible for generating pack files from specified set of objects from repository. This implementation produce pack files in format version 2.
Source of objects may be specified in two ways:
- (usually) by providing sets of interesting and uninteresting objects in repository - all interesting objects and their ancestors except uninteresting objects and their ancestors will be included in pack, or
- by providing iterator of
RevObject
specifying exact list and order of objects in pack
Typical usage consists of creating an instance, configuring options, preparing the list of objects by calling
preparePack(Iterator)
orpreparePack(ProgressMonitor, Set, Set)
, and streaming withwritePack(ProgressMonitor, ProgressMonitor, OutputStream)
. If the pack is being stored as a file the matching index can be written out after writing the pack bywriteIndex(OutputStream)
. An optional bitmap index can be made by callingprepareBitmapIndex(ProgressMonitor)
followed bywriteBitmapIndex(OutputStream)
.Class provide set of configurable options and
ProgressMonitor
support, as operations may take a long time for big repositories. Deltas searching algorithm is NOT IMPLEMENTED yet - this implementation relies only on deltas and objects reuse.This class is not thread safe. It is intended to be used in one thread as a single pass to produce one pack. Invoking methods multiple times or out of order is not supported as internal data structures are destroyed during certain phases to save memory when packing large repositories.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description private static class
PackWriter.DepthAwareVisitationPolicy
A visitation policy which uses the depth at which the object is seen to decide if re-traversal is necessary.private class
PackWriter.MutableState
static class
PackWriter.PackfileUriConfig
Configuration related to the packfile URI feature.static class
PackWriter.PackingPhase
Possible states that a PackWriter can be in.class
PackWriter.State
Summary of the current state of a PackWriter.
-
Field Summary
Fields Modifier and Type Field Description private java.util.List<CachedPack>
cachedPacks
private ObjectCountCallback
callback
private boolean
canBuildBitmaps
(package private) PackConfig
config
private java.util.zip.CRC32
crc32
private boolean
deltaBaseAsOffset
private int
depth
private java.util.List<ObjectToPack>
edgeObjects
private java.util.Set<? extends ObjectId>
excludeFromBitmapSelection
private ObjectIdSet
excludeInPackLast
private ObjectIdSet[]
excludeInPacks
private FilterSpec
filterSpec
private BitmapIndex.BitmapBuilder
haveObjects
private boolean
ignoreMissingUninteresting
private boolean
indexDisabled
private static java.util.Map<java.lang.ref.WeakReference<PackWriter>,java.lang.Boolean>
instances
private static java.lang.Iterable<PackWriter>
instancesIterable
private java.util.zip.Deflater
myDeflater
static java.util.Set<ObjectId>
NONE
Empty set of objects forpreparePack()
.(package private) BlockList<ObjectToPack>[]
objectsLists
private ObjectIdOwnerMap<ObjectToPack>
objectsMap
private static int
PACK_VERSION_GENERATED
private byte[]
packcsum
private PackWriter.PackfileUriConfig
packfileUriConfig
private boolean
pruneCurrentObjectList
private ObjectReader
reader
private boolean
reuseDeltaCommits
private boolean
reuseDeltas
private ObjectReuseAsIs
reuseSupport
reader
recast to the reuse interface, if it supports it.private boolean
reuseValidate
private java.lang.ref.WeakReference<PackWriter>
selfRef
private boolean
shallowPack
private java.util.List<ObjectToPack>
sortedByName
private PackWriter.MutableState
state
private PackStatistics.Accumulator
stats
private java.util.Set<ObjectId>
tagTargets
private boolean
thin
private PackStatistics.ObjectType.Accumulator
typeStats
private java.util.Collection<? extends ObjectId>
unshallowObjects
private boolean
useBitmaps
private boolean
useCachedPacks
private PackBitmapIndexBuilder
writeBitmaps
-
Constructor Summary
Constructors Constructor Description PackWriter(ObjectReader reader)
Create a writer to load objects from the specified reader.PackWriter(Repository repo)
Create writer for specified repository.PackWriter(Repository repo, ObjectReader reader)
Create writer for specified repository.PackWriter(PackConfig config, ObjectReader reader)
Create writer with a specified configuration.PackWriter(PackConfig config, ObjectReader reader, PackStatistics.Accumulator statsAccumulator)
Create writer with a specified configuration.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description private void
addObject(AnyObjectId src, int type, int pathHashCode)
void
addObject(RevObject object)
Include one object to the output file.private void
addObject(RevObject object, int pathHashCode)
private void
beginPhase(PackWriter.PackingPhase phase, ProgressMonitor monitor, long cnt)
private byte[]
buffer(AnyObjectId objId)
(package private) static byte[]
buffer(PackConfig config, ObjectReader or, AnyObjectId objId)
void
close()
ObjectId
computeName()
Computes SHA-1 of lexicographically sorted objects ids written in this pack, as used to name a pack file in repository.private void
cutDeltaChains(BlockList<ObjectToPack> list)
private java.util.zip.Deflater
deflater()
private TemporaryBuffer.Heap
delta(ObjectToPack otp)
private boolean
depthSkip(RevObject obj, ObjectWalk walker)
Determines if the object should be omitted from the pack as a result of its depth (probably because of the tree:filter). private void
endPhase(ProgressMonitor monitor)
private boolean
exclude(AnyObjectId objectId)
void
excludeObjects(ObjectIdSet idx)
Add a pack index whose contents should be excluded from the result.private void
filterAndAddObject(AnyObjectId src, int type, int pathHashCode, java.util.Set<? extends AnyObjectId> want)
private int
findObjectsNeedingDelta(ObjectToPack[] list, int cnt, int type)
private void
findObjectsToPack(ProgressMonitor countingMonitor, ObjectWalk walker, java.util.Set<? extends ObjectId> want, java.util.Set<? extends ObjectId> have, java.util.Set<? extends ObjectId> noBitmaps)
private void
findObjectsToPackUsingBitmaps(BitmapWalker bitmapWalker, java.util.Set<? extends ObjectId> want, java.util.Set<? extends ObjectId> have)
ObjectToPack
get(AnyObjectId id)
Lookup the ObjectToPack object for a given ObjectId.int
getIndexVersion()
Returns the index format version that will be written.static java.lang.Iterable<PackWriter>
getInstances()
Get all allocated, non-released PackWriters instances.long
getObjectCount()
Returns objects number in a pack file that was created by this writer.ObjectIdOwnerMap<ObjectIdOwnerMap.Entry>
getObjectSet()
Returns the object ids in the pack file that was created by this writer.private ObjectWalk
getObjectWalk()
PackWriter.State
getState()
Get snapshot of the current state of this PackWriter.PackStatistics
getStatistics()
Get statistics of what this PackWriter did in order to create the final pack stream.private long
getUnoffloadedObjectCount()
private boolean
have(ObjectToPack ptr, AnyObjectId objectId)
boolean
isDeltaBaseAsOffset()
Check whether writer can store delta base as an offset (new style reducing pack size) or should store it as an object id (legacy style, compatible with old readers).boolean
isIgnoreMissingUninteresting()
Whether to ignore missing uninteresting objectsboolean
isIndexDisabled()
Whether the index file cannot be created by this PackWriter.boolean
isReuseDeltaCommits()
Check if the writer will reuse commits that are already stored as deltas.boolean
isReuseValidatingObjects()
Check if the writer validates objects before copying them.boolean
isThin()
Whether this writer is producing a thin pack.boolean
isUseBitmaps()
Whether to use bitmapsboolean
isUseCachedPacks()
Whether to reuse cached packs.private void
parallelDeltaSearch(ProgressMonitor monitor, ObjectToPack[] list, int cnt, int threads)
boolean
prepareBitmapIndex(ProgressMonitor pm)
Prepares the bitmaps to be written to the bitmap index file.void
preparePack(java.util.Collection<? extends CachedPack> c)
Prepare the list of objects to be written to the pack stream.void
preparePack(java.util.Iterator<RevObject> objectsSource)
Prepare the list of objects to be written to the pack stream.void
preparePack(ProgressMonitor countingMonitor, java.util.Set<? extends ObjectId> want, java.util.Set<? extends ObjectId> have)
Prepare the list of objects to be written to the pack stream.void
preparePack(ProgressMonitor countingMonitor, java.util.Set<? extends ObjectId> want, java.util.Set<? extends ObjectId> have, java.util.Set<? extends ObjectId> shallow)
Prepare the list of objects to be written to the pack stream.void
preparePack(ProgressMonitor countingMonitor, java.util.Set<? extends ObjectId> want, java.util.Set<? extends ObjectId> have, java.util.Set<? extends ObjectId> shallow, java.util.Set<? extends ObjectId> noBitmaps)
Prepare the list of objects to be written to the pack stream.void
preparePack(ProgressMonitor countingMonitor, ObjectWalk walk, java.util.Set<? extends ObjectId> interestingObjects, java.util.Set<? extends ObjectId> uninterestingObjects, java.util.Set<? extends ObjectId> noBitmaps)
Prepare the list of objects to be written to the pack stream.private static void
pruneEdgesFromObjectList(java.util.List<ObjectToPack> list)
private void
reselectNonDelta(ObjectToPack otp)
private boolean
reuseDeltaFor(ObjectToPack otp)
private static void
runTasks(java.util.concurrent.ExecutorService pool, ThreadSafeProgressMonitor pm, DeltaTask.Block tb, java.util.List<java.lang.Throwable> errors)
private void
searchForDeltas(ProgressMonitor monitor)
private void
searchForDeltas(ProgressMonitor monitor, ObjectToPack[] list, int cnt)
private void
searchForReuse(ProgressMonitor monitor)
private void
searchForReuse(ProgressMonitor monitor, java.util.List<ObjectToPack> list)
void
select(ObjectToPack otp, StoredObjectRepresentation next)
Select an object representation for this writer.void
setClientShallowCommits(java.util.Set<ObjectId> clientShallowCommits)
Records the set of shallow commits in the client.void
setDeltaBaseAsOffset(boolean deltaBaseAsOffset)
Set writer delta base format.void
setFilterSpec(FilterSpec filter)
void
setIgnoreMissingUninteresting(boolean ignore)
Whether writer should ignore non existing uninteresting objectsvoid
setIndexDisabled(boolean noIndex)
Whether to disable creation of the index file.PackWriter
setObjectCountCallback(ObjectCountCallback callback)
Set theObjectCountCallback
.void
setPackfileUriConfig(PackWriter.PackfileUriConfig config)
void
setReuseDeltaCommits(boolean reuse)
Set the writer to reuse existing delta versions of commits.void
setReuseValidatingObjects(boolean validate)
Enable (or disable) object validation during packing.void
setShallowPack(int depth, java.util.Collection<? extends ObjectId> unshallow)
Configure this pack for a shallow clone.void
setTagTargets(java.util.Set<ObjectId> objects)
Set the tag targets that should be hoisted earlier during packing.void
setThin(boolean packthin)
Whether writer may pack objects with delta base object not within set of objects to packvoid
setUseBitmaps(boolean useBitmaps)
Whether to use bitmapsvoid
setUseCachedPacks(boolean useCached)
Whether to use cached packsprivate void
singleThreadDeltaSearch(ProgressMonitor monitor, ObjectToPack[] list, int cnt)
private java.util.List<ObjectToPack>
sortByName()
boolean
willInclude(AnyObjectId id)
Determine if the pack file will contain the requested object.private void
writeBase(PackOutputStream out, ObjectToPack base)
void
writeBitmapIndex(java.io.OutputStream bitmapIndexStream)
Create a bitmap index file to match the pack file just written.private void
writeChecksum(PackOutputStream out)
private void
writeDeltaObjectDeflate(PackOutputStream out, ObjectToPack otp)
void
writeIndex(java.io.OutputStream indexStream)
Create an index file to match the pack file just written.(package private) void
writeObject(PackOutputStream out, ObjectToPack otp)
private void
writeObjectImpl(PackOutputStream out, ObjectToPack otp)
private void
writeObjects(PackOutputStream out)
private void
writeObjects(PackOutputStream out, java.util.List<ObjectToPack> list)
void
writePack(ProgressMonitor compressMonitor, ProgressMonitor writeMonitor, java.io.OutputStream packStream)
Write the prepared pack to the supplied stream.private void
writeWholeObjectDeflate(PackOutputStream out, ObjectToPack otp)
-
-
-
Field Detail
-
PACK_VERSION_GENERATED
private static final int PACK_VERSION_GENERATED
- See Also:
- Constant Field Values
-
NONE
public static final java.util.Set<ObjectId> NONE
Empty set of objects forpreparePack()
.
-
instances
private static final java.util.Map<java.lang.ref.WeakReference<PackWriter>,java.lang.Boolean> instances
-
instancesIterable
private static final java.lang.Iterable<PackWriter> instancesIterable
-
objectsLists
BlockList<ObjectToPack>[] objectsLists
-
objectsMap
private ObjectIdOwnerMap<ObjectToPack> objectsMap
-
edgeObjects
private java.util.List<ObjectToPack> edgeObjects
-
haveObjects
private BitmapIndex.BitmapBuilder haveObjects
-
cachedPacks
private java.util.List<CachedPack> cachedPacks
-
tagTargets
private java.util.Set<ObjectId> tagTargets
-
excludeFromBitmapSelection
private java.util.Set<? extends ObjectId> excludeFromBitmapSelection
-
excludeInPacks
private ObjectIdSet[] excludeInPacks
-
excludeInPackLast
private ObjectIdSet excludeInPackLast
-
myDeflater
private java.util.zip.Deflater myDeflater
-
reader
private final ObjectReader reader
-
reuseSupport
private final ObjectReuseAsIs reuseSupport
reader
recast to the reuse interface, if it supports it.
-
config
final PackConfig config
-
stats
private final PackStatistics.Accumulator stats
-
state
private final PackWriter.MutableState state
-
selfRef
private final java.lang.ref.WeakReference<PackWriter> selfRef
-
typeStats
private PackStatistics.ObjectType.Accumulator typeStats
-
sortedByName
private java.util.List<ObjectToPack> sortedByName
-
packcsum
private byte[] packcsum
-
deltaBaseAsOffset
private boolean deltaBaseAsOffset
-
reuseDeltas
private boolean reuseDeltas
-
reuseDeltaCommits
private boolean reuseDeltaCommits
-
reuseValidate
private boolean reuseValidate
-
thin
private boolean thin
-
useCachedPacks
private boolean useCachedPacks
-
useBitmaps
private boolean useBitmaps
-
ignoreMissingUninteresting
private boolean ignoreMissingUninteresting
-
pruneCurrentObjectList
private boolean pruneCurrentObjectList
-
shallowPack
private boolean shallowPack
-
canBuildBitmaps
private boolean canBuildBitmaps
-
indexDisabled
private boolean indexDisabled
-
depth
private int depth
-
unshallowObjects
private java.util.Collection<? extends ObjectId> unshallowObjects
-
writeBitmaps
private PackBitmapIndexBuilder writeBitmaps
-
crc32
private java.util.zip.CRC32 crc32
-
callback
private ObjectCountCallback callback
-
filterSpec
private FilterSpec filterSpec
-
packfileUriConfig
private PackWriter.PackfileUriConfig packfileUriConfig
-
-
Constructor Detail
-
PackWriter
public PackWriter(Repository repo)
Create writer for specified repository.Objects for packing are specified in
preparePack(Iterator)
orpreparePack(ProgressMonitor, Set, Set)
.- Parameters:
repo
- repository where objects are stored.
-
PackWriter
public PackWriter(ObjectReader reader)
Create a writer to load objects from the specified reader.Objects for packing are specified in
preparePack(Iterator)
orpreparePack(ProgressMonitor, Set, Set)
.- Parameters:
reader
- reader to read from the repository with.
-
PackWriter
public PackWriter(Repository repo, ObjectReader reader)
Create writer for specified repository.Objects for packing are specified in
preparePack(Iterator)
orpreparePack(ProgressMonitor, Set, Set)
.- Parameters:
repo
- repository where objects are stored.reader
- reader to read from the repository with.
-
PackWriter
public PackWriter(PackConfig config, ObjectReader reader)
Create writer with a specified configuration.Objects for packing are specified in
preparePack(Iterator)
orpreparePack(ProgressMonitor, Set, Set)
.- Parameters:
config
- configuration for the pack writer.reader
- reader to read from the repository with.
-
PackWriter
public PackWriter(PackConfig config, ObjectReader reader, @Nullable PackStatistics.Accumulator statsAccumulator)
Create writer with a specified configuration.Objects for packing are specified in
preparePack(Iterator)
orpreparePack(ProgressMonitor, Set, Set)
.- Parameters:
config
- configuration for the pack writer.reader
- reader to read from the repository with.statsAccumulator
- accumulator for statics
-
-
Method Detail
-
getInstances
public static java.lang.Iterable<PackWriter> getInstances()
Get all allocated, non-released PackWriters instances.- Returns:
- all allocated, non-released PackWriters instances.
-
setObjectCountCallback
public PackWriter setObjectCountCallback(ObjectCountCallback callback)
Set theObjectCountCallback
.It should be set before calling
writePack(ProgressMonitor, ProgressMonitor, OutputStream)
.- Parameters:
callback
- the callback to set- Returns:
- this object for chaining.
-
setClientShallowCommits
public void setClientShallowCommits(java.util.Set<ObjectId> clientShallowCommits)
Records the set of shallow commits in the client.- Parameters:
clientShallowCommits
- the shallow commits in the client
-
isDeltaBaseAsOffset
public boolean isDeltaBaseAsOffset()
Check whether writer can store delta base as an offset (new style reducing pack size) or should store it as an object id (legacy style, compatible with old readers). Default setting: false- Returns:
- true if delta base is stored as an offset; false if it is stored as an object id.
-
setDeltaBaseAsOffset
public void setDeltaBaseAsOffset(boolean deltaBaseAsOffset)
Set writer delta base format. Delta base can be written as an offset in a pack file (new approach reducing file size) or as an object id (legacy approach, compatible with old readers). Default setting: false- Parameters:
deltaBaseAsOffset
- boolean indicating whether delta base can be stored as an offset.
-
isReuseDeltaCommits
public boolean isReuseDeltaCommits()
Check if the writer will reuse commits that are already stored as deltas.- Returns:
- true if the writer would reuse commits stored as deltas, assuming delta reuse is already enabled.
-
setReuseDeltaCommits
public void setReuseDeltaCommits(boolean reuse)
Set the writer to reuse existing delta versions of commits.- Parameters:
reuse
- if true, the writer will reuse any commits stored as deltas. By default the writer does not reuse delta commits.
-
isReuseValidatingObjects
public boolean isReuseValidatingObjects()
Check if the writer validates objects before copying them.- Returns:
- true if validation is enabled; false if the reader will handle object validation as a side-effect of it consuming the output.
-
setReuseValidatingObjects
public void setReuseValidatingObjects(boolean validate)
Enable (or disable) object validation during packing.- Parameters:
validate
- if true the pack writer will validate an object before it is put into the output. This additional validation work may be necessary to avoid propagating corruption from one local pack file to another local pack file.
-
isThin
public boolean isThin()
Whether this writer is producing a thin pack.- Returns:
- true if this writer is producing a thin pack.
-
setThin
public void setThin(boolean packthin)
Whether writer may pack objects with delta base object not within set of objects to pack- Parameters:
packthin
- a boolean indicating whether writer may pack objects with delta base object not within set of objects to pack, but belonging to party repository (uninteresting/boundary) as determined by set; this kind of pack is used only for transport; true - to produce thin pack, false - otherwise.
-
isUseCachedPacks
public boolean isUseCachedPacks()
Whether to reuse cached packs.- Returns:
true
to reuse cached packs. If true index creation isn't available.
-
setUseCachedPacks
public void setUseCachedPacks(boolean useCached)
Whether to use cached packs- Parameters:
useCached
- if set totrue
and a cached pack is present, it will be appended onto the end of a thin-pack, reducing the amount of working set space and CPU used by PackWriter. Enabling this feature prevents PackWriter from creating an index for the newly created pack, so its only suitable for writing to a network client, where the client will make the index.
-
isUseBitmaps
public boolean isUseBitmaps()
Whether to use bitmaps- Returns:
true
to use bitmaps for ObjectWalks, if available.
-
setUseBitmaps
public void setUseBitmaps(boolean useBitmaps)
Whether to use bitmaps- Parameters:
useBitmaps
- if set to true, bitmaps will be used when preparing a pack.
-
isIndexDisabled
public boolean isIndexDisabled()
Whether the index file cannot be created by this PackWriter.- Returns:
true
if the index file cannot be created by this PackWriter.
-
setIndexDisabled
public void setIndexDisabled(boolean noIndex)
Whether to disable creation of the index file.- Parameters:
noIndex
-true
to disable creation of the index file.
-
isIgnoreMissingUninteresting
public boolean isIgnoreMissingUninteresting()
Whether to ignore missing uninteresting objects- Returns:
true
to ignore objects that are uninteresting and also not found on local disk; false to throw aMissingObjectException
out ofpreparePack(ProgressMonitor, Set, Set)
if an uninteresting object is not in the source repository. By default, true, permitting gracefully ignoring of uninteresting objects.
-
setIgnoreMissingUninteresting
public void setIgnoreMissingUninteresting(boolean ignore)
Whether writer should ignore non existing uninteresting objects- Parameters:
ignore
-true
if writer should ignore non existing uninteresting objects during construction set of objects to pack; false otherwise - non existing uninteresting objects may causeMissingObjectException
-
setTagTargets
public void setTagTargets(java.util.Set<ObjectId> objects)
Set the tag targets that should be hoisted earlier during packing.Callers may put objects into this set before invoking any of the preparePack methods to influence where an annotated tag's target is stored within the resulting pack. Typically these will be clustered together, and hoisted earlier in the file even if they are ancient revisions, allowing readers to find tag targets with better locality.
- Parameters:
objects
- objects that annotated tags point at.
-
setShallowPack
public void setShallowPack(int depth, java.util.Collection<? extends ObjectId> unshallow)
Configure this pack for a shallow clone.- Parameters:
depth
- maximum depth of history to return. 1 means return only the "wants".unshallow
- objects which used to be shallow on the client, but are being extended as part of this fetch
-
setFilterSpec
public void setFilterSpec(@NonNull FilterSpec filter)
- Parameters:
filter
- the filter which indicates what and what not this writer should include
-
setPackfileUriConfig
public void setPackfileUriConfig(PackWriter.PackfileUriConfig config)
- Parameters:
config
- configuration related to packfile URIs- Since:
- 5.5
-
getObjectCount
public long getObjectCount() throws java.io.IOException
Returns objects number in a pack file that was created by this writer.- Returns:
- number of objects in pack.
- Throws:
java.io.IOException
- a cached pack cannot supply its object count.
-
getUnoffloadedObjectCount
private long getUnoffloadedObjectCount() throws java.io.IOException
- Throws:
java.io.IOException
-
getObjectSet
public ObjectIdOwnerMap<ObjectIdOwnerMap.Entry> getObjectSet() throws java.io.IOException
Returns the object ids in the pack file that was created by this writer.This method can only be invoked after
writePack(ProgressMonitor, ProgressMonitor, OutputStream)
has been invoked and completed successfully.- Returns:
- set of objects in pack.
- Throws:
java.io.IOException
- a cached pack cannot supply its object ids.
-
excludeObjects
public void excludeObjects(ObjectIdSet idx)
Add a pack index whose contents should be excluded from the result.- Parameters:
idx
- objects in this index will not be in the output pack.
-
preparePack
public void preparePack(@NonNull java.util.Iterator<RevObject> objectsSource) throws java.io.IOException
Prepare the list of objects to be written to the pack stream.Iterator exactly determines which objects are included in a pack and order they appear in pack (except that objects order by type is not needed at input). This order should conform general rules of ordering objects in git - by recency and path (type and delta-base first is internally secured) and responsibility for guaranteeing this order is on a caller side. Iterator must return each id of object to write exactly once.
- Parameters:
objectsSource
- iterator of object to store in a pack; order of objects within each type is important, ordering by type is not needed; allowed types for objects areConstants.OBJ_COMMIT
,Constants.OBJ_TREE
,Constants.OBJ_BLOB
andConstants.OBJ_TAG
; objects returned by iterator may be later reused by caller as object id and type are internally copied in each iteration.- Throws:
java.io.IOException
- when some I/O problem occur during reading objects.
-
preparePack
public void preparePack(java.util.Collection<? extends CachedPack> c)
Prepare the list of objects to be written to the pack stream.PackWriter will concat and write out the specified packs as-is.
- Parameters:
c
- cached packs to be written.
-
preparePack
public void preparePack(ProgressMonitor countingMonitor, @NonNull java.util.Set<? extends ObjectId> want, @NonNull java.util.Set<? extends ObjectId> have) throws java.io.IOException
Prepare the list of objects to be written to the pack stream.Basing on these 2 sets, another set of objects to put in a pack file is created: this set consists of all objects reachable (ancestors) from interesting objects, except uninteresting objects and their ancestors. This method uses class
ObjectWalk
extensively to find out that appropriate set of output objects and their optimal order in output pack. Order is consistent with general git in-pack rules: sort by object type, recency, path and delta-base first.- Parameters:
countingMonitor
- progress during object enumeration.want
- collection of objects to be marked as interesting (start points of graph traversal). Must not benull
.have
- collection of objects to be marked as uninteresting (end points of graph traversal). PassNONE
if all objects reachable fromwant
are desired, such as when serving a clone.- Throws:
java.io.IOException
- when some I/O problem occur during reading objects.
-
preparePack
public void preparePack(ProgressMonitor countingMonitor, @NonNull java.util.Set<? extends ObjectId> want, @NonNull java.util.Set<? extends ObjectId> have, @NonNull java.util.Set<? extends ObjectId> shallow) throws java.io.IOException
Prepare the list of objects to be written to the pack stream.Like
preparePack(ProgressMonitor, Set, Set)
but also allows specifying commits that should not be walked past ("shallow" commits). The caller is responsible for filtering out commits that should not be shallow any more ("unshallow" commits as insetShallowPack(int, java.util.Collection<? extends org.eclipse.jgit.lib.ObjectId>)
) from the shallow set.- Parameters:
countingMonitor
- progress during object enumeration.want
- objects of interest, ancestors of which will be included in the pack. Must not benull
.have
- objects whose ancestors (up to and includingshallow
commits) do not need to be included in the pack because they are already available from elsewhere. Must not benull
.shallow
- commits indicating the boundary of the history marked withhave
. Shallow commits have parents but those parents are considered not to be already available. Parents ofshallow
commits and earlier generations will be included in the pack if requested bywant
. Must not benull
.- Throws:
java.io.IOException
- an I/O problem occurred while reading objects.
-
preparePack
public void preparePack(ProgressMonitor countingMonitor, @NonNull java.util.Set<? extends ObjectId> want, @NonNull java.util.Set<? extends ObjectId> have, @NonNull java.util.Set<? extends ObjectId> shallow, @NonNull java.util.Set<? extends ObjectId> noBitmaps) throws java.io.IOException
Prepare the list of objects to be written to the pack stream.Like
preparePack(ProgressMonitor, Set, Set)
but also allows specifying commits that should not be walked past ("shallow" commits). The caller is responsible for filtering out commits that should not be shallow any more ("unshallow" commits as insetShallowPack(int, java.util.Collection<? extends org.eclipse.jgit.lib.ObjectId>)
) from the shallow set.- Parameters:
countingMonitor
- progress during object enumeration.want
- objects of interest, ancestors of which will be included in the pack. Must not benull
.have
- objects whose ancestors (up to and includingshallow
commits) do not need to be included in the pack because they are already available from elsewhere. Must not benull
.shallow
- commits indicating the boundary of the history marked withhave
. Shallow commits have parents but those parents are considered not to be already available. Parents ofshallow
commits and earlier generations will be included in the pack if requested bywant
. Must not benull
.noBitmaps
- collection of objects to be excluded from bitmap commit selection.- Throws:
java.io.IOException
- an I/O problem occurred while reading objects.
-
getObjectWalk
private ObjectWalk getObjectWalk()
-
preparePack
public void preparePack(ProgressMonitor countingMonitor, @NonNull ObjectWalk walk, @NonNull java.util.Set<? extends ObjectId> interestingObjects, @NonNull java.util.Set<? extends ObjectId> uninterestingObjects, @NonNull java.util.Set<? extends ObjectId> noBitmaps) throws java.io.IOException
Prepare the list of objects to be written to the pack stream.Basing on these 2 sets, another set of objects to put in a pack file is created: this set consists of all objects reachable (ancestors) from interesting objects, except uninteresting objects and their ancestors. This method uses class
ObjectWalk
extensively to find out that appropriate set of output objects and their optimal order in output pack. Order is consistent with general git in-pack rules: sort by object type, recency, path and delta-base first.- Parameters:
countingMonitor
- progress during object enumeration.walk
- ObjectWalk to perform enumeration.interestingObjects
- collection of objects to be marked as interesting (start points of graph traversal). Must not benull
.uninterestingObjects
- collection of objects to be marked as uninteresting (end points of graph traversal). PassNONE
if all objects reachable fromwant
are desired, such as when serving a clone.noBitmaps
- collection of objects to be excluded from bitmap commit selection.- Throws:
java.io.IOException
- when some I/O problem occur during reading objects.
-
willInclude
public boolean willInclude(AnyObjectId id) throws java.io.IOException
Determine if the pack file will contain the requested object.- Parameters:
id
- the object to test the existence of.- Returns:
- true if the object will appear in the output pack file.
- Throws:
java.io.IOException
- a cached pack cannot be examined.
-
get
public ObjectToPack get(AnyObjectId id)
Lookup the ObjectToPack object for a given ObjectId.- Parameters:
id
- the object to find in the pack.- Returns:
- the object we are packing, or null.
-
computeName
public ObjectId computeName()
Computes SHA-1 of lexicographically sorted objects ids written in this pack, as used to name a pack file in repository.- Returns:
- ObjectId representing SHA-1 name of a pack that was created.
-
getIndexVersion
public int getIndexVersion()
Returns the index format version that will be written.This method can only be invoked after
writePack(ProgressMonitor, ProgressMonitor, OutputStream)
has been invoked and completed successfully.- Returns:
- the index format version.
-
writeIndex
public void writeIndex(java.io.OutputStream indexStream) throws java.io.IOException
Create an index file to match the pack file just written.Called after
writePack(ProgressMonitor, ProgressMonitor, OutputStream)
.Writing an index is only required for local pack storage. Packs sent on the network do not need to create an index.
- Parameters:
indexStream
- output for the index data. Caller is responsible for closing this stream.- Throws:
java.io.IOException
- the index data could not be written to the supplied stream.
-
writeBitmapIndex
public void writeBitmapIndex(java.io.OutputStream bitmapIndexStream) throws java.io.IOException
Create a bitmap index file to match the pack file just written.Called after
prepareBitmapIndex(ProgressMonitor)
.- Parameters:
bitmapIndexStream
- output for the bitmap index data. Caller is responsible for closing this stream.- Throws:
java.io.IOException
- the index data could not be written to the supplied stream.
-
sortByName
private java.util.List<ObjectToPack> sortByName()
-
beginPhase
private void beginPhase(PackWriter.PackingPhase phase, ProgressMonitor monitor, long cnt)
-
endPhase
private void endPhase(ProgressMonitor monitor)
-
writePack
public void writePack(ProgressMonitor compressMonitor, ProgressMonitor writeMonitor, java.io.OutputStream packStream) throws java.io.IOException
Write the prepared pack to the supplied stream.Called after
preparePack(ProgressMonitor, ObjectWalk, Set, Set, Set)
orpreparePack(ProgressMonitor, Set, Set)
.Performs delta search if enabled and writes the pack stream.
All reused objects data checksum (Adler32/CRC32) is computed and validated against existing checksum.
- Parameters:
compressMonitor
- progress monitor to report object compression work.writeMonitor
- progress monitor to report the number of objects written.packStream
- output stream of pack data. The stream should be buffered by the caller. The caller is responsible for closing the stream.- Throws:
java.io.IOException
- an error occurred reading a local object's data to include in the pack, or writing compressed object data to the output stream.WriteAbortedException
- the write operation is aborted byObjectCountCallback
.
-
getStatistics
public PackStatistics getStatistics()
Get statistics of what this PackWriter did in order to create the final pack stream.- Returns:
- description of what this PackWriter did in order to create the final pack stream. This should only be invoked after the calls to create the pack/index/bitmap have completed.
-
getState
public PackWriter.State getState()
Get snapshot of the current state of this PackWriter.- Returns:
- snapshot of the current state of this PackWriter.
-
close
public void close()
Release all resources used by this writer.
- Specified by:
close
in interfacejava.lang.AutoCloseable
-
searchForReuse
private void searchForReuse(ProgressMonitor monitor) throws java.io.IOException
- Throws:
java.io.IOException
-
searchForReuse
private void searchForReuse(ProgressMonitor monitor, java.util.List<ObjectToPack> list) throws java.io.IOException, MissingObjectException
- Throws:
java.io.IOException
MissingObjectException
-
cutDeltaChains
private void cutDeltaChains(BlockList<ObjectToPack> list) throws java.io.IOException
- Throws:
java.io.IOException
-
searchForDeltas
private void searchForDeltas(ProgressMonitor monitor) throws MissingObjectException, IncorrectObjectTypeException, java.io.IOException
- Throws:
MissingObjectException
IncorrectObjectTypeException
java.io.IOException
-
findObjectsNeedingDelta
private int findObjectsNeedingDelta(ObjectToPack[] list, int cnt, int type)
-
reselectNonDelta
private void reselectNonDelta(ObjectToPack otp) throws java.io.IOException
- Throws:
java.io.IOException
-
searchForDeltas
private void searchForDeltas(ProgressMonitor monitor, ObjectToPack[] list, int cnt) throws MissingObjectException, IncorrectObjectTypeException, LargeObjectException, java.io.IOException
- Throws:
MissingObjectException
IncorrectObjectTypeException
LargeObjectException
java.io.IOException
-
singleThreadDeltaSearch
private void singleThreadDeltaSearch(ProgressMonitor monitor, ObjectToPack[] list, int cnt) throws java.io.IOException
- Throws:
java.io.IOException
-
parallelDeltaSearch
private void parallelDeltaSearch(ProgressMonitor monitor, ObjectToPack[] list, int cnt, int threads) throws java.io.IOException
- Throws:
java.io.IOException
-
runTasks
private static void runTasks(java.util.concurrent.ExecutorService pool, ThreadSafeProgressMonitor pm, DeltaTask.Block tb, java.util.List<java.lang.Throwable> errors) throws java.io.IOException
- Throws:
java.io.IOException
-
writeObjects
private void writeObjects(PackOutputStream out) throws java.io.IOException
- Throws:
java.io.IOException
-
writeObjects
private void writeObjects(PackOutputStream out, java.util.List<ObjectToPack> list) throws java.io.IOException
- Throws:
java.io.IOException
-
writeObject
void writeObject(PackOutputStream out, ObjectToPack otp) throws java.io.IOException
- Throws:
java.io.IOException
-
writeObjectImpl
private void writeObjectImpl(PackOutputStream out, ObjectToPack otp) throws java.io.IOException
- Throws:
java.io.IOException
-
writeBase
private void writeBase(PackOutputStream out, ObjectToPack base) throws java.io.IOException
- Throws:
java.io.IOException
-
writeWholeObjectDeflate
private void writeWholeObjectDeflate(PackOutputStream out, ObjectToPack otp) throws java.io.IOException
- Throws:
java.io.IOException
-
writeDeltaObjectDeflate
private void writeDeltaObjectDeflate(PackOutputStream out, ObjectToPack otp) throws java.io.IOException
- Throws:
java.io.IOException
-
delta
private TemporaryBuffer.Heap delta(ObjectToPack otp) throws java.io.IOException
- Throws:
java.io.IOException
-
buffer
private byte[] buffer(AnyObjectId objId) throws java.io.IOException
- Throws:
java.io.IOException
-
buffer
static byte[] buffer(PackConfig config, ObjectReader or, AnyObjectId objId) throws java.io.IOException
- Throws:
java.io.IOException
-
deflater
private java.util.zip.Deflater deflater()
-
writeChecksum
private void writeChecksum(PackOutputStream out) throws java.io.IOException
- Throws:
java.io.IOException
-
findObjectsToPack
private void findObjectsToPack(@NonNull ProgressMonitor countingMonitor, @NonNull ObjectWalk walker, @NonNull java.util.Set<? extends ObjectId> want, @NonNull java.util.Set<? extends ObjectId> have, @NonNull java.util.Set<? extends ObjectId> noBitmaps) throws java.io.IOException
- Throws:
java.io.IOException
-
findObjectsToPackUsingBitmaps
private void findObjectsToPackUsingBitmaps(BitmapWalker bitmapWalker, java.util.Set<? extends ObjectId> want, java.util.Set<? extends ObjectId> have) throws MissingObjectException, IncorrectObjectTypeException, java.io.IOException
- Throws:
MissingObjectException
IncorrectObjectTypeException
java.io.IOException
-
pruneEdgesFromObjectList
private static void pruneEdgesFromObjectList(java.util.List<ObjectToPack> list)
-
addObject
public void addObject(RevObject object) throws IncorrectObjectTypeException
Include one object to the output file.Objects are written in the order they are added. If the same object is added twice, it may be written twice, creating a larger than necessary file.
- Parameters:
object
- the object to add.- Throws:
IncorrectObjectTypeException
- the object is an unsupported type.
-
addObject
private void addObject(RevObject object, int pathHashCode)
-
addObject
private void addObject(AnyObjectId src, int type, int pathHashCode)
-
depthSkip
private boolean depthSkip(@NonNull RevObject obj, ObjectWalk walker)
Determines if the object should be omitted from the pack as a result of its depth (probably because of the tree:filter). Causes
walker
to skip traversing the current tree, which ought to have just started traversal, assuming this method is called as soon as a new depth is reached.This method increments the
treesTraversed
statistic.- Parameters:
obj
- the object to check whether it should be omitted.walker
- the walker being used for traveresal.- Returns:
- whether the given object should be skipped.
-
filterAndAddObject
private void filterAndAddObject(@NonNull AnyObjectId src, int type, int pathHashCode, @NonNull java.util.Set<? extends AnyObjectId> want) throws java.io.IOException
- Throws:
java.io.IOException
-
exclude
private boolean exclude(AnyObjectId objectId)
-
select
public void select(ObjectToPack otp, StoredObjectRepresentation next)
Select an object representation for this writer.An
ObjectReader
implementation should invoke this method once for each representation available for an object, to allow the writer to find the most suitable one for the output.- Parameters:
otp
- the object being packed.next
- the next available representation from the repository.
-
have
private final boolean have(ObjectToPack ptr, AnyObjectId objectId)
-
prepareBitmapIndex
public boolean prepareBitmapIndex(ProgressMonitor pm) throws java.io.IOException
Prepares the bitmaps to be written to the bitmap index file.Bitmaps can be used to speed up fetches and clones by storing the entire object graph at selected commits. Writing a bitmap index is an optional feature that not all pack users may require.
Called after
writeIndex(OutputStream)
.To reduce memory internal state is cleared during this method, rendering the PackWriter instance useless for anything further than a call to write out the new bitmaps with
writeBitmapIndex(OutputStream)
.- Parameters:
pm
- progress monitor to report bitmap building work.- Returns:
- whether a bitmap index may be written.
- Throws:
java.io.IOException
- when some I/O problem occur during reading objects.
-
reuseDeltaFor
private boolean reuseDeltaFor(ObjectToPack otp)
-
-