aboutsummaryrefslogtreecommitdiffstats
path: root/plumbing/format/packfile
Commit message (Collapse)AuthorAgeFilesLines
* plumbing/format/idxfile: add new Index and MemoryIndexMiguel Molina2018-07-192-135/+17
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* packfile: optimise NewIndexFromIdxFile for a very common caseDavid Symonds2018-06-211-2/+12
| | | | | | | Loading from an on-disk idxfile will usually already have the idxfile entries in order, so check that before wasting time on sorting. Signed-off-by: David Symonds <dsymonds@golang.org>
* plumbing: packfile, Don't push empty objects. Fixes #840kuba--2018-06-072-4/+19
| | | | Signed-off-by: kuba-- <kuba@sourced.tech>
* packfile: improve Index memory representation to be more compactDavid Symonds2018-05-302-23/+67
| | | | | | | | | | | | | | | | | | Instead of using a map for offset indexing, use a sorted slice. Binary searching is fast, and a slice is much more compact. This has a negligible hit on speed, but has a significant impact on memory usage, especially for larger repos. benchmark old ns/op new ns/op delta BenchmarkIndexConstruction-12 15506506 14056098 -9.35% benchmark old allocs new allocs delta BenchmarkIndexConstruction-12 60764 60385 -0.62% benchmark old bytes new bytes delta BenchmarkIndexConstruction-12 4318145 3913169 -9.38% Signed-off-by: David Symonds <dsymonds@golang.org>
* *: Use CheckClose with named returnsJavi Fontan2018-03-272-4/+4
| | | | | | | | Previously some close errors were losts. This is specially problematic in go-git as lots of work is done here like generating indexes and moving packfiles. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* *: skip time consuming testsMáximo Cuadros2018-03-211-0/+9
| | | | Signed-off-by: Máximo Cuadros <mcuadros@gmail.com>
* plumbing: format/packfile, add SaveOriginalMetadata functionJavi Fontan2018-02-092-5/+9
| | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing: format/packfile, fix panic retrieving object hash.Javi Fontan2018-02-093-4/+10
| | | | | | | | | | | | | | | | In some cases the original data is not saved before it is cleaned and forces a panic when it's needed. The change adds ObjectToPack.CleanOriginal to be used to clean original object instead of: object.Original = nil Now when the Original data is freed because it's no longer in the pack window a SetOriginal call is done to make sure that Size, Hash and Size data is not lost. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing: format/packfile, check nil objects in ObjectToPackJavi Fontan2018-01-252-8/+12
| | | | | | | SetOriginal now skips setting resolved values if the provided object is nil. BackToOriginal also skips nil Original objects. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing: format/packfile, fix crash with cycle deltasJavi Fontan2018-01-244-1/+51
| | | | | | | | | | | | Resolving cycles relied on ObjectToPack objects having Original. This is no longer true with the changes from #720. This commit changes: * Save original type, hash and size in ObjectToPack * Use SetObject to set both Original and resolved type, hash and size * Restore original object before using BackToOriginal (cycle resolution) * Update encoder test to check this case Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing: packfile, Add crc check to scanner test.Javi Fontan2018-01-211-4/+75
| | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing: packfile, Add a buffer to crc writer.Javi Fontan2018-01-211-9/+31
| | | | | | | | | | | crc update with block smaller than 16 bytes uses a slower version of the function. ReadByte is heavily used by zlib inflate so most of the time crc is update byte by byte. A new Flush method is added to the scanner to flush this crc writer cache. It is only called when the Scanner reader is a teeReader. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* Modify cache to delete more than one item to free spaceJavi Fontan2018-01-161-0/+2
| | | | | | | | | | | The previous version could only delete the oldest used object. If the object to cache was bigger than the space freed it could not be added. Also the decoder adds bases to the cache when they are needed. This change increases the speed creating indexes 2x. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* Clean reconstructed objects outside pack windowJavi Fontan2018-01-111-13/+19
| | | | | | | | | | | Object walk reconstructs delta objects but these are not cleaned up after they got out the pack window. Without this change all reconstructed objects reside in memory. restoreOriginal call is moved before calling Size(). Now we can not guarantee that the object is already undeltified. Signed-off-by: Javi Fontan <javier@sourced.tech>
* Merge pull request #698 from jfontan/improvement/use-decoder-cacheMáximo Cuadros2017-12-202-17/+39
|\ | | | | plumbing: cache, enforce the use of cache in packfile decoder
| * Make DeltaBaseCache privateJavi Fontan2017-12-201-6/+13
| | | | | | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
| * Fix typo and documentation of NewDecoderForTypeJavi Fontan2017-12-201-3/+3
| | | | | | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
| * Enforce the use of cache in packfile decoderJavi Fontan2017-12-202-12/+27
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Decoder object can make use of an object cache to speed up processing. Previously the only way to specify it was changing manually the struct generated by NewDecodeForFile. This lead to some instances to be created without it and penalized performance. Now the cache should be explicitly passed to the constructor function. NewDecoder now creates objects with a cache using the default size. A new helper function was added to create cache objects with the default size as this becomes a common task now: cache.NewObjectLRUDefault() Signed-off-by: Javi Fontan <jfontan@gmail.com>
* | Improve delta reutilizationAntonio Jesus Navarro Perez2017-12-205-29/+152
|/ | | | | | | | | | | - Remove wrong 'if' on delta selector that causes poor delta reutilizations - packfile.Encoder now can write deltas and objects in a non specific order - ObjectToPack now saves the Offset on the packfile to be able to obtain base offset in a recursive manner and write them before the delta itself - Added encoder test to check cyclic delta chains - Check the output packfile hash in all encoder tests Signed-off-by: Antonio Jesus Navarro Perez <antnavper@gmail.com>
* all: gofmt -sferhat elmas2017-11-302-2/+2
|
* all: simplificationferhat elmas2017-11-294-14/+5
| | | | | | | | | | - no length for map initialization - don't check for boolean/error return - don't format string - use string method of bytes buffer instead of converting bytes to string - use `strings.Contains` instead of `strings.Index` - use `bytes.Equal` instead of `bytes.Compare`
* update to go-billy.v4 and go-git-fixtures.v3Máximo Cuadros2017-11-234-6/+5
| | | | Signed-off-by: Máximo Cuadros <mcuadros@gmail.com>
* format: packfile fix DecodeObjectAt when Decoder has typeMáximo Cuadros2017-11-192-4/+29
| | | | Signed-off-by: Máximo Cuadros <mcuadros@gmail.com>
* Merge pull request #631 from keybase/strib/use-bytes-pool-for-diffsMáximo Cuadros2017-10-311-4/+13
|\ | | | | packfile: use buffer pool for diffs
| * packfile: use buffer pool for diffsJeremy Stribling2017-10-301-4/+13
| |
* | packfile: delete index maps from memory when no longer neededJeremy Stribling2017-10-301-0/+6
|/ | | | This helps keep memory usage stable while calculating deltas.
* config: support a configurable, and turn-off-able, pack.windowJeremy Stribling2017-09-115-37/+89
| | | | | | | | | | | | | | | | | | | | One use of go-git is to transfer git data from a non-standard git repo (not stored in a file system, for example) to a "remote" backed by a standard, local .git repo. In this scenario, delta compression is not needed to reduce transfer time over the "network", because there is no network. The underlying storage layer has already taken care of the data tranfer, and sending the objects to local .git storage doesn't require compression. So this PR gives the user the option to turn off compression when it isn't needed. Of course, this results in a larger, uncompressed local .git repo, but the user can then run git gc or git repack on that repo if they care about the storage costs. Turning the pack window to 0 on reduces total push time of a 36K repo by 50 seconds (out of a pre-PR total of 3m26s).
* packfile: small optimizations for findMatch and matchLengthMiguel Molina2017-09-072-16/+38
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* packfile: parallelize deltification of objects in groupsMiguel Molina2017-09-072-21/+31
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* packfile: use a modified version of JGit DeltaIndex and DeltaIndexScannerMiguel Molina2017-09-063-40/+286
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* packfile: slightly haster hash function for chunk-offset index keyMiguel Molina2017-09-062-11/+32
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* packfile: reuse delta indexes when possibleMiguel Molina2017-09-063-10/+22
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* packfile: improve performance a little by reducing gc pressureMiguel Molina2017-09-044-6/+33
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* plumbing: use sliding window in delta calculations, like git CLJeremy Stribling2017-08-282-1/+23
| | | | | | | | | | This sets a default sliding window of 10 for the delta calculation, just like git CL: https://git-scm.com/docs/git-pack-objects#git-pack-objects---windowltngt For a big-ish repo with 35K objects (17K commits), this reduced the time for calling `deltaSelection.walk` during a push from more than 14 minutes to about a minute.
* plumbing: add IsDelta method to ObjectTypeSantiago M. Mola2017-07-271-6/+2
| | | | | ObjectType.IsDelta is a convenience function to match both offset and reference delta types.
* storage: reuse deltas from packfilesSantiago M. Mola2017-07-275-98/+293
| | | | | | | | | | | | | | | | | | | * plumbing: add DeltaObject interface for EncodedObjects that are deltas and hold additional information about them, such as the hash of the base object. * plumbing/storer: add DeltaObjectStorer interface for object storers that can return DeltaObject. Note that calls to EncodedObject will never return instances of DeltaObject. That requires explicit calls to DeltaObject. * storage/filesystem: implement DeltaObjectStorer interface. * plumbing/packfile: packfile encoder now supports reusing deltas that are already computed (e.g. from an existing packfile) if the storage implements DeltaObjectStorer. Reusing deltas boosts performance of packfile generation (e.g. on push).
* test: improve packfile.Encoder testsSantiago M. Mola2017-07-271-54/+68
| | | | | | | * Improve checks for encode/decode. * Make it easier to extend this test with more storage backends.
* storage/filesystem: reuse delta cacheSantiago M. Mola2017-07-271-10/+22
| | | | | Reuse delta base object cache for packfile decoders across multiple instances.
* plumbing/cache: use more explicit interfaceSantiago M. Mola2017-07-271-7/+6
| | | | | | * renamed Add to Put * Get returns a second bool value to indicate if there was hit or miss.
* packfile: create packfile.Index and reuse itSantiago M. Mola2017-07-264-82/+279
| | | | | | | | | | | | | | | There was an internal type (i.e. storage/filesystem.idx) to use as in-memory index for packfiles. This was not convenient to reuse in the packfile. This commit creates a new representation (format/packfile.Index) that can be converted to and from idxfile.Idxfile. A packfile.Index now contains the functionality that was scattered on storage/filesystem.idx and packfile.Decoder's internals. storage/filesystem now reuses packfile.Index instances and this also results in higher cache hit ratios when resolving deltas.
* Merge pull request #491 from smola/error-checksMáximo Cuadros2017-07-192-21/+22
|\ | | | | *: add more IO error checks
| * packfile: check close error in UpdateObjectStorageSantiago M. Mola2017-07-191-8/+14
| |
| * packfile: check zlib reader closing errorSantiago M. Mola2017-07-191-13/+8
| |
* | packfile: Avoid panics patching corrupted deltasAntonio Jesus Navarro Perez2017-07-192-16/+85
|/
* *: upgrade to go-billy.v3, mergeMáximo Cuadros2017-06-181-1/+1
|
* packfile: A copy operation cannot be bigger than 64kbAntonio Jesus Navarro Perez2017-06-012-1/+35
| | | | More info here: https://github.com/git/git/blob/f7466e94375b3be27f229c78873f0acf8301c0a5/diff-delta.c#L428
* format/packfile: improve binary delta algorithmAntonio Jesus Navarro Perez2017-05-242-441/+94
| | | | Implemented algorithm described in "File System Support for Delta Compression" paper, from "Joshua P. MacDonald".
* format/packfile: fix bug when the delta depth is equals to 50Antonio Jesus Navarro Perez2017-05-082-0/+10
|
* format/packfile: rename Seek to SeekFromStartSantiago M. Mola2017-04-263-6/+8
| | | | | | This has signature and behavior distinct from io.Seeker, go vet complains about this, so we change it to a different name to avoid confusion.
* use go-billy.v2 versionAntonio Jesus Navarro Perez2017-03-101-1/+1
|
generated by cgit (git 2.34.1) at 2024-11-11 12:10:57 +0000