aboutsummaryrefslogtreecommitdiffstats
path: root/plumbing/format/packfile
Commit message (Collapse)AuthorAgeFilesLines
* Merge pull request #631 from keybase/strib/use-bytes-pool-for-diffsMáximo Cuadros2017-10-311-4/+13
|\ | | | | packfile: use buffer pool for diffs
| * packfile: use buffer pool for diffsJeremy Stribling2017-10-301-4/+13
| |
* | packfile: delete index maps from memory when no longer neededJeremy Stribling2017-10-301-0/+6
|/ | | | This helps keep memory usage stable while calculating deltas.
* config: support a configurable, and turn-off-able, pack.windowJeremy Stribling2017-09-115-37/+89
| | | | | | | | | | | | | | | | | | | | One use of go-git is to transfer git data from a non-standard git repo (not stored in a file system, for example) to a "remote" backed by a standard, local .git repo. In this scenario, delta compression is not needed to reduce transfer time over the "network", because there is no network. The underlying storage layer has already taken care of the data tranfer, and sending the objects to local .git storage doesn't require compression. So this PR gives the user the option to turn off compression when it isn't needed. Of course, this results in a larger, uncompressed local .git repo, but the user can then run git gc or git repack on that repo if they care about the storage costs. Turning the pack window to 0 on reduces total push time of a 36K repo by 50 seconds (out of a pre-PR total of 3m26s).
* packfile: small optimizations for findMatch and matchLengthMiguel Molina2017-09-072-16/+38
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* packfile: parallelize deltification of objects in groupsMiguel Molina2017-09-072-21/+31
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* packfile: use a modified version of JGit DeltaIndex and DeltaIndexScannerMiguel Molina2017-09-063-40/+286
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* packfile: slightly haster hash function for chunk-offset index keyMiguel Molina2017-09-062-11/+32
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* packfile: reuse delta indexes when possibleMiguel Molina2017-09-063-10/+22
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* packfile: improve performance a little by reducing gc pressureMiguel Molina2017-09-044-6/+33
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* plumbing: use sliding window in delta calculations, like git CLJeremy Stribling2017-08-282-1/+23
| | | | | | | | | | This sets a default sliding window of 10 for the delta calculation, just like git CL: https://git-scm.com/docs/git-pack-objects#git-pack-objects---windowltngt For a big-ish repo with 35K objects (17K commits), this reduced the time for calling `deltaSelection.walk` during a push from more than 14 minutes to about a minute.
* plumbing: add IsDelta method to ObjectTypeSantiago M. Mola2017-07-271-6/+2
| | | | | ObjectType.IsDelta is a convenience function to match both offset and reference delta types.
* storage: reuse deltas from packfilesSantiago M. Mola2017-07-275-98/+293
| | | | | | | | | | | | | | | | | | | * plumbing: add DeltaObject interface for EncodedObjects that are deltas and hold additional information about them, such as the hash of the base object. * plumbing/storer: add DeltaObjectStorer interface for object storers that can return DeltaObject. Note that calls to EncodedObject will never return instances of DeltaObject. That requires explicit calls to DeltaObject. * storage/filesystem: implement DeltaObjectStorer interface. * plumbing/packfile: packfile encoder now supports reusing deltas that are already computed (e.g. from an existing packfile) if the storage implements DeltaObjectStorer. Reusing deltas boosts performance of packfile generation (e.g. on push).
* test: improve packfile.Encoder testsSantiago M. Mola2017-07-271-54/+68
| | | | | | | * Improve checks for encode/decode. * Make it easier to extend this test with more storage backends.
* storage/filesystem: reuse delta cacheSantiago M. Mola2017-07-271-10/+22
| | | | | Reuse delta base object cache for packfile decoders across multiple instances.
* plumbing/cache: use more explicit interfaceSantiago M. Mola2017-07-271-7/+6
| | | | | | * renamed Add to Put * Get returns a second bool value to indicate if there was hit or miss.
* packfile: create packfile.Index and reuse itSantiago M. Mola2017-07-264-82/+279
| | | | | | | | | | | | | | | There was an internal type (i.e. storage/filesystem.idx) to use as in-memory index for packfiles. This was not convenient to reuse in the packfile. This commit creates a new representation (format/packfile.Index) that can be converted to and from idxfile.Idxfile. A packfile.Index now contains the functionality that was scattered on storage/filesystem.idx and packfile.Decoder's internals. storage/filesystem now reuses packfile.Index instances and this also results in higher cache hit ratios when resolving deltas.
* Merge pull request #491 from smola/error-checksMáximo Cuadros2017-07-192-21/+22
|\ | | | | *: add more IO error checks
| * packfile: check close error in UpdateObjectStorageSantiago M. Mola2017-07-191-8/+14
| |
| * packfile: check zlib reader closing errorSantiago M. Mola2017-07-191-13/+8
| |
* | packfile: Avoid panics patching corrupted deltasAntonio Jesus Navarro Perez2017-07-192-16/+85
|/
* *: upgrade to go-billy.v3, mergeMáximo Cuadros2017-06-181-1/+1
|
* packfile: A copy operation cannot be bigger than 64kbAntonio Jesus Navarro Perez2017-06-012-1/+35
| | | | More info here: https://github.com/git/git/blob/f7466e94375b3be27f229c78873f0acf8301c0a5/diff-delta.c#L428
* format/packfile: improve binary delta algorithmAntonio Jesus Navarro Perez2017-05-242-441/+94
| | | | Implemented algorithm described in "File System Support for Delta Compression" paper, from "Joshua P. MacDonald".
* format/packfile: fix bug when the delta depth is equals to 50Antonio Jesus Navarro Perez2017-05-082-0/+10
|
* format/packfile: rename Seek to SeekFromStartSantiago M. Mola2017-04-263-6/+8
| | | | | | This has signature and behavior distinct from io.Seeker, go vet complains about this, so we change it to a different name to avoid confusion.
* use go-billy.v2 versionAntonio Jesus Navarro Perez2017-03-101-1/+1
|
* project: move imports from srcd.works to gopkg.inAntonio Jesus Navarro Perez2017-03-0714-28/+28
| | | | To be able to fix #261 we will move again to gopkg.in before v4 stable release.
* plumbing: improve documentation (Fix #242)Antonio Jesus Navarro Perez2017-02-233-5/+11
|
* cache: move package to plumbingAntonio Jesus Navarro Perez2017-02-211-1/+1
| | | | Because cache package is only intended to be used at internal level, we move it to the plumbing package.
* Merge pull request #258 from ajnavarro/fix/onf-decoderAntonio Navarro Perez2017-02-092-7/+65
|\ | | | | Fix missing objects if they where deltified using ref-delta
| * Fix missing objects if they where deltified using ref-deltaAntonio Jesus Navarro Perez2017-02-072-7/+65
| | | | | | | | | | - Deleted invalid logic that returned nil if an ref-delta was not found into the decoder index. This logic was missing objects if it was deltified using ref-deltas. - Now, to avoid that problem, index is mandatory to decode correctly a packfile of a specific type. Decoder.SetOffsets method now is called into the EncodedObjectIterator to avoid this problem.
* | package plumbing documentation improvements (#248)Máximo Cuadros2017-02-071-167/+38
|/
* new srcd.works/go-git.v4 pathMáximo Cuadros2017-01-3014-27/+27
|
* new git fixture pathMáximo Cuadros2017-01-303-3/+3
|
* rename billy importsMáximo Cuadros2017-01-301-3/+3
|
* Fix some typos in plumbing docs (#244)Sergio Arbeo2017-01-305-12/+12
|
* packfile: cache undeltified objects to improve decode performance (#218)Antonio Navarro Perez2017-01-251-8/+33
| | | | | * Simple object cache that keeps in memory the last undeltified objects. When no more objects can be kept into memory, the oldest one is deleted (FIFO). This speeds up packfile operations preventing redundant seeks and decodes.
* packfile/decoder: speed up packfile iterator when specific type (#200)Antonio Navarro Perez2017-01-122-1/+134
|
* packfile/scanner: reset zlib reader instead of new one (#201)Antonio Navarro Perez2017-01-121-5/+20
|
* server: add git server implementation (#190)Santiago M. Mola2017-01-041-0/+30
| | | | | | | | | | | | | | | * server: add generic server implementation (transport-independent), both for git-upload-pack and git-receive-pack. * server: move internal functions to internal/common. * cli: add git-receive-pack and git-upload-pack implementations. * format/packfile: add UpdateObjectStorage function, extracted from Remote. * transport: implement tranport RPC-like, only with git-upload-pack and git-receive-pack methods. Client renamed to Transport. * storer: add storer.Storer interface. * protocol/packp: add UploadPackResponse constructor with packfile. * protocol/packp: fix UploadPackResponse encoding, add tests. * protocol/packp/capability: implement All.
* Extract billy (#173)Sergio Arbeo2016-12-191-1/+2
| | | | | | | | | | | | | | | * Extract billy Billy is a new library directly extracted from go-git. It abstract several storages systems in a filesystem interface. More in github.com/src-d/billy * Fix grouping in imports block * Update billy to v1 * Re-remove fs_implementation example
* packfile: delta selection logic (#182)Antonio Navarro Perez2016-12-169-111/+473
| | | | | | | | | | * packfile: delta selection logic - Implemented logic to assign deltas to objects * Requested changes * Improved tests and fix errors
* move plumbing from top level package to plumbing (#183)Santiago M. Mola2016-12-147-52/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | * plumbing: rename Object -> EncodedObject. * plumbing/storer: rename ObjectStorer -> EncodedObjectStorer. * move difftree to plumbing/difftree. * move diff -> utils/diff * make Object/Tag/Blob/Tree/Commit/File depend on storer. * Object and its implementations now depend only on storer.EncodedObjectStorer, not git.Repository. * Tests are decoupled accordingly. * move Object/Commit/File/Tag/Tree to plumbing/object. * move Object/Commit/File/Tag/Tree to plumbing/object. * move checkClose to utils/ioutil. * move RevListObjects to plumbing/revlist.Objects. * move DiffTree to plumbing/difftree package. * rename files with plural nouns to singular * plumbing/object: add GetBlob/GetCommit/GetTag/GetTree.
* packfile: improve delta copy operation encoding code (#186)Antonio Navarro Perez2016-12-141-31/+13
|
* format/packfile: implement delta encoding (#172)Antonio Navarro Perez2016-12-145-25/+322
| | | | | | | | | | | | * format/packfile: implement delta encoding - Added all the logic to the encoder to be able to encode ref-delta and offset-delta objects - Created plumbing.ObjectToPack to handle deltas and standard objects when we are writting them into a packfile - Added specific encoder delta tests, one standard object and one delta, and one standard object and two deltas * Requested changes. * Requested changes
* packfile: delta diff implementation (#159)Antonio Navarro Perez2016-12-094-0/+637
| | | | | | | | | | * packfile: delta diff implementation - Renamed delta.go to patch_delta.go - Added diff_delta.go file - Added tests that creates a diff and then tries to patch it * Requested changes
* revision based on goreportcard.comMáximo Cuadros2016-12-061-1/+1
|
* plumbing/packfile: PACK encoder (#131)Antonio Navarro Perez2016-11-244-15/+272
| | | | | | | | | | | | * plumbing/packfile: PACK encoder - Added simple PACK encoder, deltas not supported by now * Requested changes * Requested changes * Requested changes
* plumbing: format, packfile fix issue #129, related #124, and documentation ↵Máximo Cuadros2016-11-232-51/+154
| | | | | | | | improvements (#130) * plumbing: format, packfile fix issue #129, related #124 * plumbing: format, packfile documentation improvements
generated by cgit (git 2.34.1) at 2024-09-22 02:25:08 +0000