aboutsummaryrefslogtreecommitdiffstats
path: root/plumbing/format/packfile
Commit message (Collapse)AuthorAgeFilesLines
* plumbing: format/packfile, Refactor patch deltaPaulo Gomes2023-11-062-113/+79
| | | | | | The changes aim to simplify the code and reduce duplication. Signed-off-by: Paulo Gomes <pjbgf@linux.com>
* plumbing: Optimise memory consumption for filesystem storagePaulo Gomes2023-10-282-50/+247
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, as part of building the index representation, the resolveObject func would create an interim plumbing.MemoryObject, which would then be saved into storage via storage.SetEncodedObject. This meant that objects would be unnecessarily loaded into memory, to then be saved into disk. The changes streamlines this process by: - Introducing the LazyObjectWriter interface which enables the write operation to take places directly against the filesystem-based storage. - Leverage multi-writers to process the input data once, while targeting multiple writers (e.g. hasher and storage). An additional change relates to the caching of object info children within Parser.get. The cache is now skipped when a seekable filesystem is being used. The impact of the changes can be observed when using seekable filesystem storages, especially when cloning large repositories. The stats below were captured by adapting the BenchmarkPlainClone test to clone https://github.com/torvalds/linux.git: pkg: github.com/go-git/go-git/v5 cpu: Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz │ /tmp/old │ /tmp/new │ │ sec/op │ sec/op vs base │ PlainClone-16 41.68 ± 17% 48.04 ± 9% +15.27% (p=0.015 n=6) │ /tmp/old │ /tmp/new │ │ B/op │ B/op vs base │ PlainClone-16 1127.8Mi ± 7% 256.7Mi ± 50% -77.23% (p=0.002 n=6) │ /tmp/old │ /tmp/new │ │ allocs/op │ allocs/op vs base │ PlainClone-16 3.125M ± 0% 3.800M ± 0% +21.60% (p=0.002 n=6) Notice that on average the memory consumption per operation is over 75% smaller. The time per operation increased by 15%, which may actual be less on long running applications, due to the decreased GC pressure and the garbage collection costs. Signed-off-by: Paulo Gomes <pjbgf@linux.com>
* fuzzing : fuzz testing support for oss-fuzz integrationArjun Singh2023-10-021-0/+12
| | | | Signed-off-by: Arjun Singh <ajsinghyadav00@gmail.com>
* *: Remove use of deprecated io/utilPaulo Gomes2023-05-114-13/+10
| | | | Signed-off-by: Paulo Gomes <pjbgf@linux.com>
* *: Remove redudant err nil checksPaulo Gomes2023-05-111-5/+1
| | | | Signed-off-by: Paulo Gomes <pjbgf@linux.com>
* Merge pull request #707 from pjbgf/experimental-sha256Paulo Gomes2023-04-113-7/+8
|\ | | | | *: Add support for initializing SHA256 repositories
| * *: Support variable length plumbing.HashPaulo Gomes2023-03-083-7/+8
| | | | | | | | | | | | | | | | | | | | | | | | The variable length for plumbing.Hash is defined at build time, blocked by tag sha256. This approach was a trade-off between keeping backwards compatibility while making progress towards supporting SHA256 with a small amount of changes. Relates to the SHA256 implementation, defined in #706. Signed-off-by: Paulo Gomes <pjbgf@linux.com>
* | plumbing: resolve non-external delta referencesZauberNerd2023-03-232-0/+22
|/ | | | | | | | | | | In a self-contained pack file delta references might point to base objects stored later in the file. In this case we need to replace placeholders for external refs with the actual base object and update the children references. Fixes: #484 Co-authored-by: Markus Wolf <mail@markus-wolf.de>
* sha1: Add collision resistent implementationPaulo Gomes2022-11-251-2/+3
| | | | | | | | | | | | | | | | | | | Implement the same SHA1 collision resistent algorithm used by both the Git CLI and libgit2. Only commits with input that match the unavoidable bit conditions will be further processed, which will result in different hashes. Which is the same behaviour experienced in the Git CLI and Libgit2. Users can override the hash algorithm used with: hash.RegisterHash(crypto.SHA1, sha1.New) xref links: https://github.com/libgit2/libgit2/pull/4136/commits/2dfd1294f7a694bfa9e864a9489ae3cb318a5ed0 https://github.com/git/git/commit/28dc98e343ca4eb370a29ceec4c19beac9b5c01e Signed-off-by: Paulo Gomes <pjbgf@linux.com>
* Optimize zlib reader and consolidate sync.poolsPaulo Gomes2022-11-076-98/+66
| | | | | | | | | | | | | | | | | | | Expands on the optimisations from https://github.com/fluxcd/go-git/pull/5 and ensures that zlib reader does not need to recreate a deflate dictionary at every use. The use of sync pools was consolidated into a new sync utils package. name old time/op new time/op delta Parser-16 7.51ms ± 3% 7.71ms ± 6% ~ (p=0.222 n=5+5) name old alloc/op new alloc/op delta Parser-16 4.65MB ± 3% 1.90MB ± 3% -59.06% (p=0.008 n=5+5) name old allocs/op new allocs/op delta Parser-16 3.48k ± 0% 3.32k ± 0% -4.57% (p=0.016 n=5+4) Signed-off-by: Paulo Gomes <pjbgf@linux.com>
* Use Sync.Pool pointers to optimise memory usagePaulo Gomes2022-11-073-7/+40
| | | | Signed-off-by: Paulo Gomes <pjbgf@linux.com>
* Fix typos (#532)Quanyi Ma2022-09-221-2/+2
|
* remove packfile and align to test fixturespaul.t2022-01-052-3/+2
|
* Merge branch 'master' into codecommit-ref-deltapaul.t2021-12-153-33/+0
|\
| * Merge branch 'go-git:master' into masterPaul T2021-12-153-33/+0
| |\
| | * Remove unused variables/types/functionsAbhinav Gupta2021-11-273-33/+0
| | | | | | | | | | | | | | | | | | | | | | | | [staticcheck](https://staticcheck.io/) reported a number of unused fields, functions, types, and variables across the code. Where possible, use them (assert unchecked errors in tests, for example) and otherwise remove them.
| * | resolve external reference deltaspaul.t2021-10-111-0/+11
| |/
* | include example codecommit pack filepaul.t2021-11-091-0/+0
| |
* | add codecommit packfile for testing external ref resolutionpaul.t2021-11-091-0/+14
| |
* | resolve external reference deltapaul.t2021-10-111-0/+11
|/
* plumbing: format/packfile, prevent large objects from being read into memory ↵zeripath2021-06-308-33/+405
| | | | | | | | | | | | | | | completely (#330) This PR adds code to prevent large objects from being read into memory from packfiles or the filesystem. Objects greater than 1Mb are now no longer directly stored in the cache or read completely into memory. This PR differs and improves the previous broken #323 by fixing several bugs in the reader and transparently wrapping ReaderAt as a Reader. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Revert "plumbing: format/packfile, prevent large objects from being read ↵v5.4.2zeripath2021-06-024-313/+0
| | | | | into memory completely (#303)" (#329) This reverts commit 720c192831a890d0a36b4c6720b60411fa4a0159.
* plumbing: format/packfile, prevent large objects from being read into memory ↵v5.4.0zeripath2021-05-124-0/+313
| | | | | | | | | | | completely (#303) This PR adds code to prevent large objects from being read into memory from packfiles or the filesystem. Objects greater than 1Mb are now no longer directly stored in the cache or read completely into memory. Signed-off-by: Andrew Thornton <art27@cantab.net>
* plumbing: format, use os.UserHomeDir()Máximo Cuadros2021-05-022-41/+29
|
* fix goreportcard warningsOleg Kovalov2020-07-013-6/+5
|
* Close Reader & Writer of EncodedObject after useKyungmin Bae2020-05-246-12/+41
|
* ci: based on github actionsv5.0.0Máximo Cuadros2020-03-151-10/+2
|
* *: migration from go-git-fixtures/v4 and go-git/gcfgMáximo Cuadros2020-03-105-20/+12
|
* *: migration from gopkg to go modulesMáximo Cuadros2020-03-1018-51/+51
|
* feat: avoid memory allocation on ApplyDelta, PatchDeltaNao YONASHIRO2019-07-311-15/+30
| | | | Signed-off-by: Nao YONASHIRO <owan.orisano@gmail.com>
* refactor: use bufPoolNao YONASHIRO2019-07-311-12/+3
| | | | Signed-off-by: Nao YONASHIRO <owan.orisano@gmail.com>
* feat: avoid memory allocation on resolveDeltasNao YONASHIRO2019-07-311-46/+53
| | | | Signed-off-by: Nao YONASHIRO <owan.orisano@gmail.com>
* feat: avoid ioutil.ReadAll on ApplyDeltaNao YONASHIRO2019-07-311-2/+15
| | | | Signed-off-by: Nao YONASHIRO <owan.orisano@gmail.com>
* *: added missing error checks in testsChristian Muehlhaeuser2019-07-291-0/+1
| | | | | | | | | | When we assign a value to err, make sure to also check for it being nil afterwards. If those were intentionally unchecked, we should remove the assignment in the first place. Those checks certainly never harm, but please review thoroughly and let me know. Signed-off-by: Christian Muehlhaeuser <muesli@gmail.com> (cherry picked from commit 19d6f42a4d814a50bd262fbb69a9b670db9756a2)
* plumbing: format/packfile, Fix data race and resource leak.Ed Bardsley2019-06-032-11/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Two problems are fixed: - Buffers are not returned to the pool in the case of errors. - Per https://golang.org/pkg/bytes/#Buffer.Bytes, the slice returned from bytes.Buffer.Bytes() is only valid until the next modification of the buffer, so it must be copied before the buffer is returned to the pool. Running `go test -race` detected the following: ``` ================== WARNING: DATA RACE Write at 0x00c000224020 by goroutine 15: bytes.(*Buffer).WriteByte() /usr/local/Cellar/go/1.11.5/libexec/src/bytes/buffer.go:271 +0xc8 vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.diffDelta() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/diff_delta.go:95 +0x505 vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.getDelta() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/diff_delta.go:60 +0x4ae vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*deltaSelector).tryToDeltify() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/delta_selector.go:309 +0x398 vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*deltaSelector).walk() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/delta_selector.go:262 +0x33b vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*deltaSelector).ObjectsToPack.func1() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/delta_selector.go:70 +0x6a Previous read at 0x00c000224020 by goroutine 8: runtime.slicecopy() /usr/local/Cellar/go/1.11.5/libexec/src/runtime/slice.go:221 +0x0 vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.getDelta() vendor/gopkg.in/src-d/go-git.v4/plumbing/memory.go:53 +0x5e2 vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*deltaSelector).tryToDeltify() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/delta_selector.go:309 +0x398 vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*deltaSelector).walk() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/delta_selector.go:262 +0x33b vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*deltaSelector).ObjectsToPack.func1() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/delta_selector.go:70 +0x6a Goroutine 15 (running) created at: vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*deltaSelector).ObjectsToPack() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/delta_selector.go:69 +0x761 vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Encoder).Encode() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/encoder.go:52 +0xb1 vendor/gopkg.in/src-d/go-git%2ev4.pushHashes.func1() vendor/gopkg.in/src-d/go-git.v4/remote.go:1026 +0x102 Goroutine 8 (finished) created at: vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*deltaSelector).ObjectsToPack() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/delta_selector.go:69 +0x761 vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Encoder).Encode() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/encoder.go:52 +0xb1 vendor/gopkg.in/src-d/go-git%2ev4.pushHashes.func1() vendor/gopkg.in/src-d/go-git.v4/remote.go:1026 +0x102 ================== ``` Signed-off-by: Ed Bardsley <ewb@uber.com>
* Merge pull request #1121 from filipnavara/small-deltasMáximo Cuadros2019-04-241-56/+126
|\ | | | | plumbing: packfile, apply small object reading optimization also for delta objects
| * Populate the offsetToType cache in the iterator even for the skipped objectsFilip Navara2019-04-241-0/+3
| | | | | | | | Signed-off-by: Filip Navara <navara@emclient.com>
| * Better fix for the getObjectType problem in the iteratorFilip Navara2019-04-241-6/+9
| | | | | | | | Signed-off-by: Filip Navara <navara@emclient.com>
| * Remove duplicate cache lookup in getObjectContent, it is already done in ↵Filip Navara2019-04-241-13/+0
| | | | | | | | | | | | FSObject Signed-off-by: Filip Navara <navara@emclient.com>
| * Fix an error where getObjectType would seek in file and mess up the position ↵Filip Navara2019-04-241-7/+11
| | | | | | | | | | | | used by getNextObject in the subsequent statement Signed-off-by: Filip Navara <navara@emclient.com>
| * Reduce indentationFilip Navara2019-04-231-7/+5
| | | | | | | | Signed-off-by: Filip Navara <navara@emclient.com>
| * Add one more cache check to the iteratorFilip Navara2019-04-231-2/+12
| | | | | | | | Signed-off-by: Filip Navara <navara@emclient.com>
| * Address PR feedbackFilip Navara2019-04-231-2/+2
| | | | | | | | Signed-off-by: Filip Navara <filip.navara@gmail.com>
| * Rename functionsFilip Navara2019-04-231-7/+9
| | | | | | | | Signed-off-by: Filip Navara <filip.navara@gmail.com>
| * Avoid filling up the object cache from Packfile.GetByType with typ != ↵Filip Navara2019-04-231-17/+42
| | | | | | | | | | | | plumbing.AnyObject Signed-off-by: Filip Navara <filip.navara@gmail.com>
| * Get the buffer from pool in fillOFSDeltaObjectContent instead of creating ↵Filip Navara2019-04-221-1/+1
| | | | | | | | | | | | new one Signed-off-by: Filip Navara <filip.navara@gmail.com>
| * Update commentFilip Navara2019-04-211-3/+3
| | | | | | | | Signed-off-by: Filip Navara <navara@emclient.com>
| * plumbing: packfile, apply small object reading optimization also for delta ↵Filip Navara2019-04-201-23/+61
| | | | | | | | | | | | objects Signed-off-by: Filip Navara <navara@emclient.com>
* | Merge pull request #1124 from saracen/packfile-scanner-improved-readerMáximo Cuadros2019-04-233-105/+143
|\ \ | | | | | | plumbing: packfile/scanner, readability/performance improvements, zlib pooling
| * | plumbing: packfile/scanner, readability/performance improvements, zlib poolingArran Walker2019-04-223-105/+143
| |/ | | | | | | Signed-off-by: Arran Walker <arran.walker@fiveturns.org>
generated by cgit (git 2.34.1) at 2024-11-05 22:31:59 +0000