aboutsummaryrefslogtreecommitdiffstats
path: root/plumbing/format/packfile/parser.go
Commit message (Collapse)AuthorAgeFilesLines
* plumbing: Optimise memory consumption for filesystem storagePaulo Gomes2023-10-281-50/+145
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, as part of building the index representation, the resolveObject func would create an interim plumbing.MemoryObject, which would then be saved into storage via storage.SetEncodedObject. This meant that objects would be unnecessarily loaded into memory, to then be saved into disk. The changes streamlines this process by: - Introducing the LazyObjectWriter interface which enables the write operation to take places directly against the filesystem-based storage. - Leverage multi-writers to process the input data once, while targeting multiple writers (e.g. hasher and storage). An additional change relates to the caching of object info children within Parser.get. The cache is now skipped when a seekable filesystem is being used. The impact of the changes can be observed when using seekable filesystem storages, especially when cloning large repositories. The stats below were captured by adapting the BenchmarkPlainClone test to clone https://github.com/torvalds/linux.git: pkg: github.com/go-git/go-git/v5 cpu: Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz │ /tmp/old │ /tmp/new │ │ sec/op │ sec/op vs base │ PlainClone-16 41.68 ± 17% 48.04 ± 9% +15.27% (p=0.015 n=6) │ /tmp/old │ /tmp/new │ │ B/op │ B/op vs base │ PlainClone-16 1127.8Mi ± 7% 256.7Mi ± 50% -77.23% (p=0.002 n=6) │ /tmp/old │ /tmp/new │ │ allocs/op │ allocs/op vs base │ PlainClone-16 3.125M ± 0% 3.800M ± 0% +21.60% (p=0.002 n=6) Notice that on average the memory consumption per operation is over 75% smaller. The time per operation increased by 15%, which may actual be less on long running applications, due to the decreased GC pressure and the garbage collection costs. Signed-off-by: Paulo Gomes <pjbgf@linux.com>
* *: Remove use of deprecated io/utilPaulo Gomes2023-05-111-2/+1
| | | | Signed-off-by: Paulo Gomes <pjbgf@linux.com>
* plumbing: resolve non-external delta referencesZauberNerd2023-03-231-0/+9
| | | | | | | | | | | In a self-contained pack file delta references might point to base objects stored later in the file. In this case we need to replace placeholders for external refs with the actual base object and update the children references. Fixes: #484 Co-authored-by: Markus Wolf <mail@markus-wolf.de>
* Optimize zlib reader and consolidate sync.poolsPaulo Gomes2022-11-071-8/+11
| | | | | | | | | | | | | | | | | | | Expands on the optimisations from https://github.com/fluxcd/go-git/pull/5 and ensures that zlib reader does not need to recreate a deflate dictionary at every use. The use of sync pools was consolidated into a new sync utils package. name old time/op new time/op delta Parser-16 7.51ms ± 3% 7.71ms ± 6% ~ (p=0.222 n=5+5) name old alloc/op new alloc/op delta Parser-16 4.65MB ± 3% 1.90MB ± 3% -59.06% (p=0.008 n=5+5) name old allocs/op new allocs/op delta Parser-16 3.48k ± 0% 3.32k ± 0% -4.57% (p=0.016 n=5+4) Signed-off-by: Paulo Gomes <pjbgf@linux.com>
* Merge branch 'go-git:master' into masterPaul T2021-12-151-1/+0
|\
| * Remove unused variables/types/functionsAbhinav Gupta2021-11-271-1/+0
| | | | | | | | | | | | | | | | [staticcheck](https://staticcheck.io/) reported a number of unused fields, functions, types, and variables across the code. Where possible, use them (assert unchecked errors in tests, for example) and otherwise remove them.
* | resolve external reference deltaspaul.t2021-10-111-0/+11
|/
* Close Reader & Writer of EncodedObject after useKyungmin Bae2020-05-241-5/+10
|
* *: migration from gopkg to go modulesMáximo Cuadros2020-03-101-3/+3
|
* feat: avoid memory allocation on resolveDeltasNao YONASHIRO2019-07-311-46/+53
| | | | Signed-off-by: Nao YONASHIRO <owan.orisano@gmail.com>
* plumbing: format/packfile, performance optimizations for reading large ↵Filip Navara2018-11-281-5/+1
| | | | | | commit histories (#963) Signed-off-by: Filip Navara <navara@emclient.com>
* plumbing/format/packfile: Fix broken "thin" packfile support. Fixes #991Javier Peletier2018-10-231-47/+45
| | | | Signed-off-by: Javier Peletier <jm@epiclabs.io>
* plumbing/packfile: do not compute sha1 for already undeltified objectsJavi Fontan2018-08-141-7/+9
| | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing/pacfile: tidy up objectInfo structJavi Fontan2018-08-141-36/+22
| | | | | | | | * a new hasher is created when needed * delete unused fields * base content is no longer kept in memory Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing: add buffer cache and use it in packfile parserJavi Fontan2018-08-141-14/+10
| | | | | | | It uses less memory and is faster as slices don't have to be converted from/to MemoryObject and they are indexed by offset. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing: packfile, read object content only onceMiguel Molina2018-08-091-7/+15
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* plumbing: packfile, allow non-seekable sources on ParserMiguel Molina2018-08-081-109/+202
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* *: use parser to populate non writable storages and bug fixesMiguel Molina2018-08-071-43/+87
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* plumbing, storage: integrate new indexJavi Fontan2018-07-261-5/+6
| | | | | | Now dotgit.PackWriter uses the new packfile.Parser and index. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing/packfile: add new packfile parserJavi Fontan2018-07-261-0/+359
Signed-off-by: Javi Fontan <jfontan@gmail.com>
generated by cgit (git 2.34.1) at 2024-09-30 01:19:08 +0000