aboutsummaryrefslogtreecommitdiffstats
path: root/plumbing/format/packfile/packfile.go
Commit message (Collapse)AuthorAgeFilesLines
* Optimize zlib reader and consolidate sync.poolsPaulo Gomes2022-11-071-18/+16
| | | | | | | | | | | | | | | | | | | Expands on the optimisations from https://github.com/fluxcd/go-git/pull/5 and ensures that zlib reader does not need to recreate a deflate dictionary at every use. The use of sync pools was consolidated into a new sync utils package. name old time/op new time/op delta Parser-16 7.51ms ± 3% 7.71ms ± 6% ~ (p=0.222 n=5+5) name old alloc/op new alloc/op delta Parser-16 4.65MB ± 3% 1.90MB ± 3% -59.06% (p=0.008 n=5+5) name old allocs/op new allocs/op delta Parser-16 3.48k ± 0% 3.32k ± 0% -4.57% (p=0.016 n=5+4) Signed-off-by: Paulo Gomes <pjbgf@linux.com>
* plumbing: format/packfile, prevent large objects from being read into memory ↵zeripath2021-06-301-7/+85
| | | | | | | | | | | | | | | completely (#330) This PR adds code to prevent large objects from being read into memory from packfiles or the filesystem. Objects greater than 1Mb are now no longer directly stored in the cache or read completely into memory. This PR differs and improves the previous broken #323 by fixing several bugs in the reader and transparently wrapping ReaderAt as a Reader. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Revert "plumbing: format/packfile, prevent large objects from being read ↵v5.4.2zeripath2021-06-021-73/+0
| | | | | into memory completely (#303)" (#329) This reverts commit 720c192831a890d0a36b4c6720b60411fa4a0159.
* plumbing: format/packfile, prevent large objects from being read into memory ↵v5.4.0zeripath2021-05-121-0/+73
| | | | | | | | | | | completely (#303) This PR adds code to prevent large objects from being read into memory from packfiles or the filesystem. Objects greater than 1Mb are now no longer directly stored in the cache or read completely into memory. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Close Reader & Writer of EncodedObject after useKyungmin Bae2020-05-241-1/+4
|
* *: migration from gopkg to go modulesMáximo Cuadros2020-03-101-5/+5
|
* plumbing: format/packfile, Fix data race and resource leak.Ed Bardsley2019-06-031-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Two problems are fixed: - Buffers are not returned to the pool in the case of errors. - Per https://golang.org/pkg/bytes/#Buffer.Bytes, the slice returned from bytes.Buffer.Bytes() is only valid until the next modification of the buffer, so it must be copied before the buffer is returned to the pool. Running `go test -race` detected the following: ``` ================== WARNING: DATA RACE Write at 0x00c000224020 by goroutine 15: bytes.(*Buffer).WriteByte() /usr/local/Cellar/go/1.11.5/libexec/src/bytes/buffer.go:271 +0xc8 vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.diffDelta() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/diff_delta.go:95 +0x505 vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.getDelta() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/diff_delta.go:60 +0x4ae vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*deltaSelector).tryToDeltify() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/delta_selector.go:309 +0x398 vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*deltaSelector).walk() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/delta_selector.go:262 +0x33b vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*deltaSelector).ObjectsToPack.func1() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/delta_selector.go:70 +0x6a Previous read at 0x00c000224020 by goroutine 8: runtime.slicecopy() /usr/local/Cellar/go/1.11.5/libexec/src/runtime/slice.go:221 +0x0 vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.getDelta() vendor/gopkg.in/src-d/go-git.v4/plumbing/memory.go:53 +0x5e2 vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*deltaSelector).tryToDeltify() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/delta_selector.go:309 +0x398 vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*deltaSelector).walk() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/delta_selector.go:262 +0x33b vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*deltaSelector).ObjectsToPack.func1() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/delta_selector.go:70 +0x6a Goroutine 15 (running) created at: vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*deltaSelector).ObjectsToPack() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/delta_selector.go:69 +0x761 vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Encoder).Encode() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/encoder.go:52 +0xb1 vendor/gopkg.in/src-d/go-git%2ev4.pushHashes.func1() vendor/gopkg.in/src-d/go-git.v4/remote.go:1026 +0x102 Goroutine 8 (finished) created at: vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*deltaSelector).ObjectsToPack() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/delta_selector.go:69 +0x761 vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Encoder).Encode() vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/encoder.go:52 +0xb1 vendor/gopkg.in/src-d/go-git%2ev4.pushHashes.func1() vendor/gopkg.in/src-d/go-git.v4/remote.go:1026 +0x102 ================== ``` Signed-off-by: Ed Bardsley <ewb@uber.com>
* Merge pull request #1121 from filipnavara/small-deltasMáximo Cuadros2019-04-241-56/+126
|\ | | | | plumbing: packfile, apply small object reading optimization also for delta objects
| * Populate the offsetToType cache in the iterator even for the skipped objectsFilip Navara2019-04-241-0/+3
| | | | | | | | Signed-off-by: Filip Navara <navara@emclient.com>
| * Better fix for the getObjectType problem in the iteratorFilip Navara2019-04-241-6/+9
| | | | | | | | Signed-off-by: Filip Navara <navara@emclient.com>
| * Remove duplicate cache lookup in getObjectContent, it is already done in ↵Filip Navara2019-04-241-13/+0
| | | | | | | | | | | | FSObject Signed-off-by: Filip Navara <navara@emclient.com>
| * Fix an error where getObjectType would seek in file and mess up the position ↵Filip Navara2019-04-241-7/+11
| | | | | | | | | | | | used by getNextObject in the subsequent statement Signed-off-by: Filip Navara <navara@emclient.com>
| * Reduce indentationFilip Navara2019-04-231-7/+5
| | | | | | | | Signed-off-by: Filip Navara <navara@emclient.com>
| * Add one more cache check to the iteratorFilip Navara2019-04-231-2/+12
| | | | | | | | Signed-off-by: Filip Navara <navara@emclient.com>
| * Address PR feedbackFilip Navara2019-04-231-2/+2
| | | | | | | | Signed-off-by: Filip Navara <filip.navara@gmail.com>
| * Rename functionsFilip Navara2019-04-231-7/+9
| | | | | | | | Signed-off-by: Filip Navara <filip.navara@gmail.com>
| * Avoid filling up the object cache from Packfile.GetByType with typ != ↵Filip Navara2019-04-231-17/+42
| | | | | | | | | | | | plumbing.AnyObject Signed-off-by: Filip Navara <filip.navara@gmail.com>
| * Get the buffer from pool in fillOFSDeltaObjectContent instead of creating ↵Filip Navara2019-04-221-1/+1
| | | | | | | | | | | | new one Signed-off-by: Filip Navara <filip.navara@gmail.com>
| * Update commentFilip Navara2019-04-211-3/+3
| | | | | | | | Signed-off-by: Filip Navara <navara@emclient.com>
| * plumbing: packfile, apply small object reading optimization also for delta ↵Filip Navara2019-04-201-23/+61
| | | | | | | | | | | | objects Signed-off-by: Filip Navara <navara@emclient.com>
* | filesystem: ObjectStorage, MaxOpenDescriptors optionArran Walker2019-04-221-0/+5
|/ | | | | | | | The MaxOpenDescriptors option provides a middle ground solution between keeping all packfiles open (as offered by the KeepDescriptors option) and keeping none open. Signed-off-by: Arran Walker <arran.walker@fiveturns.org>
* packfile: get object size correctly for delta objectsJeremy Stribling2019-01-311-1/+1
| | | | Signed-off-by: Jeremy Stribling <strib@alum.mit.edu>
* plumbing: format/packfile, performance optimizations for reading large ↵Filip Navara2018-11-281-23/+28
| | | | | | commit histories (#963) Signed-off-by: Filip Navara <navara@emclient.com>
* Remove unused method (#1022)Antonio Navarro Perez2018-11-161-56/+0
| | | Signed-off-by: Antonio Jesus Navarro Perez <antnavper@gmail.com>
* packfile: add comment on GetSizeByOffsetJeremy Stribling2018-10-151-0/+2
| | | | | | | Suggested by mcuadros. Issue: src-d/go-git#982 Signed-off-by: Jeremy Stribling <strib@alum.mit.edu>
* object: get object size without reading whole objectJeremy Stribling2018-10-111-0/+16
| | | | Signed-off-by: Jeremy Stribling <strib@alum.mit.edu>
* plumbing, storage: add bases to the common cacheJavi Fontan2018-08-221-0/+15
| | | | | | | | | | | | After clone only resolved deltas were added to the cache. This caused slowdowns in small repositories where most objects can be held in cache. It also makes packfiles reuse delta cache from the store. Previously it created a new delta cache each time a packfile object was created. This also slowed down a bit accessing objects and had an impact on memory consumption when bases are added to the cache. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing/idxfile: object iterators returns entries in offset orderJavi Fontan2018-08-211-1/+1
| | | | | | | | | | | In the latest change the order was changed from offset order in packfiles to hash order. This makes reading all the objects not as efficient as before. It also created problems when the previous order was expected. Also added EntriesByOffset to indexes. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing: packfile, open and close packfile on FSObject readsMiguel Molina2018-08-091-30/+39
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* plumbing: packfile, rename DiskObject to FSObjectMiguel Molina2018-08-091-2/+2
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* *: use parser to populate non writable storages and bug fixesMiguel Molina2018-08-071-35/+100
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* Merge pull request #907 from erizocosmico/feature/fix-testsMiguel Molina2018-08-011-0/+26
| | | | | plumbing: packfile, fix package tests Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* plumbing: packfile, lazy object reads with DiskObjectsMiguel Molina2018-07-271-26/+182
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* plumbing: packfile, new Packfile representationMiguel Molina2018-07-261-0/+249
Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
generated by cgit (git 2.34.1) at 2024-09-27 11:23:53 +0000