aboutsummaryrefslogtreecommitdiffstats
path: root/plumbing/format/packfile
Commit message (Collapse)AuthorAgeFilesLines
* Merge pull request #1124 from saracen/packfile-scanner-improved-readerMáximo Cuadros2019-04-233-105/+143
|\ | | | | plumbing: packfile/scanner, readability/performance improvements, zlib pooling
| * plumbing: packfile/scanner, readability/performance improvements, zlib poolingArran Walker2019-04-223-105/+143
| | | | | | | | Signed-off-by: Arran Walker <arran.walker@fiveturns.org>
* | filesystem: ObjectStorage, MaxOpenDescriptors optionArran Walker2019-04-221-0/+5
|/ | | | | | | | The MaxOpenDescriptors option provides a middle ground solution between keeping all packfiles open (as offered by the KeepDescriptors option) and keeping none open. Signed-off-by: Arran Walker <arran.walker@fiveturns.org>
* packfile: get object size correctly for delta objectsJeremy Stribling2019-01-312-1/+27
| | | | Signed-off-by: Jeremy Stribling <strib@alum.mit.edu>
* plumbing/packfile: test UpdateObjectStorage empty packfile errorJavi Fontan2018-12-031-0/+14
| | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
* git: return better error message when packfile cannot be downloadedJavi Fontan2018-11-301-1/+7
| | | | | | | | Previously the error message when the connection was closed while fetching was "object not found" and was misleading. Now when the packfile size is 0 the error "unable to fetch packfile" is returned. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing: format/packfile, performance optimizations for reading large ↵Filip Navara2018-11-285-33/+89
| | | | | | commit histories (#963) Signed-off-by: Filip Navara <navara@emclient.com>
* Merge pull request #994 from epiclabs-io/fix-thin-packMáximo Cuadros2018-11-162-47/+95
|\ | | | | plumbing/format/packfile: Fix broken "thin" packfile support. Fixes #991
| * plumbing/format/packfile: Added thin pack testJavier Peletier2018-11-121-0/+50
| | | | | | | | Signed-off-by: Javier Peletier <jm@epiclabs.io>
| * plumbing/format/packfile: Fix broken "thin" packfile support. Fixes #991Javier Peletier2018-10-231-47/+45
| | | | | | | | Signed-off-by: Javier Peletier <jm@epiclabs.io>
* | Remove unused method (#1022)Antonio Navarro Perez2018-11-161-56/+0
|/ | | Signed-off-by: Antonio Jesus Navarro Perez <antnavper@gmail.com>
* packfile: add comment on GetSizeByOffsetJeremy Stribling2018-10-151-0/+2
| | | | | | | Suggested by mcuadros. Issue: src-d/go-git#982 Signed-off-by: Jeremy Stribling <strib@alum.mit.edu>
* object: get object size without reading whole objectJeremy Stribling2018-10-111-0/+16
| | | | Signed-off-by: Jeremy Stribling <strib@alum.mit.edu>
* Expose Storage cache.kuba--2018-09-071-4/+3
| | | | Signed-off-by: kuba-- <kuba@sourced.tech>
* plumbing, storage: add bases to the common cacheJavi Fontan2018-08-222-0/+25
| | | | | | | | | | | | After clone only resolved deltas were added to the cache. This caused slowdowns in small repositories where most objects can be held in cache. It also makes packfiles reuse delta cache from the store. Previously it created a new delta cache each time a packfile object was created. This also slowed down a bit accessing objects and had an impact on memory consumption when bases are added to the cache. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing/idxfile: object iterators returns entries in offset orderJavi Fontan2018-08-211-1/+1
| | | | | | | | | | | In the latest change the order was changed from offset order in packfiles to hash order. This makes reading all the objects not as efficient as before. It also created problems when the previous order was expected. Also added EntriesByOffset to indexes. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing/packfile: do not compute sha1 for already undeltified objectsJavi Fontan2018-08-141-7/+9
| | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing/pacfile: tidy up objectInfo structJavi Fontan2018-08-141-36/+22
| | | | | | | | * a new hasher is created when needed * delete unused fields * base content is no longer kept in memory Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing: add buffer cache and use it in packfile parserJavi Fontan2018-08-141-14/+10
| | | | | | | It uses less memory and is faster as slices don't have to be converted from/to MemoryObject and they are indexed by offset. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing: packfile, open and close packfile on FSObject readsMiguel Molina2018-08-095-56/+126
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* plumbing: packfile, rename DiskObject to FSObjectMiguel Molina2018-08-092-15/+15
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* plumbing: packfile, read object content only onceMiguel Molina2018-08-092-7/+40
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* plumbing: packfile, add Parse benchmarkMiguel Molina2018-08-091-0/+30
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* plumbing: packfile, allow non-seekable sources on ParserMiguel Molina2018-08-085-177/+226
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* *: use parser to populate non writable storages and bug fixesMiguel Molina2018-08-079-1156/+489
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* Merge pull request #907 from erizocosmico/feature/fix-testsMiguel Molina2018-08-014-34/+98
|\ | | | | | | | | plumbing: packfile, fix package tests Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
| * plumbing: packfile, fix package testsMiguel Molina2018-07-303-32/+70
|/ | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* plumbing/packfile: add index generation to decoderJavi Fontan2018-07-271-7/+25
| | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing: packfile, lazy object reads with DiskObjectsMiguel Molina2018-07-274-27/+293
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* plumbing: packfile, new Packfile representationMiguel Molina2018-07-265-154/+418
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* plumbing, storage: integrate new indexJavi Fontan2018-07-262-9/+11
| | | | | | Now dotgit.PackWriter uses the new packfile.Parser and index. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing/packfile: preallocate memory in PatchDeltaJavi Fontan2018-07-261-1/+1
| | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing/packfile: disable lookup by offsetJavi Fontan2018-07-261-8/+9
| | | | | | | In one case it disables the cache and the other disables lookup when the scanner is not seekable. Could be added back later. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing/packfile: add new packfile parserJavi Fontan2018-07-262-0/+498
| | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing/format/idxfile: add new Index and MemoryIndexMiguel Molina2018-07-192-135/+17
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* packfile: optimise NewIndexFromIdxFile for a very common caseDavid Symonds2018-06-211-2/+12
| | | | | | | Loading from an on-disk idxfile will usually already have the idxfile entries in order, so check that before wasting time on sorting. Signed-off-by: David Symonds <dsymonds@golang.org>
* plumbing: packfile, Don't push empty objects. Fixes #840kuba--2018-06-072-4/+19
| | | | Signed-off-by: kuba-- <kuba@sourced.tech>
* packfile: improve Index memory representation to be more compactDavid Symonds2018-05-302-23/+67
| | | | | | | | | | | | | | | | | | Instead of using a map for offset indexing, use a sorted slice. Binary searching is fast, and a slice is much more compact. This has a negligible hit on speed, but has a significant impact on memory usage, especially for larger repos. benchmark old ns/op new ns/op delta BenchmarkIndexConstruction-12 15506506 14056098 -9.35% benchmark old allocs new allocs delta BenchmarkIndexConstruction-12 60764 60385 -0.62% benchmark old bytes new bytes delta BenchmarkIndexConstruction-12 4318145 3913169 -9.38% Signed-off-by: David Symonds <dsymonds@golang.org>
* *: Use CheckClose with named returnsJavi Fontan2018-03-272-4/+4
| | | | | | | | Previously some close errors were losts. This is specially problematic in go-git as lots of work is done here like generating indexes and moving packfiles. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* *: skip time consuming testsMáximo Cuadros2018-03-211-0/+9
| | | | Signed-off-by: Máximo Cuadros <mcuadros@gmail.com>
* plumbing: format/packfile, add SaveOriginalMetadata functionJavi Fontan2018-02-092-5/+9
| | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing: format/packfile, fix panic retrieving object hash.Javi Fontan2018-02-093-4/+10
| | | | | | | | | | | | | | | | In some cases the original data is not saved before it is cleaned and forces a panic when it's needed. The change adds ObjectToPack.CleanOriginal to be used to clean original object instead of: object.Original = nil Now when the Original data is freed because it's no longer in the pack window a SetOriginal call is done to make sure that Size, Hash and Size data is not lost. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing: format/packfile, check nil objects in ObjectToPackJavi Fontan2018-01-252-8/+12
| | | | | | | SetOriginal now skips setting resolved values if the provided object is nil. BackToOriginal also skips nil Original objects. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing: format/packfile, fix crash with cycle deltasJavi Fontan2018-01-244-1/+51
| | | | | | | | | | | | Resolving cycles relied on ObjectToPack objects having Original. This is no longer true with the changes from #720. This commit changes: * Save original type, hash and size in ObjectToPack * Use SetObject to set both Original and resolved type, hash and size * Restore original object before using BackToOriginal (cycle resolution) * Update encoder test to check this case Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing: packfile, Add crc check to scanner test.Javi Fontan2018-01-211-4/+75
| | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing: packfile, Add a buffer to crc writer.Javi Fontan2018-01-211-9/+31
| | | | | | | | | | | crc update with block smaller than 16 bytes uses a slower version of the function. ReadByte is heavily used by zlib inflate so most of the time crc is update byte by byte. A new Flush method is added to the scanner to flush this crc writer cache. It is only called when the Scanner reader is a teeReader. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* Modify cache to delete more than one item to free spaceJavi Fontan2018-01-161-0/+2
| | | | | | | | | | | The previous version could only delete the oldest used object. If the object to cache was bigger than the space freed it could not be added. Also the decoder adds bases to the cache when they are needed. This change increases the speed creating indexes 2x. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* Clean reconstructed objects outside pack windowJavi Fontan2018-01-111-13/+19
| | | | | | | | | | | Object walk reconstructs delta objects but these are not cleaned up after they got out the pack window. Without this change all reconstructed objects reside in memory. restoreOriginal call is moved before calling Size(). Now we can not guarantee that the object is already undeltified. Signed-off-by: Javi Fontan <javier@sourced.tech>
* Merge pull request #698 from jfontan/improvement/use-decoder-cacheMáximo Cuadros2017-12-202-17/+39
|\ | | | | plumbing: cache, enforce the use of cache in packfile decoder
| * Make DeltaBaseCache privateJavi Fontan2017-12-201-6/+13
| | | | | | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
generated by cgit (git 2.34.1) at 2024-11-14 11:14:59 +0000