aboutsummaryrefslogtreecommitdiffstats
path: root/storage/filesystem
Commit message (Collapse)AuthorAgeFilesLines
* Merge pull request #1123 from saracen/object-storage-open-packfileMáximo Cuadros2019-04-234-49/+138
|\ | | | | filesystem: ObjectStorage, MaxOpenDescriptors option
| * filesystem: ObjectStorage, MaxOpenDescriptors optionArran Walker2019-04-224-49/+138
| | | | | | | | | | | | | | | | The MaxOpenDescriptors option provides a middle ground solution between keeping all packfiles open (as offered by the KeepDescriptors option) and keeping none open. Signed-off-by: Arran Walker <arran.walker@fiveturns.org>
* | plumbing: format/index perf, buffered reads, reflection removalArran Walker2019-04-211-1/+2
|/ | | | | | | | | | Large performance increase by buffering reads. There were a few instances where binary.Read() would end up using reflection on &plumbing.Hash, rather than treating it as a byte slice. This has now been resolved. Signed-off-by: Arran Walker <arran.walker@fiveturns.org>
* Merge pull request #1006 from mcuadros/transactional-storageMáximo Cuadros2019-02-021-1/+2
|\ | | | | storage: transactional, new storage with transactional capabilities
| * storage: new storage.ErrReferenceHasChanged error and test for ↵Máximo Cuadros2018-12-101-1/+2
| | | | | | | | | | | | CheckAndSetReference Signed-off-by: Máximo Cuadros <mcuadros@gmail.com>
* | storage/filesystem: check file object before using cacheJavi Fontan2019-01-302-5/+21
| | | | | | | | | | | | | | | | | | | | If the cache is shared between several repositories getFromUnpacked can erroneously return an object from other repository. This decreases performance a little bit as there's an extra fs operation when the object is in the cache but is correct when the cache is shared. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* | storage/dotgit: test setRef with a non rwfsJavi Fontan2018-12-101-1/+24
| | | | | | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
* | storage/dotgit: use fs capabilities in setRefJavi Fontan2018-12-102-49/+49
| | | | | | | | | | | | | | Do not use tags to check if the filesystem is able to open files in read/write mode. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* | plumbing: format/packfile, performance optimizations for reading large ↵Filip Navara2018-11-282-28/+37
| | | | | | | | | | | | commit histories (#963) Signed-off-by: Filip Navara <navara@emclient.com>
* | storage/filesystem: Added reindex method to reindex packfilesJavier Peletier2018-11-122-0/+61
|/ | | | Signed-off-by: Javier Peletier <jm@epiclabs.io>
* filesystem: add a new test for EncodedObjectSizeJeremy Stribling2018-10-122-3/+15
| | | | | | Suggested by taruti. Signed-off-by: Jeremy Stribling <strib@alum.mit.edu>
* object: get object size without reading whole objectJeremy Stribling2018-10-112-0/+99
| | | | Signed-off-by: Jeremy Stribling <strib@alum.mit.edu>
* storage/filesystem: add more doc to NewPackfileIterJavi Fontan2018-09-211-4/+7
| | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
* storage/filesystem: keep packs open in PackfileIterJavi Fontan2018-09-202-14/+63
| | | | | | | | PackfileIter was not taking into account the option KeepDescriptors and was always closing the file. This caused "file already closed" errors when iterating packfiles in with KeepDescriptors active. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* Expose Storage cache.kuba--2018-09-076-60/+41
| | | | Signed-off-by: kuba-- <kuba@sourced.tech>
* storage/filesystem: compare files using offset in testJavi Fontan2018-09-062-3/+20
| | | | | | | | Using equals to compare files it uses diff to do so. This can potentially consume lots of ram. Changed the comparison to use file offsets. If the descriptor is reused the offset is maintained. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* storage/filesystem: add KeepDescriptors testJavi Fontan2018-09-051-0/+31
| | | | | | Also delete Close from MockObjectStorage and memory storer. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* storage/dotgit: add KeepDescriptors optionJavi Fontan2018-09-044-2/+83
| | | | | | | | | | This option maintains packfile file descriptors opened after reading objects from them. It improves performance as it does not have to be opening packfiles each time an object is needed. Also adds Close to EncodedObjectStorer to close all the files manualy. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* storage/dotgit: add ExclusiveAccess tests in dotgitJavi Fontan2018-09-041-0/+24
| | | | | | | | This functionality was already tested in storage/filesystem. The coverage tool only takes into account files from the same package of the test. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* storage/filesystem: move Options to filesytem and dotgitJavi Fontan2018-09-034-25/+50
| | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
* storage/filesystem: make Storage options privateJavi Fontan2018-09-031-2/+2
| | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing/storer: rename Static option to ExclusiveAccessJavi Fontan2018-08-312-6/+6
| | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
* dotgit: fix typo in commentJavi Fontan2018-08-301-1/+1
| | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
* git, storer: use a common storer.Options for storer and PlainOpenJavi Fontan2018-08-303-29/+17
| | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
* git: add Static option to PlainOpenJavi Fontan2018-08-303-6/+221
| | | | | | | | | | | | | | | | Also adds Static configuration to Storage and DotGit. This option means that the git repository is not expected to be modified while open and enables some optimizations. Each time a file is accessed the storer tries to open an object file for the requested hash. When this is done for a lot of objects it is expensive. With Static option a list of object files is generated the first time an object is accessed and used to check if exists instead of using system calls. A similar optimization is done for packfiles. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* storage/dotgit: use HasPrefix instead of SplitJavi Fontan2018-08-271-6/+9
| | | | | | Also reformatted function comment and fixed some typos. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* storage/dotgit: search for incoming dir only onceJavi Fontan2018-08-251-13/+28
| | | | | | | | | Search for incoming object directory was done once each time objects were accessed. This means a ReadDir of the objects path that is expensive. Now incoming directory is searched the first time an object is accessed and its name kept in DotGit to be reused. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* plumbing, storage: add bases to the common cacheJavi Fontan2018-08-221-2/+16
| | | | | | | | | | | | After clone only resolved deltas were added to the cache. This caused slowdowns in small repositories where most objects can be held in cache. It also makes packfiles reuse delta cache from the store. Previously it created a new delta cache each time a packfile object was created. This also slowed down a bit accessing objects and had an impact on memory consumption when bases are added to the cache. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* Merge pull request #887 from noxora/hook-supportMáximo Cuadros2018-08-212-3/+101
|\ | | | | added support for quarantine directory
| * dotgit: fix object delete testSantiago M. Mola2018-08-171-14/+18
| | | | | | | | Signed-off-by: Santiago M. Mola <santi@mola.io>
| * added hook supportnoxora2018-08-162-4/+98
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Signed-off-by: noxora <ldecker@indeed.com> trying a possible fix to the delete test Signed-off-by: noxora <ldecker@indeed.com> still trying to fix this test Signed-off-by: noxora <ldecker@indeed.com> fixes did not work, seems to be a windows env problem Signed-off-by: noxora <ldecker@indeed.com>
* | plumbing: packfile, open and close packfile on FSObject readsMiguel Molina2018-08-093-12/+48
| | | | | | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* | storage: filesystem, add PackfileIter benchmark reading object contentMiguel Molina2018-08-091-0/+67
| | | | | | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* | storage: filesystem, close Packfile after iterating objectsMiguel Molina2018-08-091-1/+10
| | | | | | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* | storage: filesystem, benchmark PackfileIterMiguel Molina2018-08-092-15/+94
| | | | | | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* | plumbing: packfile, allow non-seekable sources on ParserMiguel Molina2018-08-081-1/+6
| | | | | | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* | *: use parser to populate non writable storages and bug fixesMiguel Molina2018-08-072-48/+35
| | | | | | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* | storage/filesystem: remove duplicated IndexStorageJavi Fontan2018-07-271-15/+0
| | | | | | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
* | Merge pull request #899 from erizocosmico/feature/new-packfileMiguel Molina2018-07-271-0/+15
|\ \ | | | | | | plumbing: packfile, new Packfile representation
| * | plumbing: packfile, new Packfile representationMiguel Molina2018-07-261-0/+15
| | | | | | | | | | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* | | storage/filesystem: add back IndexStorageJavi Fontan2018-07-271-0/+47
|/ / | | | | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
* | plumbing, storage: integrate new indexJavi Fontan2018-07-263-36/+46
| | | | | | | | | | | | Now dotgit.PackWriter uses the new packfile.Parser and index. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* | plumbing/format/idxfile: add new Index and MemoryIndexMiguel Molina2018-07-192-48/+1
|/ | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* storage/filesystem: avoid norwfs build flagSantiago M. Mola2018-06-215-94/+82
| | | | | | | | | | | | | | | | | norwfs build flag was used to work on filesystems that do not support neither opening a file in read/write mode or renaming a file (e.f. sivafs). This had two problems: - go-git could not be compiled to work properly both with regular filesystems and limited filesystems at the same time. - the norwfs trick was not available on Windows. This PR removes the norwfs build flag, as well as the windows conditional flag on the dotgit package. For the file open mode, we use the new billy capabilities, to check at runtime if the filesystem supports opening a file in read/write mode or not. For the renaming, we just try and fallback to alternative methods if billy.ErrNotSupported is returned. Signed-off-by: Santiago M. Mola <santi@mola.io>
* storage: filesystem, make ObjectStorage constructor publicMiguel Molina2018-06-083-8/+9
| | | | Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
* dotgit: Move package outside internal.Antonio Jesus Navarro Perez2018-06-0518-9/+9
| | | | Signed-off-by: Antonio Jesus Navarro Perez <antnavper@gmail.com>
* storage: filesystem, close shallow file when readMáximo Cuadros2018-04-171-0/+2
| | | | Signed-off-by: Máximo Cuadros <mcuadros@gmail.com>
* Merge pull request #807 from keybase/strib/src-d-ignore-non-hash-filesMáximo Cuadros2018-04-172-3/+23
|\ | | | | dotgit: ignore filenames that don't match a hash
| * dotgit: add test for bad file in pack directoryJeremy Stribling2018-04-161-1/+13
| | | | | | | | | | | | | | Suggested by mcuadros. Issue: src-d/go-git#807 Signed-off-by: Jeremy Stribling <strib@alum.mit.edu>
| * dotgit: ignore filenames that don't match a hashJeremy Stribling2018-04-131-2/+10
| | | | | | | | | | | | | | For both packfiles and object files. Issue: keybase/client#11366 Signed-off-by: Jeremy Stribling <strib@alum.mit.edu>
generated by cgit (git 2.34.1) at 2024-11-10 14:34:11 +0000