aboutsummaryrefslogtreecommitdiffstats
path: root/storage
Commit message (Collapse)AuthorAgeFilesLines
* Merge pull request #1128 from pjbgf/small-fixesPaulo Gomes2024-08-091-2/+2
|\ | | | | Fix some lint warning and increase stalebot to 180 days
| * *: Small fixes to remove linter warningsPaulo Gomes2024-06-291-2/+2
| | | | | | | | Signed-off-by: Paulo Gomes <pjbgf@linux.com>
* | storage: filesystem, Fix object cache not working due to uninitialised ↵SatelliteMind2024-07-122-2/+63
| | | | | | | | objects being put into cache; The hash (which is the cache key) returned by the object at the previous put-into-cache point is always zero, so subsequent loads of the same object were never resolved from the cache.
* | storage: Fix reference updated concurrently error for the filesystem storerJavier Alvarez Garcia2024-06-292-0/+79
|/ | | | | | | | | | | | If a reference exists in the packed refs file but not in the actual refs folder, we need to fall back to reading the pack refs file when storing the reference. This happens because by the time we are storing the reference, we open the file with write permissions and read from it. At that point, if the file is empty, we still need to read the valid reference. Otherwise, it will read 0, and assume that the references do not match (because the old value was read from the packed file).
* storage: dotgit, add testAyman Bagabas2024-04-271-7/+13
|
* storage: dotgit, head reference usually comes firstAyman Bagabas2024-04-261-7/+8
| | | | | | | When reading the repository references from DotGit, the HEAD reference should always come first. This mimics the behavior of `git show-ref --head`
* plumbing: Properly detect EOF when reading index extensionsAlexander Block2024-04-051-1/+1
| | | | | | | | | Before this, go-git was relying on the peeked header to not include a valid 4 char string header. While doing this, it did not differentiate between the errournously read final hash and an unknown extension. This made it impossible to properly skip unknown optional extensions while detecting EOF early enough.
* storage: filesystem, Add option to set a specific FS for alternatesPaulo Gomes2023-12-023-53/+178
| | | | | | | | | | | Introduces the option to set a FS for alternates, enabling more flexible cross FS sharing of alternates. If none is set, falls back to the current FS used for the object storage. The changes only process a given path once, and if an alternates dir is not valid, exits with error - aligning behaviour with upstream. Signed-off-by: Paulo Gomes <paulo.gomes@suse.com>
* plumbing: Optimise memory consumption for filesystem storagePaulo Gomes2023-10-281-0/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, as part of building the index representation, the resolveObject func would create an interim plumbing.MemoryObject, which would then be saved into storage via storage.SetEncodedObject. This meant that objects would be unnecessarily loaded into memory, to then be saved into disk. The changes streamlines this process by: - Introducing the LazyObjectWriter interface which enables the write operation to take places directly against the filesystem-based storage. - Leverage multi-writers to process the input data once, while targeting multiple writers (e.g. hasher and storage). An additional change relates to the caching of object info children within Parser.get. The cache is now skipped when a seekable filesystem is being used. The impact of the changes can be observed when using seekable filesystem storages, especially when cloning large repositories. The stats below were captured by adapting the BenchmarkPlainClone test to clone https://github.com/torvalds/linux.git: pkg: github.com/go-git/go-git/v5 cpu: Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz │ /tmp/old │ /tmp/new │ │ sec/op │ sec/op vs base │ PlainClone-16 41.68 ± 17% 48.04 ± 9% +15.27% (p=0.015 n=6) │ /tmp/old │ /tmp/new │ │ B/op │ B/op vs base │ PlainClone-16 1127.8Mi ± 7% 256.7Mi ± 50% -77.23% (p=0.002 n=6) │ /tmp/old │ /tmp/new │ │ allocs/op │ allocs/op vs base │ PlainClone-16 3.125M ± 0% 3.800M ± 0% +21.60% (p=0.002 n=6) Notice that on average the memory consumption per operation is over 75% smaller. The time per operation increased by 15%, which may actual be less on long running applications, due to the decreased GC pressure and the garbage collection costs. Signed-off-by: Paulo Gomes <pjbgf@linux.com>
* git: clone --shared implementedenverbisevac2023-10-084-1/+44
|
* storage: filesystem/dotgit, add support for tmp_objdir prefixSavely Krasovsky2023-07-202-3/+26
|
* *: Remove use of deprecated io/utilPaulo Gomes2023-05-114-11/+7
| | | | Signed-off-by: Paulo Gomes <pjbgf@linux.com>
* storage: filesystem, Populate index before use. Fixes #148Arieh Schneier2023-05-042-0/+25
| | | | Signed-off-by: Arieh Schneier <15041913+AriehSchneier@users.noreply.github.com>
* storage: filesystem/dotgit, Improve load packed-refsForce Charlie2023-04-251-42/+38
|
* Merge pull request #707 from pjbgf/experimental-sha256Paulo Gomes2023-04-112-7/+10
|\ | | | | *: Add support for initializing SHA256 repositories
| * *: Support variable length plumbing.HashPaulo Gomes2023-03-082-7/+10
| | | | | | | | | | | | | | | | | | | | | | | | The variable length for plumbing.Hash is defined at build time, blocked by tag sha256. This approach was a trade-off between keeping backwards compatibility while making progress towards supporting SHA256 with a small amount of changes. Relates to the SHA256 implementation, defined in #706. Signed-off-by: Paulo Gomes <pjbgf@linux.com>
* | dotgit: fix deleted references test in windowsJavi Fontan2023-03-071-2/+3
| | | | | | | | Signed-off-by: Javi Fontan <jfontan@gmail.com>
* | dotgit: test skip deleted referencesJavi Fontan2023-03-071-0/+67
| | | | | | | | | | | | | | Checks that reading references it correctly skips deleted directories and files. Signed-off-by: Javi Fontan <jfontan@gmail.com>
* | dotgit: fix a filesystem race in Refs/walkReferencesTreeMichael Muré2023-01-141-0/+5
|/ | | | | | | | | | | | In walkReferencesTree(), the filesystem is browsed recursively. After a folder is listed, each child element (directory, file) are inspected. What can happen is that by the time we get to operate on the child element, it might have been deleted from the filesystem and we would get a PathError. In the case of directories there was already a case to avoid bubbling up the error (we consider that there is no ref there, which is correct), but that was missing for files. This commit just apply the same logic.
* Use Sync.Pool pointers to optimise memory usagePaulo Gomes2022-11-071-1/+13
| | | | Signed-off-by: Paulo Gomes <pjbgf@linux.com>
* minor grammatical fixesJon Eskin2022-09-221-1/+1
|
* Merge pull request #425 from abhinav/error-stringsMáximo Cuadros2021-12-112-2/+2
|\ | | | | error strings: Don't capitalize, use periods, or newlines
| * error strings: Don't capitalize, use periods, or newlinesAbhinav Gupta2021-12-042-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Per [Go Code Review Comments][1], > Error strings should not be capitalized (unless beginning with proper > nouns or acronyms) or end with punctuation staticcheck's [ST1005][2] also complains about these. For example, ``` object_walker.go:63:10: error strings should not be capitalized (ST1005) object_walker.go:101:10: error strings should not be capitalized (ST1005) object_walker.go:101:10: error strings should not end with punctuation or a newline (ST1005) plumbing/format/commitgraph/file.go:17:26: error strings should not be capitalized (ST1005) ``` This fixes all instances of this issue reported by staticcheck. [1]: https://github.com/golang/go/wiki/CodeReviewComments#error-strings [2]: https://staticcheck.io/docs/checks/#ST1005
* | Merge pull request #418 from abhinav/unusedMáximo Cuadros2021-12-104-7/+3
|\ \ | | | | | | Remove unused vars/types/funcs/fields
| * | transactional/ReferenceStorage: Drop packRefs fieldAbhinav Gupta2021-11-271-4/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The packRefs field is unused. It is assigned to true from the `PackRefs()` method, but because the method is not on the pointer type, the assignment has no effect. var st ReferenceStorage fmt.Println(st.packRefs) // false st.PackRefs() fmt.Println(st.packRefs) // false Delete the unused field.
| * | storage/transactional: Use correct config in testAbhinav Gupta2021-11-271-1/+1
| | | | | | | | | | | | | | | | | | The `cfg` loaded from `cs.Config` was unused. It looks like this assertion intended to validate that, not `temporalCfg`, which was already checked above this block.
| * | Remove unused variables/types/functionsAbhinav Gupta2021-11-272-2/+2
| |/ | | | | | | | | | | | | | | [staticcheck](https://staticcheck.io/) reported a number of unused fields, functions, types, and variables across the code. Where possible, use them (assert unchecked errors in tests, for example) and otherwise remove them.
* / storage: filesystem, switch from os.SEEK_* to io.Seek* (#421)Abhinav Gupta2021-12-102-5/+6
|/ | | | | | | The `os.SEEK_*` constants have been deprecated since Go 1.7. It is now recommended to use the equivalent `io.Seek*` constants. Switch `os.SEEK_CUR` to `io.SeekCurrent`, and `os.SEEK_SET` to `io.SeekStart`.
* plumbing: format/packfile, prevent large objects from being read into memory ↵zeripath2021-06-304-10/+156
| | | | | | | | | | | | | | | completely (#330) This PR adds code to prevent large objects from being read into memory from packfiles or the filesystem. Objects greater than 1Mb are now no longer directly stored in the cache or read completely into memory. This PR differs and improves the previous broken #323 by fixing several bugs in the reader and transparently wrapping ReaderAt as a Reader. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Revert "plumbing: format/packfile, prevent large objects from being read ↵v5.4.2zeripath2021-06-022-87/+1
| | | | | into memory completely (#303)" (#329) This reverts commit 720c192831a890d0a36b4c6720b60411fa4a0159.
* plumbing: format/packfile, prevent large objects from being read into memory ↵v5.4.0zeripath2021-05-122-1/+87
| | | | | | | | | | | completely (#303) This PR adds code to prevent large objects from being read into memory from packfiles or the filesystem. Objects greater than 1Mb are now no longer directly stored in the cache or read completely into memory. Signed-off-by: Andrew Thornton <art27@cantab.net>
* *: use go-billy instead of os callsMáximo Cuadros2021-05-026-114/+93
|
* Merge pull request #125 from cristaloleg/minor-linter-fixesMáximo Cuadros2020-07-161-1/+1
|\ | | | | *: minor linter fixes
| * Minor linter fixesOleg Kovalov2020-07-061-1/+1
| |
* | Support partial hashes in Repository.ResolveRevision.David Symonds2020-07-164-3/+157
| | | | | | | | | | | | | | | | | | | | | | | | | | Like `git rev-parse <prefix>`, this enumerates the hashes of objects with the given prefix and adds them to the list of candidates for resolution. This has an exhaustive slow path, which requires enumerating all objects and filtering each one, but also a couple of fast paths for common cases. There's room for future work to make this faster; TODOs have been left for that. Fixes #135.
* | Use only one name for receiverOleg Kovalov2020-07-061-2/+2
|/
* fix goreportcard warningsOleg Kovalov2020-07-017-8/+13
|
* Support `.git/commondir` repository layoutTimofey Kirillov2020-06-153-0/+241
| | | | | | | | | | | | | | | | Git creates `.git/commondir` when there are custom worktrees (see "git worktree add" related commands). `.git/commondir` in such case contains a link to another dot-git repository tree, which could contain some folders like: - objects; - config; - refs; - etc. In this PR a new dotgit.RepositoryFilesystem struct is defined, which is billy.Filesystem interface compatible object-wrapper, that can handle commondir and dispatch all operations to the correct file path. `git.PlainOpen` remain unchanged, but `git.PlainOpenWithOptions` has a new option: `PlainOpenOptions.EnableDotGitCommonDir=true|false` (which is false by default). When `EnableDotGitCommonDir=true` repository-open procedure will read `.git/commondir` (if it exists) and then create dotgit.RepositoryFilesystem object initialized with 2 filesystems. This object then passed into storage and then into dotgit.DotGit as `billy.Filesystem` interface. This object will catch all filesystem operations and dispatch to the correct repository-filesystem (dot-git or common-dot-git) according to the rules described in the doc: https://git-scm.com/docs/gitrepository-layout#Documentation/gitrepository-layout.txt. EnableDotGitCommonDir option will only work with the filesystem-backed storage. Also worktree_test.go has been adopted from an older, already existing existing PR: https://github.com/src-d/go-git/pull/1098. This PR needs new fixtures added in the following PR: https://github.com/go-git/go-git-fixtures/pull/1.
* Merge pull request #73 from WKBae/close_objectv5.1.0Máximo Cuadros2020-05-242-2/+14
|\ | | | | Close Reader & Writer of EncodedObject after use
| * Close Reader & Writer of EncodedObject after useKyungmin Bae2020-05-242-2/+14
| |
* | config: ReadConfig and LoadConfig from scopesMáximo Cuadros2020-05-241-15/+2
| |
* | Revert "Merge pull request #20 from quorumcontrol/feature/other-configs"Máximo Cuadros2020-05-243-86/+22
|/ | | | | This reverts commit 3127ad9a44a2ee935502816065dfe39f494f583d, reversing changes made to 73c52edaad2dae256be61bd1dbbab08e1092f58e.
* storage/filesystem: dotgit, sanity check provided reference pathKyle Evans2020-04-232-1/+12
| | | | | | | | | | On some operating systems, read() of a directory fd may actually succeed and return garbage resembling on-disk contents. As a result, dotgit may return successful from readReferenceFile() when it should in-fact not. This fixes readReferenceFile() on FreeBSD. Signed-off-by: Kyle Evans <kevans@FreeBSD.org>
* storage/filesystem: dotgit, add a test for reading refs from a directoryKyle Evans2020-04-231-1/+13
| | | | | | Specifically, the problem demonstrated is that on some OS (e.g. FreeBSD), read() on a directory will succeed. The current implementation will accept that and return incorrect results, rather than rejecting the directory.
* Fix dotgit testsWes Morgan2020-04-061-3/+3
|
* Add Merged configWes Morgan2020-04-062-19/+83
| | | | ...for reading and writing global (~/.git/config) and reading system (/etc/gitconfig) configs in addition to local repo config
* ci: based on github actionsv5.0.0Máximo Cuadros2020-03-152-16/+4
|
* *: migration from go-git-fixtures/v4 and go-git/gcfgMáximo Cuadros2020-03-108-34/+14
|
* *: migration from gopkg to go modulesMáximo Cuadros2020-03-1033-106/+106
|
* Merge pull request #6 from go-git/pr-1153Máximo Cuadros2020-03-102-4/+10
|\ | | | | storage/filesystem: dotgit, unable to work with .git folder with temporal packfiles
generated by cgit (git 2.34.1) at 2024-11-10 17:06:56 +0000