diff options
| author | Alberto Cortés <alcortesm@gmail.com> | 2016-07-04 17:09:22 +0200 |
|---|---|---|
| committer | Máximo Cuadros <mcuadros@gmail.com> | 2016-07-04 17:09:22 +0200 |
| commit | 5e73f01cb2e027a8f02801635b79d3a9bc866914 (patch) | |
| tree | c0e7eb355c9b8633d99bab9295cb72b6c3a9c0e1 | |
| parent | 808076af869550a200a3a544c9ee2fa22a8b6a85 (diff) | |
| download | go-git-5e73f01cb2e027a8f02801635b79d3a9bc866914.tar.gz | |
Adds support to open local repositories and to use file-based object storage (#55)v3.1.0
* remove some comments
* idx writer/reader
* Shut up ssh tests, they are annoying
* Add file scheme test to clients
* Add dummy file client
* Add test fot file client
* Make tests use fixture endpoint
* add parser for packed-refs format
* add parser for packed-refs format
* WIP adding dir.Refs() tests
* Add test for fixture refs
* refs parser for the refs directory
* Documentation
* Add Capabilities to file client
* tgz.Exatract now accpets a path instead of a Reader
* fix bug in idxfile fanout calculation
* remove dead code
* packfile documentation
* clean packfile parser code
* add core.Object.Content() and returns errors for core.ObjectStorage.Iter()
* add seekable storage
* add dir repos to NewRepository
* clean prints
* Add dir client documentation to README
* Organize the README
* README
* Clean tgz package
* Clean temp dirs after tgz tests
* Gometalinter on gitdir
* Clean pattern function
* metalinter tgz
* metalinter gitdir
* gitdir coverage and remove seekable packfile filedescriptor leak
* gitdir Idxfile tests and remove file descriptor leak
* gitdir Idxfile tests when no idx is found
* clean storage/seekable/internal/index and some formats/idxfile API issues
* clean storage/seekable
* clean formats/idx
* turn packfile/doc.go into packfile/doc.txt
* move formats/packfile/reader to decoder
* fix packfile decoder error names
* improve documentation
* comment packfile decoder errors
* comment public API (format/packfile)
* remve duplicated code in packfile decoder test
* move tracking_reader into an internal package and clean it
* use iota for packfile format
* rename packfile parse.go to packfile object_at.go
* clean packfile deltas
* fix delta header size bug
* improve delta documentation
* clean packfile deltas
* clean packfiles deltas
* clean repository.go
* Remove go 1.5 from Travis CI
Because go 1.5 does not suport internal packages.
* change local repo scheme to local://
* change "local://" to "file://" as the local scheme
* fix broken indentation
* shortens names of variables in short scopes
* more shortening of variable names
* more shortening of variable names
* Rename git dir client to "file", as the scheme used for it
* Fix file format ctor name, now that the package name has change
* Sortcut local repo constructor to not use remotes
The object storage is build directly in the repository ctor, instead
of creating a remote and waiting for the user to pull it.
* update README and fix some errors in it
* remove file scheme client
* Local respositories has now a new ctor
This is, they are no longer identified by the scheme of the URL, but are
created different from inception.
* remove unused URL field form Repository
* move all git dir logic to seekable sotrage ctor
* fix documentation
* Make formats/file/dir an internal package to storage/seekable
* change package storage/seekable to storage/fs
* clean storage/fs
* overall storage/fs clean
* more cleaning
* some metalinter fixes
* upgrade cshared to last changes
* remove dead code
* fix test error info
* remove file scheme check from clients
* fix test error message
* fix test error message
* fix error messages
* style changes
* fix comments everywhere
* style changes
* style changes
* scaffolding and tests for local packfiles without ifx files
* outsource index building from packfile to the packfile decoder
* refactor packfile header reading into a new function
* move code to generate index from packfile back to index package
* add header parsing
* fix documentation errata
* add undeltified and OFS delta support for index building from the packfile
* add tests for packfile with ref-deltas
* support for packfiles with ref-deltas and no idx
* refactor packfile format parser to reuse code
* refactor packfile format parser to reuse code
* refactor packfile format parser to reuse code
* refactor packfile format parser to reuse code
* refactor packfile format parser to reuse code
* WIP refactor packfile format parser to reuse code
* refactor packfile format parser to reuse code
* remove prints from tests
* remove prints from tests
* refactor packfile.core into packfile.parser
* rename packfile reader to something that shows it is a recaller
* rename cannot recall error
* rename packfile.Reader to packfile.ReadRecaller and document
* speed up test by using StreamReader instead of SeekableReader when possible
* clean packfile StreamReader
* stream_reader tests
* refactor packfile.StreamReader into packfile.StreamReadRecaller
* refactor packfile.SeekableReader into packfile.SeekableReadRecaller and document it
* generalize packfile.StreamReadRecaller test to all packfile.ReadRecaller implementations
* speed up storage/fs tests
* speed up tests in . by loading packfiles in memory
* speed up repository tests by using and smaller fixture
* restore doc.go files
* rename packfile.ReadRecaller implementations to shorter names
* update comments to type changes
* packfile.Parser test (WIP)
* packfile.Parser tests and add ForgetAll() to packfile.ReadRecaller
* add test for packfile.ReadRecaller.ForgetAll()
* clarify seekable being able to recallByOffset forgetted objects
* use better names for internal maps
* metalinter packfile package
* speed up some tests
* documentation fixes
* change storage.fs package name to storage.proxy to avoid confusion with new filesystem support
* New fs package and os transparent implementation
Now NewRepositoryFromFS receives a fs and a path and tests are
modified accordingly, but it is still not using for anything.
* add fs to gitdir and proxy.store
* reduce fs interface for easier implementation
* remove garbage dirs from tgz tests
* change file name gitdir/dir.go to gitdir/gitdir.go
* fs.OS tests
* metalinter utils/fs
* add NewRepositoryFromFS documentation to README
* Readability fixes to README
* move tgz to an external dependency
* move filesystem impl. example to example dir
* rename proxy/store.go to proxy/storage.go for coherence with memory/storage.go
* rename proxy package to seekable
58 files changed, 6630 insertions, 924 deletions
diff --git a/.travis.yml b/.travis.yml index 388b676..ee769d2 100644 --- a/.travis.yml +++ b/.travis.yml @@ -2,7 +2,6 @@ language: go go: - 1.4 - - 1.5 - 1.6 - tip @@ -49,7 +49,10 @@ if err := r.PullDefault(); err != nil { panic(err) } -iter := r.Commits() +iter, err := r.Commits() +if err != nil { + panic(err) +} defer iter.Close() for { @@ -112,6 +115,61 @@ if err != nil { fmt.Println(commit) ``` +Creating a repository from an ordinary local git directory (that has been +previously prepared by running `git gc` on it). + +```go +// Download any git repository and prepare it as as follows: +// +// $ git clone https://github.com/src-d/go-git /tmp/go-git +// $ pushd /tmp/go-git ; git gc ; popd +// +// Then, create a go-git repository from the local content +// and print its commits as follows: + +package main + +import ( + "fmt" + "io" + + "gopkg.in/src-d/go-git.v3" + "gopkg.in/src-d/go-git.v3/utils/fs" +) + +func main() { + fs := fs.NewOS() // a simple proxy for the local host filesystem + path := "/tmp/go-git/.git" + + repo, err := git.NewRepositoryFromFS(fs, path) + if err != nil { + panic(err) + } + + iter, err := repo.Commits() + if err != nil { + panic(err) + } + defer iter.Close() + + for { + commit, err := iter.Next() + if err != nil { + if err == io.EOF { + break + } + panic(err) + } + + fmt.Println(commit) + } +} +``` + +Implementing your own filesystem will let you access repositories stored on +remote services (e.g. amazon S3), see the +[examples](https://github.com/src-d/go-git/tree/master/examples/fs_implementation/) +directory for a simple filesystem implementation and usage. Wrapping -------- diff --git a/blame_test.go b/blame_test.go index 880b0ad..40ae0b0 100644 --- a/blame_test.go +++ b/blame_test.go @@ -1,6 +1,8 @@ package git import ( + "bytes" + "io/ioutil" "os" "gopkg.in/src-d/go-git.v3/core" @@ -19,24 +21,23 @@ var _ = Suite(&BlameCommon{}) func (s *BlameCommon) SetUpSuite(c *C) { s.repos = make(map[string]*Repository, 0) for _, fixRepo := range fixtureRepos { - repo := NewPlainRepository() - repo.URL = fixRepo.url + r := NewPlainRepository() - d, err := os.Open(fixRepo.packfile) + f, err := os.Open(fixRepo.packfile) c.Assert(err, IsNil) - r := packfile.NewReader(d) - // TODO: how to know the format of a pack file ahead of time? - // Some info at: - // https://codewords.recurse.com/issues/three/unpacking-git-packfiles - r.Format = packfile.OFSDeltaFormat + data, err := ioutil.ReadAll(f) + c.Assert(err, IsNil) + + stream := packfile.NewStream(bytes.NewReader(data)) - _, err = r.Read(repo.Storage) + d := packfile.NewDecoder(stream) + err = d.Decode(r.Storage) c.Assert(err, IsNil) - c.Assert(d.Close(), IsNil) + c.Assert(f.Close(), IsNil) - s.repos[fixRepo.url] = repo + s.repos[fixRepo.url] = r } } @@ -48,22 +49,22 @@ type blameTest struct { } func (s *BlameCommon) mockBlame(t blameTest, c *C) (blame *Blame) { - repo, ok := s.repos[t.repo] + r, ok := s.repos[t.repo] c.Assert(ok, Equals, true) - commit, err := repo.Commit(core.NewHash(t.rev)) - c.Assert(err, IsNil, Commentf("%v: repo=%s, rev=%s", err, repo, t.rev)) + commit, err := r.Commit(core.NewHash(t.rev)) + c.Assert(err, IsNil, Commentf("%v: repo=%s, rev=%s", err, r, t.rev)) - file, err := commit.File(t.path) + f, err := commit.File(t.path) c.Assert(err, IsNil) - lines, err := file.Lines() + lines, err := f.Lines() c.Assert(err, IsNil) c.Assert(len(t.blames), Equals, len(lines), Commentf( "repo=%s, path=%s, rev=%s: the number of lines in the file and the number of expected blames differ (len(blames)=%d, len(lines)=%d)\nblames=%#q\nlines=%#q", t.repo, t.path, t.rev, len(t.blames), len(lines), t.blames, lines)) blamedLines := make([]*line, 0, len(t.blames)) for i := range t.blames { - commit, err := repo.Commit(core.NewHash(t.blames[i])) + commit, err := r.Commit(core.NewHash(t.blames[i])) c.Assert(err, IsNil) l := &line{ author: commit.Author.Email, @@ -82,17 +83,17 @@ func (s *BlameCommon) mockBlame(t blameTest, c *C) (blame *Blame) { // run a blame on all the suite's tests func (s *BlameCommon) TestBlame(c *C) { for _, t := range blameTests { - expected := s.mockBlame(t, c) + exp := s.mockBlame(t, c) - repo, ok := s.repos[t.repo] + r, ok := s.repos[t.repo] c.Assert(ok, Equals, true) - commit, err := repo.Commit(core.NewHash(t.rev)) + commit, err := r.Commit(core.NewHash(t.rev)) c.Assert(err, IsNil) - obtained, err := commit.Blame(t.path) + obt, err := commit.Blame(t.path) c.Assert(err, IsNil) - c.Assert(obtained, DeepEquals, expected) + c.Assert(obt, DeepEquals, exp) } } @@ -105,16 +106,18 @@ func repeat(s string, n int) []string { for i := 0; i < n; i++ { r = append(r, s) } + return r } // utility function to concat slices func concat(vargs ...[]string) []string { - var result []string + var r []string for _, ss := range vargs { - result = append(result, ss...) + r = append(r, ss...) } - return result + + return r } var blameTests = [...]blameTest{ diff --git a/clients/common.go b/clients/common.go index 55b3b4b..b6da656 100644 --- a/clients/common.go +++ b/clients/common.go @@ -55,10 +55,10 @@ func NewGitUploadPackService(repoURL string) (common.GitUploadPackService, error if err != nil { return nil, fmt.Errorf("invalid url %q", repoURL) } - service, ok := KnownProtocols[u.Scheme] + s, ok := KnownProtocols[u.Scheme] if !ok { return nil, fmt.Errorf("unsupported scheme %q", u.Scheme) } - return service, nil + return s, nil } diff --git a/clients/common/common.go b/clients/common/common.go index 5aa6269..7280450 100644 --- a/clients/common/common.go +++ b/clients/common/common.go @@ -236,11 +236,7 @@ func (r *GitUploadPackInfo) decodeHeaderLine(line string) { } func (r *GitUploadPackInfo) isValidLine(line string) bool { - if line[0] == '#' { - return false - } - - return true + return line[0] != '#' } func (r *GitUploadPackInfo) readLine(line string) { diff --git a/clients/common_test.go b/clients/common_test.go index 88f66ab..c571f04 100644 --- a/clients/common_test.go +++ b/clients/common_test.go @@ -3,23 +3,41 @@ package clients import ( "fmt" "io" + "os" "testing" - . "gopkg.in/check.v1" "gopkg.in/src-d/go-git.v3/clients/common" + + "github.com/alcortesm/tgz" + . "gopkg.in/check.v1" ) func Test(t *testing.T) { TestingT(t) } -type SuiteCommon struct{} +type SuiteCommon struct { + dirFixturePath string +} var _ = Suite(&SuiteCommon{}) +const fixtureTGZ = "../storage/seekable/internal/gitdir/fixtures/spinnaker-gc.tgz" + +func (s *SuiteCommon) SetUpSuite(c *C) { + var err error + s.dirFixturePath, err = tgz.Extract(fixtureTGZ) + c.Assert(err, IsNil) +} + +func (s *SuiteCommon) TearDownSuite(c *C) { + err := os.RemoveAll(s.dirFixturePath) + c.Assert(err, IsNil) +} + func (s *SuiteCommon) TestNewGitUploadPackService(c *C) { var tests = [...]struct { - input string - err bool - expected string + input string + err bool + exp string }{ {"://example.com", true, "<nil>"}, {"badscheme://github.com/src-d/go-git", true, "<nil>"}, @@ -30,8 +48,10 @@ func (s *SuiteCommon) TestNewGitUploadPackService(c *C) { for i, t := range tests { output, err := NewGitUploadPackService(t.input) - c.Assert(err != nil, Equals, t.err, Commentf("%d) %q: wrong error value", i, t.input)) - c.Assert(typeAsString(output), Equals, t.expected, Commentf("%d) %q: wrong type", i, t.input)) + c.Assert(err != nil, Equals, t.err, + Commentf("%d) %q: wrong error value (was: %s)", i, t.input, err)) + c.Assert(typeAsString(output), Equals, t.exp, + Commentf("%d) %q: wrong type", i, t.input)) } } @@ -70,7 +90,6 @@ func (s *SuiteCommon) TestInstallProtocol(c *C) { for i, t := range tests { if t.panic { - fmt.Println(t.service == nil) c.Assert(func() { InstallProtocol(t.scheme, t.service) }, PanicMatches, `nil service`) continue } diff --git a/clients/ssh/git_upload_pack_test.go b/clients/ssh/git_upload_pack_test.go index 4b50c4c..07aa816 100644 --- a/clients/ssh/git_upload_pack_test.go +++ b/clients/ssh/git_upload_pack_test.go @@ -1,3 +1,5 @@ +// +build ssh + package ssh import ( diff --git a/common_test.go b/common_test.go index 53160a7..edc41a9 100644 --- a/common_test.go +++ b/common_test.go @@ -1,7 +1,9 @@ package git import ( + "bytes" "io" + "io/ioutil" "os" "testing" @@ -29,21 +31,22 @@ func (s *MockGitUploadPackService) ConnectWithAuth(url common.Endpoint, auth com } func (s *MockGitUploadPackService) Info() (*common.GitUploadPackInfo, error) { - hash := core.NewHash("6ecf0ef2c2dffb796033e5a02219af86ec6584e5") + h := core.NewHash("6ecf0ef2c2dffb796033e5a02219af86ec6584e5") - cap := common.NewCapabilities() - cap.Decode("6ecf0ef2c2dffb796033e5a02219af86ec6584e5 HEADmulti_ack thin-pack side-band side-band-64k ofs-delta shallow no-progress include-tag multi_ack_detailed no-done symref=HEAD:refs/heads/master agent=git/2:2.4.8~dbussink-fix-enterprise-tokens-compilation-1167-gc7006cf") + c := common.NewCapabilities() + c.Decode("6ecf0ef2c2dffb796033e5a02219af86ec6584e5 HEADmulti_ack thin-pack side-band side-band-64k ofs-delta shallow no-progress include-tag multi_ack_detailed no-done symref=HEAD:refs/heads/master agent=git/2:2.4.8~dbussink-fix-enterprise-tokens-compilation-1167-gc7006cf") return &common.GitUploadPackInfo{ - Capabilities: cap, - Head: hash, - Refs: map[string]core.Hash{"refs/heads/master": hash}, + Capabilities: c, + Head: h, + Refs: map[string]core.Hash{"refs/heads/master": h}, }, nil } func (s *MockGitUploadPackService) Fetch(*common.GitUploadPackRequest) (io.ReadCloser, error) { var err error s.RC, err = os.Open("formats/packfile/fixtures/git-fixture.ref-delta") + return s.RC, err } @@ -65,20 +68,28 @@ func unpackFixtures(c *C, fixtures ...[]packedFixture) map[string]*Repository { if _, existing := repos[fixture.url]; existing { continue } - repos[fixture.url] = NewPlainRepository() - d, err := os.Open(fixture.packfile) - c.Assert(err, IsNil) + comment := Commentf("fixture packfile: %q", fixture.packfile) - r := packfile.NewReader(d) - r.Format = packfile.OFSDeltaFormat // This is hardcoded because we don't have a good way to sniff the format + repos[fixture.url] = NewPlainRepository() - _, err = r.Read(repos[fixture.url].Storage) + f, err := os.Open(fixture.packfile) + c.Assert(err, IsNil, comment) + + // increase memory consumption to speed up tests + data, err := ioutil.ReadAll(f) c.Assert(err, IsNil) + memStream := bytes.NewReader(data) + r := packfile.NewStream(memStream) + + d := packfile.NewDecoder(r) + err = d.Decode(repos[fixture.url].Storage) + c.Assert(err, IsNil, comment) - c.Assert(d.Close(), IsNil) + c.Assert(f.Close(), IsNil, comment) } } + return repos } diff --git a/core/object.go b/core/object.go index 9dd906b..4610c45 100644 --- a/core/object.go +++ b/core/object.go @@ -34,16 +34,16 @@ type Object interface { SetType(ObjectType) Size() int64 SetSize(int64) + Content() []byte Reader() (ObjectReader, error) Writer() (ObjectWriter, error) } // ObjectStorage generic storage of objects type ObjectStorage interface { - New() (Object, error) Set(Object) (Hash, error) Get(Hash) (Object, error) - Iter(ObjectType) ObjectIter + Iter(ObjectType) (ObjectIter, error) } // ObjectType internal object type's diff --git a/cshared/repository_cshared.go b/cshared/repository_cshared.go index 6ca600f..1c8fa9d 100644 --- a/cshared/repository_cshared.go +++ b/cshared/repository_cshared.go @@ -5,8 +5,8 @@ import ( "C" "gopkg.in/src-d/go-git.v3" - "gopkg.in/src-d/go-git.v3/core" "gopkg.in/src-d/go-git.v3/clients/common" + "gopkg.in/src-d/go-git.v3/core" ) //export c_Repository @@ -90,25 +90,6 @@ func c_Repository_set_Storage(r uint64, val uint64) { repo.Storage = *obj.(*core.ObjectStorage) } -//export c_Repository_get_URL -func c_Repository_get_URL(r uint64) *C.char { - obj, ok := GetObject(Handle(r)) - if !ok { - return nil - } - return C.CString(obj.(*git.Repository).URL) -} - -//export c_Repository_set_URL -func c_Repository_set_URL(r uint64, val string) { - obj, ok := GetObject(Handle(r)) - if !ok { - return - } - repo := obj.(*git.Repository) - repo.URL = CopyString(val) -} - //export c_Repository_Pull func c_Repository_Pull(r uint64, remoteName, branch string) (int, *C.char) { obj, ok := GetObject(Handle(r)) @@ -150,20 +131,23 @@ func c_Repository_Commit(r uint64, h []byte) (uint64, int, *C.char) { if err != nil { return IH, ErrorCodeInternal, C.CString(err.Error()) } - commit_handle := RegisterObject(commit) + commit_handle := RegisterObject(commit) return uint64(commit_handle), ErrorCodeSuccess, nil } //export c_Repository_Commits -func c_Repository_Commits(r uint64) uint64 { +func c_Repository_Commits(r uint64) (uint64, int, *C.char) { obj, ok := GetObject(Handle(r)) if !ok { - return IH + return IH, ErrorCodeNotFound, C.CString(MessageNotFound) } repo := obj.(*git.Repository) - iter := repo.Commits() + iter, err := repo.Commits() + if err != nil { + return IH, ErrorCodeInternal, C.CString(err.Error()) + } iter_handle := RegisterObject(iter) - return uint64(iter_handle) + return uint64(iter_handle), ErrorCodeSuccess, nil } //export c_Repository_Tree @@ -218,15 +202,18 @@ func c_Repository_Tag(r uint64, h []byte) (uint64, int, *C.char) { } //export c_Repository_Tags -func c_Repository_Tags(r uint64) uint64 { +func c_Repository_Tags(r uint64) (uint64, int, *C.char) { obj, ok := GetObject(Handle(r)) if !ok { - return IH + return IH, ErrorCodeNotFound, C.CString(MessageNotFound) } repo := obj.(*git.Repository) - iter := repo.Tags() + iter, err := repo.Tags() + if err != nil { + return IH, ErrorCodeInternal, C.CString(err.Error()) + } iter_handle := RegisterObject(iter) - return uint64(iter_handle) + return uint64(iter_handle), ErrorCodeSuccess, nil } //export c_Repository_Object diff --git a/diff/diff_ext_test.go b/diff/diff_ext_test.go index aa94ca2..1fa3a78 100644 --- a/diff/diff_ext_test.go +++ b/diff/diff_ext_test.go @@ -52,17 +52,17 @@ func (s *suiteCommon) TestAll(c *C) { var doTests = [...]struct { src, dst string - expected []diffmatchpatch.Diff + exp []diffmatchpatch.Diff }{ { - src: "", - dst: "", - expected: []diffmatchpatch.Diff{}, + src: "", + dst: "", + exp: []diffmatchpatch.Diff{}, }, { src: "a", dst: "a", - expected: []diffmatchpatch.Diff{ + exp: []diffmatchpatch.Diff{ { Type: 0, Text: "a", @@ -72,7 +72,7 @@ var doTests = [...]struct { { src: "", dst: "abc\ncba", - expected: []diffmatchpatch.Diff{ + exp: []diffmatchpatch.Diff{ { Type: 1, Text: "abc\ncba", @@ -82,7 +82,7 @@ var doTests = [...]struct { { src: "abc\ncba", dst: "", - expected: []diffmatchpatch.Diff{ + exp: []diffmatchpatch.Diff{ { Type: -1, Text: "abc\ncba", @@ -92,7 +92,7 @@ var doTests = [...]struct { { src: "abc\nbcd\ncde", dst: "000\nabc\n111\nBCD\n", - expected: []diffmatchpatch.Diff{ + exp: []diffmatchpatch.Diff{ {Type: 1, Text: "000\n"}, {Type: 0, Text: "abc\n"}, {Type: -1, Text: "bcd\ncde"}, @@ -104,6 +104,6 @@ var doTests = [...]struct { func (s *suiteCommon) TestDo(c *C) { for i, t := range doTests { diffs := diff.Do(t.src, t.dst) - c.Assert(diffs, DeepEquals, t.expected, Commentf("subtest %d", i)) + c.Assert(diffs, DeepEquals, t.exp, Commentf("subtest %d", i)) } } diff --git a/examples/basic/main.go b/examples/basic/main.go index f5a1c8a..652ebf1 100644 --- a/examples/basic/main.go +++ b/examples/basic/main.go @@ -15,11 +15,14 @@ func main() { panic(err) } - if err := r.PullDefault(); err != nil { + if err = r.PullDefault(); err != nil { panic(err) } - iter := r.Commits() + iter, err := r.Commits() + if err != nil { + panic(err) + } defer iter.Close() for { diff --git a/examples/fs_implementation/main.go b/examples/fs_implementation/main.go new file mode 100644 index 0000000..0371f7f --- /dev/null +++ b/examples/fs_implementation/main.go @@ -0,0 +1,103 @@ +package main + +import ( + "fmt" + "io" + "io/ioutil" + "os" + "path/filepath" + "strings" + + "gopkg.in/src-d/go-git.v3" + gogitFS "gopkg.in/src-d/go-git.v3/utils/fs" +) + +func main() { + if len(os.Args) != 2 { + usage() + os.Exit(1) + } + + fs := newFS(os.Args[1]) + + repo, err := git.NewRepositoryFromFS(fs, ".git") + if err != nil { + fmt.Fprint(os.Stderr, err) + os.Exit(1) + } + + iter, err := repo.Commits() + if err != nil { + fmt.Fprint(os.Stderr, err) + os.Exit(1) + } + defer iter.Close() + + for { + commit, err := iter.Next() + if err != nil { + if err == io.EOF { + break + } + + fmt.Fprint(os.Stderr, err) + os.Exit(1) + } + + fmt.Println(commit) + } +} + +func usage() { + fmt.Fprintf(os.Stderr, "%s <path to .git dir>", os.Args[0]) +} + +// A simple proxy filesystem example: It mimics local filesystems, using +// 'base' as its root and a funny path separator ("--"). +// +// Example: when constructed with 'newFS("tmp")', a path like 'foo--bar' +// will represent the local path "/tmp/foo/bar". +type fs struct { + base string +} + +const separator = "--" + +func newFS(path string) *fs { + return &fs{ + base: path, + } +} + +func (fs *fs) Stat(path string) (info os.FileInfo, err error) { + f, err := os.Open(fs.ToReal(path)) + if err != nil { + return nil, err + } + + defer func() { + errClose := f.Close() + if err == nil { + err = errClose + } + }() + + return f.Stat() +} + +func (fs *fs) ToReal(path string) string { + parts := strings.Split(path, separator) + return filepath.Join(fs.base, filepath.Join(parts...)) +} + +func (fs *fs) Open(path string) (gogitFS.ReadSeekCloser, error) { + return os.Open(fs.ToReal(path)) +} + +func (fs *fs) ReadDir(path string) ([]os.FileInfo, error) { + return ioutil.ReadDir(fs.ToReal(path)) +} + +func (fs *fs) Join(elem ...string) string { + return strings.Join(elem, separator) +} diff --git a/examples/fs_implementation/main_test.go b/examples/fs_implementation/main_test.go new file mode 100644 index 0000000..4f138c2 --- /dev/null +++ b/examples/fs_implementation/main_test.go @@ -0,0 +1,195 @@ +package main + +import ( + "io/ioutil" + "os" + "path/filepath" + "reflect" + "regexp" + "testing" + + "github.com/alcortesm/tgz" +) + +func TestMain(m *testing.M) { + setUp() + rval := m.Run() + tearDown() + os.Exit(rval) +} + +func setUp() { + var err error + repo, err = tgz.Extract("../../storage/seekable/internal/gitdir/fixtures/spinnaker-gc.tgz") + if err != nil { + panic(err) + } +} + +var repo string + +func tearDown() { + err := os.RemoveAll(repo) + if err != nil { + panic(err) + } +} + +func TestJoin(t *testing.T) { + fs := newFS("") + for i, test := range [...]struct { + input []string + expected string + }{ + { + input: []string{}, + expected: "", + }, { + input: []string{"a"}, + expected: "a", + }, { + input: []string{"a", "b"}, + expected: "a--b", + }, { + input: []string{"a", "b", "c"}, + expected: "a--b--c", + }, + } { + obtained := fs.Join(test.input...) + if obtained != test.expected { + t.Fatalf("test %d:\n\tinput = %v\n\tobtained = %v\n\texpected = %v\n", + i, test.input, obtained, test.expected) + } + } +} + +func TestStat(t *testing.T) { + fs := newFS(filepath.Join(repo, ".git/")) + for i, path := range [...]string{ + "index", + "info--refs", + "objects--pack--pack-584416f86235cac0d54bfabbdc399fb2b09a5269.pack", + } { + real, err := os.Open(fs.ToReal(path)) + if err != nil { + t.Fatalf("test %d: openning real: %s", err) + } + + expected, err := real.Stat() + if err != nil { + t.Fatalf("test %d: stat on real: %s", err) + } + + obtained, err := fs.Stat(path) + if err != nil { + t.Fatalf("test %d: fs.Stat unexpected error: %s", i, err) + } + + if !reflect.DeepEqual(obtained, expected) { + t.Fatalf("test %d:\n\tinput = %s\n\tobtained = %v\n\texpected = %v\n", + i, path, obtained, expected) + } + + err = real.Close() + if err != nil { + t.Fatalf("test %d: closing real: %s", i, err) + } + } +} + +func TestStatErrors(t *testing.T) { + fs := newFS(filepath.Join(repo, ".git/")) + for i, test := range [...]struct { + input string + errRegExp string + }{ + { + input: "bla", + errRegExp: ".*bla: no such file or directory", + }, { + input: "bla--foo", + errRegExp: ".*bla/foo: no such file or directory", + }, + } { + expected := regexp.MustCompile(test.errRegExp) + + _, err := fs.Stat(test.input) + if err == nil { + t.Fatalf("test %d: no error returned", i) + } + if !expected.MatchString(err.Error()) { + t.Fatalf("test %d: error missmatch\n\tobtained = %q\n\texpected regexp = %q\n", + i, err.Error(), test.errRegExp) + } + } +} + +func TestOpen(t *testing.T) { + fs := newFS(filepath.Join(repo, ".git/")) + for i, path := range [...]string{ + "index", + "info--refs", + "objects--pack--pack-584416f86235cac0d54bfabbdc399fb2b09a5269.pack", + } { + real, err := os.Open(fs.ToReal(path)) + if err != nil { + t.Fatalf("test %d: openning real: %s", err) + } + + realData, err := ioutil.ReadAll(real) + if err != nil { + t.Fatal("test %d: ioutil.ReadAll on real: %s", err) + } + + err = real.Close() + if err != nil { + t.Fatal("test %d: closing real: %s", err) + } + + obtained, err := fs.Open(path) + if err != nil { + t.Fatalf("test %d: fs.Open unexpected error: %s", i, err) + } + + obtainedData, err := ioutil.ReadAll(obtained) + if err != nil { + t.Fatal("test %d: ioutil.ReadAll on obtained: %s", err) + } + + err = obtained.Close() + if err != nil { + t.Fatal("test %d: closing obtained: %s", err) + } + + if !reflect.DeepEqual(obtainedData, realData) { + t.Fatalf("test %d:\n\tinput = %s\n\tobtained = %v\n\texpected = %v\n", + i, path, obtainedData, realData) + } + } +} + +func TestReadDir(t *testing.T) { + fs := newFS(filepath.Join(repo, ".git/")) + for i, path := range [...]string{ + "info", + ".", + "", + "objects", + "objects--pack", + } { + expected, err := ioutil.ReadDir(fs.ToReal(path)) + if err != nil { + t.Fatalf("test %d: real ReadDir: %s", err) + } + + obtained, err := fs.ReadDir(path) + if err != nil { + t.Fatalf("test %d: fs.ReadDir unexpected error: %s", i, err) + } + + if !reflect.DeepEqual(obtained, expected) { + t.Fatalf("test %d:\n\tinput = %s\n\tobtained = %v\n\texpected = %v\n", + i, path, obtained, expected) + } + } +} diff --git a/examples/latest/latest.go b/examples/latest/latest.go index 84aaf48..b443abe 100644 --- a/examples/latest/latest.go +++ b/examples/latest/latest.go @@ -14,7 +14,7 @@ func main() { panic(err) } - if err := r.Pull(git.DefaultRemoteName, "refs/heads/master"); err != nil { + if err = r.Pull(git.DefaultRemoteName, "refs/heads/master"); err != nil { panic(err) } diff --git a/file_test.go b/file_test.go index 2f457f4..ad5abb3 100644 --- a/file_test.go +++ b/file_test.go @@ -55,14 +55,14 @@ func (s *SuiteFile) TestIter(c *C) { iter := NewFileIter(r, commit.Tree()) for k := 0; k < len(t.files); k++ { - expected := t.files[k] + exp := t.files[k] file, err := iter.Next() c.Assert(err, IsNil, Commentf("subtest %d, iter %d, err=%v", i, k, err)) c.Assert(file.Mode.String(), Equals, "-rw-r--r--") c.Assert(file.Hash.IsZero(), Equals, false) c.Assert(file.Hash, Equals, file.ID()) - c.Assert(file.Name, Equals, expected.Name, Commentf("subtest %d, iter %d, name=%s, expected=%s", i, k, file.Name, expected.Hash)) - c.Assert(file.Hash.String(), Equals, expected.Hash, Commentf("subtest %d, iter %d, hash=%v, expected=%s", i, k, file.Hash.String(), expected.Hash)) + c.Assert(file.Name, Equals, exp.Name, Commentf("subtest %d, iter %d, name=%s, expected=%s", i, k, file.Name, exp.Hash)) + c.Assert(file.Hash.String(), Equals, exp.Hash, Commentf("subtest %d, iter %d, hash=%v, expected=%s", i, k, file.Hash.String(), exp.Hash)) } _, err = iter.Next() c.Assert(err, Equals, io.EOF) diff --git a/formats/idxfile/decoder.go b/formats/idxfile/decoder.go new file mode 100644 index 0000000..72a9338 --- /dev/null +++ b/formats/idxfile/decoder.go @@ -0,0 +1,160 @@ +package idxfile + +import ( + "bytes" + "encoding/binary" + "errors" + "io" + + "gopkg.in/src-d/go-git.v3/core" +) + +var ( + // ErrUnsupportedVersion is returned by Decode when the idx file version + // is not supported. + ErrUnsupportedVersion = errors.New("Unsuported version") + // ErrMalformedIdxFile is returned by Decode when the idx file is corrupted. + ErrMalformedIdxFile = errors.New("Malformed IDX file") +) + +// A Decoder reads and decodes idx files from an input stream. +type Decoder struct { + io.Reader +} + +// NewDecoder returns a new decoder that reads from r. +func NewDecoder(r io.Reader) *Decoder { + return &Decoder{r} +} + +// Decode reads the whole idx object from its input and stores it in the +// value pointed to by idx. +func (d *Decoder) Decode(idx *Idxfile) error { + if err := validateHeader(d); err != nil { + return err + } + + flow := []func(*Idxfile, io.Reader) error{ + readVersion, + readFanout, + readObjectNames, + readCRC32, + readOffsets, + readChecksums, + } + + for _, f := range flow { + if err := f(idx, d); err != nil { + return err + } + } + + if !idx.isValid() { + return ErrMalformedIdxFile + } + + return nil +} + +func validateHeader(r io.Reader) error { + var h = make([]byte, 4) + if _, err := r.Read(h); err != nil { + return err + } + + if !bytes.Equal(h, idxHeader) { + return ErrMalformedIdxFile + } + + return nil +} + +func readVersion(idx *Idxfile, r io.Reader) error { + v, err := readInt32(r) + if err != nil { + return err + } + + if v > VersionSupported { + return ErrUnsupportedVersion + } + + idx.Version = v + + return nil +} + +func readFanout(idx *Idxfile, r io.Reader) error { + var err error + + for i := 0; i < 255; i++ { + idx.Fanout[i], err = readInt32(r) + if err != nil { + return err + } + } + + idx.ObjectCount, err = readInt32(r) + + return err +} + +func readObjectNames(idx *Idxfile, r io.Reader) error { + c := int(idx.ObjectCount) + for i := 0; i < c; i++ { + var ref core.Hash + if _, err := r.Read(ref[:]); err != nil { + return err + } + + idx.Entries = append(idx.Entries, Entry{Hash: ref}) + } + + return nil +} + +func readCRC32(idx *Idxfile, r io.Reader) error { + c := int(idx.ObjectCount) + for i := 0; i < c; i++ { + if _, err := r.Read(idx.Entries[i].CRC32[:]); err != nil { + return err + } + } + + return nil +} + +func readOffsets(idx *Idxfile, r io.Reader) error { + c := int(idx.ObjectCount) + for i := 0; i < c; i++ { + o, err := readInt32(r) + if err != nil { + return err + } + + idx.Entries[i].Offset = uint64(o) + } + + return nil +} + +func readChecksums(idx *Idxfile, r io.Reader) error { + if _, err := r.Read(idx.PackfileChecksum[:]); err != nil { + return err + } + + if _, err := r.Read(idx.IdxChecksum[:]); err != nil { + return err + } + + return nil +} + +func readInt32(r io.Reader) (uint32, error) { + var v uint32 + if err := binary.Read(r, binary.BigEndian, &v); err != nil { + return 0, err + } + + return v, nil +} diff --git a/formats/idxfile/decoder_test.go b/formats/idxfile/decoder_test.go new file mode 100644 index 0000000..597a002 --- /dev/null +++ b/formats/idxfile/decoder_test.go @@ -0,0 +1,40 @@ +package idxfile + +import ( + "fmt" + "os" + "testing" + + . "gopkg.in/check.v1" +) + +func Test(t *testing.T) { TestingT(t) } + +type IdxfileSuite struct{} + +var _ = Suite(&IdxfileSuite{}) + +func (s *IdxfileSuite) TestDecode(c *C) { + f, err := os.Open("fixtures/git-fixture.idx") + c.Assert(err, IsNil) + + d := NewDecoder(f) + idx := &Idxfile{} + err = d.Decode(idx) + c.Assert(err, IsNil) + + err = f.Close() + c.Assert(err, IsNil) + + c.Assert(int(idx.ObjectCount), Equals, 31) + c.Assert(idx.Entries, HasLen, 31) + c.Assert(idx.Entries[0].Hash.String(), Equals, + "1669dce138d9b841a518c64b10914d88f5e488ea") + c.Assert(idx.Entries[0].Offset, Equals, uint64(615)) + + c.Assert(fmt.Sprintf("%x", idx.IdxChecksum), Equals, + "bba9b7a9895724819225a044c857d391bb9d61d9") + c.Assert(fmt.Sprintf("%x", idx.PackfileChecksum), Equals, + "54bb61360ab2dad1a3e344a8cd3f82b848518cba") + +} diff --git a/formats/idxfile/doc.go b/formats/idxfile/doc.go new file mode 100644 index 0000000..74149ab --- /dev/null +++ b/formats/idxfile/doc.go @@ -0,0 +1,130 @@ +/* +== Original (version 1) pack-*.idx files have the following format: + + - The header consists of 256 4-byte network byte order + integers. N-th entry of this table records the number of + objects in the corresponding pack, the first byte of whose + object name is less than or equal to N. This is called the + 'first-level fan-out' table. + + - The header is followed by sorted 24-byte entries, one entry + per object in the pack. Each entry is: + + 4-byte network byte order integer, recording where the + object is stored in the packfile as the offset from the + beginning. + + 20-byte object name. + + - The file is concluded with a trailer: + + A copy of the 20-byte SHA1 checksum at the end of + corresponding packfile. + + 20-byte SHA1-checksum of all of the above. + +Pack Idx file: + + -- +--------------------------------+ +fanout | fanout[0] = 2 (for example) |-. +table +--------------------------------+ | + | fanout[1] | | + +--------------------------------+ | + | fanout[2] | | + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | + | fanout[255] = total objects |---. + -- +--------------------------------+ | | +main | offset | | | +index | object name 00XXXXXXXXXXXXXXXX | | | +table +--------------------------------+ | | + | offset | | | + | object name 00XXXXXXXXXXXXXXXX | | | + +--------------------------------+<+ | + .-| offset | | + | | object name 01XXXXXXXXXXXXXXXX | | + | +--------------------------------+ | + | | offset | | + | | object name 01XXXXXXXXXXXXXXXX | | + | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | + | | offset | | + | | object name FFXXXXXXXXXXXXXXXX | | + --| +--------------------------------+<--+ +trailer | | packfile checksum | + | +--------------------------------+ + | | idxfile checksum | + | +--------------------------------+ + .-------. + | +Pack file entry: <+ + + packed object header: + 1-byte size extension bit (MSB) + type (next 3 bit) + size0 (lower 4-bit) + n-byte sizeN (as long as MSB is set, each 7-bit) + size0..sizeN form 4+7+7+..+7 bit integer, size0 + is the least significant part, and sizeN is the + most significant part. + packed object data: + If it is not DELTA, then deflated bytes (the size above + is the size before compression). + If it is REF_DELTA, then + 20-byte base object name SHA1 (the size above is the + size of the delta data that follows). + delta data, deflated. + If it is OFS_DELTA, then + n-byte offset (see below) interpreted as a negative + offset from the type-byte of the header of the + ofs-delta entry (the size above is the size of + the delta data that follows). + delta data, deflated. + + offset encoding: + n bytes with MSB set in all but the last one. + The offset is then the number constructed by + concatenating the lower 7 bit of each byte, and + for n >= 2 adding 2^7 + 2^14 + ... + 2^(7*(n-1)) + to the result. + + + +== Version 2 pack-*.idx files support packs larger than 4 GiB, and + have some other reorganizations. They have the format: + + - A 4-byte magic number '\377tOc' which is an unreasonable + fanout[0] value. + + - A 4-byte version number (= 2) + + - A 256-entry fan-out table just like v1. + + - A table of sorted 20-byte SHA1 object names. These are + packed together without offset values to reduce the cache + footprint of the binary search for a specific object name. + + - A table of 4-byte CRC32 values of the packed object data. + This is new in v2 so compressed data can be copied directly + from pack to pack during repacking without undetected + data corruption. + + - A table of 4-byte offset values (in network byte order). + These are usually 31-bit pack file offsets, but large + offsets are encoded as an index into the next table with + the msbit set. + + - A table of 8-byte offset entries (empty for pack files less + than 2 GiB). Pack files are organized with heavily used + objects toward the front, so most object references should + not need to refer to this table. + + - The same trailer as a v1 pack file: + + A copy of the 20-byte SHA1 checksum at the end of + corresponding packfile. + + 20-byte SHA1-checksum of all of the above. + +From: +https://www.kernel.org/pub/software/scm/git/docs/v1.7.5/technical/pack-protocol.txt +*/ +package idxfile diff --git a/formats/idxfile/encoder.go b/formats/idxfile/encoder.go new file mode 100644 index 0000000..f85ff84 --- /dev/null +++ b/formats/idxfile/encoder.go @@ -0,0 +1,124 @@ +package idxfile + +import ( + "crypto/sha1" + "encoding/binary" + "hash" + "io" +) + +// An Encoder writes idx files to an output stream. +type Encoder struct { + io.Writer + hash hash.Hash +} + +// NewEncoder returns a new encoder that writes to w. +func NewEncoder(w io.Writer) *Encoder { + h := sha1.New() + mw := io.MultiWriter(w, h) + return &Encoder{mw, h} +} + +// Encode writes the idx in an idx file format to the stream of the encoder. +func (e *Encoder) Encode(idx *Idxfile) (int, error) { + flow := []func(*Idxfile) (int, error){ + e.encodeHeader, + e.encodeFanout, + e.encodeHashes, + e.encodeCRC32, + e.encodeOffsets, + e.encodeChecksums, + } + + sz := 0 + for _, f := range flow { + i, err := f(idx) + sz += i + + if err != nil { + return sz, err + } + } + + return sz, nil +} + +func (e *Encoder) encodeHeader(idx *Idxfile) (int, error) { + c, err := e.Write(idxHeader) + if err != nil { + return c, err + } + + return c + 4, e.writeInt32(idx.Version) +} + +func (e *Encoder) encodeFanout(idx *Idxfile) (int, error) { + fanout := idx.calculateFanout() + for _, c := range fanout { + if err := e.writeInt32(c); err != nil { + return 0, err + } + } + + return 1024, nil +} + +func (e *Encoder) encodeHashes(idx *Idxfile) (int, error) { + return e.encodeEntryField(idx, true) +} + +func (e *Encoder) encodeCRC32(idx *Idxfile) (int, error) { + return e.encodeEntryField(idx, false) +} + +func (e *Encoder) encodeEntryField(idx *Idxfile, isHash bool) (int, error) { + sz := 0 + for _, ent := range idx.Entries { + var data []byte + if isHash { + data = ent.Hash[:] + } else { + data = ent.CRC32[:] + } + i, err := e.Write(data) + sz += i + + if err != nil { + return sz, err + } + } + + return sz, nil +} + +func (e *Encoder) encodeOffsets(idx *Idxfile) (int, error) { + sz := 0 + for _, ent := range idx.Entries { + if err := e.writeInt32(uint32(ent.Offset)); err != nil { + return sz, err + } + + sz += 4 + + } + + return sz, nil +} + +func (e *Encoder) encodeChecksums(idx *Idxfile) (int, error) { + if _, err := e.Write(idx.PackfileChecksum[:]); err != nil { + return 0, err + } + + copy(idx.IdxChecksum[:], e.hash.Sum(nil)[:20]) + if _, err := e.Write(idx.IdxChecksum[:]); err != nil { + return 0, err + } + + return 40, nil +} + +func (e *Encoder) writeInt32(value uint32) error { + return binary.Write(e, binary.BigEndian, value) +} diff --git a/formats/idxfile/encoder_test.go b/formats/idxfile/encoder_test.go new file mode 100644 index 0000000..bfb9f91 --- /dev/null +++ b/formats/idxfile/encoder_test.go @@ -0,0 +1,47 @@ +package idxfile + +import ( + "bytes" + "io" + "os" + + . "gopkg.in/check.v1" +) + +func (s *IdxfileSuite) TestEncode(c *C) { + for i, path := range [...]string{ + "fixtures/git-fixture.idx", + "../packfile/fixtures/spinnaker-spinnaker.idx", + } { + com := Commentf("subtest %d: path = %s", i, path) + + exp, idx, err := decode(path) + c.Assert(err, IsNil, com) + + obt := new(bytes.Buffer) + e := NewEncoder(obt) + size, err := e.Encode(idx) + c.Assert(err, IsNil, com) + + c.Assert(size, Equals, exp.Len(), com) + c.Assert(obt, DeepEquals, exp, com) + } +} + +func decode(path string) (*bytes.Buffer, *Idxfile, error) { + f, err := os.Open(path) + if err != nil { + return nil, nil, err + } + + cont := new(bytes.Buffer) + tee := io.TeeReader(f, cont) + + d := NewDecoder(tee) + idx := &Idxfile{} + if err = d.Decode(idx); err != nil { + return nil, nil, err + } + + return cont, idx, f.Close() +} diff --git a/formats/idxfile/idxfile.go b/formats/idxfile/idxfile.go new file mode 100644 index 0000000..5f12dad --- /dev/null +++ b/formats/idxfile/idxfile.go @@ -0,0 +1,61 @@ +package idxfile + +import "gopkg.in/src-d/go-git.v3/core" + +const ( + // VersionSupported is the only idx version supported. + VersionSupported = 2 +) + +var ( + idxHeader = []byte{255, 't', 'O', 'c'} +) + +// An Idxfile represents an idx file in memory. +type Idxfile struct { + Version uint32 + Fanout [255]uint32 + ObjectCount uint32 + Entries []Entry + PackfileChecksum [20]byte + IdxChecksum [20]byte +} + +// An Entry represents data about an object in the packfile: its hash, +// offset and CRC32 checksum. +type Entry struct { + Hash core.Hash + CRC32 [4]byte + Offset uint64 +} + +func (idx *Idxfile) isValid() bool { + fanout := idx.calculateFanout() + for k, c := range idx.Fanout { + if fanout[k] != c { + return false + } + } + + return true +} + +func (idx *Idxfile) calculateFanout() [256]uint32 { + fanout := [256]uint32{} + var c uint32 + for _, e := range idx.Entries { + c++ + fanout[e.Hash[0]] = c + } + + var i uint32 + for k, c := range fanout { + if c != 0 { + i = c + } + + fanout[k] = i + } + + return fanout +} diff --git a/formats/objfile/reader_test.go b/formats/objfile/reader_test.go index caebb60..b2c3d0c 100644 --- a/formats/objfile/reader_test.go +++ b/formats/objfile/reader_test.go @@ -19,16 +19,16 @@ var _ = Suite(&SuiteReader{}) func (s *SuiteReader) TestReadObjfile(c *C) { for k, fixture := range objfileFixtures { - comment := fmt.Sprintf("test %d: ", k) + com := fmt.Sprintf("test %d: ", k) hash := core.NewHash(fixture.hash) content, _ := base64.StdEncoding.DecodeString(fixture.content) data, _ := base64.StdEncoding.DecodeString(fixture.data) - testReader(c, bytes.NewReader(data), hash, fixture.t, content, comment) + testReader(c, bytes.NewReader(data), hash, fixture.t, content, com) } } -func testReader(c *C, source io.Reader, hash core.Hash, typ core.ObjectType, content []byte, comment string) { +func testReader(c *C, source io.Reader, hash core.Hash, typ core.ObjectType, content []byte, com string) { r, err := NewReader(source) c.Assert(err, IsNil) c.Assert(r.Type(), Equals, typ) diff --git a/formats/objfile/writer_test.go b/formats/objfile/writer_test.go index 0061f3f..160491c 100644 --- a/formats/objfile/writer_test.go +++ b/formats/objfile/writer_test.go @@ -16,20 +16,20 @@ var _ = Suite(&SuiteWriter{}) func (s *SuiteWriter) TestWriteObjfile(c *C) { for k, fixture := range objfileFixtures { - comment := fmt.Sprintf("test %d: ", k) + com := fmt.Sprintf("test %d: ", k) hash := core.NewHash(fixture.hash) content, _ := base64.StdEncoding.DecodeString(fixture.content) buffer := new(bytes.Buffer) // Write the data out to the buffer - testWriter(c, buffer, hash, fixture.t, content, comment) + testWriter(c, buffer, hash, fixture.t, content, com) // Read the data back in from the buffer to be sure it matches - testReader(c, buffer, hash, fixture.t, content, comment) + testReader(c, buffer, hash, fixture.t, content, com) } } -func testWriter(c *C, dest io.Writer, hash core.Hash, typ core.ObjectType, content []byte, comment string) { +func testWriter(c *C, dest io.Writer, hash core.Hash, typ core.ObjectType, content []byte, com string) { length := int64(len(content)) w, err := NewWriter(dest, typ, length) c.Assert(err, IsNil) diff --git a/formats/packfile/common.go b/formats/packfile/common.go deleted file mode 100644 index b5f8de2..0000000 --- a/formats/packfile/common.go +++ /dev/null @@ -1,63 +0,0 @@ -package packfile - -import ( - "bufio" - "fmt" - "io" -) - -type trackingReader struct { - r io.Reader - position int64 -} - -func NewTrackingReader(r io.Reader) *trackingReader { - return &trackingReader{ - r: bufio.NewReader(r), - } -} - -func (t *trackingReader) Read(p []byte) (n int, err error) { - n, err = t.r.Read(p) - if err != nil { - return 0, err - } - - t.position += int64(n) - return n, err -} - -func (t *trackingReader) ReadByte() (c byte, err error) { - var p [1]byte - n, err := t.r.Read(p[:]) - if err != nil { - return 0, err - } - - if n > 1 { - return 0, fmt.Errorf("read %d bytes, should have read just 1", n) - } - - t.position++ - return p[0], nil -} - -// checkClose is used with defer to close the given io.Closer and check its -// returned error value. If Close returns an error and the given *error -// is not nil, *error is set to the error returned by Close. -// -// checkClose is typically used with named return values like so: -// -// func do(obj *Object) (err error) { -// w, err := obj.Writer() -// if err != nil { -// return nil -// } -// defer checkClose(w, &err) -// // work with w -// } -func checkClose(c io.Closer, err *error) { - if cerr := c.Close(); cerr != nil && *err == nil { - *err = cerr - } -} diff --git a/formats/packfile/decoder.go b/formats/packfile/decoder.go new file mode 100644 index 0000000..e8c5c6a --- /dev/null +++ b/formats/packfile/decoder.go @@ -0,0 +1,116 @@ +package packfile + +import ( + "io" + + "gopkg.in/src-d/go-git.v3/core" +) + +// Format specifies if the packfile uses ref-deltas or ofs-deltas. +type Format int + +// Possible values of the Format type. +const ( + UnknownFormat Format = iota + OFSDeltaFormat + REFDeltaFormat +) + +var ( + // ErrMaxObjectsLimitReached is returned by Decode when the number + // of objects in the packfile is higher than + // Decoder.MaxObjectsLimit. + ErrMaxObjectsLimitReached = NewError("max. objects limit reached") + + // ErrInvalidObject is returned by Decode when an invalid object is + // found in the packfile. + ErrInvalidObject = NewError("invalid git object") + + // ErrPackEntryNotFound is returned by Decode when a reference in + // the packfile references and unknown object. + ErrPackEntryNotFound = NewError("can't find a pack entry") + + // ErrZLib is returned by Decode when there was an error unzipping + // the packfile contents. + ErrZLib = NewError("zlib reading error") +) + +const ( + // DefaultMaxObjectsLimit is the maximum amount of objects the + // decoder will decode before returning ErrMaxObjectsLimitReached. + DefaultMaxObjectsLimit = 1 << 20 +) + +// Decoder reads and decodes packfiles from an input stream. +type Decoder struct { + // MaxObjectsLimit is the limit of objects to be load in the packfile, if + // a packfile excess this number an error is throw, the default value + // is defined by DefaultMaxObjectsLimit, usually the default limit is more + // than enough to work with any repository, with higher values and huge + // repositories you can run out of memory. + MaxObjectsLimit uint32 + + p *Parser + s core.ObjectStorage +} + +// NewDecoder returns a new Decoder that reads from r. +func NewDecoder(r ReadRecaller) *Decoder { + return &Decoder{ + MaxObjectsLimit: DefaultMaxObjectsLimit, + + p: NewParser(r), + } +} + +// Decode reads a packfile and stores it in the value pointed to by s. +func (d *Decoder) Decode(s core.ObjectStorage) error { + d.s = s + + count, err := d.p.ReadHeader() + if err != nil { + return err + } + + if count > d.MaxObjectsLimit { + return ErrMaxObjectsLimitReached.AddDetails("%d", count) + } + + err = d.readObjects(count) + + return err +} + +func (d *Decoder) readObjects(count uint32) error { + // This code has 50-80 µs of overhead per object not counting zlib inflation. + // Together with zlib inflation, it's 400-410 µs for small objects. + // That's 1 sec for ~2450 objects, ~4.20 MB, or ~250 ms per MB, + // of which 12-20 % is _not_ zlib inflation (ie. is our code). + for i := 0; i < int(count); i++ { + start, err := d.p.Offset() + if err != nil { + return err + } + + obj, err := d.p.ReadObject() + if err != nil { + if err == io.EOF { + break + } + + return err + } + + err = d.p.Remember(start, obj) + if err != nil { + return err + } + + _, err = d.s.Set(obj) + if err == io.EOF { + break + } + } + + return nil +} diff --git a/formats/packfile/reader_test.go b/formats/packfile/decoder_test.go index 9ae569d..0c471a2 100644 --- a/formats/packfile/reader_test.go +++ b/formats/packfile/decoder_test.go @@ -26,15 +26,15 @@ var packFileWithEmptyObjects = "UEFDSwAAAAIAAAALnw54nKXMQWoDMQxA0b1PoX2hSLIm44FS func (s *ReaderSuite) TestReadPackfile(c *C) { data, _ := base64.StdEncoding.DecodeString(packFileWithEmptyObjects) - d := bytes.NewReader(data) + f := bytes.NewReader(data) + r := NewStream(f) + d := NewDecoder(r) - r := NewReader(d) - - storage := memory.NewObjectStorage() - _, err := r.Read(storage) + sto := memory.NewObjectStorage() + err := d.Decode(sto) c.Assert(err, IsNil) - AssertObjects(c, storage, []string{ + AssertObjects(c, sto, []string{ "778c85ff95b5514fea0ba4c7b6a029d32e2c3b96", "db4002e880a08bf6cc7217512ad937f1ac8824a2", "551fe11a9ef992763b7e0be4500cf7169f2f8575", @@ -57,18 +57,17 @@ func (s *ReaderSuite) TestReadPackfileREFDelta(c *C) { s.testReadPackfileGitFixture(c, "fixtures/git-fixture.ref-delta", REFDeltaFormat) } -func (s *ReaderSuite) testReadPackfileGitFixture(c *C, file string, f Format) { - d, err := os.Open(file) +func (s *ReaderSuite) testReadPackfileGitFixture(c *C, file string, format Format) { + f, err := os.Open(file) c.Assert(err, IsNil) + r := NewSeekable(f) + d := NewDecoder(r) - r := NewReader(d) - r.Format = f - - storage := memory.NewObjectStorage() - _, err = r.Read(storage) + sto := memory.NewObjectStorage() + err = d.Decode(sto) c.Assert(err, IsNil) - AssertObjects(c, storage, []string{ + AssertObjects(c, sto, []string{ "918c48b83bd081e863dbe1b80f8998f058cd8294", "af2d6a6954d532f8ffb47615169c8fdf9d383a1a", "1669dce138d9b841a518c64b10914d88f5e488ea", @@ -102,10 +101,10 @@ func (s *ReaderSuite) testReadPackfileGitFixture(c *C, file string, f Format) { func AssertObjects(c *C, s *memory.ObjectStorage, expects []string) { c.Assert(len(expects), Equals, len(s.Objects)) - for _, expected := range expects { - obtained, err := s.Get(core.NewHash(expected)) + for _, exp := range expects { + obt, err := s.Get(core.NewHash(exp)) c.Assert(err, IsNil) - c.Assert(obtained.Hash().String(), Equals, expected) + c.Assert(obt.Hash().String(), Equals, exp) } } @@ -139,12 +138,12 @@ func (s *ReaderSuite) BenchmarkGit(c *C) { } } -func (s *ReaderSuite) _TestMemoryOFS(c *C) { +func (s *ReaderSuite) _testMemory(c *C, format Format) { var b, a runtime.MemStats start := time.Now() runtime.ReadMemStats(&b) - p := readFromFile(c, "/tmp/symfony.ofs-delta", OFSDeltaFormat) + p := readFromFile(c, "/tmp/symfony.ofs-delta", format) runtime.ReadMemStats(&a) fmt.Println("OFS--->") @@ -157,34 +156,23 @@ func (s *ReaderSuite) _TestMemoryOFS(c *C) { fmt.Println("time", time.Since(start)) } -func (s *ReaderSuite) _TestMemoryREF(c *C) { - var b, a runtime.MemStats - - start := time.Now() - runtime.ReadMemStats(&b) - p := readFromFile(c, "/tmp/symonfy", REFDeltaFormat) - runtime.ReadMemStats(&a) - - fmt.Println("REF--->") - fmt.Println("Alloc", a.Alloc-b.Alloc, humanize.Bytes(a.Alloc-b.Alloc)) - fmt.Println("TotalAlloc", a.TotalAlloc-b.TotalAlloc, humanize.Bytes(a.TotalAlloc-b.TotalAlloc)) - fmt.Println("HeapAlloc", a.HeapAlloc-b.HeapAlloc, humanize.Bytes(a.HeapAlloc-b.HeapAlloc)) - fmt.Println("HeapSys", a.HeapSys, humanize.Bytes(a.HeapSys-b.HeapSys)) +func (s *ReaderSuite) _TestMemoryOFS(c *C) { + s._testMemory(c, OFSDeltaFormat) +} - fmt.Println("objects", len(p.Objects)) - fmt.Println("time", time.Since(start)) +func (s *ReaderSuite) _TestMemoryREF(c *C) { + s._testMemory(c, REFDeltaFormat) } -func readFromFile(c *C, file string, f Format) *memory.ObjectStorage { - d, err := os.Open(file) +func readFromFile(c *C, file string, format Format) *memory.ObjectStorage { + f, err := os.Open(file) c.Assert(err, IsNil) + r := NewSeekable(f) + d := NewDecoder(r) - r := NewReader(d) - r.Format = f - - storage := memory.NewObjectStorage() - _, err = r.Read(storage) + sto := memory.NewObjectStorage() + err = d.Decode(sto) c.Assert(err, IsNil) - return storage + return sto } diff --git a/formats/packfile/delta.go b/formats/packfile/delta.go index 571ccf8..e0bbb65 100644 --- a/formats/packfile/delta.go +++ b/formats/packfile/delta.go @@ -1,117 +1,148 @@ package packfile -import "io" +// See https://github.com/git/git/blob/49fa3dc76179e04b0833542fa52d0f287a4955ac/delta.h +// https://github.com/git/git/blob/c2c5f6b1e479f2c38e0e01345350620944e3527f/patch-delta.c, +// and https://github.com/tarruda/node-git-core/blob/master/src/js/delta.js +// for details about the delta format. const deltaSizeMin = 4 -func deltaHeaderSize(b []byte) (uint, []byte) { - var size, j uint - var cmd byte - for { - cmd = b[j] - size |= (uint(cmd) & 0x7f) << (j * 7) - j++ - if uint(cmd)&0xb80 == 0 || j == uint(len(b)) { - break - } - } - return size, b[j:] -} - -func patchDelta(src, delta []byte) []byte { +// PatchDelta returns the result of applying the modification deltas in delta to src. +func PatchDelta(src, delta []byte) []byte { if len(delta) < deltaSizeMin { return nil } - size, delta := deltaHeaderSize(delta) - if size != uint(len(src)) { + + srcSz, delta := decodeLEB128(delta) + if srcSz != uint(len(src)) { return nil } - size, delta = deltaHeaderSize(delta) - origSize := size - dest := make([]byte, 0) + targetSz, delta := decodeLEB128(delta) + remainingTargetSz := targetSz - // var offset uint + var dest []byte var cmd byte for { cmd = delta[0] delta = delta[1:] - if (cmd & 0x80) != 0 { - var cp_off, cp_size uint - if (cmd & 0x01) != 0 { - cp_off = uint(delta[0]) - delta = delta[1:] - } - if (cmd & 0x02) != 0 { - cp_off |= uint(delta[0]) << 8 - delta = delta[1:] - } - if (cmd & 0x04) != 0 { - cp_off |= uint(delta[0]) << 16 - delta = delta[1:] - } - if (cmd & 0x08) != 0 { - cp_off |= uint(delta[0]) << 24 - delta = delta[1:] - } - - if (cmd & 0x10) != 0 { - cp_size = uint(delta[0]) - delta = delta[1:] - } - if (cmd & 0x20) != 0 { - cp_size |= uint(delta[0]) << 8 - delta = delta[1:] - } - if (cmd & 0x40) != 0 { - cp_size |= uint(delta[0]) << 16 - delta = delta[1:] - } - if cp_size == 0 { - cp_size = 0x10000 - } - if cp_off+cp_size < cp_off || - cp_off+cp_size > uint(len(src)) || - cp_size > origSize { + if isCopyFromSrc(cmd) { + var offset, sz uint + offset, delta = decodeOffset(cmd, delta) + sz, delta = decodeSize(cmd, delta) + if invalidSize(sz, targetSz) || + invalidOffsetSize(offset, sz, srcSz) { break } - dest = append(dest, src[cp_off:cp_off+cp_size]...) - size -= cp_size - } else if cmd != 0 { - if uint(cmd) > origSize { + dest = append(dest, src[offset:offset+sz]...) + remainingTargetSz -= sz + } else if isCopyFromDelta(cmd) { + sz := uint(cmd) // cmd is the size itself + if invalidSize(sz, targetSz) { break } - dest = append(dest, delta[0:uint(cmd)]...) - size -= uint(cmd) - delta = delta[uint(cmd):] + dest = append(dest, delta[0:sz]...) + remainingTargetSz -= sz + delta = delta[sz:] } else { return nil } - if size <= 0 { + + if remainingTargetSz <= 0 { break } } + return dest } -func decodeOffset(src io.ByteReader, steps int64) (int64, error) { - b, err := src.ReadByte() - if err != nil { - return 0, err - } +// Decodes a number encoded as an unsigned LEB128 at the start of some +// binary data and returns the decoded number and the rest of the +// stream. +// +// This must be called twice on the delta data buffer, first to get the +// expected source buffer size, and again to get the target buffer size. +func decodeLEB128(input []byte) (uint, []byte) { + var num, sz uint + var b byte + for { + b = input[sz] + num |= (uint(b) & payload) << (sz * 7) // concats 7 bits chunks + sz++ - var offset = int64(b & 0x7f) - for (b & 0x80) != 0 { - offset++ // WHY? - b, err = src.ReadByte() - if err != nil { - return 0, err + if uint(b)&continuation == 0 || sz == uint(len(input)) { + break } + } + + return num, input[sz:] +} + +const ( + payload = 0x7f // 0111 1111 + continuation = 0x80 // 1000 0000 +) - offset = (offset << 7) + int64(b&0x7f) +func isCopyFromSrc(cmd byte) bool { + return (cmd & 0x80) != 0 +} + +func isCopyFromDelta(cmd byte) bool { + return (cmd&0x80) == 0 && cmd != 0 +} + +func decodeOffset(cmd byte, delta []byte) (uint, []byte) { + var offset uint + if (cmd & 0x01) != 0 { + offset = uint(delta[0]) + delta = delta[1:] + } + if (cmd & 0x02) != 0 { + offset |= uint(delta[0]) << 8 + delta = delta[1:] + } + if (cmd & 0x04) != 0 { + offset |= uint(delta[0]) << 16 + delta = delta[1:] + } + if (cmd & 0x08) != 0 { + offset |= uint(delta[0]) << 24 + delta = delta[1:] + } + + return offset, delta +} + +func decodeSize(cmd byte, delta []byte) (uint, []byte) { + var sz uint + if (cmd & 0x10) != 0 { + sz = uint(delta[0]) + delta = delta[1:] + } + if (cmd & 0x20) != 0 { + sz |= uint(delta[0]) << 8 + delta = delta[1:] + } + if (cmd & 0x40) != 0 { + sz |= uint(delta[0]) << 16 + delta = delta[1:] + } + if sz == 0 { + sz = 0x10000 } - // offset needs to be aware of the bytes we read for `o.typ` and `o.size` - offset += steps - return -offset, nil + return sz, delta +} + +func invalidSize(sz, targetSz uint) bool { + return sz > targetSz +} + +func invalidOffsetSize(offset, sz, srcSz uint) bool { + return sumOverflows(offset, sz) || + offset+sz > srcSz +} + +func sumOverflows(a, b uint) bool { + return a+b < a } diff --git a/formats/packfile/doc.go b/formats/packfile/doc.go index cb3f542..c79c180 100644 --- a/formats/packfile/doc.go +++ b/formats/packfile/doc.go @@ -1,165 +1,168 @@ -package packfile +// Package packfile documentation: +/* + +GIT pack format +=============== + +== pack-*.pack files have the following format: + + - A header appears at the beginning and consists of the following: + + 4-byte signature: + The signature is: {'P', 'A', 'C', 'K'} + + 4-byte version number (network byte order): + GIT currently accepts version number 2 or 3 but + generates version 2 only. + + 4-byte number of objects contained in the pack (network byte order) + + Observation: we cannot have more than 4G versions ;-) and + more than 4G objects in a pack. + + - The header is followed by number of object entries, each of + which looks like this: + + (undeltified representation) + n-byte type and length (3-bit type, (n-1)*7+4-bit length) + compressed data + + (deltified representation) + n-byte type and length (3-bit type, (n-1)*7+4-bit length) + 20-byte base object name + compressed delta data + + Observation: length of each object is encoded in a variable + length format and is not constrained to 32-bit or anything. + + - The trailer records 20-byte SHA1 checksum of all of the above. + +== Original (version 1) pack-*.idx files have the following format: + + - The header consists of 256 4-byte network byte order + integers. N-th entry of this table records the number of + objects in the corresponding pack, the first byte of whose + object name is less than or equal to N. This is called the + 'first-level fan-out' table. + + - The header is followed by sorted 24-byte entries, one entry + per object in the pack. Each entry is: + + 4-byte network byte order integer, recording where the + object is stored in the packfile as the offset from the + beginning. + + 20-byte object name. + + - The file is concluded with a trailer: + + A copy of the 20-byte SHA1 checksum at the end of + corresponding packfile. -// GIT pack format -// =============== -// -// == pack-*.pack files have the following format: -// -// - A header appears at the beginning and consists of the following: -// -// 4-byte signature: -// The signature is: {'P', 'A', 'C', 'K'} -// -// 4-byte version number (network byte order): -// GIT currently accepts version number 2 or 3 but -// generates version 2 only. -// -// 4-byte number of objects contained in the pack (network byte order) -// -// Observation: we cannot have more than 4G versions ;-) and -// more than 4G objects in a pack. -// -// - The header is followed by number of object entries, each of -// which looks like this: -// -// (undeltified representation) -// n-byte type and length (3-bit type, (n-1)*7+4-bit length) -// compressed data -// -// (deltified representation) -// n-byte type and length (3-bit type, (n-1)*7+4-bit length) -// 20-byte base object name -// compressed delta data -// -// Observation: length of each object is encoded in a variable -// length format and is not constrained to 32-bit or anything. -// -// - The trailer records 20-byte SHA1 checksum of all of the above. -// -// == Original (version 1) pack-*.idx files have the following format: -// -// - The header consists of 256 4-byte network byte order -// integers. N-th entry of this table records the number of -// objects in the corresponding pack, the first byte of whose -// object name is less than or equal to N. This is called the -// 'first-level fan-out' table. -// -// - The header is followed by sorted 24-byte entries, one entry -// per object in the pack. Each entry is: -// -// 4-byte network byte order integer, recording where the -// object is stored in the packfile as the offset from the -// beginning. -// -// 20-byte object name. -// -// - The file is concluded with a trailer: -// -// A copy of the 20-byte SHA1 checksum at the end of -// corresponding packfile. -// -// 20-byte SHA1-checksum of all of the above. -// -// Pack Idx file: -// -// -- +--------------------------------+ -// fanout | fanout[0] = 2 (for example) |-. -// table +--------------------------------+ | -// | fanout[1] | | -// +--------------------------------+ | -// | fanout[2] | | -// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | -// | fanout[255] = total objects |---. -// -- +--------------------------------+ | | -// main | offset | | | -// index | object name 00XXXXXXXXXXXXXXXX | | | -// table +--------------------------------+ | | -// | offset | | | -// | object name 00XXXXXXXXXXXXXXXX | | | -// +--------------------------------+<+ | -// .-| offset | | -// | | object name 01XXXXXXXXXXXXXXXX | | -// | +--------------------------------+ | -// | | offset | | -// | | object name 01XXXXXXXXXXXXXXXX | | -// | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | -// | | offset | | -// | | object name FFXXXXXXXXXXXXXXXX | | -// --| +--------------------------------+<--+ -// trailer | | packfile checksum | -// | +--------------------------------+ -// | | idxfile checksum | -// | +--------------------------------+ -// .-------. -// | -// Pack file entry: <+ -// -// packed object header: -// 1-byte size extension bit (MSB) -// type (next 3 bit) -// size0 (lower 4-bit) -// n-byte sizeN (as long as MSB is set, each 7-bit) -// size0..sizeN form 4+7+7+..+7 bit integer, size0 -// is the least significant part, and sizeN is the -// most significant part. -// packed object data: -// If it is not DELTA, then deflated bytes (the size above -// is the size before compression). -// If it is REF_DELTA, then -// 20-byte base object name SHA1 (the size above is the -// size of the delta data that follows). -// delta data, deflated. -// If it is OFS_DELTA, then -// n-byte offset (see below) interpreted as a negative -// offset from the type-byte of the header of the -// ofs-delta entry (the size above is the size of -// the delta data that follows). -// delta data, deflated. -// -// offset encoding: -// n bytes with MSB set in all but the last one. -// The offset is then the number constructed by -// concatenating the lower 7 bit of each byte, and -// for n >= 2 adding 2^7 + 2^14 + ... + 2^(7*(n-1)) -// to the result. -// -// -// -// == Version 2 pack-*.idx files support packs larger than 4 GiB, and -// have some other reorganizations. They have the format: -// -// - A 4-byte magic number '\377tOc' which is an unreasonable -// fanout[0] value. -// -// - A 4-byte version number (= 2) -// -// - A 256-entry fan-out table just like v1. -// -// - A table of sorted 20-byte SHA1 object names. These are -// packed together without offset values to reduce the cache -// footprint of the binary search for a specific object name. -// -// - A table of 4-byte CRC32 values of the packed object data. -// This is new in v2 so compressed data can be copied directly -// from pack to pack during repacking without undetected -// data corruption. -// -// - A table of 4-byte offset values (in network byte order). -// These are usually 31-bit pack file offsets, but large -// offsets are encoded as an index into the next table with -// the msbit set. -// -// - A table of 8-byte offset entries (empty for pack files less -// than 2 GiB). Pack files are organized with heavily used -// objects toward the front, so most object references should -// not need to refer to this table. -// -// - The same trailer as a v1 pack file: -// -// A copy of the 20-byte SHA1 checksum at the end of -// corresponding packfile. -// -// 20-byte SHA1-checksum of all of the above. -// -// From: -// https://www.kernel.org/pub/software/scm/git/docs/v1.7.5/technical/pack-protocol.txt + 20-byte SHA1-checksum of all of the above. + +Pack Idx file: + + -- +--------------------------------+ +fanout | fanout[0] = 2 (for example) |-. +table +--------------------------------+ | + | fanout[1] | | + +--------------------------------+ | + | fanout[2] | | + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | + | fanout[255] = total objects |---. + -- +--------------------------------+ | | +main | offset | | | +index | object name 00XXXXXXXXXXXXXXXX | | | +table +--------------------------------+ | | + | offset | | | + | object name 00XXXXXXXXXXXXXXXX | | | + +--------------------------------+<+ | + .-| offset | | + | | object name 01XXXXXXXXXXXXXXXX | | + | +--------------------------------+ | + | | offset | | + | | object name 01XXXXXXXXXXXXXXXX | | + | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | + | | offset | | + | | object name FFXXXXXXXXXXXXXXXX | | + --| +--------------------------------+<--+ +trailer | | packfile checksum | + | +--------------------------------+ + | | idxfile checksum | + | +--------------------------------+ + .-------. + | +Pack file entry: <+ + + packed object header: + 1-byte size extension bit (MSB) + type (next 3 bit) + size0 (lower 4-bit) + n-byte sizeN (as long as MSB is set, each 7-bit) + size0..sizeN form 4+7+7+..+7 bit integer, size0 + is the least significant part, and sizeN is the + most significant part. + packed object data: + If it is not DELTA, then deflated bytes (the size above + is the size before compression). + If it is REF_DELTA, then + 20-byte base object name SHA1 (the size above is the + size of the delta data that follows). + delta data, deflated. + If it is OFS_DELTA, then + n-byte offset (see below) interpreted as a negative + offset from the type-byte of the header of the + ofs-delta entry (the size above is the size of + the delta data that follows). + delta data, deflated. + + offset encoding: + n bytes with MSB set in all but the last one. + The offset is then the number constructed by + concatenating the lower 7 bit of each byte, and + for n >= 2 adding 2^7 + 2^14 + ... + 2^(7*(n-1)) + to the result. + + + +== Version 2 pack-*.idx files support packs larger than 4 GiB, and + have some other reorganizations. They have the format: + + - A 4-byte magic number '\377tOc' which is an unreasonable + fanout[0] value. + + - A 4-byte version number (= 2) + + - A 256-entry fan-out table just like v1. + + - A table of sorted 20-byte SHA1 object names. These are + packed together without offset values to reduce the cache + footprint of the binary search for a specific object name. + + - A table of 4-byte CRC32 values of the packed object data. + This is new in v2 so compressed data can be copied directly + from pack to pack during repacking without undetected + data corruption. + + - A table of 4-byte offset values (in network byte order). + These are usually 31-bit pack file offsets, but large + offsets are encoded as an index into the next table with + the msbit set. + + - A table of 8-byte offset entries (empty for pack files less + than 2 GiB). Pack files are organized with heavily used + objects toward the front, so most object references should + not need to refer to this table. + + - The same trailer as a v1 pack file: + + A copy of the 20-byte SHA1 checksum at the end of + corresponding packfile. + + 20-byte SHA1-checksum of all of the above. + +From: +https://www.kernel.org/pub/software/scm/git/docs/v1.7.5/technical/pack-protocol.txt +*/ +package packfile diff --git a/formats/packfile/error.go b/formats/packfile/error.go new file mode 100644 index 0000000..c0b9163 --- /dev/null +++ b/formats/packfile/error.go @@ -0,0 +1,30 @@ +package packfile + +import "fmt" + +// Error specifies errors returned during packfile parsing. +type Error struct { + reason, details string +} + +// NewError returns a new error. +func NewError(reason string) *Error { + return &Error{reason: reason} +} + +// Error returns a text representation of the error. +func (e *Error) Error() string { + if e.details == "" { + return e.reason + } + + return fmt.Sprintf("%s: %s", e.reason, e.details) +} + +// AddDetails adds details to an error, with additional text. +func (e *Error) AddDetails(format string, args ...interface{}) *Error { + return &Error{ + reason: e.reason, + details: fmt.Sprintf(format, args...), + } +} diff --git a/formats/packfile/parser.go b/formats/packfile/parser.go new file mode 100644 index 0000000..d3463bd --- /dev/null +++ b/formats/packfile/parser.go @@ -0,0 +1,353 @@ +package packfile + +import ( + "bytes" + "compress/zlib" + "encoding/binary" + "fmt" + "io" + + "gopkg.in/src-d/go-git.v3/core" + "gopkg.in/src-d/go-git.v3/storage/memory" +) + +var ( + // ErrEmptyPackfile is returned by ReadHeader when no data is found in the packfile + ErrEmptyPackfile = NewError("empty packfile") + // ErrBadSignature is returned by ReadHeader when the signature in the packfile is incorrect. + ErrBadSignature = NewError("malformed pack file signature") + // ErrUnsupportedVersion is returned by ReadHeader when the packfile version is + // different than VersionSupported. + ErrUnsupportedVersion = NewError("unsupported packfile version") +) + +const ( + // VersionSupported is the packfile version supported by this parser. + VersionSupported = 2 +) + +// A Parser is a collection of functions to read and process data form a packfile. +// Values from this type are not zero-value safe. See the NewParser function bellow. +type Parser struct { + ReadRecaller +} + +// NewParser returns a new Parser that reads from the packfile represented by r. +func NewParser(r ReadRecaller) *Parser { + return &Parser{ReadRecaller: r} +} + +// ReadInt32 reads 4 bytes and returns them as a Big Endian int32. +func (p Parser) readInt32() (uint32, error) { + var v uint32 + if err := binary.Read(p, binary.BigEndian, &v); err != nil { + return 0, err + } + + return v, nil +} + +// ReadSignature reads an returns the signature field in the packfile. +func (p *Parser) ReadSignature() ([]byte, error) { + var sig = make([]byte, 4) + if _, err := io.ReadFull(p, sig); err != nil { + return []byte{}, err + } + + return sig, nil +} + +// IsValidSignature returns if sig is a valid packfile signature. +func (p Parser) IsValidSignature(sig []byte) bool { + return bytes.Equal(sig, []byte{'P', 'A', 'C', 'K'}) +} + +// ReadVersion reads and returns the version field of a packfile. +func (p *Parser) ReadVersion() (uint32, error) { + return p.readInt32() +} + +// IsSupportedVersion returns whether version v is supported by the parser. +// The current supported version is VersionSupported, defined above. +func (p *Parser) IsSupportedVersion(v uint32) bool { + return v == VersionSupported +} + +// ReadCount reads and returns the count of objects field of a packfile. +func (p *Parser) ReadCount() (uint32, error) { + return p.readInt32() +} + +// ReadHeader reads the whole packfile header (signature, version and +// object count). It returns the object count and performs checks on the +// validity of the signature and the version fields. +func (p Parser) ReadHeader() (uint32, error) { + sig, err := p.ReadSignature() + if err != nil { + if err == io.EOF { + return 0, ErrEmptyPackfile + } + return 0, err + } + + if !p.IsValidSignature(sig) { + return 0, ErrBadSignature + } + + ver, err := p.ReadVersion() + if err != nil { + return 0, err + } + + if !p.IsSupportedVersion(ver) { + return 0, ErrUnsupportedVersion.AddDetails("%d", ver) + } + + count, err := p.ReadCount() + if err != nil { + return 0, err + } + + return count, nil +} + +// ReadObjectTypeAndLength reads and returns the object type and the +// length field from an object entry in a packfile. +func (p Parser) ReadObjectTypeAndLength() (core.ObjectType, int64, error) { + t, c, err := p.readType() + if err != nil { + return t, 0, err + } + + l, err := p.readLength(c) + + return t, l, err +} + +func (p Parser) readType() (core.ObjectType, byte, error) { + var c byte + var err error + if c, err = p.ReadByte(); err != nil { + return core.ObjectType(0), 0, err + } + typ := parseType(c) + + return typ, c, nil +} + +var ( + maskContinue = uint8(128) // 1000 0000 + maskType = uint8(112) // 0111 0000 + maskFirstLength = uint8(15) // 0000 1111 + firstLengthBits = uint8(4) // the first byte has 4 bits to store the length + maskLength = uint8(127) // 0111 1111 + lengthBits = uint8(7) // subsequent bytes has 7 bits to store the length +) + +func parseType(b byte) core.ObjectType { + return core.ObjectType((b & maskType) >> firstLengthBits) +} + +// the length is codified in the last 4 bits of the first byte and in +// the last 7 bits of subsequent bytes. Last byte has a 0 MSB. +func (p Parser) readLength(first byte) (int64, error) { + length := int64(first & maskFirstLength) + + c := first + shift := firstLengthBits + var err error + for moreBytesInLength(c) { + if c, err = p.ReadByte(); err != nil { + return 0, err + } + + length += int64(c&maskLength) << shift + shift += lengthBits + } + + return length, nil +} + +func moreBytesInLength(c byte) bool { + return c&maskContinue > 0 +} + +// ReadObject reads and returns a git object from an object entry in the packfile. +// Non-deltified and deltified objects are supported. +func (p Parser) ReadObject() (core.Object, error) { + start, err := p.Offset() + if err != nil { + return nil, err + } + + var typ core.ObjectType + typ, _, err = p.ReadObjectTypeAndLength() + if err != nil { + return nil, err + } + + var cont []byte + switch typ { + case core.CommitObject, core.TreeObject, core.BlobObject, core.TagObject: + cont, err = p.ReadNonDeltaObjectContent() + case core.REFDeltaObject: + cont, typ, err = p.ReadREFDeltaObjectContent() + case core.OFSDeltaObject: + cont, typ, err = p.ReadOFSDeltaObjectContent(start) + default: + err = ErrInvalidObject.AddDetails("tag %q", typ) + } + if err != nil { + return nil, err + } + + return memory.NewObject(typ, int64(len(cont)), cont), nil +} + +// ReadNonDeltaObjectContent reads and returns a non-deltified object +// from it zlib stream in an object entry in the packfile. +func (p Parser) ReadNonDeltaObjectContent() ([]byte, error) { + return p.readZip() +} + +func (p Parser) readZip() ([]byte, error) { + buf := bytes.NewBuffer(nil) + err := p.inflate(buf) + + return buf.Bytes(), err +} + +func (p Parser) inflate(w io.Writer) (err error) { + zr, err := zlib.NewReader(p) + if err != nil { + if err != zlib.ErrHeader { + return fmt.Errorf("zlib reading error: %s", err) + } + } + + defer func() { + closeErr := zr.Close() + if err == nil { + err = closeErr + } + }() + + _, err = io.Copy(w, zr) + + return err +} + +// ReadREFDeltaObjectContent reads and returns an object specified by a +// REF-Delta entry in the packfile, form the hash onwards. +func (p Parser) ReadREFDeltaObjectContent() ([]byte, core.ObjectType, error) { + refHash, err := p.ReadHash() + if err != nil { + return nil, core.ObjectType(0), err + } + + refObj, err := p.RecallByHash(refHash) + if err != nil { + return nil, core.ObjectType(0), err + } + + content, err := p.ReadSolveDelta(refObj.Content()) + if err != nil { + return nil, refObj.Type(), err + } + + return content, refObj.Type(), nil +} + +// ReadHash reads a hash. +func (p Parser) ReadHash() (core.Hash, error) { + var h core.Hash + if _, err := io.ReadFull(p, h[:]); err != nil { + return core.ZeroHash, err + } + + return h, nil +} + +// ReadSolveDelta reads and returns the base patched with the contents +// of a zlib compressed diff data in the delta portion of an object +// entry in the packfile. +func (p Parser) ReadSolveDelta(base []byte) ([]byte, error) { + diff, err := p.readZip() + if err != nil { + return nil, err + } + + return PatchDelta(base, diff), nil +} + +// ReadOFSDeltaObjectContent reads an returns an object specified by an +// OFS-delta entry in the packfile from it negative offset onwards. The +// start parameter is the offset of this particular object entry (the +// current offset minus the already processed type and length). +func (p Parser) ReadOFSDeltaObjectContent(start int64) ( + []byte, core.ObjectType, error) { + + jump, err := p.ReadNegativeOffset() + if err != nil { + return nil, core.ObjectType(0), err + } + + ref, err := p.RecallByOffset(start + jump) + if err != nil { + return nil, core.ObjectType(0), err + } + + content, err := p.ReadSolveDelta(ref.Content()) + if err != nil { + return nil, ref.Type(), err + } + + return content, ref.Type(), nil +} + +// ReadNegativeOffset reads and returns an offset from a OFS DELTA +// object entry in a packfile. OFS DELTA offsets are specified in Git +// VLQ special format: +// +// Ordinary VLQ has some redundancies, example: the number 358 can be +// encoded as the 2-octet VLQ 0x8166 or the 3-octet VLQ 0x808166 or the +// 4-octet VLQ 0x80808166 and so forth. +// +// To avoid these redundancies, the VLQ format used in Git removes this +// prepending redundancy and extends the representable range of shorter +// VLQs by adding an offset to VLQs of 2 or more octets in such a way +// that the lowest possible value for such an (N+1)-octet VLQ becomes +// exactly one more than the maximum possible value for an N-octet VLQ. +// In particular, since a 1-octet VLQ can store a maximum value of 127, +// the minimum 2-octet VLQ (0x8000) is assigned the value 128 instead of +// 0. Conversely, the maximum value of such a 2-octet VLQ (0xff7f) is +// 16511 instead of just 16383. Similarly, the minimum 3-octet VLQ +// (0x808000) has a value of 16512 instead of zero, which means +// that the maximum 3-octet VLQ (0xffff7f) is 2113663 instead of +// just 2097151. And so forth. +// +// This is how the offset is saved in C: +// +// dheader[pos] = ofs & 127; +// while (ofs >>= 7) +// dheader[--pos] = 128 | (--ofs & 127); +// +func (p Parser) ReadNegativeOffset() (int64, error) { + var c byte + var err error + + if c, err = p.ReadByte(); err != nil { + return 0, err + } + + var offset = int64(c & maskLength) + for moreBytesInLength(c) { + offset++ + if c, err = p.ReadByte(); err != nil { + return 0, err + } + offset = (offset << lengthBits) + int64(c&maskLength) + } + + return -offset, nil +} diff --git a/formats/packfile/parser_test.go b/formats/packfile/parser_test.go new file mode 100644 index 0000000..12d5f0d --- /dev/null +++ b/formats/packfile/parser_test.go @@ -0,0 +1,412 @@ +package packfile + +import ( + "bytes" + "io" + "io/ioutil" + "os" + + . "gopkg.in/check.v1" + "gopkg.in/src-d/go-git.v3/core" + "gopkg.in/src-d/go-git.v3/storage/memory" +) + +const ( + sigOffset = 0 + verOffset = 4 + countOffset = 8 +) + +type ParserSuite struct { + fixtures map[string]*fix +} + +type fix struct { + path string + parser *Parser + seekable io.Seeker +} + +func newFix(path string) (*fix, error) { + fix := new(fix) + fix.path = path + + f, err := os.Open(path) + if err != nil { + return nil, err + } + + data, err := ioutil.ReadAll(f) + if err != nil { + return nil, err + } + + if err = f.Close(); err != nil { + return nil, err + } + + seekable := NewSeekable(bytes.NewReader(data)) + fix.seekable = seekable + fix.parser = NewParser(seekable) + + return fix, nil +} + +func (f *fix) seek(o int64) error { + _, err := f.seekable.Seek(o, os.SEEK_SET) + return err +} + +var _ = Suite(&ParserSuite{}) + +func (s *ParserSuite) SetUpSuite(c *C) { + s.fixtures = make(map[string]*fix) + for _, fixData := range []struct { + id string + path string + }{ + {"ofs-deltas", "fixtures/alcortesm-binary-relations.pack"}, + {"ref-deltas", "fixtures/git-fixture.ref-delta"}, + } { + fix, err := newFix(fixData.path) + c.Assert(err, IsNil, + Commentf("setting up fixture id %s: %s", fixData.id, err)) + + _, ok := s.fixtures[fixData.id] + c.Assert(ok, Equals, false, + Commentf("duplicated fixture id: %s", fixData.id)) + + s.fixtures[fixData.id] = fix + } +} + +func (s *ParserSuite) TestSignature(c *C) { + for id, fix := range s.fixtures { + com := Commentf("fixture id = %s", id) + err := fix.seek(sigOffset) + c.Assert(err, IsNil, com) + p := fix.parser + + sig, err := p.ReadSignature() + c.Assert(err, IsNil, com) + c.Assert(p.IsValidSignature(sig), Equals, true, com) + } +} + +func (s *ParserSuite) TestVersion(c *C) { + for i, test := range [...]struct { + fixID string + expected uint32 + }{ + { + fixID: "ofs-deltas", + expected: uint32(2), + }, { + fixID: "ref-deltas", + expected: uint32(2), + }, + } { + com := Commentf("test %d) fixture id = %s", i, test.fixID) + fix, ok := s.fixtures[test.fixID] + c.Assert(ok, Equals, true, com) + + err := fix.seek(verOffset) + c.Assert(err, IsNil, com) + p := fix.parser + + v, err := p.ReadVersion() + c.Assert(err, IsNil, com) + c.Assert(v, Equals, test.expected, com) + c.Assert(p.IsSupportedVersion(v), Equals, true, com) + } +} + +func (s *ParserSuite) TestCount(c *C) { + for i, test := range [...]struct { + fixID string + expected uint32 + }{ + { + fixID: "ofs-deltas", + expected: uint32(0x50), + }, { + fixID: "ref-deltas", + expected: uint32(0x1c), + }, + } { + com := Commentf("test %d) fixture id = %s", i, test.fixID) + fix, ok := s.fixtures[test.fixID] + c.Assert(ok, Equals, true, com) + + err := fix.seek(countOffset) + c.Assert(err, IsNil, com) + p := fix.parser + + count, err := p.ReadCount() + c.Assert(err, IsNil, com) + c.Assert(count, Equals, test.expected, com) + } +} + +func (s *ParserSuite) TestReadObjectTypeAndLength(c *C) { + for i, test := range [...]struct { + fixID string + offset int64 + expType core.ObjectType + expLength int64 + }{ + { + fixID: "ofs-deltas", + offset: 12, + expType: core.CommitObject, + expLength: 342, + }, { + fixID: "ofs-deltas", + offset: 1212, + expType: core.OFSDeltaObject, + expLength: 104, + }, { + fixID: "ofs-deltas", + offset: 3193, + expType: core.TreeObject, + expLength: 226, + }, { + fixID: "ofs-deltas", + offset: 3639, + expType: core.BlobObject, + expLength: 90, + }, { + fixID: "ofs-deltas", + offset: 4504, + expType: core.BlobObject, + expLength: 7107, + }, { + fixID: "ref-deltas", + offset: 84849, + expType: core.REFDeltaObject, + expLength: 6, + }, { + fixID: "ref-deltas", + offset: 85070, + expType: core.REFDeltaObject, + expLength: 8, + }, + } { + com := Commentf("test %d) fixture id = %s", i, test.fixID) + fix, ok := s.fixtures[test.fixID] + c.Assert(ok, Equals, true, com) + + err := fix.seek(test.offset) + c.Assert(err, IsNil, com) + p := fix.parser + + typ, length, err := p.ReadObjectTypeAndLength() + c.Assert(err, IsNil, com) + c.Assert(typ, Equals, test.expType, com) + c.Assert(length, Equals, test.expLength, com) + } +} + +func (s *ParserSuite) TestReadNonDeltaObjectContent(c *C) { + for i, test := range [...]struct { + fixID string + offset int64 + expected []byte + }{ + { + fixID: "ofs-deltas", + offset: 12, + expected: []byte("tree 87c87d16e815a43e4e574dd8edd72c5450ac3a8e\nparent a87d72684d1cf68099ce6e9f68689e25e645a14c\nauthor Gorka Guardiola <Gorka Guardiola Múzquiz> 1450265632 +0100\ncommitter Gorka Guardiola <Gorka Guardiola Múzquiz> 1450265632 +0100\n\nChanged example to use dot.\nI did not remove the original files outside of the\ntex, I leave that to alcortes.\n"), + }, { + fixID: "ofs-deltas", + offset: 1610, + expected: []byte("tree 4b4f0d9a07109ef0b8a3051138cc20cdb47fa513\nparent b373f85fa2594d7dcd9989f4a5858a81647fb8ea\nauthor Alberto Cortés <alberto@sourced.tech> 1448017995 +0100\ncommitter Alberto Cortés <alberto@sourced.tech> 1448018112 +0100\n\nMove generated images to it own dir (img/)\n\nFixes #1.\n"), + }, { + fixID: "ofs-deltas", + offset: 10566, + expected: []byte("40000 map-slice\x00\x00\xce\xfb\x8ew\xf7\xa8\xc6\x1b\x99\xdd$\x91\xffH\xa3\xb0\xb1fy40000 simple-arrays\x00\x9a7\x81\xb7\xfd\x9d(Q\xe2\xa4H\x8c\x03^٬\x90Z\xecy"), + }, + } { + com := Commentf("test %d) fixture id = %s", i, test.fixID) + fix, ok := s.fixtures[test.fixID] + c.Assert(ok, Equals, true, com) + + err := fix.seek(test.offset) + c.Assert(err, IsNil, com) + p := fix.parser + + _, _, err = p.ReadObjectTypeAndLength() + c.Assert(err, IsNil, com) + + cont, err := p.ReadNonDeltaObjectContent() + c.Assert(err, IsNil, com) + c.Assert(cont, DeepEquals, test.expected, com) + } +} + +func (s *ParserSuite) TestReadOFSDeltaObjectContent(c *C) { + for i, test := range [...]struct { + fixID string + offset int64 + expOffset int64 + expType core.ObjectType + expContent []byte + }{ + { + fixID: "ofs-deltas", + offset: 1212, + expOffset: -212, + expType: core.CommitObject, + expContent: []byte("tree c4573589ce78ac63769c20742b9a970f6e274a38\nparent 4571a24948494ebe1cb3dc18ca5a9286e79705ae\nauthor Alberto Cortés <alberto@sourced.tech> 1448139640 +0100\ncommitter Alberto Cortés <alberto@sourced.tech> 1448139640 +0100\n\nUpdate reference to binrels module\n"), + }, { + fixID: "ofs-deltas", + offset: 3514, + expOffset: -102, + expType: core.TreeObject, + expContent: []byte("100644 .gitignore\x00\u007fA\x90[Mw\xabJ\x9a-3O\xcd\x0f\xb5\xdbn\x8e!\x83100644 .gitmodules\x00\xd4`\xa8>\x15\xcfd\x05\x81B7_\xc4\v\x04\xa7\xa9A\x85\n100644 Makefile\x00-ҭ\x8c\x14\xdef\x12\xed\x15\x816y\xa6UK\xad\x993\v100644 binary-relations.tex\x00\x802\x05@\x11'^ \xf5<\xf7\xfd\x81%3\xd1o\xa9_$40000 graphs\x00\xdehu\x16\xc6\x0e\\H\x8e\xe9\xa1JIXE\xbaڽg\xc540000 imgs-gen\x00\xeb\"\xddhzg\xa3\x1f\xc8j\xc5\xfc豢\xe9\x96\xce\xce^40000 src\x00\x895\x11t\xff\x86\xa7\xea\xa6\xc0v%\x11E\x10f,ݒ\x1a"), + }, { + fixID: "ofs-deltas", + offset: 9806, + expOffset: -6613, + expType: core.TreeObject, + expContent: []byte("100644 .gitignore\x00\u007fA\x90[Mw\xabJ\x9a-3O\xcd\x0f\xb5\xdbn\x8e!\x83100644 .gitmodules\x00\xd4`\xa8>\x15\xcfd\x05\x81B7_\xc4\v\x04\xa7\xa9A\x85\n100644 Makefile\x00-ҭ\x8c\x14\xdef\x12\xed\x15\x816y\xa6UK\xad\x993\v100644 binary-relations.tex\x00I\x13~\xb8کEU\x9f\x99#\xc4E.\x9d>\uef1e\xad40000 graphs\x00\xb9\x00\xf34\xde\xff\xce@+\xbd\xf8 9\xb8=\xc1\xb9\x00\x84]40000 imgs-gen\x00\xeb\"\xddhzg\xa3\x1f\xc8j\xc5\xfc豢\xe9\x96\xce\xce^40000 src\x00\x895\x11t\xff\x86\xa7\xea\xa6\xc0v%\x11E\x10f,ݒ\x1a"), + }, + } { + com := Commentf("test %d) fixture id = %s", i, test.fixID) + fix, ok := s.fixtures[test.fixID] + c.Assert(ok, Equals, true, com) + + err := fix.seek(test.offset) + c.Assert(err, IsNil, com) + p := fix.parser + + _, _, err = p.ReadObjectTypeAndLength() + c.Assert(err, IsNil, com) + + beforeJumpSize, err := p.Offset() + c.Assert(err, IsNil, com) + + jump, err := p.ReadNegativeOffset() + c.Assert(err, IsNil, com) + c.Assert(jump, Equals, test.expOffset, com) + + err = fix.seek(beforeJumpSize) + c.Assert(err, IsNil, com) + + cont, typ, err := p.ReadOFSDeltaObjectContent(test.offset) + c.Assert(err, IsNil, com) + c.Assert(typ, Equals, test.expType, com) + c.Assert(cont, DeepEquals, test.expContent, com) + } +} + +func (s *ParserSuite) TestReadREFDeltaObjectContent(c *C) { + for i, test := range [...]struct { + fixID string + offset int64 + deps map[int64]core.Object + expHash core.Hash + expType core.ObjectType + expContent []byte + }{ + { + fixID: "ref-deltas", + offset: 84849, + deps: map[int64]core.Object{ + 83607: newObject(core.TreeObject, []byte("100644 .gitignore\x002\x85\x8a\xad<8>\xd1\xff\n\x0f\x9b\xdf#\x1dT\xa0\f\x9e\x88100644 CHANGELOG\x00\xd3\xffS\xe0VJ\x9f\x87\xd8\xe8Kn(\xe5\x06\x0eQp\b\xaa100644 LICENSE\x00\xc1\x92\xbdj$\xea\x1a\xb0\x1dxhnA|\x8b\xdc|=\x19\u007f100644 binary.jpg\x00\xd5\xc0\xf4\xab\x81\x18\x97\xca\xdf\x03\xae\xc3X\xae`\xd2\x1f\x91\xc5\r40000 go\x00\xa3\x97q\xa7e\x1f\x97\xfa\xf5\xc7.\b\"M\x85\u007f\xc3Q3\xdb40000 json\x00Z\x87~j\x90j'C\xadnEٜ\x17\x93d*\xaf\x8e\xda40000 php\x00Xj\xf5gл^w\x1eI\xbd\xd9CO^\x0f\xb7m%\xfa40000 vendor\x00\xcfJ\xa3\xb3\x89t\xfb}\x81\xf3g\xc0\x83\x0f}x\xd6Z\xb8k")), + }, + expHash: core.NewHash("a8d315b2b1c615d43042c3a62402b8a54288cf5c"), + expType: core.TreeObject, + expContent: []byte("100644 .gitignore\x002\x85\x8a\xad<8>\xd1\xff\n\x0f\x9b\xdf#\x1dT\xa0\f\x9e\x88100644 CHANGELOG\x00\xd3\xffS\xe0VJ\x9f\x87\xd8\xe8Kn(\xe5\x06\x0eQp\b\xaa100644 LICENSE\x00\xc1\x92\xbdj$\xea\x1a\xb0\x1dxhnA|\x8b\xdc|=\x19\u007f100644 binary.jpg\x00\xd5\xc0\xf4\xab\x81\x18\x97\xca\xdf\x03\xae\xc3X\xae`\xd2\x1f\x91\xc5\r40000 go\x00\xa3\x97q\xa7e\x1f\x97\xfa\xf5\xc7.\b\"M\x85\u007f\xc3Q3\xdb40000 json\x00Z\x87~j\x90j'C\xadnEٜ\x17\x93d*\xaf\x8e\xda40000 php\x00Xj\xf5gл^w\x1eI\xbd\xd9CO^\x0f\xb7m%\xfa"), + }, { + fixID: "ref-deltas", + offset: 85070, + deps: map[int64]core.Object{ + 84922: newObject(core.TreeObject, []byte("100644 .gitignore\x002\x85\x8a\xad<8>\xd1\xff\n\x0f\x9b\xdf#\x1dT\xa0\f\x9e\x88100644 CHANGELOG\x00\xd3\xffS\xe0VJ\x9f\x87\xd8\xe8Kn(\xe5\x06\x0eQp\b\xaa100644 LICENSE\x00\xc1\x92\xbdj$\xea\x1a\xb0\x1dxhnA|\x8b\xdc|=\x19\u007f100644 binary.jpg\x00\xd5\xc0\xf4\xab\x81\x18\x97\xca\xdf\x03\xae\xc3X\xae`\xd2\x1f\x91\xc5\r")), + 84849: newObject(core.TreeObject, []byte("100644 .gitignore\x002\x85\x8a\xad<8>\xd1\xff\n\x0f\x9b\xdf#\x1dT\xa0\f\x9e\x88100644 CHANGELOG\x00\xd3\xffS\xe0VJ\x9f\x87\xd8\xe8Kn(\xe5\x06\x0eQp\b\xaa100644 LICENSE\x00\xc1\x92\xbdj$\xea\x1a\xb0\x1dxhnA|\x8b\xdc|=\x19\u007f100644 binary.jpg\x00\xd5\xc0\xf4\xab\x81\x18\x97\xca\xdf\x03\xae\xc3X\xae`\xd2\x1f\x91\xc5\r40000 go\x00\xa3\x97q\xa7e\x1f\x97\xfa\xf5\xc7.\b\"M\x85\u007f\xc3Q3\xdb40000 json\x00Z\x87~j\x90j'C\xadnEٜ\x17\x93d*\xaf\x8e\xda40000 php\x00Xj\xf5gл^w\x1eI\xbd\xd9CO^\x0f\xb7m%\xfa")), + 83607: newObject(core.TreeObject, []byte("100644 .gitignore\x002\x85\x8a\xad<8>\xd1\xff\n\x0f\x9b\xdf#\x1dT\xa0\f\x9e\x88100644 CHANGELOG\x00\xd3\xffS\xe0VJ\x9f\x87\xd8\xe8Kn(\xe5\x06\x0eQp\b\xaa100644 LICENSE\x00\xc1\x92\xbdj$\xea\x1a\xb0\x1dxhnA|\x8b\xdc|=\x19\u007f100644 binary.jpg\x00\xd5\xc0\xf4\xab\x81\x18\x97\xca\xdf\x03\xae\xc3X\xae`\xd2\x1f\x91\xc5\r40000 go\x00\xa3\x97q\xa7e\x1f\x97\xfa\xf5\xc7.\b\"M\x85\u007f\xc3Q3\xdb40000 json\x00Z\x87~j\x90j'C\xadnEٜ\x17\x93d*\xaf\x8e\xda40000 php\x00Xj\xf5gл^w\x1eI\xbd\xd9CO^\x0f\xb7m%\xfa40000 vendor\x00\xcfJ\xa3\xb3\x89t\xfb}\x81\xf3g\xc0\x83\x0f}x\xd6Z\xb8k")), + }, + expHash: core.NewHash("eba74343e2f15d62adedfd8c883ee0262b5c8021"), + expType: core.TreeObject, + expContent: []byte("100644 .gitignore\x002\x85\x8a\xad<8>\xd1\xff\n\x0f\x9b\xdf#\x1dT\xa0\f\x9e\x88100644 LICENSE\x00\xc1\x92\xbdj$\xea\x1a\xb0\x1dxhnA|\x8b\xdc|=\x19\u007f100644 binary.jpg\x00\xd5\xc0\xf4\xab\x81\x18\x97\xca\xdf\x03\xae\xc3X\xae`\xd2\x1f\x91\xc5\r"), + }, + } { + com := Commentf("test %d) fixture id = %s", i, test.fixID) + fix, ok := s.fixtures[test.fixID] + c.Assert(ok, Equals, true, com) + + err := fix.seek(test.offset) + c.Assert(err, IsNil, com) + p := fix.parser + for k, v := range test.deps { + err = p.Remember(k, v) + c.Assert(err, IsNil, com) + } + + _, _, err = p.ReadObjectTypeAndLength() + c.Assert(err, IsNil, com) + + beforeHash, err := p.Offset() + c.Assert(err, IsNil, com) + + hash, err := p.ReadHash() + c.Assert(err, IsNil, com) + c.Assert(hash, Equals, test.expHash, com) + + err = fix.seek(beforeHash) + c.Assert(err, IsNil, com) + + cont, typ, err := p.ReadREFDeltaObjectContent() + c.Assert(err, IsNil, com) + c.Assert(typ, Equals, test.expType, com) + c.Assert(cont, DeepEquals, test.expContent, com) + + p.ForgetAll() + } +} + +func newObject(t core.ObjectType, c []byte) *memory.Object { + return memory.NewObject(t, int64(len(c)), c) +} + +func (s *ParserSuite) TestReadHeaderBadSignatureError(c *C) { + data := []byte{ + 0x50, 0x42, 0x43, 0x4b, 0x00, 0x00, 0x00, 0x02, + 0x00, 0x00, 0x00, 0x50, + } + p := NewParser(NewSeekable(bytes.NewReader(data))) + + _, err := p.ReadHeader() + c.Assert(err, ErrorMatches, ErrBadSignature.Error()) +} + +func (s *ParserSuite) TestReadHeaderEmptyPackfileError(c *C) { + data := []byte{} + p := NewParser(NewSeekable(bytes.NewReader(data))) + + _, err := p.ReadHeader() + c.Assert(err, ErrorMatches, ErrEmptyPackfile.Error()) +} + +func (s *ParserSuite) TestReadHeaderUnsupportedVersionError(c *C) { + data := []byte{ + 0x50, 0x41, 0x43, 0x4b, 0x00, 0x00, 0x00, 0x01, + 0x00, 0x00, 0x00, 0x50, + } + p := NewParser(NewSeekable(bytes.NewReader(data))) + + _, err := p.ReadHeader() + c.Assert(err, ErrorMatches, ErrUnsupportedVersion.Error()+".*") +} + +func (s *ParserSuite) TestReadHeader(c *C) { + data := []byte{ + 0x50, 0x41, 0x43, 0x4b, 0x00, 0x00, 0x00, 0x02, + 0x00, 0x00, 0x00, 0x50, + } + p := NewParser(NewSeekable(bytes.NewReader(data))) + + count, err := p.ReadHeader() + c.Assert(err, IsNil) + c.Assert(count, Equals, uint32(0x50)) +} diff --git a/formats/packfile/read_recaller.go b/formats/packfile/read_recaller.go new file mode 100644 index 0000000..92ab1b2 --- /dev/null +++ b/formats/packfile/read_recaller.go @@ -0,0 +1,39 @@ +package packfile + +import "gopkg.in/src-d/go-git.v3/core" + +var ( + // ErrDuplicatedObject is returned by Remember if an object appears several + // times in a packfile. + ErrDuplicatedObject = NewError("duplicated object") + // ErrCannotRecall is returned by RecallByOffset or RecallByHash if the object + // to recall cannot be returned. + ErrCannotRecall = NewError("cannot recall object") +) + +// The ReadRecaller interface has all the functions needed by a packfile +// Parser to operate. We provide two very different implementations: +// Seekable and Stream. +type ReadRecaller interface { + // Read reads up to len(p) bytes into p. + Read(p []byte) (int, error) + // ReadByte is needed because of these: + // - https://github.com/golang/go/commit/7ba54d45732219af86bde9a5b73c145db82b70c6 + // - https://groups.google.com/forum/#!topic/golang-nuts/fWTRdHpt0QI + // - https://gowalker.org/compress/zlib#NewReader + ReadByte() (byte, error) + // Offset returns the number of bytes parsed so far from the + // packfile. + Offset() (int64, error) + // Remember ask the ReadRecaller to remember the offset and hash for + // an object, so you can later call RecallByOffset and RecallByHash. + Remember(int64, core.Object) error + // ForgetAll forgets all previously remembered objects. + ForgetAll() + // RecallByOffset returns the previously processed object found at a + // given offset. + RecallByOffset(int64) (core.Object, error) + // RecallByHash returns the previously processed object with the + // given hash. + RecallByHash(core.Hash) (core.Object, error) +} diff --git a/formats/packfile/read_recaller_impl_test.go b/formats/packfile/read_recaller_impl_test.go new file mode 100644 index 0000000..438439d --- /dev/null +++ b/formats/packfile/read_recaller_impl_test.go @@ -0,0 +1,296 @@ +package packfile + +import ( + "bytes" + "fmt" + "io/ioutil" + "os" + + "gopkg.in/src-d/go-git.v3/core" + "gopkg.in/src-d/go-git.v3/storage/memory" + + . "gopkg.in/check.v1" +) + +type ReadRecallerImplSuite struct{} + +var _ = Suite(&ReadRecallerImplSuite{}) + +type implFn func([]byte) ReadRecaller + +func newStream(data []byte) ReadRecaller { + buf := bytes.NewBuffer(data) + return NewStream(buf) +} + +func newSeekable(data []byte) ReadRecaller { + buf := bytes.NewReader(data) + return NewSeekable(buf) +} + +func (s *ReadRecallerImplSuite) TestRead(c *C) { + for _, impl := range []struct { + id string + newFn implFn + }{ + {id: "stream", newFn: newStream}, + {id: "seekable", newFn: newSeekable}, + } { + com := Commentf("implementation %s", impl.id) + data := []byte{0, 1, 2, 3, 4, 5, 7, 8, 9, 10} + sr := impl.newFn(data) + all := make([]byte, 0, len(data)) + + for len(all) < len(data) { + tmp := make([]byte, 3) + nr, err := sr.Read(tmp) + c.Assert(err, IsNil, com) + all = append(all, tmp[:nr]...) + } + c.Assert(data, DeepEquals, all, com) + } +} + +func (s *ReadRecallerImplSuite) TestReadbyte(c *C) { + for _, impl := range []struct { + id string + newFn implFn + }{ + {id: "stream", newFn: newStream}, + {id: "seekable", newFn: newSeekable}, + } { + com := Commentf("implementation %s", impl.id) + data := []byte{0, 1, 2, 3, 4, 5, 7, 8, 9, 10} + sr := impl.newFn(data) + all := make([]byte, 0, len(data)) + + for len(all) < len(data) { + b, err := sr.ReadByte() + c.Assert(err, IsNil, com) + all = append(all, b) + } + c.Assert(data, DeepEquals, all, com) + } +} + +func (s *ReadRecallerImplSuite) TestOffsetWithRead(c *C) { + for _, impl := range []struct { + id string + newFn implFn + }{ + {id: "stream", newFn: newStream}, + {id: "seekable", newFn: newSeekable}, + } { + com := Commentf("implementation %s", impl.id) + data := []byte{0, 1, 2, 3, 4, 5, 7, 8, 9, 10} + sr := impl.newFn(data) + all := make([]byte, 0, len(data)) + + for len(all) < len(data) { + tmp := make([]byte, 3) + nr, err := sr.Read(tmp) + c.Assert(err, IsNil, com) + all = append(all, tmp[:nr]...) + + off, err := sr.Offset() + c.Assert(err, IsNil, com) + c.Assert(off, Equals, int64(len(all)), com) + } + } +} + +func (s *ReadRecallerImplSuite) TestOffsetWithReadByte(c *C) { + for _, impl := range []struct { + id string + newFn implFn + }{ + {id: "stream", newFn: newStream}, + {id: "seekable", newFn: newSeekable}, + } { + com := Commentf("implementation %s", impl.id) + data := []byte{0, 1, 2, 3, 4, 5, 7, 8, 9, 10} + sr := impl.newFn(data) + all := make([]byte, 0, len(data)) + + for len(all) < len(data) { + b, err := sr.ReadByte() + c.Assert(err, IsNil, com) + all = append(all, b) + + off, err := sr.Offset() + c.Assert(err, IsNil, com) + c.Assert(off, Equals, int64(len(all)), com) + } + } +} + +func (s *ReadRecallerImplSuite) TestRememberRecall(c *C) { + packfile := "fixtures/spinnaker-spinnaker.pack" + f, err := os.Open(packfile) + c.Assert(err, IsNil) + defer func() { + err = f.Close() + c.Assert(err, IsNil) + }() + + data, err := ioutil.ReadAll(f) + c.Assert(err, IsNil) + + for _, impl := range []struct { + id string + newFn implFn + }{ + {id: "stream", newFn: newStream}, + {id: "seekable", newFn: newSeekable}, + } { + sr := impl.newFn(data) + for i, test := range [...]struct { + off int64 + obj core.Object + err string // error regexp + ignore string // ignore this test for this implementation + }{ + { + off: 12, + obj: newObj(core.CommitObject, []byte("tree 44a1cdf21c791867c51caad8f1b77e6baee6f462\nparent 87fe6e7c6b1b89519fe3a03a8961c5aa14d4cc68\nparent 9244ee648182b91a63d8cc4cbe4b9ac2a27c0492\nauthor Matt Duftler <duftler@google.com> 1448290941 -0500\ncommitter Matt Duftler <duftler@google.com> 1448290941 -0500\n\nMerge pull request #615 from ewiseblatt/create_dev\n\nPreserve original credentials of spinnaker-local.yml when transforming it.")), + }, { + off: 3037, + obj: newObj(core.TagObject, []byte("object e0005f50e22140def60260960b21667f1fdfff80\ntype commit\ntag v0.10.0\ntagger cfieber <cfieber@netflix.com> 1447687536 -0800\n\nRelease of 0.10.0\n\n- e0005f50e22140def60260960b21667f1fdfff80: Merge pull request #553 from ewiseblatt/rendezvous\n- e1a2b26b784179e6903a7ae967c037c721899eba: Wait for cassandra before starting spinnaker\n- c756e09461d071e98b8660818cf42d90c90f2854: Merge pull request #552 from duftler/google-c2d-tweaks\n- 0777fadf4ca6f458d7071de414f9bd5417911037: Fix incorrect config prop names: s/SPINNAKER_GOOGLE_PROJECT_DEFAULT_REGION/SPINNAKER_GOOGLE_DEFAULT_REGION s/SPINNAKER_GOOGLE_PROJECT_DEFAULT_ZONE/SPINNAKER_GOOGLE_DEFAULT_ZONE Hardcode profile name in generated ~/.aws/credentials to [default]. Restart all of spinnaker after updating cassandra and reconfiguring spinnaker, instead of just restarting clouddriver.\n- d8d031c1ac45801074418c43424a6f2c0dff642c: Merge pull request #551 from kenzanmedia/fixGroup\n- 626d23075f9e92aad19015f2964c95d45f41fa3a: Put in correct block for public image. Delineate cloud provider.\n")), + }, { + off: 157625, + obj: newObj(core.BlobObject, []byte(".gradle\nbuild/\n*.iml\n.idea\n*.pyc\n*~\n#*\nconfig/spinnaker-local.yml\n.DS_Store\npacker/ami_table.md\npacker/ami_table.json\npacker/example_output.txt")), + }, { + off: 1234, + obj: newObj(core.BlobObject, []byte(".gradle\nbuild/\n*.iml\n.idea\n*.pyc\n*~\n#*\nconfig/spinnaker-local.yml\n.DS_Store\npacker/ami_table.md\npacker/ami_table.json\npacker/example_output.txt")), + err: "duplicated object: with hash .*", + }, { + off: 3037, + obj: newObj(core.BlobObject, []byte("")), + err: "duplicated object: with offset 3037", + ignore: "seekable", + // seekable can not check if the offset has already been added + // for performance reasons. + }, + } { + if test.ignore == impl.id { + continue + } + com := Commentf("subtest %d) implementation %s", i, impl.id) + + err := sr.Remember(test.off, test.obj) + if test.err != "" { + c.Assert(err, ErrorMatches, test.err, com) + continue + } + c.Assert(err, IsNil, com) + + result, err := sr.RecallByHash(test.obj.Hash()) + c.Assert(err, IsNil, com) + c.Assert(result, DeepEquals, test.obj, com) + + result, err = sr.RecallByOffset(test.off) + c.Assert(err, IsNil, com) + c.Assert(result, DeepEquals, test.obj, com) + } + } +} + +func newObj(typ core.ObjectType, cont []byte) core.Object { + return memory.NewObject(typ, int64(len(cont)), cont) +} + +func (s *ReadRecallerImplSuite) TestRecallByHashErrors(c *C) { + for _, impl := range []struct { + id string + newFn implFn + }{ + {id: "stream", newFn: newStream}, + {id: "seekable", newFn: newSeekable}, + } { + com := Commentf("implementation %s", impl.id) + sr := impl.newFn([]byte{}) + obj := newObj(core.CommitObject, []byte{}) + + _, err := sr.RecallByHash(obj.Hash()) + c.Assert(err, ErrorMatches, ErrCannotRecall.Error()+".*", com) + + err = rememberSomeObjects(sr) + c.Assert(err, IsNil) + + _, err = sr.RecallByHash(obj.Hash()) + c.Assert(err, ErrorMatches, ErrCannotRecall.Error()+".*", com) + } +} + +func (s *ReadRecallerImplSuite) TestRecallByOffsetErrors(c *C) { + for _, impl := range []struct { + id string + newFn implFn + }{ + {id: "stream", newFn: newStream}, + // seekalbe allways recall every object in the packfile + } { + com := Commentf("implementation %s", impl.id) + sr := impl.newFn([]byte{}) + + _, err := sr.RecallByOffset(15) + c.Assert(err, ErrorMatches, ErrCannotRecall.Error()+".*", com) + + err = rememberSomeObjects(sr) + c.Assert(err, IsNil) + + _, err = sr.RecallByOffset(15) + c.Assert(err, ErrorMatches, ErrCannotRecall.Error()+".*", com) + } +} + +func rememberSomeObjects(sr ReadRecaller) error { + for i, init := range [...]struct { + off int64 + obj core.Object + }{ + {off: 0, obj: newObj(core.CommitObject, []byte{'a'})}, // 93114cce67ec23976d15199514399203f69cc676 + {off: 10, obj: newObj(core.CommitObject, []byte{'b'})}, // 2bb767097e479f668f0ebdabe88df11337bd8f19 + {off: 20, obj: newObj(core.CommitObject, []byte{'c'})}, // 2f8096005677370e6446541a50e074299d43d468 + } { + err := sr.Remember(init.off, init.obj) + if err != nil { + return fmt.Errorf("cannot ask StreamReader to Remember item %d", i) + } + } + + return nil +} + +func (s *ReadRecallerImplSuite) TestForgetAll(c *C) { + for _, impl := range []struct { + id string + newFn implFn + }{ + {id: "stream", newFn: newStream}, + {id: "seekable", newFn: newSeekable}, + } { + com := Commentf("implementation %s", impl.id) + sr := impl.newFn([]byte{}) + + err := rememberSomeObjects(sr) + c.Assert(err, IsNil) + + sr.ForgetAll() + + if impl.id != "seekable" { // for efficiency, seekable always finds objects by offset + _, err = sr.RecallByOffset(0) + c.Assert(err, ErrorMatches, ErrCannotRecall.Error()+".*", com) + _, err = sr.RecallByOffset(10) + c.Assert(err, ErrorMatches, ErrCannotRecall.Error()+".*", com) + _, err = sr.RecallByOffset(20) + c.Assert(err, ErrorMatches, ErrCannotRecall.Error()+".*", com) + } + _, err = sr.RecallByHash(core.NewHash("93114cce67ec23976d15199514399203f69cc676")) + c.Assert(err, ErrorMatches, ErrCannotRecall.Error()+".*", com) + _, err = sr.RecallByHash(core.NewHash("2bb767097e479f668f0ebdabe88df11337bd8f19")) + c.Assert(err, ErrorMatches, ErrCannotRecall.Error()+".*", com) + _, err = sr.RecallByHash(core.NewHash("2f8096005677370e6446541a50e074299d43d468")) + c.Assert(err, ErrorMatches, ErrCannotRecall.Error()+".*", com) + } +} diff --git a/formats/packfile/reader.go b/formats/packfile/reader.go deleted file mode 100644 index 3f7081b..0000000 --- a/formats/packfile/reader.go +++ /dev/null @@ -1,338 +0,0 @@ -package packfile - -import ( - "bytes" - "encoding/binary" - "fmt" - "io" - "io/ioutil" - - "gopkg.in/src-d/go-git.v3/core" - - "github.com/klauspost/compress/zlib" -) - -type Format int - -var ( - EmptyRepositoryErr = newError("empty repository") - UnsupportedVersionErr = newError("unsupported packfile version") - MaxObjectsLimitReachedErr = newError("max. objects limit reached") - MalformedPackfileErr = newError("malformed pack file, does not start with 'PACK'") - InvalidObjectErr = newError("invalid git object") - PatchingErr = newError("patching error") - PackEntryNotFoundErr = newError("can't find a pack entry") - ErrObjectNotFound = newError("can't find a object") - ZLibErr = newError("zlib reading error") -) - -const ( - DefaultMaxObjectsLimit = 1 << 20 - - VersionSupported = 2 - UnknownFormat Format = 0 - OFSDeltaFormat Format = 1 - REFDeltaFormat Format = 2 -) - -// Reader reads a packfile from a binary string splitting it on objects -type Reader struct { - // MaxObjectsLimit is the limit of objects to be load in the packfile, if - // a packfile excess this number an error is throw, the default value - // is defined by DefaultMaxObjectsLimit, usually the default limit is more - // than enough to work with any repository, working extremely big repositories - // where the number of object is bigger the memory can be exhausted. - MaxObjectsLimit uint32 - - // Format specifies if we are using ref-delta's or ofs-delta's, choosing the - // correct format the memory usage is optimized - // https://github.com/git/git/blob/8d530c4d64ffcc853889f7b385f554d53db375ed/Documentation/technical/protocol-capabilities.txt#L154 - Format Format - - r *trackingReader - s core.ObjectStorage - offsets map[int64]core.Hash -} - -// NewReader returns a new Reader that reads from a io.Reader -func NewReader(r io.Reader) *Reader { - return &Reader{ - MaxObjectsLimit: DefaultMaxObjectsLimit, - - r: NewTrackingReader(r), - offsets: make(map[int64]core.Hash, 0), - } -} - -// Read reads the objects and stores it at the ObjectStorage -func (r *Reader) Read(s core.ObjectStorage) (int64, error) { - r.s = s - if err := r.validateHeader(); err != nil { - if err == io.EOF { - return -1, EmptyRepositoryErr - } - - return -1, err - } - - version, err := r.readInt32() - if err != nil { - return -1, err - } - - if version > VersionSupported { - return -1, UnsupportedVersionErr - } - - count, err := r.readInt32() - if err != nil { - return -1, err - } - - if count > r.MaxObjectsLimit { - return -1, MaxObjectsLimitReachedErr - } - - return r.r.position, r.readObjects(count) -} - -func (r *Reader) validateHeader() error { - var header = make([]byte, 4) - if _, err := io.ReadFull(r.r, header); err != nil { - return err - } - - if !bytes.Equal(header, []byte{'P', 'A', 'C', 'K'}) { - return MalformedPackfileErr - } - - return nil -} - -func (r *Reader) readInt32() (uint32, error) { - var value uint32 - if err := binary.Read(r.r, binary.BigEndian, &value); err != nil { - return 0, err - } - - return value, nil -} - -func (r *Reader) readObjects(count uint32) error { - // This code has 50-80 µs of overhead per object not counting zlib inflation. - // Together with zlib inflation, it's 400-410 µs for small objects. - // That's 1 sec for ~2450 objects, ~4.20 MB, or ~250 ms per MB, - // of which 12-20 % is _not_ zlib inflation (ie. is our code). - for i := 0; i < int(count); i++ { - start := r.r.position - obj, err := r.newObject() - if err != nil && err != io.EOF { - return err - } - - if r.Format == UnknownFormat || r.Format == OFSDeltaFormat { - r.offsets[start] = obj.Hash() - } - - r.s.Set(obj) - if err == io.EOF { - break - } - } - - return nil -} - -func (r *Reader) newObject() (core.Object, error) { - raw, err := r.s.New() - if err != nil { - return nil, err - } - var steps int64 - - var buf [1]byte - if _, err := r.r.Read(buf[:]); err != nil { - return nil, err - } - - typ := core.ObjectType((buf[0] >> 4) & 7) - size := int64(buf[0] & 15) - steps++ // byte we just read to get `o.typ` and `o.size` - - var shift uint = 4 - for buf[0]&0x80 == 0x80 { - if _, err := r.r.Read(buf[:]); err != nil { - return nil, err - } - - size += int64(buf[0]&0x7f) << shift - steps++ // byte we just read to update `o.size` - shift += 7 - } - - raw.SetType(typ) - raw.SetSize(size) - - switch raw.Type() { - case core.REFDeltaObject: - err = r.readREFDelta(raw) - case core.OFSDeltaObject: - err = r.readOFSDelta(raw, steps) - case core.CommitObject, core.TreeObject, core.BlobObject, core.TagObject: - err = r.readObject(raw) - default: - err = InvalidObjectErr.n("tag %q", raw.Type) - } - - return raw, err -} - -func (r *Reader) readREFDelta(raw core.Object) (err error) { - var ref core.Hash - if _, err := io.ReadFull(r.r, ref[:]); err != nil { - return err - } - - buf := bytes.NewBuffer(nil) - if err := r.inflate(buf); err != nil { - return err - } - - referenced, err := r.s.Get(ref) - if err != nil { - if err == core.ErrObjectNotFound { - return ErrObjectNotFound.n("%s", ref) - } - return err - } - - reader, err := referenced.Reader() - if err != nil { - return err - } - defer checkClose(reader, &err) - - d, err := ioutil.ReadAll(reader) - if err != nil { - return err - } - - patched := patchDelta(d, buf.Bytes()) - if patched == nil { - return PatchingErr.n("hash %q", ref) - } - - raw.SetType(referenced.Type()) - raw.SetSize(int64(len(patched))) - - writer, err := raw.Writer() - if err != nil { - return err - } - defer checkClose(writer, &err) - - writer.Write(patched) - - return nil -} - -func (r *Reader) readOFSDelta(raw core.Object, steps int64) (err error) { - start := r.r.position - offset, err := decodeOffset(r.r, steps) - if err != nil { - return err - } - - buf := bytes.NewBuffer(nil) - if err = r.inflate(buf); err != nil { - return err - } - - ref, ok := r.offsets[start+offset] - if !ok { - return PackEntryNotFoundErr.n("offset %d", start+offset) - } - - referenced, err := r.s.Get(ref) - if err != nil { - return err - } - - reader, err := referenced.Reader() - if err != nil { - return err - } - defer checkClose(reader, &err) - - d, err := ioutil.ReadAll(reader) - if err != nil { - return err - } - - patched := patchDelta(d, buf.Bytes()) - if patched == nil { - return PatchingErr.n("hash %q", ref) - } - - raw.SetType(referenced.Type()) - raw.SetSize(int64(len(patched))) - - writer, err := raw.Writer() - if err != nil { - return err - } - defer checkClose(writer, &err) - - writer.Write(patched) - - return nil -} - -func (r *Reader) readObject(raw core.Object) (err error) { - writer, err := raw.Writer() - if err != nil { - return err - } - defer checkClose(writer, &err) - - return r.inflate(writer) -} - -func (r *Reader) inflate(w io.Writer) error { - zr, err := zlib.NewReader(r.r) - if err != nil { - if err == zlib.ErrHeader { - return zlib.ErrHeader - } - - return ZLibErr.n("%s", err) - } - - defer zr.Close() - - _, err = io.Copy(w, zr) - return err -} - -type ReaderError struct { - reason, additional string -} - -func newError(reason string) *ReaderError { - return &ReaderError{reason: reason} -} - -func (e *ReaderError) Error() string { - if e.additional == "" { - return e.reason - } - - return fmt.Sprintf("%s: %s", e.reason, e.additional) -} - -func (e *ReaderError) n(format string, args ...interface{}) *ReaderError { - return &ReaderError{ - reason: e.reason, - additional: fmt.Sprintf(format, args...), - } -} diff --git a/formats/packfile/seekable.go b/formats/packfile/seekable.go new file mode 100644 index 0000000..ea1c501 --- /dev/null +++ b/formats/packfile/seekable.go @@ -0,0 +1,108 @@ +package packfile + +import ( + "io" + "os" + + "gopkg.in/src-d/go-git.v3/core" +) + +// Seekable implements ReadRecaller for the io.ReadSeeker of a packfile. +// Remembering does not actually stores any reference to the remembered +// objects; the object offset is remebered instead and the packfile is +// read again everytime a recall operation is requested. This saves +// memory buy can be very slow if the associated io.ReadSeeker is slow +// (like a hard disk). +type Seekable struct { + io.ReadSeeker + HashToOffset map[core.Hash]int64 +} + +// NewSeekable returns a new Seekable that reads form r. +func NewSeekable(r io.ReadSeeker) *Seekable { + return &Seekable{ + r, + make(map[core.Hash]int64), + } +} + +// Read reads up to len(p) bytes into p. +func (r *Seekable) Read(p []byte) (int, error) { + return r.ReadSeeker.Read(p) +} + +// ReadByte reads a byte. +func (r *Seekable) ReadByte() (byte, error) { + var p [1]byte + _, err := r.ReadSeeker.Read(p[:]) + if err != nil { + return 0, err + } + + return p[0], nil +} + +// Offset returns the offset for the next Read or ReadByte. +func (r *Seekable) Offset() (int64, error) { + return r.Seek(0, os.SEEK_CUR) +} + +// Remember stores the offset of the object and its hash, but not the +// object itself. This implementation does not check for already stored +// offsets, as it is too expensive to build this information from an +// index every time a get operation is performed on the SeekableReadRecaller. +func (r *Seekable) Remember(o int64, obj core.Object) error { + h := obj.Hash() + if _, ok := r.HashToOffset[h]; ok { + return ErrDuplicatedObject.AddDetails("with hash %s", h) + } + + r.HashToOffset[h] = o + + return nil +} + +// ForgetAll forgets all previously remembered objects. For efficiency +// reasons RecallByOffset always find objects, even if they have been +// forgetted or were never remembered. +func (r *Seekable) ForgetAll() { + r.HashToOffset = make(map[core.Hash]int64) +} + +// RecallByHash returns the object for a given hash by looking for it again in +// the io.ReadeSeerker. +func (r *Seekable) RecallByHash(h core.Hash) (core.Object, error) { + o, ok := r.HashToOffset[h] + if !ok { + return nil, ErrCannotRecall.AddDetails("hash not found: %s", h) + } + + return r.RecallByOffset(o) +} + +// RecallByOffset returns the object for a given offset by looking for it again in +// the io.ReadeSeerker. For efficiency reasons, this method always find objects by +// offset, even if they have not been remembered or if they have been forgetted. +func (r *Seekable) RecallByOffset(o int64) (obj core.Object, err error) { + // remember current offset + beforeJump, err := r.Offset() + if err != nil { + return nil, err + } + + defer func() { + // jump back + _, seekErr := r.Seek(beforeJump, os.SEEK_SET) + if err == nil { + err = seekErr + } + }() + + // jump to requested offset + _, err = r.Seek(o, os.SEEK_SET) + if err != nil { + return nil, err + } + + return NewParser(r).ReadObject() +} diff --git a/formats/packfile/stream.go b/formats/packfile/stream.go new file mode 100644 index 0000000..41266b1 --- /dev/null +++ b/formats/packfile/stream.go @@ -0,0 +1,95 @@ +package packfile + +import ( + "io" + + "gopkg.in/src-d/go-git.v3/core" +) + +// Stream implements ReadRecaller for the io.Reader of a packfile. This +// implementation keeps all remembered objects referenced in maps for +// quick access. +type Stream struct { + io.Reader + count int64 + offsetToObject map[int64]core.Object + hashToObject map[core.Hash]core.Object +} + +// NewStream returns a new Stream that reads form r. +func NewStream(r io.Reader) *Stream { + return &Stream{ + Reader: r, + count: 0, + hashToObject: make(map[core.Hash]core.Object, 0), + offsetToObject: make(map[int64]core.Object, 0), + } +} + +// Read reads up to len(p) bytes into p. +func (r *Stream) Read(p []byte) (n int, err error) { + n, err = r.Reader.Read(p) + r.count += int64(n) + + return +} + +// ReadByte reads a byte. +func (r *Stream) ReadByte() (byte, error) { + var p [1]byte + _, err := r.Reader.Read(p[:]) + r.count++ + + return p[0], err +} + +// Offset returns the number of bytes read. +func (r *Stream) Offset() (int64, error) { + return r.count, nil +} + +// Remember stores references to the passed object to be used later by +// RecalByHash and RecallByOffset. It receives the object and the offset +// of its object entry in the packfile. +func (r *Stream) Remember(o int64, obj core.Object) error { + h := obj.Hash() + if _, ok := r.hashToObject[h]; ok { + return ErrDuplicatedObject.AddDetails("with hash %s", h) + } + r.hashToObject[h] = obj + + if _, ok := r.offsetToObject[o]; ok { + return ErrDuplicatedObject.AddDetails("with offset %d", o) + } + r.offsetToObject[o] = obj + + return nil +} + +// ForgetAll forgets all previously remembered objects. +func (r *Stream) ForgetAll() { + r.hashToObject = make(map[core.Hash]core.Object) + r.offsetToObject = make(map[int64]core.Object) +} + +// RecallByHash returns an object that has been previously Remember-ed by +// its hash. +func (r *Stream) RecallByHash(h core.Hash) (core.Object, error) { + obj, ok := r.hashToObject[h] + if !ok { + return nil, ErrCannotRecall.AddDetails("by hash %s", h) + } + + return obj, nil +} + +// RecallByOffset returns an object that has been previously Remember-ed by +// the offset of its object entry in the packfile. +func (r *Stream) RecallByOffset(o int64) (core.Object, error) { + obj, ok := r.offsetToObject[o] + if !ok { + return nil, ErrCannotRecall.AddDetails("no object found at offset %d", o) + } + + return obj, nil +} diff --git a/objects_test.go b/objects_test.go index c48b768..4beeeba 100644 --- a/objects_test.go +++ b/objects_test.go @@ -4,9 +4,10 @@ import ( "io/ioutil" "time" - . "gopkg.in/check.v1" "gopkg.in/src-d/go-git.v3/core" "gopkg.in/src-d/go-git.v3/storage/memory" + + . "gopkg.in/check.v1" ) type ObjectsSuite struct { @@ -18,9 +19,11 @@ var _ = Suite(&ObjectsSuite{}) func (s *ObjectsSuite) SetUpTest(c *C) { var err error s.r, err = NewRepository(RepositoryFixture, nil) + c.Assert(err, IsNil) + s.r.Remotes["origin"].upSrv = &MockGitUploadPackService{} - s.r.Pull("origin", "refs/heads/master") + err = s.r.Pull("origin", "refs/heads/master") c.Assert(err, IsNil) } diff --git a/references_test.go b/references_test.go index 4bd91d2..7e003bd 100644 --- a/references_test.go +++ b/references_test.go @@ -20,17 +20,15 @@ var _ = Suite(&ReferencesSuite{}) // create the repositories of the fixtures func (s *ReferencesSuite) SetUpSuite(c *C) { s.repos = make(map[string]*Repository, 0) - for _, fixRepo := range fixtureRepos { - s.repos[fixRepo.url] = NewPlainRepository() + for _, fix := range fixtureRepos { + s.repos[fix.url] = NewPlainRepository() - d, err := os.Open(fixRepo.packfile) - defer d.Close() + f, err := os.Open(fix.packfile) + defer f.Close() c.Assert(err, IsNil) - - r := packfile.NewReader(d) - r.Format = packfile.OFSDeltaFormat // TODO: how to know the format of a pack file ahead of time? - - _, err = r.Read(s.repos[fixRepo.url].Storage) + r := packfile.NewSeekable(f) + d := packfile.NewDecoder(r) + err = d.Decode(s.repos[fix.url].Storage) c.Assert(err, IsNil) } } @@ -339,10 +337,10 @@ func compareSideBySide(a []string, b []*Commit) string { var buf bytes.Buffer buf.WriteString("\t EXPECTED OBTAINED ") var sep string - var obtained string + var obt string for i := range a { - obtained = b[i].Hash.String() - if a[i] != obtained { + obt = b[i].Hash.String() + if a[i] != obt { sep = "------" } else { sep = " " @@ -351,7 +349,7 @@ func compareSideBySide(a []string, b []*Commit) string { buf.WriteString(sep) buf.WriteString(a[i]) buf.WriteString(sep) - buf.WriteString(obtained) + buf.WriteString(obt) } return buf.String() } diff --git a/remote_test.go b/remote_test.go index 85b464c..381609e 100644 --- a/remote_test.go +++ b/remote_test.go @@ -16,10 +16,10 @@ var _ = Suite(&SuiteRemote{}) const RepositoryFixture = "https://github.com/tyba/git-fixture" func (s *SuiteRemote) TestNewAuthenticatedRemote(c *C) { - auth := &http.BasicAuth{} - r, err := NewAuthenticatedRemote(RepositoryFixture, auth) + a := &http.BasicAuth{} + r, err := NewAuthenticatedRemote(RepositoryFixture, a) c.Assert(err, IsNil) - c.Assert(r.Auth, Equals, auth) + c.Assert(r.Auth, Equals, a) } func (s *SuiteRemote) TestConnect(c *C) { @@ -56,12 +56,13 @@ func (s *SuiteRemote) TestFetchDefaultBranch(c *C) { reader, err := r.FetchDefaultBranch() c.Assert(err, IsNil) - pr := packfile.NewReader(reader) + packfileReader := packfile.NewStream(reader) + d := packfile.NewDecoder(packfileReader) - storage := memory.NewObjectStorage() - _, err = pr.Read(storage) + sto := memory.NewObjectStorage() + err = d.Decode(sto) c.Assert(err, IsNil) - c.Assert(storage.Objects, HasLen, 28) + c.Assert(sto.Objects, HasLen, 28) } func (s *SuiteRemote) TestHead(c *C) { diff --git a/repository.go b/repository.go index 65e0a5d..c05afdb 100644 --- a/repository.go +++ b/repository.go @@ -8,6 +8,8 @@ import ( "gopkg.in/src-d/go-git.v3/core" "gopkg.in/src-d/go-git.v3/formats/packfile" "gopkg.in/src-d/go-git.v3/storage/memory" + "gopkg.in/src-d/go-git.v3/storage/seekable" + "gopkg.in/src-d/go-git.v3/utils/fs" ) var ( @@ -24,29 +26,39 @@ const ( type Repository struct { Remotes map[string]*Remote Storage core.ObjectStorage - URL string } // NewRepository creates a new repository setting remote as default remote func NewRepository(url string, auth common.AuthMethod) (*Repository, error) { - var remote *Remote - var err error - - if auth == nil { - remote, err = NewRemote(url) - } else { - remote, err = NewAuthenticatedRemote(url, auth) - } + repo := NewPlainRepository() + r, err := NewAuthenticatedRemote(url, auth) + repo.Remotes[DefaultRemoteName] = r if err != nil { return nil, err } - r := NewPlainRepository() - r.Remotes[DefaultRemoteName] = remote - r.URL = url + return repo, nil +} + +// NewRepositoryFromFS creates a new repository from an standard git +// repository on disk. +// +// Repositories created like this don't hold a local copy of the +// original repository objects, instead all queries are resolved by +// looking at the original repository packfile. This is very cheap in +// terms of memory and allows to process repositories bigger than your +// memory. +// +// To be able to use git repositories this way, you must run "git gc" on +// them beforehand. +func NewRepositoryFromFS(fs fs.FS, path string) (*Repository, error) { + repo := NewPlainRepository() - return r, nil + var err error + repo.Storage, err = seekable.New(fs, path) + + return repo, err } // NewPlainRepository creates a new repository without remotes @@ -66,7 +78,7 @@ func (r *Repository) Pull(remoteName, branch string) (err error) { return fmt.Errorf("unable to find remote %q", remoteName) } - if err := remote.Connect(); err != nil { + if err = remote.Connect(); err != nil { return err } @@ -89,13 +101,12 @@ func (r *Repository) Pull(remoteName, branch string) (err error) { return err } defer checkClose(reader, &err) + stream := packfile.NewStream(reader) - pr := packfile.NewReader(reader) - if _, err = pr.Read(r.Storage); err != nil { - return err - } + d := packfile.NewDecoder(stream) + err = d.Decode(r.Storage) - return nil + return err } // PullDefault like Pull but retrieve the default branch from the default remote @@ -118,8 +129,13 @@ func (r *Repository) Commit(h core.Hash) (*Commit, error) { } // Commits decode the objects into commits -func (r *Repository) Commits() *CommitIter { - return NewCommitIter(r, r.Storage.Iter(core.CommitObject)) +func (r *Repository) Commits() (*CommitIter, error) { + iter, err := r.Storage.Iter(core.CommitObject) + if err != nil { + return nil, err + } + + return NewCommitIter(r, iter), nil } // Tree return the tree with the given hash @@ -160,14 +176,19 @@ func (r *Repository) Tag(h core.Hash) (*Tag, error) { return nil, err } - tag := &Tag{r: r} - return tag, tag.Decode(obj) + t := &Tag{r: r} + return t, t.Decode(obj) } // Tags returns a TagIter that can step through all of the annotated tags // in the repository. -func (r *Repository) Tags() *TagIter { - return NewTagIter(r, r.Storage.Iter(core.TagObject)) +func (r *Repository) Tags() (*TagIter, error) { + iter, err := r.Storage.Iter(core.TagObject) + if err != nil { + return nil, err + } + + return NewTagIter(r, iter), nil } // Object returns an object with the given hash. diff --git a/repository_test.go b/repository_test.go index 1f57b18..7d4abb5 100644 --- a/repository_test.go +++ b/repository_test.go @@ -2,28 +2,60 @@ package git import ( "fmt" + "os" "gopkg.in/src-d/go-git.v3/clients/http" "gopkg.in/src-d/go-git.v3/core" + "gopkg.in/src-d/go-git.v3/storage/seekable" + "gopkg.in/src-d/go-git.v3/utils/fs" + "github.com/alcortesm/tgz" . "gopkg.in/check.v1" ) +var dirFixtures = [...]struct { + name string + tgz string +}{ + { + name: "binrels", + tgz: "storage/seekable/internal/gitdir/fixtures/alcortesm-binary-relations.tgz", + }, +} + type SuiteRepository struct { - repos map[string]*Repository + repos map[string]*Repository + dirFixturePaths map[string]string } var _ = Suite(&SuiteRepository{}) func (s *SuiteRepository) SetUpSuite(c *C) { s.repos = unpackFixtures(c, tagFixtures, treeWalkerFixtures) + + s.dirFixturePaths = make(map[string]string, len(dirFixtures)) + for _, fix := range dirFixtures { + com := Commentf("fixture name = %s\n", fix.name) + + path, err := tgz.Extract(fix.tgz) + c.Assert(err, IsNil, com) + + s.dirFixturePaths[fix.name] = path + } +} + +func (s *SuiteRepository) TearDownSuite(c *C) { + for name, path := range s.dirFixturePaths { + err := os.RemoveAll(path) + c.Assert(err, IsNil, Commentf("cannot delete tmp dir for fixture %s: %s\n", + name, path)) + } } func (s *SuiteRepository) TestNewRepository(c *C) { r, err := NewRepository(RepositoryFixture, nil) c.Assert(err, IsNil) c.Assert(r.Remotes["origin"].Auth, IsNil) - c.Assert(r.URL, Equals, RepositoryFixture) } func (s *SuiteRepository) TestNewRepositoryWithAuth(c *C) { @@ -33,6 +65,22 @@ func (s *SuiteRepository) TestNewRepositoryWithAuth(c *C) { c.Assert(r.Remotes["origin"].Auth, Equals, auth) } +func (s *SuiteRepository) TestNewRepositoryFromFS(c *C) { + for name, path := range s.dirFixturePaths { + fs := fs.NewOS() + gitPath := fs.Join(path, ".git/") + com := Commentf("dir fixture %q → %q\n", name, gitPath) + repo, err := NewRepositoryFromFS(fs, gitPath) + c.Assert(err, IsNil, com) + + err = repo.PullDefault() + c.Assert(err, ErrorMatches, `unable to find remote "origin"`) + + c.Assert(repo.Storage, NotNil, com) + c.Assert(repo.Storage, FitsTypeOf, &seekable.ObjectStorage{}, com) + } +} + func (s *SuiteRepository) TestPull(c *C) { r, err := NewRepository(RepositoryFixture, nil) r.Remotes["origin"].upSrv = &MockGitUploadPackService{} @@ -87,7 +135,8 @@ func (s *SuiteRepository) TestCommits(c *C) { c.Assert(r.Pull("origin", "refs/heads/master"), IsNil) count := 0 - commits := r.Commits() + commits, err := r.Commits() + c.Assert(err, IsNil) for { commit, err := commits.Next() if err != nil { @@ -109,11 +158,11 @@ func (s *SuiteRepository) TestTag(c *C) { r, ok := s.repos[t.repo] c.Assert(ok, Equals, true) k := 0 - for hashString, expected := range t.tags { + for hashString, exp := range t.tags { hash := core.NewHash(hashString) tag, err := r.Tag(hash) c.Assert(err, IsNil) - testTagExpected(c, tag, hash, expected, fmt.Sprintf("subtest %d, tag %d: ", i, k)) + testTagExpected(c, tag, hash, exp, fmt.Sprintf("subtest %d, tag %d: ", i, k)) k++ } } @@ -123,7 +172,9 @@ func (s *SuiteRepository) TestTags(c *C) { for i, t := range tagTests { r, ok := s.repos[t.repo] c.Assert(ok, Equals, true) - testTagIter(c, r.Tags(), t.tags, fmt.Sprintf("subtest %d, ", i)) + tagsIter, err := r.Tags() + c.Assert(err, IsNil) + testTagIter(c, tagsIter, t.tags, fmt.Sprintf("subtest %d, ", i)) } } @@ -132,13 +183,13 @@ func (s *SuiteRepository) TestObject(c *C) { r, ok := s.repos[t.repo] c.Assert(ok, Equals, true) for k := 0; k < len(t.objs); k++ { - comment := fmt.Sprintf("subtest %d, tag %d", i, k) + com := fmt.Sprintf("subtest %d, tag %d", i, k) info := t.objs[k] hash := core.NewHash(info.Hash) obj, err := r.Object(hash) - c.Assert(err, IsNil, Commentf(comment)) - c.Assert(obj.Type(), Equals, info.Kind, Commentf(comment)) - c.Assert(obj.ID(), Equals, hash, Commentf(comment)) + c.Assert(err, IsNil, Commentf(com)) + c.Assert(obj.Type(), Equals, info.Kind, Commentf(com)) + c.Assert(obj.ID(), Equals, hash, Commentf(com)) } } } @@ -150,6 +201,7 @@ func (s *SuiteRepository) TestCommitIterClosePanic(c *C) { c.Assert(err, IsNil) c.Assert(r.Pull("origin", "refs/heads/master"), IsNil) - commits := r.Commits() + commits, err := r.Commits() + c.Assert(err, IsNil) commits.Close() } diff --git a/storage/memory/object.go b/storage/memory/object.go index 125e887..1720ebd 100644 --- a/storage/memory/object.go +++ b/storage/memory/object.go @@ -9,19 +9,29 @@ import ( // Object on memory core.Object implementation type Object struct { - t core.ObjectType - h core.Hash - content []byte - size int64 + t core.ObjectType + h core.Hash + cont []byte + sz int64 +} + +// NewObject creates a new object with the given type and content +func NewObject(typ core.ObjectType, size int64, cont []byte) *Object { + return &Object{ + t: typ, + h: core.ComputeHash(typ, cont), + cont: cont, + sz: int64(len(cont)), + } } // Hash return the object Hash, the hash is calculated on-the-fly the first -// time is called, the subsequent calls the same Hash is returned even in the +// time is called, the subsequent calls the same Hash is returned even if the // type or the content has changed. The Hash is only generated if the size of // the content is exactly the Object.Size func (o *Object) Hash() core.Hash { - if o.h == core.ZeroHash && int64(len(o.content)) == o.size { - o.h = core.ComputeHash(o.t, o.content) + if o.h == core.ZeroHash && int64(len(o.cont)) == o.sz { + o.h = core.ComputeHash(o.t, o.cont) } return o.h @@ -34,14 +44,17 @@ func (o *Object) Type() core.ObjectType { return o.t } func (o *Object) SetType(t core.ObjectType) { o.t = t } // Size return the size of the object -func (o *Object) Size() int64 { return o.size } +func (o *Object) Size() int64 { return o.sz } // SetSize set the object size, the given size should be written afterwards -func (o *Object) SetSize(s int64) { o.size = s } +func (o *Object) SetSize(s int64) { o.sz = s } + +// Content returns the contents of the object +func (o *Object) Content() []byte { return o.cont } // Reader returns a core.ObjectReader used to read the object's content. func (o *Object) Reader() (core.ObjectReader, error) { - return ioutil.NopCloser(bytes.NewBuffer(o.content)), nil + return ioutil.NopCloser(bytes.NewBuffer(o.cont)), nil } // Writer returns a core.ObjectWriter used to write the object's content. @@ -50,7 +63,7 @@ func (o *Object) Writer() (core.ObjectWriter, error) { } func (o *Object) Write(p []byte) (n int, err error) { - o.content = append(o.content, p...) + o.cont = append(o.cont, p...) return len(p), nil } diff --git a/storage/memory/object_test.go b/storage/memory/object_test.go index f2873fa..3271254 100644 --- a/storage/memory/object_test.go +++ b/storage/memory/object_test.go @@ -49,7 +49,7 @@ func (s *ObjectSuite) TestSize(c *C) { } func (s *ObjectSuite) TestReader(c *C) { - o := &Object{content: []byte("foo")} + o := &Object{cont: []byte("foo")} reader, err := o.Reader() c.Assert(err, IsNil) @@ -71,5 +71,5 @@ func (s *ObjectSuite) TestWriter(c *C) { c.Assert(err, IsNil) c.Assert(n, Equals, 3) - c.Assert(o.content, DeepEquals, []byte("foo")) + c.Assert(o.cont, DeepEquals, []byte("foo")) } diff --git a/storage/memory/storage.go b/storage/memory/storage.go index fbf4bc4..84de980 100644 --- a/storage/memory/storage.go +++ b/storage/memory/storage.go @@ -28,11 +28,6 @@ func NewObjectStorage() *ObjectStorage { } } -// New returns a new empty memory.Object -func (o *ObjectStorage) New() (core.Object, error) { - return &Object{}, nil -} - // Set stores an object, the object should be properly filled before set it. func (o *ObjectStorage) Set(obj core.Object) (core.Hash, error) { h := obj.Hash() @@ -65,7 +60,7 @@ func (o *ObjectStorage) Get(h core.Hash) (core.Object, error) { } // Iter returns a core.ObjectIter for the given core.ObjectTybe -func (o *ObjectStorage) Iter(t core.ObjectType) core.ObjectIter { +func (o *ObjectStorage) Iter(t core.ObjectType) (core.ObjectIter, error) { var series []core.Object switch t { case core.CommitObject: @@ -77,7 +72,7 @@ func (o *ObjectStorage) Iter(t core.ObjectType) core.ObjectIter { case core.TagObject: series = flattenObjectMap(o.Tags) } - return core.NewObjectSliceIter(series) + return core.NewObjectSliceIter(series), nil } func flattenObjectMap(m map[core.Hash]core.Object) []core.Object { diff --git a/storage/memory/storage_test.go b/storage/memory/storage_test.go index df6d5cf..19f4476 100644 --- a/storage/memory/storage_test.go +++ b/storage/memory/storage_test.go @@ -9,20 +9,10 @@ type ObjectStorageSuite struct{} var _ = Suite(&ObjectStorageSuite{}) -func (s *ObjectStorageSuite) TestNew(c *C) { - os := NewObjectStorage() - - o, err := os.New() - c.Assert(err, IsNil) - c.Assert(o.Size(), Equals, int64(0)) -} - func (s *ObjectStorageSuite) TestSet(c *C) { os := NewObjectStorage() - o, err := os.New() - c.Assert(err, IsNil) - + o := &Object{} o.SetType(core.CommitObject) o.SetSize(3) @@ -40,9 +30,7 @@ func (s *ObjectStorageSuite) TestSet(c *C) { func (s *ObjectStorageSuite) TestGet(c *C) { os := NewObjectStorage() - o, err := os.New() - c.Assert(err, IsNil) - + o := &Object{} o.SetType(core.CommitObject) o.SetSize(3) diff --git a/storage/seekable/internal/gitdir/gitdir.go b/storage/seekable/internal/gitdir/gitdir.go new file mode 100644 index 0000000..bfdf030 --- /dev/null +++ b/storage/seekable/internal/gitdir/gitdir.go @@ -0,0 +1,145 @@ +package gitdir + +import ( + "errors" + "io/ioutil" + "os" + "strings" + + "gopkg.in/src-d/go-git.v3/clients/common" + "gopkg.in/src-d/go-git.v3/core" + "gopkg.in/src-d/go-git.v3/utils/fs" +) + +const ( + suffix = ".git" + packedRefsPath = "packed-refs" +) + +var ( + // ErrNotFound is returned by New when the path is not found. + ErrNotFound = errors.New("path not found") + // ErrIdxNotFound is returned by Idxfile when the idx file is not found on the + // repository. + ErrIdxNotFound = errors.New("idx file not found") + // ErrPackfileNotFound is returned by Packfile when the packfile is not found + // on the repository. + ErrPackfileNotFound = errors.New("packfile not found") +) + +// The GitDir type represents a local git repository on disk. This +// type is not zero-value-safe, use the New function to initialize it. +type GitDir struct { + fs fs.FS + path string + refs map[string]core.Hash + packDir string +} + +// New returns a GitDir value ready to be used. The path argument must +// be the absolute path of a git repository directory (e.g. +// "/foo/bar/.git"). +func New(fs fs.FS, path string) (*GitDir, error) { + d := &GitDir{} + d.fs = fs + d.path = path + d.packDir = d.fs.Join(d.path, "objects", "pack") + + if _, err := fs.Stat(path); err != nil { + if os.IsNotExist(err) { + return nil, ErrNotFound + } + return nil, err + } + + return d, nil +} + +// Refs scans the git directory collecting references, which it returns. +// Symbolic references are resolved and included in the output. +func (d *GitDir) Refs() (map[string]core.Hash, error) { + var err error + + d.refs = make(map[string]core.Hash) + + if err = d.addRefsFromPackedRefs(); err != nil { + return nil, err + } + + if err = d.addRefsFromRefDir(); err != nil { + return nil, err + } + + return d.refs, err +} + +// Capabilities scans the git directory collection capabilities, which it returns. +func (d *GitDir) Capabilities() (*common.Capabilities, error) { + c := common.NewCapabilities() + + err := d.addSymRefCapability(c) + + return c, err +} + +func (d *GitDir) addSymRefCapability(cap *common.Capabilities) (err error) { + f, err := d.fs.Open(d.fs.Join(d.path, "HEAD")) + if err != nil { + return err + } + + defer func() { + errClose := f.Close() + if err == nil { + err = errClose + } + }() + + b, err := ioutil.ReadAll(f) + if err != nil { + return err + } + data := strings.TrimSpace(string(b)) + + c := "symref" + ref := strings.TrimPrefix(data, symRefPrefix) + cap.Set(c, "HEAD:"+ref) + + return nil +} + +// Packfile returns the path of the packfile (really, it returns the +// path of the first file in the "objects/pack/" directory with a +// ".pack" extension. +func (d *GitDir) Packfile() (fs.FS, string, error) { + files, err := d.fs.ReadDir(d.packDir) + if err != nil { + return nil, "", err + } + + for _, f := range files { + if strings.HasSuffix(f.Name(), ".pack") { + return d.fs, d.fs.Join(d.packDir, f.Name()), nil + } + } + + return nil, "", ErrPackfileNotFound +} + +// Packfile returns the path of the idx file (really, it returns the +// path of the first file in the "objects/pack/" directory with an +// ".idx" extension. +func (d *GitDir) Idxfile() (fs.FS, string, error) { + files, err := d.fs.ReadDir(d.packDir) + if err != nil { + return nil, "", err + } + + for _, f := range files { + if strings.HasSuffix(f.Name(), ".idx") { + return d.fs, d.fs.Join(d.packDir, f.Name()), nil + } + } + + return nil, "", ErrIdxNotFound +} diff --git a/storage/seekable/internal/gitdir/gitdir_test.go b/storage/seekable/internal/gitdir/gitdir_test.go new file mode 100644 index 0000000..7504119 --- /dev/null +++ b/storage/seekable/internal/gitdir/gitdir_test.go @@ -0,0 +1,263 @@ +package gitdir + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "gopkg.in/src-d/go-git.v3/clients/common" + "gopkg.in/src-d/go-git.v3/core" + "gopkg.in/src-d/go-git.v3/utils/fs" + + "github.com/alcortesm/tgz" + . "gopkg.in/check.v1" +) + +func Test(t *testing.T) { TestingT(t) } + +var initFixtures = [...]struct { + name string + tgz string + capabilities [][2]string + packfile string + idxfile string +}{ + { + name: "spinnaker", + tgz: "fixtures/spinnaker-gc.tgz", + capabilities: [][2]string{ + {"symref", "HEAD:refs/heads/master"}, + }, + packfile: "objects/pack/pack-584416f86235cac0d54bfabbdc399fb2b09a5269.pack", + idxfile: "objects/pack/pack-584416f86235cac0d54bfabbdc399fb2b09a5269.idx", + }, { + name: "no-packfile-no-idx", + tgz: "fixtures/no-packfile-no-idx.tgz", + }, { + name: "empty", + tgz: "fixtures/empty-gitdir.tgz", + }, +} + +type fixture struct { + installDir string + fs fs.FS + path string // repo names to paths of the extracted tgz + capabilities *common.Capabilities // expected capabilities + packfile string // path of the packfile + idxfile string // path of the idxfile +} + +type SuiteGitDir struct { + fixtures map[string]fixture +} + +var _ = Suite(&SuiteGitDir{}) + +func (s *SuiteGitDir) SetUpSuite(c *C) { + s.fixtures = make(map[string]fixture, len(initFixtures)) + + for _, init := range initFixtures { + com := Commentf("fixture name = %s\n", init.name) + + path, err := tgz.Extract(init.tgz) + c.Assert(err, IsNil, com) + + f := fixture{} + + f.installDir = path + f.fs = fs.NewOS() + f.path = f.fs.Join(path, ".git") + + f.capabilities = common.NewCapabilities() + for _, pair := range init.capabilities { + f.capabilities.Add(pair[0], pair[1]) + } + + f.packfile = init.packfile + f.idxfile = init.idxfile + + s.fixtures[init.name] = f + } +} + +func (s *SuiteGitDir) TearDownSuite(c *C) { + for n, f := range s.fixtures { + err := os.RemoveAll(f.installDir) + c.Assert(err, IsNil, Commentf("cannot delete tmp dir for fixture %s: %s\n", + n, f.installDir)) + } +} + +func (s *SuiteGitDir) TestNewErrors(c *C) { + for i, test := range [...]struct { + input string + err error + }{ + { + input: "./tmp/foo", + err: ErrNotFound, + }, { + input: "./tmp/foo/.git", + err: ErrNotFound, + }, + } { + com := Commentf("subtest %d", i) + + _, err := New(fs.NewOS(), test.input) + c.Assert(err, Equals, test.err, com) + } +} + +func (s *SuiteGitDir) TestRefs(c *C) { + for i, test := range [...]struct { + fixture string + refs map[string]core.Hash + }{ + { + fixture: "spinnaker", + refs: map[string]core.Hash{ + "refs/heads/master": core.NewHash("409db80e56365049edb704f2ecbd449ddf64dc0d"), + "refs/remotes/origin/HEAD": core.NewHash("409db80e56365049edb704f2ecbd449ddf64dc0d"), + "refs/remotes/origin/explicit-machine-type": core.NewHash("f262e833a215c90b703115691f03f182c1be4b91"), + "refs/remotes/origin/fix-aws-creds-copy": core.NewHash("871cf4d673e0d94c6eb2558bfc7a525c2bc7e538"), + "refs/remotes/origin/kubernetes-no-gcloud": core.NewHash("0b553b5b6fa773f3d7a38b229d9f75627c0762aa"), + "refs/remotes/origin/lwander-patch-igor": core.NewHash("9c987f44908bc9aa05e950347cd03228ba199630"), + "refs/remotes/origin/master": core.NewHash("409db80e56365049edb704f2ecbd449ddf64dc0d"), + "refs/remotes/origin/revert-898-codelab-script-fix": core.NewHash("426cd84d1741d0ff68bad646bc8499b1f163a893"), + "refs/remotes/origin/terraform-aws-prototype": core.NewHash("a34445e7d2e758a8c953fa3a357198ec09fcba88"), + "refs/remotes/origin/typo": core.NewHash("86b48b962e599c096a5870cd8047778bb32a6e1e"), + "refs/tags/v0.10.0": core.NewHash("d081d66c2a76d04ff479a3431dc36e44116fde40"), + "refs/tags/v0.11.0": core.NewHash("3e349f806a0d02bf658c3544c46a0a7a9ee78673"), + "refs/tags/v0.12.0": core.NewHash("82562fa518f0a2e2187ea2604b07b67f2e7049ae"), + "refs/tags/v0.13.0": core.NewHash("48b655898fa9c72d62e8dd73b022ecbddd6e4cc2"), + "refs/tags/v0.14.0": core.NewHash("7ecc2ad58e24a5b52504985467a10c6a3bb85b9b"), + "refs/tags/v0.15.0": core.NewHash("740e3adff4c350899db7772f8f537d1d0d96ec75"), + "refs/tags/v0.16.0": core.NewHash("466ca58a3129f1b2ead117a43535ecb410d621ac"), + "refs/tags/v0.17.0": core.NewHash("48020cb7a45603d47e6041de072fe0665e47676f"), + "refs/tags/v0.18.0": core.NewHash("6fcb9036ab4d921dbdab41baf923320484a11188"), + "refs/tags/v0.19.0": core.NewHash("a2ce1f4c9d0bde4e93dfcb90a445ed069030640c"), + "refs/tags/v0.20.0": core.NewHash("974f476f0ec5a9dcc4bb005384d449f0a5122da4"), + "refs/tags/v0.21.0": core.NewHash("e08e3917f3a0487e33cd6dcef24fe03e570b73f5"), + "refs/tags/v0.22.0": core.NewHash("834612b4f181171d5e1e263b4e7e55d609ab19f5"), + "refs/tags/v0.23.0": core.NewHash("65558da39c07a6f9104651281c226981e880b49c"), + "refs/tags/v0.24.0": core.NewHash("5c97aa1f2f784e92f065055f9e79df83fac7a4aa"), + "refs/tags/v0.25.0": core.NewHash("d6e696f9d5e2dac968638665886e2300ae15709a"), + "refs/tags/v0.26.0": core.NewHash("974861702abd8388e0507cf3f348d6d3c40acef4"), + "refs/tags/v0.27.0": core.NewHash("65771ef145b3e07e130abc84fb07f0b8044fcf59"), + "refs/tags/v0.28.0": core.NewHash("5d86433d6dc4358277a5e9a834948f0822225a6d"), + "refs/tags/v0.29.0": core.NewHash("c1582497c23d81e61963841861c5aebbf10e12ab"), + "refs/tags/v0.3.0": core.NewHash("8b6002b614b454d45bafbd244b127839421f92ff"), + "refs/tags/v0.30.0": core.NewHash("b0f26484aab0afe2f342be84583213c3c64b7eb3"), + "refs/tags/v0.31.0": core.NewHash("8a2da11c9d29e3a879a068c197568c108b9e5f88"), + "refs/tags/v0.32.0": core.NewHash("5c5fc48a1506bb4609ca5588f90cf021a29a4a37"), + "refs/tags/v0.33.0": core.NewHash("d443f1f61e23411d9ac08f0fc6bbeb8e4c46ee39"), + "refs/tags/v0.34.0": core.NewHash("0168d74697d65cde65f931254c09a6bd7ff4f0d5"), + "refs/tags/v0.35.0": core.NewHash("a46303084ad9decf71a8ea9fd1529e22c6fdd2c4"), + "refs/tags/v0.36.0": core.NewHash("4da0d7bb89e85bd5f14ff36d983a0ae773473b2d"), + "refs/tags/v0.37.0": core.NewHash("85ec60477681933961c9b64c18ada93220650ac5"), + "refs/tags/v0.4.0": core.NewHash("95ee6e6c750ded1f4dc5499bad730ce3f58c6c3a"), + "refs/tags/v0.5.0": core.NewHash("0a3fb06ff80156fb153bcdcc58b5e16c2d27625c"), + "refs/tags/v0.6.0": core.NewHash("dc22e2035292ccf020c30d226f3cc2da651773f6"), + "refs/tags/v0.7.0": core.NewHash("3f36d8f1d67538afd1f089ffd0d242fc4fda736f"), + "refs/tags/v0.8.0": core.NewHash("8526c58617f68de076358873b8aa861a354b48a9"), + "refs/tags/v0.9.0": core.NewHash("776914ef8a097f5683957719c49215a5db17c2cb"), + }, + }, + } { + com := Commentf("subtest %d", i) + _, d := s.newFixtureDir(c, test.fixture) + + refs, err := d.Refs() + c.Assert(err, IsNil, com) + c.Assert(refs, DeepEquals, test.refs, com) + } +} + +func (s *SuiteGitDir) newFixtureDir(c *C, fixName string) (*fixture, *GitDir) { + f, ok := s.fixtures[fixName] + c.Assert(ok, Equals, true) + + d, err := New(fs.NewOS(), f.path) + c.Assert(err, IsNil) + + return &f, d +} + +func (s *SuiteGitDir) TestCapabilities(c *C) { + for i, test := range [...]struct { + fixture string + capabilities *common.Capabilities + }{ + { + fixture: "spinnaker", + }, + } { + com := Commentf("subtest %d", i) + f, d := s.newFixtureDir(c, test.fixture) + + caps, err := d.Capabilities() + c.Assert(err, IsNil, com) + c.Assert(caps, DeepEquals, f.capabilities, com) + } +} + +func (s *SuiteGitDir) TestPackfile(c *C) { + packfile := func(d *GitDir) (fs.FS, string, error) { + return d.Packfile() + } + idxfile := func(d *GitDir) (fs.FS, string, error) { + return d.Idxfile() + } + for _, test := range [...]struct { + fixture string + fn getPathFn + err string // error regexp + }{ + { + fixture: "spinnaker", + fn: packfile, + }, { + fixture: "spinnaker", + fn: idxfile, + }, { + fixture: "empty", + fn: packfile, + err: ".* no such file or directory", + }, { + fixture: "empty", + fn: idxfile, + err: ".* no such file or directory", + }, { + fixture: "no-packfile-no-idx", + fn: packfile, + err: "packfile not found", + }, { + fixture: "no-packfile-no-idx", + fn: idxfile, + err: "idx file not found", + }, + } { + com := Commentf("fixture = %s", test.fixture) + + fix, dir := s.newFixtureDir(c, test.fixture) + + _, path, err := test.fn(dir) + + if test.err != "" { + c.Assert(err, ErrorMatches, test.err, com) + } else { + c.Assert(err, IsNil, com) + c.Assert(strings.HasSuffix(noExt(path), noExt(fix.packfile)), + Equals, true, com) + } + } +} + +type getPathFn func(*GitDir) (fs.FS, string, error) + +func noExt(path string) string { + ext := filepath.Ext(path) + return path[0 : len(path)-len(ext)] +} diff --git a/storage/seekable/internal/gitdir/refs.go b/storage/seekable/internal/gitdir/refs.go new file mode 100644 index 0000000..9c2e8fb --- /dev/null +++ b/storage/seekable/internal/gitdir/refs.go @@ -0,0 +1,152 @@ +package gitdir + +import ( + "bufio" + "errors" + "io/ioutil" + "os" + "strings" + + "gopkg.in/src-d/go-git.v3/core" +) + +var ( + // ErrPackedRefsDuplicatedRef is returned when a duplicated + // reference is found in the packed-ref file. This is usually the + // case for corrupted git repositories. + ErrPackedRefsDuplicatedRef = errors.New("duplicated ref found in packed-ref file") + // ErrPackedRefsBadFormat is returned when the packed-ref file + // corrupt. + ErrPackedRefsBadFormat = errors.New("malformed packed-ref") + // ErrSymRefTargetNotFound is returned when a symbolic reference is + // targeting a non-existing object. This usually means the + // repository is corrupt. + ErrSymRefTargetNotFound = errors.New("symbolic reference target not found") +) + +const ( + symRefPrefix = "ref: " +) + +func (d *GitDir) addRefsFromPackedRefs() (err error) { + path := d.fs.Join(d.path, packedRefsPath) + f, err := d.fs.Open(path) + if err != nil { + if err == os.ErrNotExist { + return nil + } + return err + } + defer func() { + errClose := f.Close() + if err == nil { + err = errClose + } + }() + + s := bufio.NewScanner(f) + for s.Scan() { + line := s.Text() + if err = d.processLine(line); err != nil { + return err + } + } + + return s.Err() +} + +// process lines from a packed-refs file +func (d *GitDir) processLine(line string) error { + switch line[0] { + case '#': // comment - ignore + return nil + case '^': // annotated tag commit of the previous line - ignore + return nil + default: + ws := strings.Split(line, " ") // hash then ref + if len(ws) != 2 { + return ErrPackedRefsBadFormat + } + h, r := ws[0], ws[1] + + if _, ok := d.refs[r]; ok { + return ErrPackedRefsDuplicatedRef + } + d.refs[r] = core.NewHash(h) + } + + return nil +} + +func (d *GitDir) addRefsFromRefDir() error { + return d.walkTree("refs") +} + +func (d *GitDir) walkTree(relPath string) error { + files, err := d.fs.ReadDir(d.fs.Join(d.path, relPath)) + if err != nil { + return err + } + + for _, f := range files { + newRelPath := d.fs.Join(relPath, f.Name()) + + if f.IsDir() { + if err = d.walkTree(newRelPath); err != nil { + return err + } + } else { + filePath := d.fs.Join(d.path, newRelPath) + h, err := d.readHashFile(filePath) + if err != nil { + return err + } + d.refs[newRelPath] = h + } + } + + return nil +} + +// ReadHashFile reads a single hash from a file. If a symbolic +// reference is found instead of a hash, the reference is resolved and +// the proper hash is returned. +func (d *GitDir) readHashFile(path string) (h core.Hash, err error) { + f, err := d.fs.Open(path) + if err != nil { + return core.ZeroHash, err + } + defer func() { + errClose := f.Close() + if err == nil { + err = errClose + } + }() + + b, err := ioutil.ReadAll(f) + if err != nil { + return core.ZeroHash, err + } + line := strings.TrimSpace(string(b)) + + if isSymRef(line) { + return d.resolveSymRef(line) + } + + return core.NewHash(line), nil +} + +func isSymRef(contents string) bool { + return strings.HasPrefix(contents, symRefPrefix) +} + +func (d *GitDir) resolveSymRef(symRef string) (core.Hash, error) { + ref := strings.TrimPrefix(symRef, symRefPrefix) + + hash, ok := d.refs[ref] + if !ok { + return core.ZeroHash, ErrSymRefTargetNotFound + } + + return hash, nil +} diff --git a/storage/seekable/internal/index/index.go b/storage/seekable/internal/index/index.go new file mode 100644 index 0000000..8e041bd --- /dev/null +++ b/storage/seekable/internal/index/index.go @@ -0,0 +1,92 @@ +package index + +import ( + "fmt" + "io" + + "gopkg.in/src-d/go-git.v3/core" + "gopkg.in/src-d/go-git.v3/formats/idxfile" + "gopkg.in/src-d/go-git.v3/formats/packfile" +) + +// Index is a database of objects and their offset in a packfile. +// Objects are identified by their hash. +type Index map[core.Hash]int64 + +// NewFromIdx returns a new index from an idx file reader. +func NewFromIdx(r io.Reader) (Index, error) { + d := idxfile.NewDecoder(r) + idx := &idxfile.Idxfile{} + err := d.Decode(idx) + if err != nil { + return nil, err + } + + ind := make(Index) + for _, e := range idx.Entries { + if _, ok := ind[e.Hash]; ok { + return nil, fmt.Errorf("duplicated hash: %s", e.Hash) + } + ind[e.Hash] = int64(e.Offset) + } + + return ind, nil +} + +// NewFrompackfile returns a new index from a packfile reader. +func NewFromPackfile(rs io.ReadSeeker) (Index, error) { + index := make(Index) + + r := packfile.NewSeekable(rs) + p := packfile.NewParser(r) + + count, err := p.ReadHeader() + if err != nil { + return nil, err + } + + for i := 0; i < int(count); i++ { + offset, err := r.Offset() + if err != nil { + return nil, err + } + + obj, err := p.ReadObject() + if err != nil { + return nil, err + } + + err = r.Remember(offset, obj) + if err != nil { + return nil, err + } + + err = index.Set(obj.Hash(), offset) + if err != nil { + return nil, err + } + } + + return index, nil +} + +// Get returns the offset that an object has the packfile. +func (i Index) Get(h core.Hash) (int64, error) { + o, ok := i[h] + if !ok { + return 0, core.ErrObjectNotFound + } + + return o, nil +} + +// Set adds a new hash-offset pair to the index, or substitutes an existing one. +func (i Index) Set(h core.Hash, o int64) error { + if _, ok := i[h]; ok { + return fmt.Errorf("index.Set failed: duplicated key: %s", h) + } + + i[h] = o + + return nil +} diff --git a/storage/seekable/internal/index/index_test.go b/storage/seekable/internal/index/index_test.go new file mode 100644 index 0000000..49b3a9d --- /dev/null +++ b/storage/seekable/internal/index/index_test.go @@ -0,0 +1,2289 @@ +package index + +import ( + "os" + "testing" + + "gopkg.in/src-d/go-git.v3/core" + "gopkg.in/src-d/go-git.v3/formats/idxfile" + + . "gopkg.in/check.v1" +) + +func Test(t *testing.T) { TestingT(t) } + +type SuiteIndex struct{} + +var _ = Suite(&SuiteIndex{}) + +func (s *SuiteIndex) TestNewFromIdx(c *C) { + for i, test := range [...]struct { + idxPath string + errRegexp string + }{ + { + idxPath: "../../../../formats/packfile/fixtures/spinnaker-spinnaker.idx", + }, { + idxPath: "../../../../formats/packfile/fixtures/invalid.idx", + errRegexp: idxfile.ErrMalformedIdxFile.Error(), + }, + } { + com := Commentf("subtest %d) idxPath = %s", i, test.idxPath) + + idx, err := os.Open(test.idxPath) + c.Assert(err, IsNil, com) + + index, err := NewFromIdx(idx) + if test.errRegexp != "" { + c.Assert(err, ErrorMatches, test.errRegexp, com) + } else { + c.Assert(err, IsNil, com) + c.Assert(index, DeepEquals, expectedIndexes[test.idxPath], com) + + err = idx.Close() + c.Assert(err, IsNil, com) + } + } +} + +func (s *SuiteIndex) TestGet(c *C) { + for i, test := range [...]struct { + idx string + hash core.Hash + err error + }{ + { + idx: "../../../../formats/packfile/fixtures/spinnaker-spinnaker.idx", + hash: core.NewHash("92ce4b53184a00f4168c66f1c66868a2706a3c1c"), + }, { + idx: "../../../../formats/packfile/fixtures/spinnaker-spinnaker.idx", + hash: core.NewHash("9244ee648182b91a63d8cc4cbe4b9ac2a27c0492"), + }, { + idx: "../../../../formats/packfile/fixtures/spinnaker-spinnaker.idx", + hash: core.NewHash("bfc5e621b13f5acdd1e118603bbb33a9074a40b5"), + }, { + idx: "../../../../formats/packfile/fixtures/spinnaker-spinnaker.idx", + hash: core.NewHash("8e6b090adae2a84beefa47822dfe646dc2ded4ea"), + }, { + idx: "../../../../formats/packfile/fixtures/spinnaker-spinnaker.idx", + hash: core.NewHash("3bdab2c86819b4bd5b631a2d446d73a1f84ade8d"), + }, { + idx: "../../../../formats/packfile/fixtures/spinnaker-spinnaker.idx", + hash: core.NewHash("1111111111111111111111111111111111111111"), + err: core.ErrObjectNotFound, + }, + } { + com := Commentf("subtest %d", i) + + idx, err := os.Open(test.idx) + c.Assert(err, IsNil, com) + + index, err := NewFromIdx(idx) + c.Assert(err, IsNil, com) + + obt, err := index.Get(test.hash) + if test.err != nil { + c.Assert(err, Equals, test.err, com) + } else { + c.Assert(err, IsNil) + c.Assert(obt, Equals, expectedIndexes[test.idx][test.hash], com) + } + + err = idx.Close() + c.Assert(err, IsNil, com) + } +} + +var benchmarkResult = make([]Index, 0) + +func (s *SuiteIndex) BenchmarkFromIdx(c *C) { + path := "../../../../formats/packfile/fixtures/spinnaker-spinnaker.idx" + idx, err := os.Open(path) + c.Assert(err, IsNil) + + var indexes = make([]Index, 0) // to prevent optimization of the benchmark + c.ResetTimer() + + for i := 0; i < c.N; i++ { + c.StartTimer() + index, _ := NewFromIdx(idx) + c.StopTimer() + indexes = append(indexes, index) + } + + err = idx.Close() + c.Assert(err, IsNil) + + copy(benchmarkResult, indexes) +} + +// how to extract offsets from an idx file: +// git verify-pack -v spinnaker-spinnaker.idx | sed -e's/ */ /g' +// format: SHA-1 type size size-in-packfile offset-in-packfile depth base-SHA-1 +var expectedIndexes = map[string]Index{ + "../../../../formats/packfile/fixtures/spinnaker-spinnaker.idx": Index{ + core.NewHash("92ce4b53184a00f4168c66f1c66868a2706a3c1c"): 12, + core.NewHash("9244ee648182b91a63d8cc4cbe4b9ac2a27c0492"): 275, + core.NewHash("87fe6e7c6b1b89519fe3a03a8961c5aa14d4cc68"): 555, + core.NewHash("9a399fd62ecf19b6cda56fb13ad560a0985f677f"): 782, + core.NewHash("98eb2457f4dad2d72f874a682c8ebfa56bd2c7ac"): 998, + core.NewHash("7611b64a58dffb89dbf1233e5583fe5a80d88401"): 1228, + core.NewHash("f39d86f59a0781f130e8de6b2115329c1fbe9545"): 1397, + core.NewHash("838aed816872c52ed435e4876a7b64dba0bed500"): 1641, + core.NewHash("bd87d73416fe156337ee40e377a8cf7795862086"): 2028, + core.NewHash("07d52b701de6e22250a7f69fd67669511d7abbb7"): 2258, + core.NewHash("7c8d9a6081d9cb7a56c479bfe64d70540ea32795"): 2509, + core.NewHash("cb44aca3f3401964ba79aba4694172425e3f89af"): 2685, + core.NewHash("7ecc2ad58e24a5b52504985467a10c6a3bb85b9b"): 2875, + core.NewHash("d081d66c2a76d04ff479a3431dc36e44116fde40"): 3037, + core.NewHash("3e349f806a0d02bf658c3544c46a0a7a9ee78673"): 3657, + core.NewHash("82562fa518f0a2e2187ea2604b07b67f2e7049ae"): 4929, + core.NewHash("48b655898fa9c72d62e8dd73b022ecbddd6e4cc2"): 6940, + core.NewHash("8b6002b614b454d45bafbd244b127839421f92ff"): 7486, + core.NewHash("95ee6e6c750ded1f4dc5499bad730ce3f58c6c3a"): 7738, + core.NewHash("0a3fb06ff80156fb153bcdcc58b5e16c2d27625c"): 7974, + core.NewHash("dc22e2035292ccf020c30d226f3cc2da651773f6"): 10514, + core.NewHash("3f36d8f1d67538afd1f089ffd0d242fc4fda736f"): 11058, + core.NewHash("8526c58617f68de076358873b8aa861a354b48a9"): 11269, + core.NewHash("776914ef8a097f5683957719c49215a5db17c2cb"): 11561, + core.NewHash("a77d88e40e86ae81b3ce1c19d04fd73f473f5644"): 11862, + core.NewHash("a174b873e97fb9a2d551d007c92aa5889c081a99"): 12124, + core.NewHash("8586b7cd3f70fe63053fd5fa321bc86c6b803622"): 12378, + core.NewHash("3525c938ab51af81cff2448c8c784b925af2fd0f"): 12607, + core.NewHash("b1b5146a77d363e136336923429134d0759eb9c8"): 12866, + core.NewHash("1ef157853d770a26e7682e543ac42de485b34f77"): 13072, + core.NewHash("1ea743cd62e8e60f97f55a434a3f46400b49f606"): 13248, + core.NewHash("8fe3f13ad04ee25fde0add4ed19d29acd49a5916"): 13491, + core.NewHash("855e3b979f1d65fbfbcc68df905dafb9945f3825"): 13780, + core.NewHash("2b1ab713af3789204594a45f265cc93858807e98"): 14014, + core.NewHash("52831ed7689ab0f481486f62e81d2b4e9e1c535b"): 14252, + core.NewHash("1b44b5467f78a3f5e1915b6fe78f7d0814c29427"): 14432, + core.NewHash("637ba49300f701cfbd859c1ccf13c4f39a9ba1c8"): 14668, + core.NewHash("4f3c7375fa7c661735a6a69beeeeac1aaa43f7c9"): 14833, + core.NewHash("a74422026841e05debdcc417190428b419a99f39"): 15090, + core.NewHash("d73f9cee49a5ad27a42a6e18af7c49a8f28ad8a8"): 15313, + core.NewHash("b260ce026a2505037876b4c21c0985882ff373b7"): 15558, + core.NewHash("608976766959bdb1b18eaa53b3ca33ee6782bc3c"): 15789, + core.NewHash("cfdd19354e2a3981484a7cfe4b0d95c9abce9296"): 16024, + core.NewHash("8ef83dd443a05e9122681950399edaa58a38d466"): 16252, + core.NewHash("769ce2a32e60bf2219ffb5b8467d62f71f1e4877"): 16483, + core.NewHash("b2c7142082d52b09ca20228606c31c7479c0833e"): 16746, + core.NewHash("d25148149d6a67989be79cdb7452cdab8d2f1a4b"): 16964, + core.NewHash("c89dab0d42f1856d157357e9010f8cc6a12f5b1f"): 17221, + core.NewHash("8a9804234551d61209f67b3c89f7706f248ae805"): 17429, + core.NewHash("b45ffa99a6daaf045043ab0b0d8bcf823f10e157"): 17650, + core.NewHash("827682091dd09c1887e82686e36822695b88bb1e"): 17887, + core.NewHash("4f9cd01b6e533c3b1261660b9cc3302879e5b303"): 18135, + core.NewHash("d1ff4e13e9e0b500821aa558373878f93487e34b"): 18368, + core.NewHash("1c370109898641253617a4d48d77f2c9b0a4ccf5"): 18600, + core.NewHash("8d1e069744321ff97cbefeaba593c778105c3aad"): 18837, + core.NewHash("dd7e66c862209e8b912694a582a09c0db3227f0d"): 19023, + core.NewHash("4cce5f988005be72dca910fb53e4b2f5802bf7cf"): 19191, + core.NewHash("0ae9771322873f03893180d90b0af5e3b30154e9"): 19364, + core.NewHash("e805183c72f0426fb073728c01901c2fd2db1da6"): 19594, + core.NewHash("f98b6099746b849abfb9d5b1db7e861363747be2"): 19787, + core.NewHash("52edbd4c10193f87f8f9768c92789637bfedb867"): 19968, + core.NewHash("d7a3eedbf9fa133d7c4366afae555a2ed46d4849"): 20184, + core.NewHash("9944d6cf72b8f82d622d85dad7434472bc8f397d"): 20405, + core.NewHash("6694fb99ca6fbf469798f1fb9386b55ff80f0128"): 20567, + core.NewHash("2b28ea424acc8f2817d3298c143fae68bcad91a7"): 20796, + core.NewHash("206033f8afb2609982fdc6e929a94a340bc80054"): 20998, + core.NewHash("6ea37d18b706aab813532254ce0d412843c68782"): 21227, + core.NewHash("fad219f07e362f97eda945790320f1f0552a919c"): 21461, + core.NewHash("9414750a933037ec4f0bc42af7ad81ec4f360c0a"): 21719, + core.NewHash("d6e6fe0194447cc280f942d6a2e0521b68ea7796"): 21956, + core.NewHash("e259e024b1c7a221e8329fb942a4992738bc81af"): 22136, + core.NewHash("b32b2aecae2cfca4840dd480f8082da206a538da"): 22311, + core.NewHash("8eed01ff4f2ef7c9c68ab031b54e0cf84a0b1cc9"): 22567, + core.NewHash("66ee9032d57be4bac236edec0e501aaa0501a57d"): 22739, + core.NewHash("24551a5d486969a2972ee05e87f16444890f9555"): 22977, + core.NewHash("d4b48a39aba7d3bd3e8abef2274a95b112d1ae73"): 23137, + core.NewHash("5ad50e028c59d67ae5d8160e685947582dc68f36"): 23389, + core.NewHash("9a06d3f20eabb254d0a1e2ff7735ef007ccd595e"): 23611, + core.NewHash("811795c8a185e88f5d269195cb68b29c8d0fe170"): 23777, + core.NewHash("c0a70a0f5aa494f0ae01c55ba191f2325556489a"): 23963, + core.NewHash("d6905eab6fec1841c7cf8e4484499f5c8d7d423e"): 24151, + core.NewHash("f5300bb86b22eda66eb4baef6b2a211c85f14690"): 24335, + core.NewHash("d3046b5b2f7aafa0832da6806ee8c7dab7d0da9e"): 24569, + core.NewHash("ca87222cb609773c56d43c960e8f0ade554fc138"): 24807, + core.NewHash("bd42370d3fe8d410e78acb96f81cb3d838ad1c21"): 24988, + core.NewHash("67f0a0f488b3592bb611391150f2e1d0ee037231"): 25160, + core.NewHash("638f61b3331695f46f1a88095e26dea0f09f176b"): 25400, + core.NewHash("09a4ea729b25714b6368959eea5113c99938f7b6"): 25576, + core.NewHash("8731e9edc1619e798a76fedb30b26cf48fa62897"): 25778, + core.NewHash("bcbbd656c19dbc47ffd5b247927ea99f3949c78a"): 25996, + core.NewHash("4584fab37e93d66fd1896d07fa3427f8056711bc"): 26157, + core.NewHash("e0005f50e22140def60260960b21667f1fdfff80"): 26328, + core.NewHash("e1a2b26b784179e6903a7ae967c037c721899eba"): 26569, + core.NewHash("c756e09461d071e98b8660818cf42d90c90f2854"): 26747, + core.NewHash("0777fadf4ca6f458d7071de414f9bd5417911037"): 26979, + core.NewHash("d8d031c1ac45801074418c43424a6f2c0dff642c"): 27331, + core.NewHash("626d23075f9e92aad19015f2964c95d45f41fa3a"): 27587, + core.NewHash("c24f0caac157254e480055fb605a71465d13bc00"): 27785, + core.NewHash("7622add2bc8c47d1a37244f39b94bcc187bf671d"): 28012, + core.NewHash("a57b08a9072f6a865f760551be2a4944f72f804a"): 28245, + core.NewHash("f69376bd065db787894bd2775d447c8d87d3b50c"): 28410, + core.NewHash("3b0f2a5fbc354b116452e9f3e366af74ce1f1321"): 28646, + core.NewHash("4bbcad219ec55a465fb48ce236cb10ca52d43b1f"): 28883, + core.NewHash("0ce1393c24c7083ec7f9f04b4cf461c047ad2192"): 29058, + core.NewHash("dd2d03c19658ff96d371aef00e75e2e54702da0e"): 29275, + core.NewHash("46670eb6477c353d837dbaba3cf36c5f8b86f037"): 29432, + core.NewHash("2b20a9a5149deadbe43227b70445bf6699fd3a3a"): 29695, + core.NewHash("99280af2aaf171fe056400938ae2dbf6d93d3736"): 29933, + core.NewHash("495c7118e7cf757aa04eab410b64bfb5b5149ad2"): 30176, + core.NewHash("a47d0aaeda421f06df248ad65bd58230766bf118"): 30358, + core.NewHash("079e42e7c979541b6fab7343838f7b9fd4a360cd"): 30570, + core.NewHash("b7b9e7c464c3c343133ed17e778a2f600b5863b8"): 30752, + core.NewHash("03e24883d2f0a60419b0d43074aa2b3341bb2a97"): 30919, + core.NewHash("237166c72299ec287d1f6ab96aea7af07a2df160"): 31152, + core.NewHash("855c220530cb8aa8e9ff2598fc873240bf4a543b"): 31325, + core.NewHash("cacc42e050181fd1f74069a41c642f038d395c2d"): 31557, + core.NewHash("d98e03d4dc5d87aa5a1b2a5dd74feb14de965128"): 31733, + core.NewHash("82c28940a4b2d6a7e03c9349a7c2a37c9e164810"): 31960, + core.NewHash("bb702a749521496ea7e542df78806671d8d8c657"): 32124, + core.NewHash("023d4fb17b76e0fe0764971df8b8538b735a1d67"): 32363, + core.NewHash("125eceff9807da34b6e6ad7888441e7d9b7d629b"): 32544, + core.NewHash("8a594011096b65f5b455254f95d2c7d99ec64c11"): 32762, + core.NewHash("01575d8fc3845c69bbf522f93cc4189f436eaf8a"): 32981, + core.NewHash("b41d7c0e5b20bbe7c8eb6606731a3ff68f4e3941"): 33160, + core.NewHash("66d1c8f2fa2e32c2c936679c8b10e2134b2ac187"): 33361, + core.NewHash("6eb5d9c5225224bfe59c401182a2939d6c27fc00"): 33584, + core.NewHash("46ac02f5fbeb5e9c026bd85ee56d828836e0c323"): 33759, + core.NewHash("23da1763950b26aaa23551d798e3a52f1526fcc6"): 33999, + core.NewHash("36152fb0265180a42b7a79be31848de9845a81b5"): 34184, + core.NewHash("ba486de7c025457963701114c683dcd4708e1dee"): 34445, + core.NewHash("743a148328362ff93312329de0165fab07641546"): 34670, + core.NewHash("c4a9091e4076cb740fa46e790dd5b658e19012ad"): 34893, + core.NewHash("b5c6053a46993b20d1b91e7b7206bffa54669ad7"): 35062, + core.NewHash("505577dc87d300cf562dc4702a05a5615d90d855"): 35221, + core.NewHash("921a8a191aff8b0333c08ab78803878fdc26e9f5"): 35377, + core.NewHash("370d61cdbc1f3c90db6759f1599ccbabd40ad6c1"): 35625, + core.NewHash("ebe1cd8da4246d8b9b3f1c4717e99309a00490f6"): 35804, + core.NewHash("9467ec579708b3c71dd9e5b3906772841c144a30"): 36041, + core.NewHash("88e841aad37b71b78a8fb88bc75fe69499d527c7"): 36215, + core.NewHash("bbeb98f59f4f0b373c7d764964d8c23522804ef9"): 36387, + core.NewHash("8eb116de9128c314ac8a6f5310ca500b8c74f5db"): 36612, + core.NewHash("26a83567b8d80ed7523fc1d5d1e13d3f095bb70d"): 36797, + core.NewHash("8980daf661408a3faa1f22c225702a5c1d11d5c9"): 37041, + core.NewHash("66ac94f0b4442707fb6f695fbed91d62b3bd9d4a"): 37383, + core.NewHash("a596972a661d9a7deca8abd18b52ce1a39516e89"): 37628, + core.NewHash("0a67d98c7a0eaa27bf6b62450f6f54aadbb961ed"): 37825, + core.NewHash("3de4f77c105f700f50d9549d32b9a05a01b46c4b"): 38083, + core.NewHash("7119ad9cf7d4e4d8b059e5337374baae4adc7458"): 38286, + core.NewHash("d4553dac205023fa77652308af1a2d1cf52138fb"): 38447, + core.NewHash("304cac16bddf7bfbcc1663bf408ac452d29762f2"): 38626, + core.NewHash("23a14bd9cbe1808001a88ce8218d5b6d0948fa8a"): 38851, + core.NewHash("01e65d67eed8afcb67a6bdf1c962541f62b299c9"): 39081, + core.NewHash("18fc95490bcee25a4669d9ab7640e729cef32df4"): 39253, + core.NewHash("889c1f2bafc2f74258f608a9beab14dd4a70edb9"): 39476, + core.NewHash("a6a4e6112a5009e53a37f7325620a93db7eadd9c"): 39695, + core.NewHash("e51871f45f3848ec1ed37aab052277198c98fff1"): 39914, + core.NewHash("6986d885626792dee4ef6b7474dfc9230c5bda54"): 40132, + core.NewHash("2c748387f5e9c35d001de3c9ba3072d0b3f10a72"): 40297, + core.NewHash("0c6968e24796a67fa602c94d7a82fd4fd375ec59"): 40465, + core.NewHash("3ce7b902a51bac2f10994f7d1f251b616c975e54"): 40629, + core.NewHash("65e37611b1ff9cb589e3060507427a9a2645907e"): 40788, + core.NewHash("bc02440df2ff95a014a7b3cb11b98c3a2bded777"): 40964, + core.NewHash("791bcd1592828d9d5d16e83f3a825fb08b0ba22d"): 41127, + core.NewHash("2ca8a27c3a580c6cdc8a2b2f125505c1dd8e9608"): 41295, + core.NewHash("202a9c720b3ba8106e022a0ad027ebe279040c78"): 41529, + core.NewHash("f374398787c77063419102cf148496536b14f098"): 41696, + core.NewHash("6e00810b67f04ca530bafaed08e38c37a62f3447"): 41940, + core.NewHash("f4b2d0d49b7acbfdb49dce383467cec2736fdb6b"): 42178, + core.NewHash("95b2293acb7cb0a669dd7b406b70288366485fd9"): 42401, + core.NewHash("50d61c1635571240580957f943083e491bfdde30"): 42562, + core.NewHash("781139444c98e0227b15c7b8c79ef99840fde175"): 42786, + core.NewHash("090f5a9c367921c9af8cef692027a9f45f80994f"): 42946, + core.NewHash("2557c02dc8e270887db688e0598525b6da478de7"): 43169, + core.NewHash("63f18233017af47a00dbf235a1f08274094ad15a"): 43329, + core.NewHash("6e799cdf0cd0cbfb35585864ede4b8979a8b6bdc"): 43548, + core.NewHash("702d5f2e04f078041e7fad888d695e6bb6d3a094"): 43707, + core.NewHash("d2f6214b625db706384b378a29cc4c22237db97a"): 43950, + core.NewHash("38d6211f73a9d991d9950e1da2161ebafaa868bc"): 44133, + core.NewHash("9e74d009894d73dd07773ea6b3bdd8323db980f7"): 44378, + core.NewHash("2a6288be1c8ea160c443ca3cd0fe826ff2387d37"): 44542, + core.NewHash("18526c447f5174d33c96aac6d6433318b0e2021c"): 44703, + core.NewHash("dc9d797155e27a1a62817b98908a584923879b46"): 44869, + core.NewHash("a3cdf880826b4d9af42b93f4a2df29a91ab31d35"): 45097, + core.NewHash("cdefbd8d7fced00e76ffc72a1cc1557af817466f"): 45264, + core.NewHash("c9c2a0ec03968ab17e8b16fdec9661eb1dbea173"): 45435, + core.NewHash("0c5bb1e4392e751f884f3c57de5d4aee72c40031"): 45596, + core.NewHash("582da9622e3a72a19cd261a017276d72b5b0051a"): 45764, + core.NewHash("2e63008dd04d1b4779d29c2fc9379444cfa6c83f"): 45935, + core.NewHash("d0e741a6343281c56a18cd68c611aa575e52a740"): 46201, + core.NewHash("d5558a62594379115b42412732313cd14ca34890"): 46428, + core.NewHash("c0ddf58fe5514f0a2f65059e30461f2358916e25"): 46665, + core.NewHash("89114d46ab89409d8c6c1751bf33a88bac328ef8"): 46927, + core.NewHash("7dddc9ce367d42af4b0541107e7aaa6106c90a8a"): 47163, + core.NewHash("b69b79bbc407ef76a2b34d509679fc8874600b01"): 47374, + core.NewHash("b7015a5d36990d69a054482556127b9c7404a24a"): 47576, + core.NewHash("c9a07affe4fa11d3f00d46f4c306f2c079338636"): 47749, + core.NewHash("3dbceafa3bb0c54d544363b82be2e2b3df6d220d"): 47979, + core.NewHash("4fe77d22ebca2b26235bcde046d517fb66770bda"): 48217, + core.NewHash("0e6568d7a5026b1c9cc8fbc52c2ace83a5555931"): 48444, + core.NewHash("ed887f6547d7cd2b2d741184a06f97a0a704152b"): 48676, + core.NewHash("4c13a32356813c6496d2310f31c7cda0ae903408"): 48843, + core.NewHash("3bcf0480d28e3616902264ca339c9f3581770a30"): 49010, + core.NewHash("687bfdc568c00a29d74f00e3af496243ca029129"): 49255, + core.NewHash("6dd6ee984c27853b62f14225552fbcf80b844de6"): 49431, + core.NewHash("84f0f65ce6d8e9d1e5e3b248370f7566321ccc69"): 49668, + core.NewHash("e3ac51979fe937a7d9fdd989d314c2ca7d784242"): 49831, + core.NewHash("845105f45b3f31be1a80553e0ba1d1ea58df31b5"): 50017, + core.NewHash("c407c48de6273729c624c9dc8426427309d1f66c"): 50283, + core.NewHash("de21ed0a0d27df1512e22f5226fb7626a30d6499"): 50486, + core.NewHash("23673af3ad70b50bba7fdafadc2323302f5ba520"): 50711, + core.NewHash("a4381090e10bd6363a4b89e871560fdfcc88fbdb"): 50877, + core.NewHash("f480763f8e9638e0c99b4a6ddfd8842e65371c8b"): 51094, + core.NewHash("11d6c1020b1765e236ca65b2709d37b5bfdba0f4"): 51313, + core.NewHash("3b0ca37e568ac8389141240c770889388d3065e9"): 51478, + core.NewHash("a4bc54c9ce60fffd750cb8f1d31034c431bbc67a"): 51739, + core.NewHash("974b775a8978b120ff710cac93a21c7387b914c9"): 51986, + core.NewHash("d2838db9f6ef9628645e7d04cd9658a83e8708ea"): 52181, + core.NewHash("1f9684d0e81b4c80400677e029a5d483ddfb2027"): 52359, + core.NewHash("6d38cc001b806d0161d18f4837b9e5dfcf29a89e"): 52603, + core.NewHash("ce9f123d790717599aaeb76bc62510de437761be"): 52842, + core.NewHash("d375f1994ff4d0bdc32d614e698f1b50e1093f14"): 53025, + core.NewHash("be54b2b7561c3d9fb8768f524fa197ca1dad981f"): 53354, + core.NewHash("b5d999e2986e190d81767cd3cfeda0260f9f6fb8"): 53599, + core.NewHash("cda6cf2be5027889bf94bd4d1c5a171422bf566c"): 53790, + core.NewHash("d33c2d1e350b03fb989eefc612e8c9d5fa7cadc2"): 54035, + core.NewHash("9a8fda3ad2dd768bda41b7f1d6ed5d2eca464f24"): 54222, + core.NewHash("ddaae195b628150233b0a48f50a1674fd9d1a924"): 54471, + core.NewHash("b1e656daad82952f628b730534e2cc4ae0a3c6ef"): 54654, + core.NewHash("58cbd51bd8f22392cae99e5eda1feb60bba305bb"): 54858, + core.NewHash("5a7e23fa48950d7d07dd861b63f0f2740a93a78a"): 55046, + core.NewHash("19260d46e6cad0b18357d330b9548cf636e4ed54"): 55304, + core.NewHash("d00d6ae4e4d35cba850681f55b334fbf86bce02f"): 55547, + core.NewHash("d46ae8f231382e59fff0312af5ac1567d7cf019f"): 55778, + core.NewHash("2010f3b711bac4bd7941d861402647b36db15d95"): 55932, + core.NewHash("01171a8a2e843bef3a574ba73b258ac29e5d5405"): 56130, + core.NewHash("3e207d5bd4dbce737066682d0e5897cd1d917fdd"): 56309, + core.NewHash("a3d4e3b16b4a467fb8c002faac8704deb8383aad"): 56482, + core.NewHash("c197da249529232a016dbf36c28c18dc39561e15"): 56666, + core.NewHash("d1b8f41e176ddec23a42ff63e9e7b809ebb1c1a9"): 56905, + core.NewHash("0bcfdfef2319a79225dc702949d34d138d9b0534"): 57092, + core.NewHash("69fc14f0cd6ae7d6efcc578a3140663e48531593"): 57288, + core.NewHash("a80d310d9ca42c3422627f2ecca7ee1dbefa602a"): 57545, + core.NewHash("007c4e0b3f8b4c5a8a61a6390bf2ad53aa7a9b8d"): 57734, + core.NewHash("f3967099c5c45f56ff135616819bee61caf5a554"): 57892, + core.NewHash("9ab77e0f8a9b430759c9cf66d3d95563ef03b210"): 58052, + core.NewHash("8c87c202e31ba19221f3f6f5c8837980f8ebcabc"): 58232, + core.NewHash("d198aac3885de411b767ef506e251c5537c79b60"): 58417, + core.NewHash("80abec748bbd8b39f3a3129c4f019012bfb5c2b3"): 58659, + core.NewHash("c8d88bb7714e45f865962cf3458fae45e1d7609d"): 58893, + core.NewHash("d2b3448bd1b5f77aae9a9fc4907507f22a09a1fb"): 59136, + core.NewHash("6a31f5d219766b0cec4ea4fbbbfe47bdcdb0ab8e"): 59330, + core.NewHash("4704b1e5f09d7203b00a242a7df8020e7ce6adc0"): 59561, + core.NewHash("1f26252e27cc14d8cf2f611e4bf2b27efa3f71d6"): 59800, + core.NewHash("5344429749e8b68b168d2707b7903692436cc2ea"): 59973, + core.NewHash("ffb4b8a8a84e0a25ee8f1f70ee21a00c8a3529ca"): 60137, + core.NewHash("0b9a123c7a68f0cb072d0e589b21f3859ae1557d"): 60334, + core.NewHash("76abc6eb9859494825a4c3a59527240de57189c7"): 60530, + core.NewHash("bf3fcc0f0a052d531119b63ad45c4dc6e498cd8a"): 60715, + core.NewHash("a016421fce2defa9a9e72ef1ef7416c34eeae232"): 60893, + core.NewHash("98be8de120604c7ad42829d9073e17588cf5aeea"): 61092, + core.NewHash("2c588826ca3e15c36960fcf7ef982cfc04f62763"): 61289, + core.NewHash("427af6949a88f076bb0cd6925071c21be66b41a5"): 61457, + core.NewHash("41e96c54a478e5d09dd07ed7feb2d8d08d8c7e3c"): 61704, + core.NewHash("e3f31c79bba1da4d60f259089199036f6dde355d"): 61908, + core.NewHash("5a2a845bc08974a36d599a4a4b7e25be833823b0"): 62140, + core.NewHash("a56ccc92c9f7b0f9beb0905fbaedbdc5516ca0a3"): 62313, + core.NewHash("f98965a8f42037bd038b86c3401da7e6dfbf4f2e"): 62520, + core.NewHash("7d74ab70f670c01b404e2f38590b114656c5dfd9"): 62698, + core.NewHash("b95e442c064935709e789fa02126f17ddceef10b"): 62858, + core.NewHash("0d9c9cef53af38cefcb6801bb492aaed3f2c9a42"): 63049, + core.NewHash("56dc238f6f397e93f1d1aad702976889c830e8bf"): 63712, + core.NewHash("fbd869e597e89cc50d3b5ccdac99c9fb6273c77d"): 63955, + core.NewHash("7ef99b88389c744524e123b5053061768c9383df"): 64129, + core.NewHash("66f1c938c380a4096674b27540086656076a597f"): 64355, + core.NewHash("7682dff881029c722d893a112a64fea6849a0428"): 64522, + core.NewHash("57c59e7144354a76e1beba69ae2f85db6b1727af"): 64675, + core.NewHash("fda357835d889595dc39dfebc6181d863cce7d4f"): 64848, + core.NewHash("b0c2aa0faeacbf9fbc144e1a375571f3d3031715"): 65016, + core.NewHash("b80c66c904860d2c400a191b2892fa473c4735c2"): 65233, + core.NewHash("1d7d03a6cf8e17ded81e24e654a2ecc30050453d"): 65440, + core.NewHash("31ff9c2f840cad649832b8138c188740d193b4b5"): 65622, + core.NewHash("01e89f023761dbc1552629fede1094de6a265abd"): 65848, + core.NewHash("1e3d328a2cabda5d0aaddc5dec65271343e0dc37"): 66018, + core.NewHash("d287c606d356e8d978b9673f5445b27a74ea8721"): 66193, + core.NewHash("dd52703a50e71891f63fcf05df1f69836f4e7056"): 66421, + core.NewHash("eb81417e29bf40bf358231ac601e2141f7a38051"): 66600, + core.NewHash("eb4faf67a8b775d7985d07a708e3ffeac4273580"): 66860, + core.NewHash("c48cf60958007d45f9c4c8746f00f342236825ee"): 67054, + core.NewHash("cbb13e91c2167a4c311f6759fe9df27344417b23"): 67283, + core.NewHash("93ed7a014cb979b66fe288065c9ce3a38674a555"): 67442, + core.NewHash("e3620408a7776039f7853a21748921ea2a281953"): 67596, + core.NewHash("0f84364305a6bb71290c6698e9fd45aa35a935be"): 67870, + core.NewHash("491dcace3776dc1f5f1d13a0c8d933ed0e4f79e1"): 68131, + core.NewHash("46c9dcbb55ca3f4735e82ad006e8cae2fdd050d9"): 68362, + core.NewHash("ae8576d17ed71715823ea35217f0460cceebdd65"): 68527, + core.NewHash("37f94770d81232b1895fca447878f68d65aac652"): 68781, + core.NewHash("706a6768f677c653420ba61eb3bd5c42113e609d"): 68978, + core.NewHash("c57553ee668b101d554ccef700b2cb7536c243af"): 69241, + core.NewHash("98fa29596e058d0cf253fcdfbc769775e1e5f876"): 69429, + core.NewHash("f42771ba298b93a7c4f5b16c5b30ab96c15305a8"): 69684, + core.NewHash("e9e1dcea75aeb328ffb7fde8c2ef6762fa8c32a0"): 69876, + core.NewHash("01abdb429572c3cda9da8ed6ac06ca5dd5298252"): 70127, + core.NewHash("75103952c05a04b4671a5315ec7bf54c227abc8d"): 70320, + core.NewHash("5ed9f4d0241e410596e86f3f6cc68e3bf249f231"): 70576, + core.NewHash("781281256abd0250ab5714094c0e6bfccdfb4a67"): 70754, + core.NewHash("1d6def8ab09cac9f46d0155be7f19d2a2faefcaa"): 70927, + core.NewHash("61cc6aa50a45f25bca0c6b14f8e16103d0aceb06"): 71083, + core.NewHash("8092635aa6b22f371d055b191664c700742567c5"): 71275, + core.NewHash("007b3dabe336c08181da1298fd2f4f8f13221689"): 71499, + core.NewHash("2865290e6d7cbc7dd3571e104e9e685d250e1200"): 71760, + core.NewHash("28043eabad05ba3fd2bb10e72487b8d514583532"): 71942, + core.NewHash("387dd11027dfad5f594a2de749ad957e3b4a4b2f"): 72104, + core.NewHash("c419210f896eacc981a03a74c10ed4b611974e7c"): 72313, + core.NewHash("2abcb2a8001fe9205bdc94e1732ff68bf1e0d620"): 72488, + core.NewHash("dbce4d7cf9cb27a3af33f01a682407dc5064c8a6"): 72681, + core.NewHash("3d3819741bada73fb950f14309c97e1f63492ec6"): 72874, + core.NewHash("a92c3ea045e8338c461b0d8cd769c7c5e8abe9f2"): 73078, + core.NewHash("f9594594c18bcb7b3610eea25056eb1844d5131e"): 73288, + core.NewHash("caf6d62e8285d4681514dd8027356fb019bc97ff"): 73550, + core.NewHash("95826fef343fac115001ce83c7f18e8dedc9e618"): 73744, + core.NewHash("ba10a5d5615f68eb9115cada1d639066a53ddc4d"): 73980, + core.NewHash("18d3d4c3c2099e3436ddc5c3f5f417e447c71ac1"): 74148, + core.NewHash("90feff1b58170390bf2ef196a3b29be25e5dc99a"): 74374, + core.NewHash("13ad4df676a16caf2ff1ca216be615d5aee37db3"): 74607, + core.NewHash("4890f706f8083138e3c316a8331fcab4db910d06"): 74780, + core.NewHash("eb40d65e5d93d7c147b753f67d0acce4aa672fc2"): 75025, + core.NewHash("b192881117651465df2385ef89344dd5dd4810f3"): 75286, + core.NewHash("7aef2b4078a4cfe7f582ce9e8592e096e13eb684"): 75487, + core.NewHash("7cc54563eea18c0b815a13fca3a56f5507023067"): 75717, + core.NewHash("3165ac99ecf6f7bc07cc6c87104cc00dc571a6c2"): 75966, + core.NewHash("6c11c723343b15a5a8c55441c241e44aae6ee4df"): 76151, + core.NewHash("c74e6f2189aee4fd3af86e7863cb26e9c7f8cab9"): 76367, + core.NewHash("1313ac890348258e9bf67d11ed10a5c8ac8f8e7a"): 76553, + core.NewHash("7ff85f71710dc4c1f1013b1f8d13ff6c8cefac12"): 76774, + core.NewHash("188041dd2cf20f0a2c8cf9e5cffd6c5806f8a816"): 76934, + core.NewHash("7cea14a31ec3cfd0a14faea81497b2bd0eeeb418"): 77173, + core.NewHash("add0c67315a29542fc2c74ecd76ca1955b5f8249"): 77339, + core.NewHash("fcc6154b2e702dca3079e2c7d5acab8612019191"): 77570, + core.NewHash("3efad7ca4e1136ab34a16e7009b0200310716d80"): 77743, + core.NewHash("deaf78f003697e2e44dbf9022ec6b817e950e384"): 77969, + core.NewHash("8f92c6f3acb04dc686681e3a5a089cbcf6a71ac8"): 78123, + core.NewHash("365bfc30eb2b494d3598459a2d93ce0364a7fc9c"): 78348, + core.NewHash("cd8a29d2d728fc3d2161db801f8abb2bc2430d57"): 78501, + core.NewHash("fc28a378558cdb5bbc08b6dcb96ee77c5b716760"): 78719, + core.NewHash("0a287f95cf1788d7e262532cb10037c3ddcf03c8"): 78869, + core.NewHash("5d3a01867d89982baaed276c10c2c34cb8ae9c89"): 79089, + core.NewHash("4e9b6422e4c3c604a66436a47d2d47349ef92463"): 79243, + core.NewHash("6d6e3750cb43d482fe545721754d57e42578d6bc"): 79485, + core.NewHash("74573f07269d97a8d8c6c0bcba413a2cd46e7a50"): 79731, + core.NewHash("707bcdce04eabdb0549868ad1a8efa2d76b9bdf1"): 79956, + core.NewHash("df5a121bb5b57e4abc96fdf688cec4cedc325730"): 80143, + core.NewHash("0c7cfc51c1da280742bf7179d6d8816e97ffd832"): 80369, + core.NewHash("a92fccda2221d9f8849d07bc0a31ea9979e3618d"): 80581, + core.NewHash("6a7df07a9dcb59e5a0053cd2da5331fb5e85164d"): 80766, + core.NewHash("3981a29179150646363bf1fc5aea600d76ddacd6"): 80924, + core.NewHash("eba39b8b58087a5b77303b5b968db58b9b8fa559"): 81159, + core.NewHash("f2081c9ffc8ae0d1298757265a11708d48722ac4"): 81329, + core.NewHash("f52151597c84fdb45feec61e55ffd08c35bcd413"): 81504, + core.NewHash("89eac7e84400db93b750414a5d52569694b9ed13"): 81657, + core.NewHash("d14f793a6cd7169ef708a4fc276ad876bd3edd4e"): 81875, + core.NewHash("99534ecc895fe17a1d562bb3049d4168a04d0865"): 82102, + core.NewHash("edf909edb9319c5e615e4ce73da47bbdca388ebe"): 82325, + core.NewHash("a24001f6938d425d0e7504bdf5d27fc866a85c3d"): 82507, + core.NewHash("ae904e8d60228c21c47368f6a10f1cc9ca3aeebf"): 84110, + core.NewHash("97561116dadfcfec7818e34b4235352e3b71d6ed"): 85802, + core.NewHash("ee6789891a4b7348e7c47f6fb85a2ad57ce16c57"): 86053, + core.NewHash("b79b41d9d0050b120669391a2c88cb1afa73789a"): 86398, + core.NewHash("fe4dde1cfd490bb6e826dc15e75f20f7e552a408"): 86582, + core.NewHash("31952f367d97daace08e8cc1c3716df2994c9adc"): 86815, + core.NewHash("83a0780f14f82e4b21ee2740af3eb2547f4dfbc4"): 87202, + core.NewHash("e02fef1104928efe1c069e6f0e92cdf41cb05395"): 87356, + core.NewHash("395daf1f76399ecc382d2fdb420de95b44c23697"): 87533, + core.NewHash("78926630ea276e2f56d7aa6cf498e72d76a6407a"): 88182, + core.NewHash("4ec7ac74dfc619af598a0e4be6d2803efa4d209d"): 88427, + core.NewHash("26f8d1f21d74cab5d8244899155578053f966c31"): 88614, + core.NewHash("17d5474689ae2ea2b306f258521c4f2bb6dc0bee"): 88828, + core.NewHash("dcd0570f7cb1f67080b30c8f87aadaff3bbe1d32"): 88974, + core.NewHash("9ee355c816e98398c7f2793d8c18a994ed38ebc8"): 89190, + core.NewHash("3cd71a4351ed5ae2cb87f226aa9d5ddc322c7c23"): 89452, + core.NewHash("2afb036947b1e7387af3918f04569e120f3eb50d"): 89688, + core.NewHash("e1eb63e490e837bbdd7fa6123780043ff8a875d7"): 89860, + core.NewHash("654fbe6c7beb51c17cb6fc4b9761c22dfc79faa4"): 90165, + core.NewHash("f7bbb3204807a1ff36955e0ef57d14688d303e80"): 90415, + core.NewHash("92084867f31e52fd269bb4a4dfcfef2ab2dc7954"): 90654, + core.NewHash("948a2765d59f04fc2deeb3d7c69b08c5f19aae94"): 90875, + core.NewHash("ed7776861ba44a6c4f8db22413018cb439ee6635"): 91119, + core.NewHash("d493494dd9480567b27d51b8277e677e327d8880"): 91356, + core.NewHash("c4feb443f683fffd330a1d8b157f605310358528"): 91527, + core.NewHash("a1618fb227af4861127f1662bb9ab1a0120f04c9"): 91769, + core.NewHash("55178fa0dc982da9d82962142f03e0887b4fbf1a"): 91940, + core.NewHash("db51150770b43cba2432c7f9b7a8ba95d7b372ab"): 92122, + core.NewHash("7957b63ba852e308640fdf10b30f52c8d12fe94c"): 92344, + core.NewHash("24acae3d38ad90aec908ab4119bc326950724276"): 92504, + core.NewHash("d935ee02d8458ba13f0a71841e2c237b0bd57ea8"): 92717, + core.NewHash("c07c5c0ed831a8cc37d4de8c55168718e724dc39"): 93180, + core.NewHash("30c8555d0800223e07cad213b8f391053eed5cb8"): 93428, + core.NewHash("7c92f19677b71c86d0e12ab663397995bef5514e"): 93823, + core.NewHash("4e815dd30eca232793d0172a549c8229658ac8e5"): 94064, + core.NewHash("0891b49db89b76fba0af78e1b7ee4f31d45e8a00"): 94232, + core.NewHash("4aee88ecb1f0e302829d3a8efec7412e2e96747d"): 94457, + core.NewHash("76b38ca947c89202ac4023064101979ed6a6dba6"): 94679, + core.NewHash("215fc72e255d17ad2db89e5eda91f6d596eda6f2"): 95205, + core.NewHash("f2b2935a3b5a17b50323b1bf6f7e2abe2219f33a"): 95425, + core.NewHash("2c9bfce112f45124ea72d7c0c5d78445801395db"): 95661, + core.NewHash("5b20fa56654b452203393787b3f0b2336fa93f2e"): 95839, + core.NewHash("a902f28b8a599ecbbf43cdbfd1de3138586cedec"): 96069, + core.NewHash("f3e972d2431670f72e1491b93269e836c4df4df8"): 97164, + core.NewHash("8e7d3e6c8d28b056d8f58e5b97f931e6664a05e9"): 97333, + core.NewHash("a8d3afbfc9f103a7c3e07af01c914205411cbf38"): 97556, + core.NewHash("772fd498643a03225dc54a2b6d6a62a050bfbcdc"): 97728, + core.NewHash("28ca0139a298f4817323e395dca68c294637a643"): 98220, + core.NewHash("b025f2f61176f133968abcb2bec7f469fc501ac8"): 98440, + core.NewHash("42c5b4a6d7ab60e51b223b87c043cd28ee2a5b56"): 98674, + core.NewHash("0627041659ec67227bba9eab7c8111c1d4faed41"): 99201, + core.NewHash("743b666fe84abf88dbd94ab0a9e3c0f9b5d02095"): 99443, + core.NewHash("10a6351bd5f76b7ae26796caf85464a2800e7067"): 99629, + core.NewHash("e397c0f180213ed3a671aaf45103e03ca0887f98"): 99853, + core.NewHash("0ca9f292091ae9801a41193ff79bb09e3225494c"): 100023, + core.NewHash("9657e8bbf97d5db64b30d2206ac5a76b5ded863c"): 100558, + core.NewHash("ad6c622d0f1f517eaae7a16b355ea24c019fb37c"): 100801, + core.NewHash("02f616225492ef685ac73d1b4069c1d3dc8e5616"): 101049, + core.NewHash("4f44d5e20a5bf8d5d3fee9d9667c7c3352f8778c"): 101281, + core.NewHash("0c29fad64e95ee215c61e373b49919b8e65b0db4"): 101955, + core.NewHash("590100881e7db3ea5dd7ba9c3d4c79049a67fbd4"): 102853, + core.NewHash("38c1c9275d9aeb76e5c893e9861332aa0deae248"): 103110, + core.NewHash("77cd24c745c175b90eae2caf89afb35ad9e05b74"): 103314, + core.NewHash("3c503defe4289f08e0612dd24c2a5a38981bb05c"): 103561, + core.NewHash("f544460397571d9bc0a32abc29f7523155023b4b"): 103745, + core.NewHash("ebe96fbd57da19e3d7c736c96b347cd1435cc35e"): 103999, + core.NewHash("732822e51ea396ae3be0f108ec5f89460031ac7d"): 104194, + core.NewHash("86741e24a84d3df5c07b6a5990a212d2d9711e84"): 104436, + core.NewHash("712b6f62d321702d7b4adbf2ea60fbc7819beaf3"): 104620, + core.NewHash("41d080fe4f9907d72be2e8548abd3d8decb9202d"): 104843, + core.NewHash("0b14e52ec7a8b6c4ba882152f35a1144c776a6b1"): 105077, + core.NewHash("0fab83b9f425edf77eb0c427e5a032356f19f7c7"): 105328, + core.NewHash("e1900abcda8e2c0ae43745070be05999831224ad"): 105571, + core.NewHash("2aa2a3a6cc247786e5794be1dfca3ce89da8b82d"): 105761, + core.NewHash("ce3a710d1d965f6f0b27f910d4e8bcaded5e4421"): 105985, + core.NewHash("b358803ecbad68c71d0e1864956fb4f80565e014"): 106146, + core.NewHash("e24e5ce81c529e1df4445f0138e691c69f8b354e"): 106396, + core.NewHash("32a0da9dd2805c2fddeeec7d912c84a195949729"): 106555, + core.NewHash("79fcb63e5cf7fe69480b3cd098b01a91651f648a"): 106794, + core.NewHash("13dd4d00208a7292d5e7f1f04c1984398d947edf"): 107020, + core.NewHash("eaf32b12380ed5d732f1cbdde7a9e112f404cb37"): 107338, + core.NewHash("0f8aa7151b12324f103143bde7a677f6af40bc30"): 107570, + core.NewHash("0285f896581afdae3a7f651330927e28235fb46d"): 107742, + core.NewHash("ec22b5a7b80db8cd319ce7656b536c7e3d94ce03"): 107959, + core.NewHash("aad9e27c6959ec0d40c1a983994c3080179503bf"): 108107, + core.NewHash("c688c6777a29018ca498a811a4e081f3e7a5108b"): 108361, + core.NewHash("75a1157ed447eb50d7bca00a2422e30807a707db"): 108877, + core.NewHash("3aa2dc855b3413b0ac77cac630e36d5f35f5ef7c"): 109099, + core.NewHash("3d485aea182864ec06ed40a3d7ad11b2aeeaf343"): 109254, + core.NewHash("0b4fccbb021c54a2ff9dd0145b9c17bdc2c3517f"): 109498, + core.NewHash("e46035abac831d960fdef6c3093cbb2fb3f2f659"): 109691, + core.NewHash("ec53a79f0d0d78f112de63b65ef963d4bc3b4f22"): 109920, + core.NewHash("4190cc9721162831c88f705bf78affda29d39e78"): 110254, + core.NewHash("f1769fcd9e138c714638faa18b098678aa69d081"): 110502, + core.NewHash("1a36355cf487b619f78a8b77bc298647aed2f93c"): 110863, + core.NewHash("fc96dd5f8c43c211ebdc12435ebae814cb0b6f14"): 111094, + core.NewHash("cec0197d85ce66b57015e383e49873eb1e0e4c11"): 111320, + core.NewHash("950d580008b14e85e575e62e9bab70cc7c2fe304"): 111493, + core.NewHash("f24f6059f0ab66bb1305e38cd9f7f019adcb35b6"): 111733, + core.NewHash("50a1c011871b3308466f70e75af14a1f9a7710cf"): 111913, + core.NewHash("ab60631006ac2b80b6a1e4b049acf4ad76f801c4"): 112135, + core.NewHash("ae9648e3141c029ef296052e359d459c57cc4efe"): 112299, + core.NewHash("00644b8d802c41af834fb47f654db37852f85714"): 112517, + core.NewHash("7fbe35a33b39504f4479e5909699f6edb9296be3"): 112675, + core.NewHash("1f314cd47ddd9759eb588521831ca148e5ad86a8"): 112903, + core.NewHash("961a7fa45290ce2ebc4175435880348353d4a7ed"): 113161, + core.NewHash("c2d498e471789824efebb4bfe134a04062543cf0"): 113400, + core.NewHash("88dafdfcc65f033bb61150804e55c6ddeaa5f9e4"): 113703, + core.NewHash("ac64b14edef612dbf5df002c63105681c664f388"): 113949, + core.NewHash("f773aebd781933b2d14ebdd92c3e86a80a8db4be"): 114125, + core.NewHash("6ec7ba107d8c163b70f8e0393debef2f96971433"): 114350, + core.NewHash("cb402a712eca3ee749803c8f0a22e923ce77c92d"): 114521, + core.NewHash("411a5cf5d22ac5931f260dd57d124f2ed63539ca"): 115281, + core.NewHash("5aba13ac173bc134dd438ca456a49215a7a4a781"): 115530, + core.NewHash("06f01ff1d2b7d503eb7402c7ce3c82ba904751b7"): 116035, + core.NewHash("05240995fcc556b5435c26f8a8d74ff8563d493d"): 116297, + core.NewHash("64e30ddf971b6b8000c2d0c385b1e388e5291af2"): 116479, + core.NewHash("aefd3ea7d786d4c539b63d9eb685b898e35c97de"): 116709, + core.NewHash("8c9cd1399bb7fc3099798c70de08fc4409df2890"): 116900, + core.NewHash("cd4bb35a91a40740e600bbad7023929b94350702"): 117478, + core.NewHash("52259c872b323eecad6b758868f59cb93512f20b"): 117657, + core.NewHash("145be49695eacdbfd43ef2cc6390472c1986f9bc"): 117892, + core.NewHash("3cd084d57a011945f5b3477e0363fac655ba275a"): 118224, + core.NewHash("348cb5136b959c53ef2e3e7d7678af1054c2c141"): 118485, + core.NewHash("e7220b6455102145c001f3a985152ce27a1d704b"): 118850, + core.NewHash("8a8ba3394d9457b61ec7c984af13d0f58b31fe58"): 119051, + core.NewHash("232a7a26a5b39195994aabe7c2e8a54a51aeca14"): 119209, + core.NewHash("2aae294033ddd003be7dcc25ffe8cbe43869f76a"): 119374, + core.NewHash("1c5542ff92a88695ae377fa36459174d03490d84"): 119560, + core.NewHash("aa0c40e18510a1e3ad3f6fd7b913a09ffedd5490"): 119783, + core.NewHash("5170abf34bbf490cd6d74fbeeea18506fb3bf1a4"): 119941, + core.NewHash("b42f76ad3c2a6684625d76712be5e91cd48cee62"): 120168, + core.NewHash("acd586068aca04af20e217ebb788d30d7b6c19ad"): 120328, + core.NewHash("560a2d6cb40ac6b52167ce5e593dcd3a51907837"): 120569, + core.NewHash("b4126478ad41e20a2a79f9f522cdf85dcdbe408b"): 120743, + core.NewHash("2b3fac174db42aa7944d6e606a17d5ca1ae66715"): 120904, + core.NewHash("44a1cdf21c791867c51caad8f1b77e6baee6f462"): 121055, + core.NewHash("ecca0a3da863192881bd0d9b13d0990f5c08be77"): 121805, + core.NewHash("53de2d16e7a9d828e51c898020d3da85d39ab07e"): 121931, + core.NewHash("c7466f51b88b9ceec551601d6f73ded0595f6f2f"): 122303, + core.NewHash("f5ae52381dd77d19ab516e522d83616856f353ce"): 122799, + core.NewHash("48112f982a38f0be464dbe1b10c37bf665a4b329"): 122850, + core.NewHash("da7f9a6c663f911b310c11c59e7525b1e701d6e2"): 122902, + core.NewHash("ae95633f19d834978696f52621f158d9d8cea4dd"): 122950, + core.NewHash("79c65af2cada2d4842020d1e67251294d6154310"): 123002, + core.NewHash("71585dc412e0af8ef8c3e6658502ddb898975f6b"): 123082, + core.NewHash("2b5c476fbd968005a84004468dc2c203ae0b5259"): 123236, + core.NewHash("fb6b9703dc3ee4ea6b9f5a05e35401310134e1d5"): 123358, + core.NewHash("74ed41cc3e3f02f4d84b8a4be4ba1d357474a18d"): 123448, + core.NewHash("d03a21efb29de842d455df7752af14970dc4bfd8"): 123593, + core.NewHash("e4b77202deff7124e4f06173bf54a41f07c119cf"): 123702, + core.NewHash("198e5a9fabf29f69357c68b4627d3743ac31f851"): 123776, + core.NewHash("5fba7b5285ff9bb2e369dda3f4279cb53b755808"): 124436, + core.NewHash("89da6259bc845338dd0a667586abc40ef69e3bf3"): 124539, + core.NewHash("bc7ae878d5cac271ad993dd08afbad7e10e23dd3"): 124592, + core.NewHash("05ade58fe7d02805698eb680e54e446ab54640fd"): 124635, + core.NewHash("8126934a2241f12a6ad7a80dd10184939077de2e"): 124884, + core.NewHash("991a96ab17855977be9b138808b01f985d15f47e"): 124958, + core.NewHash("6109d0b7176ce2e8875856ac6fb7c5a2e23880ed"): 125032, + core.NewHash("1d86c8c4a18ebc98b8e24ec13706904227b6e0f5"): 125211, + core.NewHash("b9e1f25425699f639f73689feabd3aecf082a8c3"): 125285, + core.NewHash("feb4beca463565d164d688cc1814c8c157d72d00"): 125359, + core.NewHash("15f40c5c384735465c7c61719673c164100c72cb"): 125540, + core.NewHash("b252fda1554bed0786e07e06a61d1f57f4865e79"): 125614, + core.NewHash("1aeb739c529a0018ba1b264a8d925b8f1aea07ae"): 125772, + core.NewHash("9557711f5a026af26a44af79fe665b950ecbb518"): 125851, + core.NewHash("88dc0431a60ad8189f4008f7cbd10a1b8377c2c2"): 125971, + core.NewHash("002f5e15b428af761690be5baffeb1e402182c58"): 126129, + core.NewHash("383d556717c865eecc29019661ffec941cda02e6"): 126203, + core.NewHash("4a3b28ff261e00073e757d5b030a9b7d90f87dc1"): 126541, + core.NewHash("1e673727d23768987e51b67feaf053bc3ce46e40"): 126615, + core.NewHash("203a2fee60ed4e117ede27b4fc4e7eb38841c28f"): 126694, + core.NewHash("ccc6bc2afc4f528a070ac09b45a41b0ca2853248"): 126814, + core.NewHash("3f54f0556f6ec6740f244f934a105564e2864e7e"): 127342, + core.NewHash("569eafe9e54a82f3fba107909ee8bc9df38d8fa4"): 127501, + core.NewHash("7206e45663c49e54d85ad85f083000af5e543fc8"): 127575, + core.NewHash("80ba87afe057db52739550073b184f94a2e923cb"): 128325, + core.NewHash("4ee568e1206c8e4291d9f0e1e22d025de2378c01"): 128399, + core.NewHash("ef36f3001a5deb486d396859ee4bce374be8830f"): 128452, + core.NewHash("ef54a5ac0686e736a634c35b1f701a361b16be96"): 128495, + core.NewHash("4911a7a517ae0fe6f5d4542b70aabb6bd7fc9593"): 128586, + core.NewHash("144f93dc1f187a2a7677fc1fa3f6eb4df3534f38"): 128829, + core.NewHash("ac24d50e8d987a9ebe0f472d76360cbb334423fd"): 128872, + core.NewHash("1bce6ec6d84143c4d28031798c0f03ba54eeb888"): 128963, + core.NewHash("753c40e928df990b12b4bb45c08819a5aff5e05f"): 129112, + core.NewHash("c9f2e3dcabb0600ccb74012b72cbdd03aee1c750"): 129206, + core.NewHash("dcd6f30eeb43a31ccea0b76d16ffa63d3bcebe55"): 129250, + core.NewHash("b1c92c2c49f373d51cdc5fad192ef3f1a2d1685f"): 129299, + core.NewHash("2aefc1578f687d26983cff0317f31544964597f2"): 129352, + core.NewHash("3f2d3955e557f86b036bdebea13260f54065bdfb"): 129397, + core.NewHash("5dc8542b710ce2ca028c39a035ee0783e9591f26"): 129474, + core.NewHash("0a6800614d09ed20475a89351af0dc766758efbf"): 129513, + core.NewHash("7af836ee10e098936f4b7743253e67783dfacfc9"): 129556, + core.NewHash("711957a8ae7b18569600145c7f9ac54662f1c4aa"): 129602, + core.NewHash("a79e8f494e0e3aaa8b470ceb5c9281a77c22b4eb"): 129655, + core.NewHash("d3a06382dadcabd7ac3f87525126ad9d430b0441"): 129811, + core.NewHash("bf1046d1cf38e6b49b58d0985a4e9be488ace2cc"): 129930, + core.NewHash("9db6e0846d313e7bdc5f65105bdf854a23476d41"): 130004, + core.NewHash("c016a4fd9f57b8b230745c7187e8e27183326e10"): 130112, + core.NewHash("77fb10aaef90dbf1acf2da9048a69f9ca35b2e43"): 130221, + core.NewHash("158f8df35f0c88e3f5e384dac9c27be997fabd04"): 130324, + core.NewHash("f403c26945b5a27a6e3af917cc53737f24e8029c"): 130372, + core.NewHash("95d381877c016138eb9b6f070336037a27eb5bea"): 130491, + core.NewHash("03b40d18ee5bb4bc0c6a993ed4fdabd10209ae71"): 130565, + core.NewHash("3580bd0fec45863f7e4d88a24d7c3a7e86f48752"): 130674, + core.NewHash("972792d0fe893657c276fd71e05fde2956d4c749"): 130783, + core.NewHash("f284c81fe5f0e83f09b25931a73bf8fb6943427c"): 130834, + core.NewHash("567143ffd6cd1437f18ef5c1b85ed6ef0ae70ae4"): 130887, + core.NewHash("cd6c6a0a00e7a7556c44178b37260c2480e10af5"): 130996, + core.NewHash("0cf35a170385b04dba41dd778ac8b0572f43a90c"): 131105, + core.NewHash("8138d3da87e6b7ce91e9502220464e41e25585b6"): 131214, + core.NewHash("e72f2cbfe26e0091257ae4796d9e619eaf07cb55"): 131318, + core.NewHash("ed2082cdf3e59dc4ea96989adeb64548bd7ed65c"): 131371, + core.NewHash("79ace1c9df2ad44f3762e9dcdb36c11991301b28"): 132091, + core.NewHash("fe5ade52fe4309dfe9c7f99503f3ae9e51ef4ef5"): 132165, + core.NewHash("41445ac1668219fda440439642b6f07eb8eef45a"): 132284, + core.NewHash("56ae27829ca515c0082d133d14777689fc0751d1"): 132405, + core.NewHash("54b8de974cb86bfb6a02133dfbd352d06c8e7fee"): 132767, + core.NewHash("7089fb47a45b8475d6518241bbfb06ed0d656f62"): 132886, + core.NewHash("ca6ad95ad5b18666afd77ee67774f7af22a35bb4"): 133478, + core.NewHash("b25efb85a7fbc7f5eff81d4068fe44c203cb6325"): 133565, + core.NewHash("b2390f7da5faa2869eda1da4fa81cbc9c9872be0"): 133685, + core.NewHash("becfb55198354699c11a7348845e597320200e68"): 133804, + core.NewHash("5dde08ac64585d6728d07378865be51f70a6604e"): 134175, + core.NewHash("bb4cb245638dd055642c1227759623ad59066f38"): 134294, + core.NewHash("b3e3f0f9e295ef646b8ef0a4a840b6e99365ffc1"): 134414, + core.NewHash("12c94cd162ccec87f3f1d942bdf623778bb19095"): 134501, + core.NewHash("5841265142ced8ec8a071ab52242d0c1933bef56"): 134619, + core.NewHash("1428bfd22b232d1df52b48d59cd70530235f2097"): 134829, + core.NewHash("9c281ad7d3ee027bd6b57c6118c7bd1b07469799"): 134947, + core.NewHash("ce0e0a255d54675aca30a122bfbe350535914177"): 135033, + core.NewHash("6ba6371da72ffe409d73ee0b060f63b273091c60"): 135136, + core.NewHash("160d23e6cee9a4524f2a5d681697f90e34a139d8"): 135184, + core.NewHash("c63ca76161f584d243d62e77a33a7bf9b6d028ab"): 135545, + core.NewHash("9d815e1f6e88c47c71429930896f423906856100"): 135619, + core.NewHash("249bf5e5bc2176067842cd39ab04298161a05409"): 135799, + core.NewHash("debd6cdd661607f88c4f4b86c06bb4da8e0a21d7"): 136160, + core.NewHash("070751fc9bfa9dc751cbf3fb1af9f3a3d60951b9"): 136246, + core.NewHash("b3550d44ac544b771060a01970517b106e8d0cb5"): 136334, + core.NewHash("84f1886f8b7404ecadfd51e6f85c92e4726e0c21"): 136421, + core.NewHash("39cea4c10af5678de778991ea4176a4fac097fa1"): 136541, + core.NewHash("ebd49b6cade098f65e98d948782a2ab6038661d3"): 136628, + core.NewHash("482d5130216b88ebbd4591cfcef796ef7e3ca752"): 136715, + core.NewHash("c23a92f6d049fd3ab532847db6507b47543a3bd4"): 136835, + core.NewHash("8f1f88e3350a0a9d0140610cf56b6f1925484b3e"): 136921, + core.NewHash("6f7c25e7d7cf88f4dd8ebb870cdf2dac659027c6"): 137041, + core.NewHash("d6b91dc1e893854f4f9db14c02677c5fc3da186f"): 137128, + core.NewHash("de855412762d6d2f410d9c32d59e7375107293e8"): 137214, + core.NewHash("22bd307a576bc5ebbb8dbc9415077e1e959286b4"): 137309, + core.NewHash("3c96d98c33b0fcf005850aaa7b25d60d4a082bf9"): 137357, + core.NewHash("cb6af3440a767192c1b582fe133832b44d5f7102"): 137720, + core.NewHash("33658bda6fc931ea27cab73ab061873b3c6159a4"): 138090, + core.NewHash("cbec3379d61191d5321a3592b54fbedae8328459"): 138244, + core.NewHash("2113c60441aa223a3ce189bff0f83f00419d243b"): 138318, + core.NewHash("3d5fb0b629d0bb25711cc44c91ff4db8e9df5b73"): 138636, + core.NewHash("07621ff0615d497fa34c23635b1d22d99d6de6af"): 138697, + core.NewHash("4b59c9cecaf7f1e134125bf48cdefe3861f88e6c"): 138771, + core.NewHash("9c5a177a893dce1dee39e759ba6ad63bae15d049"): 138885, + core.NewHash("276435c07d770b5cb4b4646c48a46cb55f5fc354"): 138976, + core.NewHash("857beded90cec3c2f6a3adc29ec13f2099550f00"): 139601, + core.NewHash("af9a535c26ef83f482669bdb205b37246d5cbcf2"): 139675, + core.NewHash("c1e918cdff20ffed4780b0dc7e0f976c4cd99c59"): 140356, + core.NewHash("da164afc6a2b7724e2e8ab80ec8f4c9055c8652e"): 140470, + core.NewHash("147c0e8a7ff38658313c31c398584d600e494c1a"): 140771, + core.NewHash("cee4cb0673a892e3bb9c25d5ce4a07f3689cfe98"): 140885, + core.NewHash("ee0c1fa713b1bbc4bf5a463766add8d9f89bcaa2"): 141323, + core.NewHash("3ae84a70ce4e3bef80245119b4938b481dd61100"): 141437, + core.NewHash("31963d3869024a726c2f6f1f51d84f63141387e7"): 141736, + core.NewHash("31f4da0ca6d28558da1eb40787120c1b81105475"): 142025, + core.NewHash("7c872c9e138ce7112483763b7357ed2648334a46"): 142117, + core.NewHash("5bcdc6dcabe43e56b97104bc1a8167478346563d"): 142417, + core.NewHash("45d5119dd9a1510718d75b88352591389a05b670"): 142531, + core.NewHash("fc4761a802819187bb628e644065be840d384b50"): 142972, + core.NewHash("46ffd1af9585777529c10aeb4434b8dbe6142b18"): 143271, + core.NewHash("20f35733375c05323d2c385852c4344ab2b56216"): 143658, + core.NewHash("862111f9070fd2f0355075fa6c7e123996e5e05a"): 143771, + core.NewHash("984129c9588aaa05089024cdf7b379ae9185b671"): 144062, + core.NewHash("ce65aa49bc4a01b18bea779f7b655e8c8ca14148"): 144153, + core.NewHash("0da37734df87583b46829cc928054f14eb54234a"): 144452, + core.NewHash("14e575615f50276898ca693909343783bf4383bc"): 145132, + core.NewHash("8a6a923d5db4115dc367ad2ccc6f41c54fc4136e"): 145193, + core.NewHash("c2969446dd5eacffd403f3b4d18b784327fe9c7a"): 145455, + core.NewHash("40f3761b2771f6740706d151e9776c65b0057f55"): 145746, + core.NewHash("53b8f944c04ea31b0b50cfc0cc6b18bcc097d535"): 146271, + core.NewHash("e3f2a0b00128b092bd8d8799ecdf9e863d67641a"): 146570, + core.NewHash("f44ec3f0ba267cc7fc5ef3b65a142d983e68c245"): 146957, + core.NewHash("483533f945eeaea440cb8bcf9267b79e2d5fa898"): 147018, + core.NewHash("e6cf1f9fbfe6ff86b100a9eeebc0c4845259bce6"): 147079, + core.NewHash("42dd09a54c43b898e3552f8c84c417828e983b16"): 147379, + core.NewHash("bada8a41419f082bc971ae98ac7dc070f5da69e8"): 147669, + core.NewHash("5ebc3ed5862907b67922945b770c1e463ac6ed1c"): 147730, + core.NewHash("0dceedf6d5c107d93009300ad27350790c852c61"): 147790, + core.NewHash("cad4f6037a53e6108194a830efdaa111d518094b"): 147851, + core.NewHash("3692c5871bfaef8e7ce936eb32eaf772a93d7237"): 148119, + core.NewHash("02878d4def198c27dc9187eecee5d659f2474144"): 148325, + core.NewHash("569b1674c7b9ae09717b536295586fea904f8f18"): 148593, + core.NewHash("e4648a55bfb2e8c2f51eb09ed750739bcde5c1c7"): 148978, + core.NewHash("270b151112ade3b3c3de0a4ab1a85f9715683e7a"): 149246, + core.NewHash("05b7ee39ac1d9aa4562eea3546098381fcc8b781"): 149627, + core.NewHash("f1eced298217d8ff98058877df78518b21659a56"): 150014, + core.NewHash("03855691a639ca7d95ccabc31073edd04c5694b3"): 150282, + core.NewHash("4c9ec4d02d8dfde7917e520ca73be1d9a8a7843e"): 150488, + core.NewHash("3f551b5c973ed93d782720c872a1cd58f36998ee"): 150637, + core.NewHash("e696a2f59d294925503740c2f4d5e9f731e5b631"): 150731, + core.NewHash("c004fd81211b236e73fabbd75991120974e18270"): 150774, + core.NewHash("a51f5e1ccd633f172d53b7c89c104d53d289b30d"): 150823, + core.NewHash("207d9b9e59048887bc151c448d700a28feec807f"): 150876, + core.NewHash("e8eb9e701dd91dd1c9983731bab1996f7abe91b0"): 150922, + core.NewHash("a4e59f1a3c05132ac114af5074821d81eee862ed"): 150999, + core.NewHash("9aee72d7e2a02c0799e364e3a95c8a23f4dea7ec"): 151108, + core.NewHash("696a1f31ebe0d5c8799693ede39604babf66a99a"): 151487, + core.NewHash("bc0a4e965e77f45144038767be1238887e224fdb"): 151635, + core.NewHash("6b8a51c89dbfb70bf9a8c65550cf3b5764c61e70"): 151729, + core.NewHash("6842f9a70ca5a107eab99cd888643f20c6d7b62b"): 151775, + core.NewHash("1490d3d7fd235f6603379cf1880dccb2d49e3484"): 151853, + core.NewHash("130d4645505847b02cdf762a15f5f052ebdb45fd"): 151932, + core.NewHash("c97445562782d57d04a5c6eea687bdd1a48222b5"): 151985, + core.NewHash("c22143c9d7aac65a775519de6eeeffa44d81775e"): 152104, + core.NewHash("4edaf6c6bfc21a943fc11f8e2bbeb66688e136c9"): 152189, + core.NewHash("0bffe1bf5d4fb1cd9fa0f12ac1788642ff1efdee"): 152274, + core.NewHash("a76ed38fc0a1cef816c92e26ed989a8b64e9536d"): 152360, + core.NewHash("d79f2736da55c123ba638710284e2856041262a5"): 152446, + core.NewHash("bb6325e4e629fc7348a6d0e6842280d5304160ff"): 152561, + core.NewHash("174bdbf9edfb0ca88415dd4a673852d5b22e7036"): 152679, + core.NewHash("376599177551c3f04ccc94d71bbb4d037dec0c3f"): 152721, + core.NewHash("50d0556563599366f29cb286525780004fa5a317"): 152792, + core.NewHash("2a3b1d3b134e937c7bafdab6cc2950e264bf5dee"): 152893, + core.NewHash("92e5c1a4fb59d01ece44004c4e1daa78fa4b7f87"): 152935, + core.NewHash("5422a86a10a8c5a1ef6728f5fc8894d9a4c54cb9"): 153046, + core.NewHash("6328ee836affafc1b52127147b5ca07300ac78e6"): 153167, + core.NewHash("ec0ff22492361ac3a9c5c6c49a223adbb4afeb7a"): 153228, + core.NewHash("f66196ceed7d6aeca313b0632657ab762487ced3"): 153330, + core.NewHash("de25f576b888569192e6442b0202d30ca7b2d8ec"): 153431, + core.NewHash("8cc2d4bdb0a15aafc7fe02cdcb03ab90c974cafa"): 153532, + core.NewHash("abad497f11a366548aa95303c8c2f165fe7ae918"): 153591, + core.NewHash("55906617fc31f6cc40855f6e0de2fbf041723afd"): 153651, + core.NewHash("5e09821cbd7d710405b61cab0a795c2982a71b9c"): 153750, + core.NewHash("87e459a9a044b3109dfeb943cc82c627b61d84a6"): 153792, + core.NewHash("739d8c6fe16edcb6ef9185dc74197de561b84315"): 153902, + core.NewHash("0f1c784670b1c410dc21c6a503a25aaa1c59a775"): 153981, + core.NewHash("124a88cfda413cb7182ca9c739a284a9e50042a1"): 154128, + core.NewHash("eaf7614cad81e8ab5c813dd4821129d0c04ea449"): 154178, + core.NewHash("9e49443da49b8c862cc140b660744f84eebcfa51"): 154245, + core.NewHash("88e60ac93f832efc2616b3c165e99a8f2ffc3e0c"): 154346, + core.NewHash("24dc2d465c85cb242262ab6bc236bde3ffbb93e0"): 154397, + core.NewHash("09c37951caa6e8f2406db63b8eb8b9f6298b3cbe"): 154459, + core.NewHash("c2b2705d11d1d718bd0b3ca44f8e8eb26afa4b86"): 154569, + core.NewHash("f0cf4c150df0c3f0a9ea5f3c06744820b61a1e1e"): 154679, + core.NewHash("1e14f94bcf82694fdc7e2dcbbfdbbed58db0f4d9"): 154741, + core.NewHash("0d6898782a5359631a47bc97ff97c83ddd6e63dd"): 154816, + core.NewHash("927a02f468eca2207f441378e3917acf44c76f46"): 154958, + core.NewHash("5bd751ac81e6c49ffd212a121ce46cd3a07521ea"): 155103, + core.NewHash("3d33ca0e51394c409137765fe6f07e6d07193b2d"): 155253, + core.NewHash("6815da76d69c051a503e1504bd1ebcb09292a798"): 155315, + core.NewHash("f9cd70860abb0c41d43210ef87ace1a4410a4210"): 155366, + core.NewHash("d97ab3dd7b906416f9e428e72afa246e5a362752"): 155467, + core.NewHash("06c059afefa0c2ff818c53af71ddc8037fcbced3"): 155563, + core.NewHash("91c9fb79cd43cafc20dbe9cd2d948f1ff543f62a"): 155613, + core.NewHash("72f33a38d9e5cbd177f79075ea662d035b555cb2"): 155761, + core.NewHash("d881d17a306158797daa840bac4f25ba7b11d1c3"): 155862, + core.NewHash("b85c1cdbd2fee945e1bc381ad907b5db69681b2e"): 156008, + core.NewHash("588b85ebc9c971781e3ab5d0bd641e6bf7536acf"): 156114, + core.NewHash("159fa0006096dc82f5c9c8b96a2bd9363750287c"): 156251, + core.NewHash("3042ee2a7948591e74decec5d9c17fba30413aaa"): 156316, + core.NewHash("8398de9475a456d42c0c367c5972ecb86af38203"): 156403, + core.NewHash("85ba55e4ae241946a6140896d75df3dbdd869c02"): 156550, + core.NewHash("b696e513a779c4627ab458e77c01c4e684218dda"): 156622, + core.NewHash("229b0fb6b3e4db26d2efcf890673b248dbb87d82"): 156707, + core.NewHash("3bfe73c2162073561c644b50a223a78463cf6a4f"): 156848, + core.NewHash("4b51f31ef6c9ac97dccf06cde0d89d6407819379"): 156913, + core.NewHash("b071b6d5e6b25e80be87b26b6ab4877d5cd3e199"): 156961, + core.NewHash("b290ea2e5671f11a3c229148efd33282783c11a7"): 157101, + core.NewHash("06aa62fbfcfc4839599836119d94775f70ca28f7"): 157149, + core.NewHash("2c698ec0ff4faa68eee9c5fdd139c7dd61304f2c"): 157231, + core.NewHash("ecbd89193551787d532fbdf9e90c43bafaadce8b"): 157339, + core.NewHash("906ba893603855e08f4cf9aafb77ddba3bd264cf"): 157484, + core.NewHash("d1a75bee9e882c0d364b413a356c76d170c70f41"): 157625, + core.NewHash("23a2676599773f7b09763083f999015399053427"): 157743, + core.NewHash("13b5ed79fa34724e67afa7166bc2ccbad91732b4"): 159498, + core.NewHash("d9e3b2d7e3e3ae3b3d39d15aa200fe71a9015ee2"): 159653, + core.NewHash("dcd58cacf540f585e412eb2ea35916f00d15ca58"): 163321, + core.NewHash("8d6e5fd9b1d521223cb65fe87a3d0c777a7efacb"): 163869, + core.NewHash("3ebde81edc5a7104bb821e6adc2f6dc734228020"): 164324, + core.NewHash("08ee331a16e6d49b836924c013e1e24f1b2cc31d"): 164349, + core.NewHash("cd4b9c4a0ce9006640acb4285e98f7bc0cb30c72"): 164372, + core.NewHash("868bfe4e8f4cdb0d96672993a51c39aa6e38a101"): 164433, + core.NewHash("1fccef2efb6840c10b4b5b600b1d4c6d7ba1136e"): 164461, + core.NewHash("3416c4ffea0143bbd592e75bcc0f82ac47ae607f"): 164489, + core.NewHash("1031b4e056c1e80a8f360a963ba349a2a7d2049e"): 164527, + core.NewHash("3dd6ba93049ca0e0dd0e630d0d4d637883de53bd"): 164716, + core.NewHash("8dada3edaf50dbc082c9a125058f25def75e625a"): 164859, + core.NewHash("0426b83280f5f7c3e8a5c0fbc3642b40aa3a562e"): 168816, + core.NewHash("7cd7dd6258f3645eae90d7698875cf84756211fc"): 172372, + core.NewHash("da0dbae43596182d2e46ca70e45bb854179cf105"): 172977, + core.NewHash("e1a3168269366acdda1bcacfd7d208b2ff526811"): 173324, + core.NewHash("66488fae4aa330a91ed8666a1fb1352b6857277f"): 173646, + core.NewHash("3687c3f4369dae75368bd748c2ffd3738f4b7dc3"): 173689, + core.NewHash("e8992a3616569d6286f8096600c302284730172f"): 173727, + core.NewHash("a779be0933ff1275eb940cee361af39dc10408e8"): 173874, + core.NewHash("01ad57ea7c52987fa0f7bebf74aec4a89514fbaa"): 174013, + core.NewHash("86ef6858fa0a23bedcb49e14b10040d939e94954"): 174158, + core.NewHash("16953349f43cbdf70a0646c38af7cc00b17acc2a"): 174206, + core.NewHash("62a60c013175421980beb59e2209316f9bafe46c"): 174554, + core.NewHash("93e786334e7dc4f4f2009b380570d9226f0576e0"): 174623, + core.NewHash("be4c60dd21af060f1adeb6579a09781ff2fcdcbb"): 175567, + core.NewHash("f96def2bcb23efb516fc1694f89c6267412f198e"): 175998, + core.NewHash("2623c9a8b9258953d11a1eddba0bb0298f4f0735"): 176429, + core.NewHash("80c950f602c7bb15492c6ef7601adeb36829a735"): 176841, + core.NewHash("08420ec5b4b252a3c2d27f87e167c75bba03374c"): 177055, + core.NewHash("29986b7dbd7b2efc0253978df5eca2ec868f6a5b"): 177244, + core.NewHash("d3207dd0815d80bf23c154cba8e491a928631e14"): 177312, + core.NewHash("43c07c42f17a50cf95cef1f36324e663430495ef"): 177416, + core.NewHash("26c42ed070dc0fc2ed1c8ed6ad9a0978a510455b"): 177578, + core.NewHash("5bb89a4e2a67658ccfceed375c68d99539536006"): 177807, + core.NewHash("0c3f8dacc94e67e688dc78cac067a6c21d8f064d"): 177858, + core.NewHash("d903adfe8c6414fdf300c04c4f02f41c63ae974f"): 177978, + core.NewHash("45a1ea5c52b1960b04cb07244bb706d8d2c41ae1"): 178101, + core.NewHash("ca63c27510717a12dd8e2bb928f57d5d89f62efc"): 178221, + core.NewHash("76ad8d3bff96bfa727e5077634d165cc8f902be6"): 178843, + core.NewHash("00986c015bd29c673e92a78aada77eb1e4dfff88"): 178871, + core.NewHash("3b28f55ee22bb869d1f220de35a0830afc8fed4f"): 178961, + core.NewHash("7edeaac2474c0bdb9ef1756494065999201b6660"): 179046, + core.NewHash("c5e2e8631a04db39bae0a891419d39eeac98e873"): 179148, + core.NewHash("c3a5e388f63c76069ea50815f62a5e52f4e54726"): 179344, + core.NewHash("240faa9f72a97516a6a904b77b916b9137f4036c"): 179366, + core.NewHash("26bf79da836d22fbf57976f0d9f935cb0565a399"): 181104, + core.NewHash("2ecaa8c8132562e9abdbe4e0b798480172d89291"): 181525, + core.NewHash("31a447417d451544a760c08fcf1932110241e420"): 181574, + core.NewHash("3749cf1e9de449d68de9ce7b89a0380536e19598"): 181592, + core.NewHash("1856e20431fc13d7d9c41736fe53c37b0268285e"): 181839, + core.NewHash("e5f4e428a95c066cc596622d53b026fee9e96734"): 182124, + core.NewHash("70bd99db1dcac31ab24ce1c7694d78e0a7f0ad79"): 182294, + core.NewHash("dd6777bb346267d6c74c0c87ae27cbe6e2546b60"): 182344, + core.NewHash("b725c0d53fca757c49b7aed558e2e95eb715f8f3"): 182706, + core.NewHash("eb34b6ccd2bcd712a998744c82fd2240b615514b"): 182814, + core.NewHash("28cf0bbf361814a70484dea63eb81b9eee6c9a11"): 182879, + core.NewHash("119a37687052e537e7135cccbdfc306c36790476"): 183157, + core.NewHash("918214f55c7a9f9d32e9de776f0525af8772e847"): 183261, + core.NewHash("fb11bdb2866a8581827b41dd9edcb0905f244bac"): 183279, + core.NewHash("2fde1baa601b5c0e317c23f3089fadf324b4021a"): 183301, + core.NewHash("f96e07e9b00cd952ebeb8cd5b7ae47a02af91944"): 183327, + core.NewHash("1054e9143203de1ea146efd5da4fc83c4da00102"): 183349, + core.NewHash("5aec7ed749cf33cd793f47436ccd69b1b68c23bf"): 183517, + core.NewHash("d23e33d6ffa22621a3e5e33d019017b1795517b0"): 183599, + core.NewHash("6bd14acfb20f7f1c247ae41e371548ea87b1bdc3"): 184546, + core.NewHash("8d3a772999d00137bd15e7923c139ebd9427ea7e"): 184965, + core.NewHash("dae2f1141a3ceaed2355dc713c34f4359dda94dc"): 185355, + core.NewHash("4c17c30b6c40b1951f57a147508dc1c3adacae53"): 185561, + core.NewHash("8341ae94b68f3a49dd6957f596a92bdf59d69059"): 185737, + core.NewHash("84d85cc845279fa5e6a78ff2d0c7242c3de0fe5c"): 185786, + core.NewHash("a41eb023136c90821e02a6a91ea27643c645db2d"): 185809, + core.NewHash("5dfd3c52ef01e88c27d26b3084b08ae82051fac2"): 185985, + core.NewHash("bb2f53ef9b8edc6ed2b1b459927a6ea33555241a"): 186133, + core.NewHash("910607fc48a7799d86b8b1de9376bfd3b081aa5d"): 186538, + core.NewHash("f4b503a7944ad43b4daa53a3d184d5305144302a"): 186687, + core.NewHash("125bd3717b2b1ceb4e2469a4e4c3243bcc3cd35e"): 186775, + core.NewHash("c0d28682fd297c8b24f3aa4d22898096e64d304b"): 189007, + core.NewHash("46c9df3b49a7456f544af5447f1247c5a189c89f"): 189230, + core.NewHash("3641c8fca3e189fda320de1234940cdf1f6a4ef9"): 189413, + core.NewHash("bb624a15200557a66fd9d6f135854f49c1e8d2ab"): 189592, + core.NewHash("2b7163e1c1abb8e348adc1f10a31792d09f7d821"): 189615, + core.NewHash("8f9d734b00e1cb824f084a2608d4e8e218b13886"): 189645, + core.NewHash("15f77e892e5afeacc0ac39c613abc7e65ccd78b1"): 189678, + core.NewHash("1b65393498d77a936874f0c3da33fc306f44ff47"): 190234, + core.NewHash("9e3862429dbefc7f64684385f99d56b4007dc7f6"): 190325, + core.NewHash("7973c1436835ffe6eed41374141a4c6199378ef5"): 190443, + core.NewHash("454e604ba254ec7980e6e7f3507cf5cf9b94fabe"): 190514, + core.NewHash("1cde567fdf50bbc38d8571bffeece7b87c1dbd92"): 190581, + core.NewHash("a049b7c082a5dbcecf30e1b34527a0f8fb5afb21"): 190674, + core.NewHash("e183108cd8d547be26bce87013eae9007aef593d"): 191071, + core.NewHash("df0a7a54e1df2c5c63abc0ca3359d053cd1503f5"): 191484, + core.NewHash("08eb667a1eaa9e403d12fa39ffd56d30d8618303"): 191514, + core.NewHash("c5c9ef8b39d36a0336fa758280644e95f25aefad"): 191732, + core.NewHash("1eaf631cc99125caedfafddc1f5a0248b8bc3e13"): 192889, + core.NewHash("e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"): 192927, + core.NewHash("35a6042bf9a39933f8a117d7713fd946f1bb1f31"): 192936, + core.NewHash("2fbd870b972111ae9c6c4f19634b175959197ca6"): 196421, + core.NewHash("406e9af99127c029b19a4a8d529fb20269bca9aa"): 196976, + core.NewHash("82d17d37e5cc9f196957b021110c386e1b733a67"): 197017, + core.NewHash("a43843f753c639599e0aa15def01738fc0dc1d42"): 197052, + core.NewHash("114b3f8a1f48b0c6ad5c6cc86fa6b3cc264e9e21"): 197198, + core.NewHash("1796fcf9e2ab7c99189bbc85a84235166a77a15a"): 197377, + core.NewHash("16b5208525b5de2d8e8a2a3fd188c76759077460"): 197477, + core.NewHash("af83d2ae8fbdcfa53b274d399046b41b90d3cd74"): 197923, + core.NewHash("708aa139e34c45805ae4e01877359f0511db8257"): 197953, + core.NewHash("1b5d1765801cb6b50c359947e254ca1e95763b15"): 198031, + core.NewHash("8ecf2f51c1e06057bd95ab20922e378281ef593c"): 198279, + core.NewHash("8da1e780b43b34e61f74a6495715a8c3f7c102ee"): 207085, + core.NewHash("39ab45b6b8fdfa6dca42476daeb3d31ac7a7d982"): 208362, + core.NewHash("58eec38faeb70d24d4f7f1f5c055addd2e8996e9"): 208635, + core.NewHash("65abc67c16c3e2ab1c4650ddffad31c002113ee8"): 211412, + core.NewHash("0a65a5a56778616c03a14ac0c573b9f2721b04e1"): 211568, + core.NewHash("6bc87884dd0975ce5ba60dcad0d41e6375d9373b"): 211665, + core.NewHash("69b655e8a38da6cb740422ec20d775921cac4722"): 211956, + core.NewHash("2093a04511313f9db0ceab913b7c820852ffd0e1"): 212057, + core.NewHash("f71e342e5297d06cb09cdd5c9b0c946c2bfae5ea"): 214487, + core.NewHash("01133dfd242170f70bc3053da977c0f1d6a030d3"): 214518, + core.NewHash("fb03617df07e6f4fc24a4dae7ed759e6b34c1b53"): 218481, + core.NewHash("ee11a55c4704d61856a3a172ba8bd46269be5893"): 222517, + core.NewHash("a1b4831723e22d7d74904c1c9b922fcbcad3a77d"): 226462, + core.NewHash("45e35644ceee0b081a51c1a2485e425149f04afd"): 230390, + core.NewHash("e5a0cd2ed7dfa974f79b169816efe1c236f0e27a"): 234232, + core.NewHash("c2d81404b5838178de55f41995080efba01a3133"): 238080, + core.NewHash("ac7b7891e3389932bdf66e60b66f159d4fa6d014"): 238147, + core.NewHash("bcdf0ea12add997c17ad7d7a104e01c0a8ee65da"): 238198, + core.NewHash("9f397f23dc19b59ef121036daa275e8d8ba141c9"): 238321, + core.NewHash("2ff35d4e08a429ee89cd65e81959ac161c550dbe"): 238479, + core.NewHash("32e37a5fb994cb7415fd0966570136e4a1704439"): 239116, + core.NewHash("44e1ea6ce525e7c36b3c0049459d266f8f154156"): 239299, + core.NewHash("8ec8e58caad97ba47ef308404b62e5465c5f3194"): 239845, + core.NewHash("afc81503c9a5cd444d3f8bd9980503a2640aab36"): 239870, + core.NewHash("a8b2d851790900f5e36d7917787e741ef99a5dd4"): 240685, + core.NewHash("8d64e24bd55ea365c25b0f4f046f72d26c680f88"): 241426, + core.NewHash("2eace03afd12517ef928d3f29b588ad015825990"): 241538, + core.NewHash("cd7dcc617f944a214ec9dcf9bd93f6b55ecb16f3"): 246159, + core.NewHash("ad9357d38064cf3f35d9b399fdc8a97e2206ee34"): 246187, + core.NewHash("ac0c4384c1697babd8cf358d7b0a4a2855c7cdd8"): 246259, + core.NewHash("46dbeccdd29242a073d86a0da311952185159f23"): 246340, + core.NewHash("9ce603d978fd230222c07f93f0365dbbe7a4dc93"): 249499, + core.NewHash("01b552bb7f6bf84941ee957e6ac8e17fa4692e2a"): 249697, + core.NewHash("ff50c250cf7d7c465ad42a6265f05b0b6a563a3f"): 250225, + core.NewHash("b1c94750deaa0c16c9517e58320e67484f1ac34f"): 250348, + core.NewHash("391a5e0f4cae1f83fdda8ef853accb7289353922"): 250416, + core.NewHash("a1ffb0966da5f9ad1624d84b2c6f2eafb5fa16a2"): 250464, + core.NewHash("d1d45e24b4baee49993dd70d614c1a7849b5771c"): 250486, + core.NewHash("36726abcee25d1b6e41aa1ecb6bfce07f7496097"): 253538, + core.NewHash("b23077de70a5420cbc061cb4d46e75f6dcd5ea32"): 253837, + core.NewHash("2931ded430b6cad4456b3e65502d088afbab7114"): 254081, + core.NewHash("2331c92095f71351d24f969a0dd34a2a456f9b0f"): 254104, + core.NewHash("f214d7713b0a46227b8c8a5f4171ace91567929b"): 254184, + core.NewHash("61ad92ad09c697a7a2a5ba49905bef9c458e4387"): 255222, + core.NewHash("5c7c6352ea3bbd5ef6f3a35059d2515e52babd4e"): 256045, + core.NewHash("b60c29daefd8b28c2fd390ca23a3ce1733e115bb"): 257240, + core.NewHash("d74e65c64c0d94f10489cd84b7243d3a1f76346f"): 258384, + core.NewHash("55b6d312b57d6532169824b46f1e825502f2020b"): 259505, + core.NewHash("4772a808ec8a84ba4fa09449766b01091f47688b"): 259538, + core.NewHash("900a53ebae32b14b0ce74e511d1d17b9ea8a4462"): 259638, + core.NewHash("ffd633d41b762fd616604a4648ba79f6e173f33c"): 259671, + core.NewHash("5dce68cc1648a5c8cc20aef4fdb1cd8a4e098983"): 259699, + core.NewHash("615f6ec3e792d7ff338e62350b26aff8cbd62864"): 259949, + core.NewHash("19a269f5f3f807eb8dea4d15a98fbd14d6b081ec"): 260199, + core.NewHash("b51aecfb55d4e54ddde4ad0f3d8f0001d74dbde5"): 261156, + core.NewHash("33392fe8b33ee258b320f858e38813fa80e58977"): 268116, + core.NewHash("782466d75dcb5671db4481703e6d6e0f1c94f888"): 268169, + core.NewHash("db40efc1057c5bec83be24938161e5aa83f32240"): 269359, + core.NewHash("95b03333bc05bddfe86f7b9851311c3e4b836297"): 270690, + core.NewHash("95b7377727d8ab66e6c8db9e43fdd528ce4dd101"): 272524, + core.NewHash("88f2e4ab34976c1a9d79103a20c46a98197a97e0"): 275795, + core.NewHash("261bd9cdf33d08e7f0e815da335cfefb7d5a56b8"): 278673, + core.NewHash("4c3f045e9539ec929b0e5edd8c49011ea2f7d532"): 281269, + core.NewHash("49ba29ddf04db22c15784ac7f45ac601039bcfbb"): 284179, + core.NewHash("cfd1f719150a9a0961e401ef0f2d889d6cff09f7"): 285307, + core.NewHash("3e14245cf2090d9a80117647e4a2150e0ed5b959"): 285371, + core.NewHash("4c94bc3f173d07280f12e2a84bd42c9760b0b0c2"): 285665, + core.NewHash("46733703592924e85bb1968e1357c2ad1a0c8bf6"): 285699, + core.NewHash("888dc5c59b7b4a89c9c717b929c0a7603879ce73"): 285802, + core.NewHash("241f81221bdbfb2720f1957212b0e461f30f4d90"): 285877, + core.NewHash("95affaf287c8199b46a889f09aa76f573ce6ee5a"): 285911, + core.NewHash("334ae27b07b58bc10b18be6102534bb60a1dc339"): 286245, + core.NewHash("dc2b8a213bfa425b7ffaac4258e692c08c905d55"): 286343, + core.NewHash("92687e65a2ecb20d0be5c8dc1a285abbdfc10532"): 286389, + core.NewHash("d79e3df187e66c1347bb84ad61c19f008b990248"): 287706, + core.NewHash("bf358f5ad609d87cc82c52631ae2c7d6aaa2af5c"): 289585, + core.NewHash("78e0e89a8157f3a9fd574863eaaadb58961b25dc"): 289672, + core.NewHash("4e105da6a96ea67315bc184fd7a1dd2caa3c1784"): 289842, + core.NewHash("8c3e3f7f257c75cf9c257aa2d65c76b65751d452"): 289921, + core.NewHash("4383febe93ee8a62be19f5779a3218f8fd067d1e"): 289944, + core.NewHash("86504e13ebbd667244a7493dd47edb9d84978efe"): 289994, + core.NewHash("3208bc8a008bf8fe7dec93cff85eab751ca1c8a7"): 290024, + core.NewHash("df81202d87c7a656c08c1d6aa9e2a5dfc0408d8d"): 290054, + core.NewHash("1c49d887d773f87b7beb15304bedde3ff5a4d7d1"): 290082, + core.NewHash("a4ab6fd7b7993968cf65c142e6ab4e77853d9a24"): 292231, + core.NewHash("66ae7567ce453d8606f08ca4b8471119f9d36162"): 293929, + core.NewHash("6675e66597bfde9788c0876041b121f62695c0f2"): 295638, + core.NewHash("47759668e29bc4af12d8081bd23c6039127fc3f8"): 297216, + core.NewHash("7a6247124e82a0684fc517e2a6d0efc0e863bad4"): 298784, + core.NewHash("8e2344558dc975f91b4100274c13e11788033675"): 299855, + core.NewHash("eb0f3ab46a9a367769ecadbede100667dd0d7cea"): 300936, + core.NewHash("a1bc8aa30183a13721ef61f356229d6d81a134eb"): 301030, + core.NewHash("d7c9ad89e016122b9aa9d1295ce3331513c5748f"): 301163, + core.NewHash("e6ec6aee0b2db79126f295dc81b6f0c498730cdb"): 301295, + core.NewHash("d7e287c6091086603849b3e7b7f42081813049de"): 301389, + core.NewHash("0c858b7cb919453fbed6f2e05c7e04472cab0e30"): 301538, + core.NewHash("417b2abd20416f75a83299d86f3871aa31a69918"): 302115, + core.NewHash("dd37226b68eff221ee75fa476a82a2a40d0bb26f"): 302154, + core.NewHash("218e6ab24c3c14c0de8f0f68df8a7308ebc4cdef"): 302184, + core.NewHash("de6059d55e83e814296b2918543335ac27396333"): 302287, + core.NewHash("7760a73e9d34548062ae8fb5aefdb7158d699dd3"): 302326, + core.NewHash("54d0d24ecaf6552eae0eabc384526aa420c53b54"): 302350, + core.NewHash("6a72690653d9ba2a8db4e2499ed49a2a394a3f2c"): 302396, + core.NewHash("61a2b8405246e46dd76cf913482207858b1fe18b"): 302442, + core.NewHash("81284249635ca9aac119c0a7ac908ff36f33a3fa"): 302488, + core.NewHash("509d1e581380b3eca7b976d72f0fa9fcf5eaaedd"): 302577, + core.NewHash("e479b686d01769e3cd641643ad6054606c8e48ba"): 302857, + core.NewHash("080f4df34e5664527e70a770910f8a2ed2478fcf"): 302898, + core.NewHash("8ea106874bea8d23e567bcfcc549cbb867c0fe92"): 302967, + core.NewHash("3d779bbd4622ef3352b32fc85bda66a1f45ff19a"): 303207, + core.NewHash("f34969f05e2487a15c964e2413e85749022052c2"): 303276, + core.NewHash("78e02ec2e775179407e2f5608ada627014322e90"): 303306, + core.NewHash("bc1ec65bc73fb4ad43178fca04a834a0fb2c1f42"): 303336, + core.NewHash("a4fd22662b6307634492630542d4f79e95f1ab9e"): 303366, + core.NewHash("71a3715213efc564e667419366e8ce0aad0867ce"): 303410, + core.NewHash("ef1e3ab538b3130ccf0df4ab53ae5efac43fd480"): 303445, + core.NewHash("1e6ebeb85e2449b8cb44523c6702b4baa7ba28c3"): 303474, + core.NewHash("7c0c0dd52cb84eeafa980d67fa5c82331c929cfe"): 303502, + core.NewHash("eb4983a177a7f8af86c94a79e02eb9ec93e01592"): 304068, + core.NewHash("67caea0d45868a2351c4b6c2e8e9122068c78b5b"): 304106, + core.NewHash("be960299f814031ee6e3151fc31341e532a7b833"): 304331, + core.NewHash("8d0b33f70f890a7497ecbb64245d45b9a8c6355f"): 304819, + core.NewHash("fdf744f01252d6bae64c799067b32c0ee2f26846"): 304845, + core.NewHash("be604bdb610fc6090c4adcce4120d44365ab3635"): 305162, + core.NewHash("d476210e9c7621369a05be2e207b4aeacabe0ca0"): 305180, + core.NewHash("cdc149b19e9c561d9dedac408175083b25f7d6ed"): 305219, + core.NewHash("1d452c616be4fb16d2cc6b8a7e7a2208a6e64d2d"): 305258, + core.NewHash("eea3127f8bbc0223fe0fbbd5338956f4bb01dc7f"): 305324, + core.NewHash("1250915d805a6868b75ff889feae98f1cbdc938e"): 305533, + core.NewHash("4b0718ab2d2accf6e5b6727a1f4f2a419caa6779"): 305574, + core.NewHash("930940ebabe48685025b48db76b36b3ed12acf0a"): 305699, + core.NewHash("3b843e72808e053a17dc28286c231ec27aebf6fd"): 305932, + core.NewHash("5f6999be7da52ad42d3a49c9d05c007e5e820df4"): 306096, + core.NewHash("0bf3e59dfeda4fd18d315f918075409b19b7a80f"): 306117, + core.NewHash("1814dce2ff2e14aff2de4ee66ce6b0bac62dd0ff"): 306156, + core.NewHash("f4f3793383f2caa9e645eed4d8e0bddd73d5adb3"): 307032, + core.NewHash("18580ffd285a508196b3816c8ef797f25396b3a5"): 307569, + core.NewHash("3462cd19180879739b490bd667c8f01b5bb529e0"): 308036, + core.NewHash("d60b1706c0ac21ef8943781fd784820bc947989b"): 308542, + core.NewHash("abd32260f32e303d714035321c154bdec977f4dd"): 308677, + core.NewHash("bcb2f0e8f30a5c136e3d3e268e224e0395768b01"): 308742, + core.NewHash("718d0a5d587c51afadbcc521e236d75ac7e94521"): 308783, + core.NewHash("72c1bb4182ae54284b1ee2c30d7540feb3fe3f26"): 308816, + core.NewHash("7787325103b12574cdd5c013d5c46ce9777c91eb"): 308831, + core.NewHash("2ee2185ad033a459f3b1bb16d00120b3d5bd700b"): 309605, + core.NewHash("9fac75b54084323f3716e36cc1e39ef8dec64f91"): 309708, + core.NewHash("a6ca853419663a9c3fe19f847a5246cf2a554f27"): 309795, + core.NewHash("6975707c11f074744b2eb2cae5889ffe00c8380e"): 310035, + core.NewHash("18f3866ec10d9b9b8a422f3fe3a64a2c86a09392"): 310639, + core.NewHash("7b84c300e330291d8d1ea762b3b083651477a553"): 311209, + core.NewHash("8464b4309b8453719a7e55bfa49c7b6d5914096c"): 311643, + core.NewHash("c362109eafaa039f9d78c63b1505e2742a33a021"): 312132, + core.NewHash("d7183fb48e87d271d34ffa8c6613b2d123d37a41"): 312244, + core.NewHash("7e47c69fcb2e4acf417781556e3592b876ef575c"): 312272, + core.NewHash("419b957252667719539e17824b4cd86f495fe6d7"): 312391, + core.NewHash("c2a06f017b89cfaac0dd7e7345282b1bc2fddc0e"): 312450, + core.NewHash("28719269f914c5939891c598e2209370e0471409"): 312768, + core.NewHash("53f737dcd139f68a9335e065259dc1ef2d062f9a"): 313173, + core.NewHash("898abf0afadd12017ce1026c3530bab10318c05f"): 313519, + core.NewHash("cab6cca57242ad7bf3792cf4e822c2ee363b055d"): 321075, + core.NewHash("f2b98fa33ba58d9650bcbc5aade6a163a4544fce"): 323163, + core.NewHash("5712504ffd8f560fb25cba73bede30b3d646ef5b"): 323575, + core.NewHash("26285bb667301101693e96c342873376f2b654c3"): 323610, + core.NewHash("4149900b48249c71fd2d14af8bd4dc8786cf2b62"): 323638, + core.NewHash("25f6ba6d0cc80e2fa74c9f1094f87d259626564f"): 331252, + core.NewHash("2b82c12ca6a5605957b0b558088a0b99ddd7115f"): 331727, + core.NewHash("83bc33eb5a64811341b5345df80ce9786204e0a2"): 333597, + core.NewHash("8482a0454318f44f94ed991b28ebcb3efae35bd9"): 335448, + core.NewHash("7e40dac6d3cb0c86bcf40e1e0fe8e9ad5d225bcb"): 335924, + core.NewHash("be77527bf202105596857f14ddc6297516504690"): 336210, + core.NewHash("2f40e0534bff9e12eefd15b5f75de9c397cf0ddb"): 336283, + core.NewHash("8aadb2f7e9ef286b31bec299d7447e14f23c9486"): 336738, + core.NewHash("576fa614a8728783ef092d912a67de81a54020c3"): 336825, + core.NewHash("9dbb0e923c3c6c60ecadd4ec043dea125d31c9e0"): 336854, + core.NewHash("4de129deb275b202e36a16900da5bdf723b88557"): 337064, + core.NewHash("6bddd5e33dba6e2b3ccf38b52a7d2e73999d3ea3"): 337268, + core.NewHash("dfb09006d334dde0f10244601d5f1c3c28754148"): 337297, + core.NewHash("802ada91192d21115b7c08aee0cda87aa9e6c2d4"): 337680, + core.NewHash("5f99776715d6630f4fcfb50395cfe68b8b3c84aa"): 337985, + core.NewHash("10a7f8c404770721501600d0512941ceecb7fc50"): 338007, + core.NewHash("1e38902f8db2e1062c054d41d3440ff457adf6e3"): 338236, + core.NewHash("9a57b7848616cefc3030b60502837180f1afe814"): 338332, + core.NewHash("a6756823316b11139b74ed975f8f4513c67ceb85"): 338359, + core.NewHash("e8ad44fdd68f3d071331f78fa99b803e2a8aeefc"): 338378, + core.NewHash("6b3b0c02ae5d4fa0ffb8f283e171b5fc19dc8c99"): 338409, + core.NewHash("d0ad5d8999fef23a64c305ec568bad15ffcdb1e8"): 338455, + core.NewHash("989ff66bff1efdb89cb709ad3cbf5fd25b690a1f"): 338579, + core.NewHash("cf8e223d6fd6b99d69aa5ddc1388a519988a2188"): 338623, + core.NewHash("13909b6f35213b75e1dedfeeb2d050cdd87fce1d"): 338667, + core.NewHash("04eec48c565f2518ed643d4d24fe88c7392d66d7"): 338737, + core.NewHash("a1afea43af1dae5c16efe347d4031d827790940b"): 338783, + core.NewHash("19fddb8db8863cd06b0d37e3efdf56921f7c8bad"): 338821, + core.NewHash("0c6fadbb09402844bfde33c625988a4f49284978"): 338890, + core.NewHash("d5479a446cbf13386c0896f8d9b7cae4e63d3e05"): 338959, + core.NewHash("8d5905476f32d0dec59d68d1514f335d8e8ba07e"): 341357, + core.NewHash("65de0f41d99e523a1f5594554ea12ce2004bbbf4"): 341403, + core.NewHash("95aad1eb799017b5811250ba05ae3c1aafe3a3db"): 344480, + core.NewHash("026ea76192791211ecff08ddab8ffeadd293bf5a"): 344520, + core.NewHash("6bd121df419951f3890726b559ad47cb7740bf7a"): 344548, + core.NewHash("ffc5b3f3b114de2e3568fec993bdbfb711afc1a6"): 344655, + core.NewHash("7eecfebb215199e908ffdf7edf933b6223cff4c9"): 348140, + core.NewHash("9054f97d595c6e932e8f499e8af91cd7a34b3f49"): 348242, + core.NewHash("4132f9489e055c8cafdb15ffade8da7a193efaa7"): 348337, + core.NewHash("1a62360ee98004089e77578d400aa04ddf8d5503"): 348426, + core.NewHash("ffb37bb17c614a4fc08179f331c1e146714d80df"): 348732, + core.NewHash("05ef575b0cd0173fc735f2857ce4bd594ce4f6bd"): 348935, + core.NewHash("895035b4c3726a935866c290ddee809839e9ece1"): 397120, + core.NewHash("68638a76d88a7edc868fa2be63021d4027f60b3e"): 397280, + core.NewHash("9aa6a3d49be24284be2f6b54204fa89747d1d6ad"): 397333, + core.NewHash("91a7e269e19dfc62e27137a0b57ef3e430cee4fd"): 397395, + core.NewHash("9d82f78915133e1c35a6ea51252590fb38efac2f"): 399459, + core.NewHash("aec99730b4e8fcd90b57a0e8e01544fea7c31a89"): 399541, + core.NewHash("db9dfd17f06d494586813764a40743a17e25592d"): 400505, + core.NewHash("dbdb987d4e43e594ec703e01ee6f4f8542117ee5"): 401325, + core.NewHash("9f4d9c5e05cf6e9708c0af6bc9954e619b23f1c7"): 401353, + core.NewHash("fcecd825be68ba99f3631428eaf65f24eeef3751"): 401394, + core.NewHash("3e0647ebce07e34b236013d4ace6c91fdbe88570"): 401456, + core.NewHash("8fe7581317d4f7795d35003ddfba0fe35b7d5412"): 401524, + core.NewHash("bce012fbfaf298d34eccb8acd35fc1eb5ea48015"): 401680, + core.NewHash("f07b5e03046bfccb2f0afb5eadced43a2ecaf418"): 402437, + core.NewHash("747760b23d4e52d9fccae88dc50c6d83d32e706e"): 402473, + core.NewHash("b68816f94e041182652afd6fe3534facd8a3e99f"): 402632, + core.NewHash("e409c43b4a55e724f384fb7375ce9ee536c24a16"): 402716, + core.NewHash("6914956b517d493a8b4eac13823654bcc255f495"): 402739, + core.NewHash("62fa35543d1c9a140a14d4805cb5493eb5cfb91e"): 402800, + core.NewHash("64dc9160c7647bbbb791211f9ed25694410adbfc"): 402852, + core.NewHash("d4af0b2d8ba3dde19620ea5a82c4899b9ac2366c"): 403014, + core.NewHash("3683219ab3c8e26a0feb8541ed304e0b2a481f25"): 403048, + core.NewHash("20bfe9df9093ae43a48d20ba9fb913276144bcc4"): 404210, + core.NewHash("9299eb855939e1e7ee4c6b1cd028019bd951560a"): 404564, + core.NewHash("8f7a749b4d4de720175caa9da6846de650466e62"): 404608, + core.NewHash("a2d7d8ec491c8038397da8721d6d560f4a3e493c"): 405823, + core.NewHash("e70886b856fa2e9ce51b22621d81032cbe07a36e"): 406090, + core.NewHash("0c1e7987143cddc64e058661bd51d7b1ee1e6571"): 406404, + core.NewHash("d3bf49b23566eb3c1c86f616ca0463048bfd3170"): 406555, + core.NewHash("221a19fa706e651de2d3be19ef7b04dfbefc1b00"): 406656, + core.NewHash("c4f6d7efb40ae71a79c06f65fc68c9e22ad6275a"): 406689, + core.NewHash("b2c928c034ac615b0ed1ef877b21447109802408"): 406712, + core.NewHash("d1df37b6ad106ea722f704ddbca57e67b8011d62"): 406778, + core.NewHash("1a2485251c33a70432394c93fb89330ef214bfc9"): 406807, + core.NewHash("02eb0a70029f52da0b2a2ad8e9db1645c0d9c40f"): 406826, + core.NewHash("3a3c1c5b30c157572644ceafb4974e11251eb8d5"): 407032, + core.NewHash("e0c17e6ba393cd4d269ed88435fe382775fed292"): 407073, + core.NewHash("6b45ffa55921843a8a75cbc64aec9a136103f292"): 407090, + core.NewHash("1a878f3ae1aa5051528b032364fa87e04d03e9a0"): 407209, + core.NewHash("8e948090dca9a5362e09f35225f2661433ead912"): 407304, + core.NewHash("7037304b65f801b0956e75a0ce80b6aa9e27ab64"): 407349, + core.NewHash("9e99989136f0523f4ce933f5cd53d120fd718e0a"): 407441, + core.NewHash("60e2a7d60174c0be1c918fb9f58c5dab354d94f0"): 407509, + core.NewHash("6ace81f9f6673b0b7b592bdeca31be9013626f58"): 407551, + core.NewHash("7e54becfd6e766232bebd36da5bdfaca60580164"): 407594, + core.NewHash("d8055889453eba8ed9cc2362b6a3afdb9f42669f"): 407664, + core.NewHash("de705bd2525c475041ef16151d0d5b036dca3192"): 407707, + core.NewHash("0de1243ead3f0069744fc0f1c1197b854ce4eaa0"): 407774, + core.NewHash("7baaf9ff34682606597848b8a6ea9d2657b58f51"): 410780, + core.NewHash("0f7d939937949e303f3ff59d5ff7e11f0d378a87"): 410814, + core.NewHash("17c5c2892788fbf6ca38edecc78f1b56aaac57ff"): 410843, + core.NewHash("264d9c8416c19ae724fa4254a6a3cc989a31bb40"): 411412, + core.NewHash("9009ac902c1ded29bb88e07731d006ed7e13cd3f"): 411506, + core.NewHash("b61a98a257c329089be50e0d6df824a0df0c4716"): 413222, + core.NewHash("24814930755b4a8d72788af8ee29e79f321dfba5"): 421623, + core.NewHash("bfc5e621b13f5acdd1e118603bbb33a9074a40b5"): 421667, + core.NewHash("331ec876d6fea71200f378cd6e43d92aa3964c50"): 421758, + core.NewHash("9be4e6b04ffa678ffcfc40d7c1776835dbf36326"): 421896, + core.NewHash("bbb6d3fbf3838d79bad0578d8e8e268179a85a89"): 422069, + core.NewHash("7e1f6278425b6af5af4df5c8fe8e24ebbfe0b5ea"): 422308, + core.NewHash("d4436d6cf2e56998307791d140a792c0253234d2"): 422333, + core.NewHash("6672acfd4bcefe5f2296161c145207e8ac73273c"): 424639, + core.NewHash("0754428152f189bc747f59d9793abb189dbdff68"): 427005, + core.NewHash("d63ec12b48b4d2902a3b49669d79964d5dfacc16"): 427086, + core.NewHash("1edb4bab36767da37b0642dabca8f51ea44e8613"): 427941, + core.NewHash("fd38d03ef4c80e5185fa973223b96cbfeb57e787"): 432328, + core.NewHash("5d07b793d12bec5e4ea20aaf9c41919cb4279a18"): 436473, + core.NewHash("81cdfe3c498d44fdf90800c00a4a55e81ddf3560"): 439254, + core.NewHash("5fec1364d606f67ccb059b5b88cce813b77c68af"): 440533, + core.NewHash("f49e846f72f79ff4cb329d765dde47d0575961a0"): 441837, + core.NewHash("ade7359e1e3100c6ee8d4060fba49590e561c171"): 442537, + core.NewHash("8b20c9ec2fc04787fef72e57189c3ef542144dc1"): 442635, + core.NewHash("b3f65ab597fe0b45b8e853bbaf8b0f0c18acf898"): 443910, + core.NewHash("7533ac9e2ab362fcfa12d17f53325dfbe69e77ed"): 444576, + core.NewHash("07cd2eee1ee995cd9b8f9b0ccc1ff2b64a2405c4"): 444599, + core.NewHash("e6609559233e402dc5728ff2817f062d507c0a25"): 444785, + core.NewHash("5c7923757dd6424563e9f7fee0493c2dac1b9237"): 444817, + core.NewHash("c650507bbcdab5308450c49d27bde42f6c6acda8"): 444935, + core.NewHash("c4af1502654363b6cfda3513aa95a695a8fd97a4"): 446726, + core.NewHash("921c0267ee5503c01fb018343e13a96ce3de9349"): 448502, + core.NewHash("fca86056505addd14dca1fa1f4453ffbce321cf8"): 448566, + core.NewHash("1e84b9ed49c6d8d5faf17f61648a66b2b17fa419"): 448611, + core.NewHash("a98949cd8e8f610ffa75f7ec1278c46ec984f704"): 448691, + core.NewHash("5db2f169967c0cdfe4b633407dfd41366b4b15ea"): 451908, + core.NewHash("5c2af91b46ce6a9ca9b34cf098c98470c3a791ed"): 452075, + core.NewHash("b4fc519fba83bde305b833db72d956adc00f3292"): 453931, + core.NewHash("1745041c917046cf757255ed52171510d1aa7d0d"): 454508, + core.NewHash("06f5cc1b0d163d05a94e4fd2528da4a7c0c4fc15"): 454531, + core.NewHash("51f52a310801b028df456d66ddcd10dd94144487"): 454609, + core.NewHash("60f2b220483352a1ee6f126c9a93d0731a4041d9"): 454706, + core.NewHash("fd8156e5bdc4f9e387a7d799b63823474ad2805c"): 455029, + core.NewHash("af27e96c49c820c8624560988999da00ece7bc9a"): 455635, + core.NewHash("050ced23f688452a6333a44de9b7cae80ec494d5"): 455788, + core.NewHash("c1edf13870ae1124379b0c816e2489aaa54e8524"): 456421, + core.NewHash("41ac9ece250dcec51c70dfff74daaec0e0e78cea"): 457061, + core.NewHash("76e19e13f1add58bab587770bf59ff2f15220b12"): 457325, + core.NewHash("06e5ac782f1ade899fe0f7e3c4f754784e498efe"): 459165, + core.NewHash("635faac3e6fea92d912b74e8bc5974650e286359"): 460258, + core.NewHash("68dcd7519288481a3de3591fde4fdf5acfeade59"): 464615, + core.NewHash("f811d2c919bf43217a6a29a8cabf24db91b3c78f"): 465075, + core.NewHash("e5bcdcccbb1f12d33fe5aa3a1c8e358ba0ba5106"): 465447, + core.NewHash("577686db5fcd55ad8124fe905c1544cc36b4e50c"): 472264, + core.NewHash("f79ad389cb6c9517e391dcd25534866bc9ccd36a"): 473019, + core.NewHash("293ff467b1c2dbf807ae7b672a98ed57f1297db1"): 473555, + core.NewHash("c4f070c41e1fb1bc01af27d69329e92dded38908"): 473783, + core.NewHash("f9e3057f33d3e0b61996a21964d89abeb254b840"): 474182, + core.NewHash("3249e6b9f51d46972769ce470e28b3b520f45900"): 477957, + core.NewHash("5f4fc70dbc0b9d0bcf690b92243d913e7ccf441c"): 479856, + core.NewHash("6b5ab87596edf7ec15d523cead7a474fdea6bdc7"): 483277, + core.NewHash("5228fad65ce200bdc6bc3e2a953280d49fc33828"): 485458, + core.NewHash("0bf1e96dc1623b34d69df91c8a93ef8a273d9430"): 494866, + core.NewHash("4d0b48a394ac8c019b401516a12f688df361cf90"): 495797, + core.NewHash("ef5a8707568808db57e5649ea8b7dc3a81c87a2e"): 496366, + core.NewHash("27084c115b0d93603dca5f39bb267be945ffa0c8"): 496411, + core.NewHash("95f2244cab7f7fce2320709e4a6de7f22f1c195a"): 497344, + core.NewHash("586ca4085e9f74e5e7ec785305426b17eb1fe895"): 497568, + core.NewHash("431993924c8670d2fb0bc4b3c2a104b208992246"): 497594, + core.NewHash("5e8a2a750bac3c47a2d70973166a5c9c5bcdc46a"): 498053, + core.NewHash("dd40efb2ca3a624bf233c04734aef8dddad44f19"): 498427, + core.NewHash("ce80e579909078e0b957f0ccd1ab0a0b1e212839"): 499192, + core.NewHash("eefc9523e66f74ea1106787a7a5627653b565b77"): 502277, + core.NewHash("4d8f93ff658e2f351743b34b41915cd4b0341e28"): 502372, + core.NewHash("50b59991537f94f77ba3b69c26a7da57af388f27"): 502780, + core.NewHash("8e6b090adae2a84beefa47822dfe646dc2ded4ea"): 503220, + core.NewHash("5ea89700cc542e69fc6dbc6b7765e6d616120e0f"): 504366, + core.NewHash("f9e59fa3cab3b021b85fea67c4e7c6cf9ebaee9a"): 504410, + core.NewHash("376ffd7c63a0700f44ff6a6f4c49aa21792de0c3"): 504588, + core.NewHash("d08580c460eafd4923aa347aff13e45b01509494"): 504671, + core.NewHash("9449f062ffeb611ee6de5bd36403618bd4221636"): 504694, + core.NewHash("accc7963e4708a0fd54e3a7b67fe73b8eaa17744"): 504932, + core.NewHash("1df70aac81fa7a9904f22d82710e70fba0e0f405"): 504958, + core.NewHash("d9c6a10e75d2cba1f7a4879cfa32d0a38ecafa68"): 504997, + core.NewHash("e5543d416a1a8ecf2ca5d495d4cc64cf7f9f012c"): 505038, + core.NewHash("d3333fed58d74bb76ed3c65467917bbf2d458d4b"): 506440, + core.NewHash("16d3d9903f4c3b13fda92d6128556613c11565a1"): 507887, + core.NewHash("f5815c1105ef7205b04ba00f28f1969ac8d6ddb3"): 508097, + core.NewHash("4894ca2de191ce1a82a7ad210cd95768f30093a9"): 508115, + core.NewHash("0eb8079217a49b3f81605734b2862b60dea1319c"): 508221, + core.NewHash("c2f5af58660398ddf898fdcddc2c3a21cba5e821"): 508241, + core.NewHash("89c1f50470180b522238a4fd474201232f0dec82"): 510209, + core.NewHash("19dbc53d03d9bbf0bca3d6a3a06ede254031c55c"): 511751, + core.NewHash("03c5d810d237794d3a3d2381dcb815306c37e140"): 513419, + core.NewHash("585fe80cc338260efde8ee4fc6bf87dfffcf36f2"): 513885, + core.NewHash("c0eae19bcf0894e45acf382436c7f535b30afe04"): 513918, + core.NewHash("ab05359d3815fa3e2a9e525435dba568837d9879"): 513952, + core.NewHash("2c75aa70cfe1cdb71b283dc4cc2f011ce2ffcd07"): 514004, + core.NewHash("dd6f40cf6e21464d09f67fed4d3a4487a200c86d"): 514037, + core.NewHash("5710b219c8b9fd2b026db7fc5d49dc89f9b51e9e"): 514084, + core.NewHash("3d2c73204644ab15e5b7e0d6a3d5ee0fe1b6c83b"): 514131, + core.NewHash("0d5651e52157e868a97a60475b08071fd3fad490"): 514178, + core.NewHash("bee9acb173d3507c9831d58e7f7a8d820b234ad0"): 514227, + core.NewHash("ae1ac56f352a3601aa41ecec22e22ea69f934fcf"): 514273, + core.NewHash("51ab29d3a435ad66192057ca7658bb811b251250"): 514431, + core.NewHash("2a5e73fcb3691f34f1e0014aedbae9a7e4453278"): 514484, + core.NewHash("0ac7c222c7e1428e6453e96043b78113516cce4e"): 514537, + core.NewHash("cd631988df43c9256143775ee5ff23e09f787d47"): 514588, + core.NewHash("d9a8130a6078a84ccd42e6f57386fead894ee3e2"): 514752, + core.NewHash("93b3205d374fd19013f6efe2031a7535177e73f8"): 514916, + core.NewHash("de9c3d5da86bc3452463c3b31d7be51ef43bda94"): 515080, + core.NewHash("45ab9917064f37684bb4304c448243930c475522"): 515245, + core.NewHash("90038b17f3ac603cac07ca0389b71de85a5da51d"): 515373, + core.NewHash("9155dd4a518303c9a73b40c7e3e8ab56cf42a9fd"): 515419, + core.NewHash("9ab674a344023e14175ad5b004a7c72efcb9b6c9"): 515523, + core.NewHash("a12e7f228496807080e4df1fad8fc73980ac14dd"): 515627, + core.NewHash("50855fa8e43fc05abdf1d58e194ba1d2eadc0964"): 515695, + core.NewHash("5d8b63843a79f526cd24030b332b56395a06aae8"): 515740, + core.NewHash("e29ff610f6856fae089dadd1fc2e4a0e3e4ef193"): 515807, + core.NewHash("d4eea296101939669ddab66c34dadb65a0379a26"): 515850, + core.NewHash("5c886b41eb66b2a1614eeebc9e33ac55b8e2ba26"): 515894, + core.NewHash("33768bb225ba3bf94a8a12d115a4e24d46f1e307"): 515935, + core.NewHash("22b1e9f02c7538f196566bfc43bffde26c5bc9e9"): 516066, + core.NewHash("a05682b8af92f19ee0400d4456b5b5ff88a72f10"): 516111, + core.NewHash("183ce359c7ae6cc56133c8345a6a81a3fbfa7a00"): 516202, + core.NewHash("fda365c769a4d0551a309e3856b66ef9cce63051"): 516248, + core.NewHash("115de13a45493e93dcfa6a403fac151519e59c39"): 516295, + core.NewHash("908a3e56bd555da86839f8456b27dd647dc64208"): 516339, + core.NewHash("14c39b6e515a277a270323758e0373ddadd0285b"): 516386, + core.NewHash("7d15eb03439b30f3f1ca707c53e963386f92b37f"): 516427, + core.NewHash("60e2d9fa1341dc624ae78cb8cd2553131c8051ec"): 516468, + core.NewHash("754f3198f787ee72ff0eca80a6b229bbcfc3f6e9"): 518243, + core.NewHash("d5bead40045be71092f2a3a66db9e72a5c22c010"): 518682, + core.NewHash("cc7358e52fad2211e9fd44f44ac0efc22bfe1fac"): 519857, + core.NewHash("c33f21b2ae9e0b28b03a26f5e080678a0f634a9a"): 520778, + core.NewHash("99716c80f51363d5abd5a553e5fa1c454c80d2dd"): 521674, + core.NewHash("d444989806cbd441cbfc9ecc3907c25c9c042629"): 522476, + core.NewHash("55c0b4400318be1992a38b8f74643a19367f2a14"): 522904, + core.NewHash("d5bd2e23afef11f62e0242497011d6abe55fd5a1"): 523131, + core.NewHash("38074ad88b991fa6791a5181007ee27a01644374"): 523293, + core.NewHash("fd3c3b65bc523eff59d8e8cac58bb949aeb0d9e2"): 523316, + core.NewHash("316951f91cf5d656b5afdf011a04a00bbc721f2e"): 523458, + core.NewHash("e357fcc03324fb6edea4bf40b64b9b7b0da39519"): 524041, + core.NewHash("a798349daee3357c8b27997d281df940a3a45cb7"): 524616, + core.NewHash("caf76bacf9ccb3ac33901b3c50b85dd11fb8de4c"): 524935, + core.NewHash("eae60124069629e129d5113c6adea7170e84c4ec"): 525074, + core.NewHash("cb3894fd3331f21a726933b3814f8c84396e5a31"): 525125, + core.NewHash("dcf40702f0b1a2a6ecf87db61c962100fdc4524c"): 525172, + core.NewHash("ace5f5630e8e11e5accbebc97c8ee856de769688"): 525198, + core.NewHash("dc6cfa8de2c7f45cc23012ea9643ea0c7627dc6b"): 525222, + core.NewHash("96edaf0d516efef6435b0cfdfb5940de18bb56b5"): 525240, + core.NewHash("83c50a1c91e0897f04e5196cfc5dbaedb3ee854b"): 525286, + core.NewHash("ee9c695bb981fb4465e33b5d2515c997c3a63590"): 525328, + core.NewHash("c2170f6a76cbc6cdd5267fd0c4d095bab4789c5c"): 525369, + core.NewHash("ab5c6049a69291cd200f799bedf84a8e7099dbc6"): 525412, + core.NewHash("202fe6c9c952762f4d27880b16a9596d2853cfb1"): 525455, + core.NewHash("bc9660ca594ab421bf3df601599a237a6f2e0d53"): 525498, + core.NewHash("db276018d2b7513cab218aa97d33da398dfbc16c"): 525543, + core.NewHash("36535b2624fbb639072e3e5667b991f714551316"): 525634, + core.NewHash("8f2b33ad53d881d2882a9d3fa8e7060cd6d08d04"): 525677, + core.NewHash("039fc384af6266a85bb0d483db966b5c6fbe24b5"): 525722, + core.NewHash("7ca1944d7a7013bbf8d721ccf424c1730c6b9acf"): 525764, + core.NewHash("e20225f7e8125e2b42767cf237e6dd982fe278c7"): 525804, + core.NewHash("709757f616893aeab555adccb4d36ec3f1cb026c"): 525846, + core.NewHash("f0cae1543fa40bf309642676f1e982a8e8a3df5f"): 526044, + core.NewHash("9af794c822f79598aacdf215f4d89fb75af30c90"): 526171, + core.NewHash("de5bee52760d7d052b47dd94093100be2272da08"): 526216, + core.NewHash("441997e1fcfb58c6d99668b464ad867664b1c286"): 526286, + core.NewHash("97b3c5aee420ebbf2025c194804d939a8f04af1c"): 526331, + core.NewHash("68f0f2b24a788b7e9d3c3f7ad146bcbf63c13faf"): 526375, + core.NewHash("47badad266097ad16719c97f6138ebc4251b4fe1"): 526453, + core.NewHash("4a5747a10ad74effcbacb708299df461fe9f61d8"): 526523, + core.NewHash("a8614b1ca49b8f3bcb05f3124ca0fed529e8f962"): 530989, + core.NewHash("2dd0284ac6b048d024a5bf1192eb6892f5464c07"): 532152, + core.NewHash("23ba018833d8c895b1f3af4e40b89b50667ea421"): 533120, + core.NewHash("953e72c32cae911698058af548015f27a9e99a97"): 534175, + core.NewHash("5599a543dbb027d08bd52e52146da2fbcd3f8f65"): 534229, + core.NewHash("50d736044565d1da35e49b9282b57f445de52f25"): 534478, + core.NewHash("ae6562ba0147b6513c2747f3552a188d1a5c86cd"): 534694, + core.NewHash("489ca9d43e4bbbf4d58f24e3f05954adfc4a66ae"): 534819, + core.NewHash("3e4f9bd5b9fd8d38570943068648e487236030d8"): 534920, + core.NewHash("01e41a18b4d6412fbb23d02bcf9480e67b13d7cc"): 534968, + core.NewHash("868a85e133e0168610adeb3461547c9ca552b8ca"): 535017, + core.NewHash("12650e8e0d7b646af910fd65a2b681f253787016"): 535354, + core.NewHash("c540cb043ecc12fd6e3b520c7e1d6289a5f803de"): 535528, + core.NewHash("05a22cbe27d6c2226cbda8ae4e136e1ba8d05d7e"): 535596, + core.NewHash("189985d8bff15cd4d61aaaf30ce8a8aef3558a0c"): 535666, + core.NewHash("449098704ccf5b663f48e787d4d9a71a87c2c16f"): 535712, + core.NewHash("56ffb8cd8176fa88842d4716141aac295b798830"): 535799, + core.NewHash("55a2614c9809f23004d7f045504998938092835b"): 535846, + core.NewHash("04caf63c18ccad007fbd3a04921e84b8c2a4aff7"): 535918, + core.NewHash("370c768921577bf9bba8cc5042f90adeac091473"): 535965, + core.NewHash("88876f22d383f41b6b4ba9b29995dd6dbd215727"): 536048, + core.NewHash("e5098f26e55c41946fe8bb452346c6b85a0dab21"): 536092, + core.NewHash("b274f346fbad86eeb90c7dacfe14fb4a89a3c7b4"): 536160, + core.NewHash("0dc0e8b45749de73596fc6b4e033fd5aa131f937"): 536255, + core.NewHash("3e97b1599e42d4e72c82d0e8b7ea8d5875566141"): 536299, + core.NewHash("89c047eefaad78b7c030cc78e48ed826da8a8c6c"): 536347, + core.NewHash("7e5df61a80bcf2b978b9557a9eb2ce1113c9a74c"): 536393, + core.NewHash("2772f7abe9fcf3ee24c6ab820ca1a336bd6054ca"): 536438, + core.NewHash("89713abcd370d9ca33638359ea2734f6dc88bfd8"): 536459, + core.NewHash("259ef7990ca3bb124dd8667f76872e2847d723cf"): 536479, + core.NewHash("cbfd5d7a5acc1c66481ab61fc31e7dd918e14270"): 536576, + core.NewHash("f3cd94e21160d2db1d9498b3d06e4b34de7bdc3b"): 536622, + core.NewHash("c63ef901170636dc95fa5655ef71596195bc7c66"): 536667, + core.NewHash("fbbc58a1addd465c527c82ab40147d0fbbd7df8f"): 536713, + core.NewHash("2606ca88dd582b5812d87a84a440eb1beb349846"): 536760, + core.NewHash("8cb17a39fbbee62588fbf7c6840c2743b66a4dc1"): 536832, + core.NewHash("d581d923dd5cb2c2de592d7f7ff4d34247b86906"): 536904, + core.NewHash("660d144c9acd7f6ea332a3f5b2e3b832c16e7da1"): 536948, + core.NewHash("92a7ed6111798949c44ddda6706b2f59095ef9cc"): 537019, + core.NewHash("222a7643890f4c49aeb9319bb006866a4de244c3"): 537065, + core.NewHash("3249f0d55c575919331f4e57a1c375d2faa2d56b"): 537109, + core.NewHash("7f29ca9dc54d863fac6ccc0d48fea1390ffb1bab"): 537157, + core.NewHash("5a436fb5d8e24e9ecad83effe0b350956cb6894e"): 537329, + core.NewHash("c839eb8c2b03b4485d5039c23ae5788ddc384a41"): 537490, + core.NewHash("81ebd6737ce86c5748c6526fd2ed45716d65fee7"): 537536, + core.NewHash("d5a88b96ef49ca176c2bc02be0ad24b082dd39dd"): 537580, + core.NewHash("dd4802ed1e3c9bcb89f340de236a7b1b9151cf8f"): 537626, + core.NewHash("249b887e938aa6c8436c828247c1ff8e3da567eb"): 537671, + core.NewHash("c55f769344fa4308e31aea554a8909cb58a4c5aa"): 537744, + core.NewHash("ce6dcd67de1f8572894fb5eef55f23fe1170790b"): 537816, + core.NewHash("6eca036a395c55808960375b857818f2c1cf2153"): 537860, + core.NewHash("1812dd4273a818e5ccd1e485e66cd4117cdf1530"): 537906, + core.NewHash("072b814d9edd1f350c081698c7e3cfcf80148a1b"): 537952, + core.NewHash("4c89ab8c8f0a6a3e333edc869edf2f7bd822651f"): 537996, + core.NewHash("aac68d8a3c87e77ba5b94b0a764c943d84cd1738"): 538072, + core.NewHash("8cdb76ec2b6994c34e56eef6ebae5f7642fe0a93"): 538120, + core.NewHash("f8e7cc41934aeb21b3cd6ccc120093e3d27dc7a4"): 538169, + core.NewHash("5790ccad8338853d8835c363dba579bd14bb21e1"): 538217, + core.NewHash("2c9c6e993be5bd3168c35e1c651d18fc65773708"): 538262, + core.NewHash("341611844dca129336126d6ed65e78dd5a32897e"): 538308, + core.NewHash("897024470b6efc064fa38be6ae302cb686d8bb23"): 538604, + core.NewHash("9ba6febc23aed10b5909d7f3320e17f1cd83870d"): 538651, + core.NewHash("f28a210d791f860a85e259cca7fe5070dff834c2"): 538696, + core.NewHash("d87f0a8544f07b709f3b082b06eaefd706912166"): 538742, + core.NewHash("3d937437b6aa1b3a4e49e045884392db46fbedc8"): 538813, + core.NewHash("be60f9f4eb6ceb1edaeb26a872e77496651c0919"): 538860, + core.NewHash("58e3efe52e59230e05e6064f8f7c938495a5c07f"): 538905, + core.NewHash("d8604fab54e4efdfcea1e5b1996740346586ed8f"): 538949, + core.NewHash("67c7aa9799cf4c050c7d1b303546913948632dd0"): 539020, + core.NewHash("b2cf042b2b0949148b44f8dae20edd588f64a2a9"): 539091, + core.NewHash("14757754db8f7185c2781bedf9912d1754d70cf3"): 539187, + core.NewHash("05cf0b166157c28030c00e82194bb8c62510d77a"): 539233, + core.NewHash("9d79069b04086b04f3914f944757ac648224fab5"): 539279, + core.NewHash("18c44559a646f3a194a7bdb92eef2bcec8b83d5b"): 539326, + core.NewHash("ed87a0a8dfdbd074ed283ec32a8fc047ea3a615a"): 539447, + core.NewHash("1f3b9f86aca722dc686e05aa5e6c575372a13eab"): 539517, + core.NewHash("e63b81143ea0872452c0dd47875d58ba1f2dba9d"): 539615, + core.NewHash("7e8e9bb7fdd1ceb65e0aefd3de77815aaec86ed1"): 539688, + core.NewHash("e2c15c001c17c4867d2154467b7a5df00dce5e62"): 539735, + core.NewHash("028988d5ff8492415202ce2f7aaeb33d149a1ee7"): 539780, + core.NewHash("b03bf9280a218d995827fd86c1524ff29ac4a0db"): 539824, + core.NewHash("f9cd14490cdebf7155988f6342b21d00a7973251"): 539891, + core.NewHash("12ee9bed49f7500b52fecf892727016c0bf053d5"): 539936, + core.NewHash("4b2fa09aadb5ca42dc495e586186f83375da4524"): 540007, + core.NewHash("eb3dd0297c2cbd820d3d1af157998f9c505ed481"): 540052, + core.NewHash("cca4f2e34b1e834a8eb4f92d971f47a0644006d9"): 540098, + core.NewHash("9f8c2c15d50baafd49ae1b41b765973cbddf87fd"): 540165, + core.NewHash("2560695887fd47cfb4d24aeda837191552336e8d"): 540237, + core.NewHash("79c3105749b8b3fafe00451093e145d0805c7bd3"): 540283, + core.NewHash("ce83f756b069a748a0ff2ecf5e804c455701a9b6"): 540354, + core.NewHash("ec99d4cffa38e9b1ed814beeef7b49599eb1de01"): 540398, + core.NewHash("fd4d5c580b864cab71ea464a1b00a2d7398ccdb0"): 540441, + core.NewHash("9bf456076d9927bef764a236499329991d1ea680"): 540487, + core.NewHash("c257ccfa87d8060a2cd3ae8178c4a9019d7860e0"): 540532, + core.NewHash("9625ea07be153a5fa7f5f3c39bc000748900126e"): 540578, + core.NewHash("90a20372abb7cc592e995a721ac381f9d4d307e9"): 540623, + core.NewHash("c6dba9119d9437905d4645d91144a9ba24f785cf"): 540668, + core.NewHash("d665eb240b7b25be20c61fe198aa486763b3ebf5"): 540711, + core.NewHash("b327f736ff1a34975c25456ac4e7a1574b4d4f97"): 540757, + core.NewHash("c8d425605357640fbfd1c48272752d881567beb7"): 540853, + core.NewHash("1a29b84c5599fe07c56fa7774be76e7601de4328"): 540949, + core.NewHash("3d4b5fa16fb902299192418358d611623ae92237"): 541019, + core.NewHash("34d424c56ee09cf044ab0e7a99f4a08c46453db8"): 541148, + core.NewHash("989de98b27bdde4c137b01952f133514e1ed6163"): 541194, + core.NewHash("dd1d84f925e9910b133697b676d3aefa1710a221"): 541239, + core.NewHash("0d6aca54462e888eb1fc5b1532654a9a354a4678"): 541311, + core.NewHash("5345a765070239ab7a23118e6794c0c42c64e3b3"): 541408, + core.NewHash("95959559449ce4c3a4f84d67a31400a7559346b0"): 544893, + core.NewHash("46a9ac55dfb33cc225cc613eae1c7ffcbefc67da"): 546074, + core.NewHash("ec96877fc1dd7f1ec13fd2b8e77b64d5d5491b53"): 547236, + core.NewHash("d7a611aa8b42ced7ec801a78a75cb61225f9b406"): 547949, + core.NewHash("c4a85a18e7a4fed394a5279e4d87a8d9213882ec"): 547989, + core.NewHash("1304803ee2b79ee3a846ba604785c5ee3588439f"): 548056, + core.NewHash("f3074cded0fef02c5bc23145ef7d105ff8f7c08e"): 548122, + core.NewHash("e2388f335d16476601e5d312e7bbf5b26d491ae7"): 548426, + core.NewHash("40581788a9d5766abd260716cb0a121eb9d9f57f"): 553606, + core.NewHash("71c556422236c346abe8301e899245afa9bb16a0"): 554828, + core.NewHash("c726c6dbd63bd3ec03cc57974aef02747e356d77"): 556018, + core.NewHash("72b5e8b54504fba5c9dbf82ce785f74872f62f8d"): 557020, + core.NewHash("4462440161bd7443ea2319ec61b855b41a6e0faf"): 557621, + core.NewHash("d149c17aaf180f73ecef0245d211bbcf46dd2aac"): 557646, + core.NewHash("2bb502cbfee59c4c4ac259db27a9c7130cb979db"): 557680, + core.NewHash("0cff7c2303538eda1231e2c45edabc0c5d3b4d5f"): 557775, + core.NewHash("9e83a789c408078906f538530f7790fae7b97a9a"): 557802, + core.NewHash("b2db18fdc342d7ca43db44a79ae7737fcaafb31b"): 558070, + core.NewHash("35b7acb4be10d13f842bc51be76e40b4e1a70590"): 558566, + core.NewHash("255a0bed248ac76a7869035809f8d991ce171cb7"): 558904, + core.NewHash("b368e30d8a470ec50b0379c69fdaefc21ed92a72"): 561002, + core.NewHash("b9742bc0123c2841c8b169e27b1a27c5348a8338"): 561037, + core.NewHash("a84fd167fd2cdb4c6d0f6c935eff261a7d16c674"): 564026, + core.NewHash("305b913715da39c53cfc8153cf5f6bf604c8f6ea"): 564869, + core.NewHash("a514c0d39067d4f57310dad4a43781f0806cb94c"): 565597, + core.NewHash("e4ea78d4d71fb5e809c9143589556fa8ff3f127e"): 565946, + core.NewHash("c63cce7b7eef4ab293b9e6a3f4531f79c081ec1a"): 568062, + core.NewHash("6e51212bfc271dd1ccf5bbd7367c807efeba7fcb"): 568221, + core.NewHash("098284866a0ffb034863895d0bd734b68c2ccabc"): 568609, + core.NewHash("3e1fff58313062ac54e28c5e9586b706ef54f97c"): 569171, + core.NewHash("55fd62a9dc27c4881b9e2bcc72ac5f6ca93091f3"): 569252, + core.NewHash("e58cf2695f54b39b66c35875a5c643348686bb34"): 569488, + core.NewHash("752e89b6656853c9ea2e3e43fceb97b8ebc20975"): 569532, + core.NewHash("a83d27bd79ce733992bb85f4c709e6ab1da3fd2c"): 571501, + core.NewHash("2769ac368ada658c615671955fff16da03b254a0"): 572247, + core.NewHash("0acb56add81ceffeb94ea4493539a8d276d62c49"): 572951, + core.NewHash("62dc14159a7f615ced0eda68d87ae4453a5809d7"): 573655, + core.NewHash("557034505f9d90cbf4644a7525e452ea25eb3927"): 574241, + core.NewHash("703d20baefa8467791d4a8d4742caf1ffc8f27a8"): 574264, + core.NewHash("3393078fc3af0fb115558847789d752f4a8f0d4d"): 574294, + core.NewHash("c46c0adce3cd35799d50f79c95e18c7b1c001ba4"): 574316, + core.NewHash("b7842ef3672119cba0e9d6afbd2222a72e4c0728"): 574350, + core.NewHash("538abe53b465fdd25aa7d438b072610957003b63"): 574388, + core.NewHash("1bdf5d7c311f833b7f5ccd14157be1e43af90372"): 574412, + core.NewHash("7595c1b3cd9c379f1db137a9968429c456ecee19"): 574474, + core.NewHash("aa503f634adda8e19b3c18e3cbe0bffc436d36b5"): 574599, + core.NewHash("a8979dabd6a83cccb4b6beb2b8c5caedf3ca5881"): 574724, + core.NewHash("1f8a92b28aa479106d252de830a31675410b2ce0"): 574849, + core.NewHash("aab92f820d8db8e93e6cfbaafa1a43c0970950e7"): 574923, + core.NewHash("41f4a3ffcc4d7ae7a7331307579f01fa11946df3"): 574997, + core.NewHash("ce9d6e4e70416ec87a3d38862f281d6fd1569c57"): 575045, + core.NewHash("548b4034140bc976e865358b4c1b1d1d4ef4da29"): 575141, + core.NewHash("9a9fae442c3f7be763d73a9338307a9d82f4d39d"): 575416, + core.NewHash("6ae4c2076d35d7069bf02cf5d9feb10c52d3eb85"): 575468, + core.NewHash("cc8a4a2810d7abc86a464e8e6ffaecfdfc8c28b4"): 575567, + core.NewHash("31fab5a22e8138f5624f3cfc9dbdaa0c66c4b7e3"): 575613, + core.NewHash("ae569734db9dfe0a2b31b689417e493552936dfd"): 575659, + core.NewHash("9c8b7a2fd3ac2856d1d032c02806bec1dcaad948"): 575771, + core.NewHash("48c5ff5060cb6b9c65c5b9f0338d6c7a99c208fb"): 575851, + core.NewHash("5c06988ed53bc0b864d7b1d78dd6674c2f16c387"): 575898, + core.NewHash("f3c1f48c13aaa2415c029a4743d401f199cd5220"): 576023, + core.NewHash("3bdd385eef93a6ad45d3cdeca24584f2251ce4f5"): 576105, + core.NewHash("84e57952849924f762529c6eded41100a320eb4a"): 576162, + core.NewHash("d84a4ad1021159d1dae521fce54daae1962c9697"): 576210, + core.NewHash("f5229ff002399261a7a60fab8ed1846c91ad83a9"): 576264, + core.NewHash("33a1ff1e20d307e41effc44337eedfa1c8f4fdca"): 576308, + core.NewHash("58fec412e0b62f3b80f3cb5c5ece2b2af43b51cd"): 576352, + core.NewHash("c2625f04372a89a6cb3587881f4452869809814b"): 576452, + core.NewHash("ca06f32fa19907b4c8cc9ee88158e29b035b735f"): 576498, + core.NewHash("041f4b74c65245de9b5d86591d5c1c36555063b0"): 576538, + core.NewHash("29cef8e7bb7ee3b2455dab14e29ee3987469f166"): 576584, + core.NewHash("2826e32bbf0e08da1de8d867f9c0af49e6723db3"): 576634, + core.NewHash("452d78bde94fd8723795af145499d246e91feef9"): 576684, + core.NewHash("84cf7c4a9ac6afa60a8d68afba4f76c9d3d82ce2"): 576730, + core.NewHash("c5d4c738e140b6e72f0f35f1995aa67cad406aa3"): 576804, + core.NewHash("3676efb45979f900e9c861f8b61cf6e4381e3b98"): 577034, + core.NewHash("bbab7a101627783a839098837d7fcfd4429dd0e2"): 577078, + core.NewHash("2e99d7becf4c999a01b5cbf449e26e1c98e51dc4"): 577100, + core.NewHash("82ad063c518a5f0a07a2249e1487c4efab0b32da"): 579787, + core.NewHash("f4b793e6455b1d7bf916c4afaff1e211edaf1cef"): 580773, + core.NewHash("b8232c337079c1d84cb43b873d0df82fe2344f54"): 581429, + core.NewHash("8238a99f2f02dd0f71ae2c3aba2bb4385bd3173e"): 581712, + core.NewHash("95f3149f85799731923aecc480491b23623dd9b1"): 581941, + core.NewHash("2521a8268800092bc54de41d3f7009170b356efc"): 581995, + core.NewHash("e5c1d91c0cae925274900c56746ec3661b287490"): 582328, + core.NewHash("1937fe10b20cdb9ab00f4bed0d877ba56cd559c8"): 582574, + core.NewHash("2db6b151348ec3265cdb86a6a3c210fd7dbebb0c"): 582734, + core.NewHash("f19ec107acee89dab21160ac4990fd7f8c17ba2c"): 582972, + core.NewHash("cdcfec1fe75b7d7d927737ac6c35719693bb5b2a"): 583118, + core.NewHash("073d036840588ee3aad6b7e89fbfbd945376fb06"): 583158, + core.NewHash("0783e26d5e6a41343c1ed197906df79e36769c04"): 583177, + core.NewHash("09f39b0873ad76986b9bc87438cb90760c86a6ed"): 583309, + core.NewHash("6d27c0c9e3ffdaa0acd1cb3d0760630bd2139854"): 583649, + core.NewHash("efcbd4e83236b54e18d61cee6c5301d55d8fb33c"): 583682, + core.NewHash("0e438063cb1e86272b8f926852f367bfa82a7854"): 583755, + core.NewHash("32ee9a042882219c85f5ea40e373ec2443a5d626"): 583778, + core.NewHash("38b0182020d5ab880e8cee6a78d2a9d6b8750d61"): 584059, + core.NewHash("0a7b268112bdfb05cf339fde7c9dae1b368f6eae"): 584087, + core.NewHash("0b72b6da07e789b22d73a2dc5feb55c2cb11929f"): 584171, + core.NewHash("a2ea873f599c382fa05e091f0a9fd28f216da2b0"): 585249, + core.NewHash("6d024b0318fd9233eb219cf4a464de10f1c6bbba"): 586279, + core.NewHash("7408cdd694d34142907835fe16d7df4bcbf59c7d"): 586330, + core.NewHash("fa8b6d7ad24dc2ac35888199c00210e331e99373"): 586353, + core.NewHash("5e5b4ce5115f4ed75bba1c05866a7299b4776cf0"): 586632, + core.NewHash("90a08eaf8c0ba6669ca1096820d3c2bdb8ca288d"): 586667, + core.NewHash("e72cb3da7fa576110e947930532410c91ce8bad1"): 586749, + core.NewHash("9108beb572c0d28a2409d75d09c25a1900cbda06"): 586784, + core.NewHash("76d6d697bfe923ec95f7607430d3be42295f61b4"): 586846, + core.NewHash("29742ca89a07f6a1c75f2758895bbfc184b28da9"): 586869, + core.NewHash("89e281c7bcd419d4eb9af1e21ecc0edf4cbeb538"): 586927, + core.NewHash("ee49204434d7da0dcf0b22fff19c609541ed6051"): 586952, + core.NewHash("62d0c8f56eb2f5f8e17bfd6c4899b1b574f78d34"): 586985, + core.NewHash("ddaa40b8fe715f93dbf02f37f0bf267ba54af48f"): 587071, + core.NewHash("745d8db34d99c30f81757b320fe5bec5a46dd0b9"): 587792, + core.NewHash("52c30cb6a835ccf0224bb709af3f3385929c29cb"): 587840, + core.NewHash("598e9a05559696ee994b8bb0287c9aca701ceb36"): 587978, + core.NewHash("764ca844b0900279666f9cf15e421183ce188d24"): 588019, + core.NewHash("0fd965be6fcc3e87e07f7faf95f43913a829f6c4"): 588060, + core.NewHash("40b8b83a6fe8c0a5e6ac23473436f87f9a2701c5"): 588112, + core.NewHash("4e77c4eb23382b0e215604147af85d9b67346315"): 588156, + core.NewHash("15f73712f5cdac2bff5473474b8add80d6f7cf76"): 588200, + core.NewHash("0087cd3b18659b5577cf6ad3ef61f8eb9416ebba"): 588222, + core.NewHash("b7612167031001b7b84baf2a959e8ea8ad03c011"): 634198, + core.NewHash("d27ddd55499608700b0e29efc88d500e1f884fb1"): 641954, + core.NewHash("33eec26740cbe78db37c70eeea3b08825d7bd5c4"): 642040, + core.NewHash("399430a5228408f583ab078a0c66405cd6b20dc1"): 642071, + core.NewHash("441f5644d245cd3687f9be8a30891d9fc2580888"): 642228, + core.NewHash("6f0a4c0a38283c2329d21ee2abba084e3f4c8d20"): 642491, + core.NewHash("b9662cd50b6acf6292735a61b48483161bea3797"): 642526, + core.NewHash("c5720672a5c4e58a5ba63c93ed581089ee8f692d"): 642692, + core.NewHash("141351a03d33e4aec9713454328397de30dd8a1a"): 642760, + core.NewHash("7e98996ae74861b8de7921633d8d078e450a5ccd"): 642799, + core.NewHash("4271a96a837b260d536cee79193a6b5b094ff71a"): 642987, + core.NewHash("e76668c0e7a2eac351ba024925969291e06ba851"): 643007, + core.NewHash("be3859ac94a96ff04a438c6741723b693e223202"): 643275, + core.NewHash("4cddd88186f9620bcca10542adbda0ea7a359b2f"): 643381, + core.NewHash("706b3eb90e6b394c8fe433947613cc3ff1dc085f"): 643485, + core.NewHash("d6f3442b800a87cea064d187db45f6579245aa46"): 643552, + core.NewHash("8cfa3558537e5e248ab3adb3597b02e5f8ed9f28"): 643692, + core.NewHash("a5909bf0f853434cb42ab446877b0646a6482ac8"): 643733, + core.NewHash("571c69e30a2a8983ba9088db8f5711d4d4067140"): 643932, + core.NewHash("c7197def095824a62574b333a27fbcc3874e66d5"): 644189, + core.NewHash("996775e04269a794d99a0dceca8b28d27c376ab0"): 644758, + core.NewHash("4091ab3d726386dd0c589e80348b69a33c4925b8"): 644902, + core.NewHash("44467e1c268286351720d89e4d704dd67ba0d1ff"): 644934, + core.NewHash("b916bb3aef605c418d277807706e9e8a928545f6"): 644978, + core.NewHash("f05247104e9e6e656d490b20b274b7a5407b19fd"): 645046, + core.NewHash("73d918541735a1aabe1d31f37f2ebb5c0f948602"): 645112, + core.NewHash("4a2b6926c956d6dbee8c234146e8b930fa44c429"): 645151, + core.NewHash("3361fada5ce3d54d4c5facdcc185b8aaaebd9ab3"): 645190, + core.NewHash("09f3f0ef0c87718940a6f4a92f7ad131311ef799"): 645305, + core.NewHash("d2a8ccfe289c9d6da783b576ff89bb0812b3156b"): 645466, + core.NewHash("8250b542ed0027397e5d3f0932ca49cf41d57cb0"): 645628, + core.NewHash("301816f9ad53e9e0716301435238d42b3c35ddb0"): 645790, + core.NewHash("c9af9f222362d7717514208262c17c1e15c0b024"): 645952, + core.NewHash("f7d03a2b012372c03026986c2788b09c87b372a7"): 646114, + core.NewHash("dfe8151a9c7638fe99f081dc060daf572e69b950"): 646276, + core.NewHash("c4de40a9632c5f2ca12ad7708a9daa4cf5f097ea"): 646319, + core.NewHash("2972f7bf8909942fc331aec6f9bdbbcf56a4c468"): 646388, + core.NewHash("472352a2218738eb5af9cf38d799fd75c8326b8a"): 646432, + core.NewHash("f8f00b5f286aad6e2a33393d517b45ec449ee930"): 646474, + core.NewHash("5007f64fb95ba86bbeea5fc53613fa3ae78fc891"): 646817, + core.NewHash("a80414ef3a5169a6a40f9cfde7441c31d525f0f2"): 646998, + core.NewHash("6aec6c9b6d763c0881931c7601ef3f463481c9a5"): 647177, + core.NewHash("98f754b1d3fe456ed1e26ea7e2b5abaaba92cac5"): 647251, + core.NewHash("b9e98ff2b1a5013a7017cf03be7f6dece47f7488"): 647300, + core.NewHash("d160bd9d5221de28e436c40fa3c096c84f222c93"): 647372, + core.NewHash("bc8c3420e76afe0326cc3c45cecac7f0973cc21d"): 647419, + core.NewHash("d7f72b78dc29417ca9c238d06f8721106cf0b30a"): 647466, + core.NewHash("d00cbfa6f3a875dbe8f17c965af985b60f457a97"): 647513, + core.NewHash("0c887c362ca184beb3e9534be6174f0a0306fb9b"): 647558, + core.NewHash("eca0c8cf343283082453dc3912f4fe3f977e21d4"): 647603, + core.NewHash("c86d2bb8129255852f0db9968a9d611f25759c1e"): 647650, + core.NewHash("b4692ef660823d1eb23c62003f3175a8d1cf9e1e"): 647747, + core.NewHash("92bba5ad7dab960292edbad09060033bf35ca333"): 647768, + core.NewHash("f202f2abae5841529551e11f0691f3d126b886eb"): 647792, + core.NewHash("1f32c1bcabf6e53b2bb68911dcad93532042a9c3"): 647845, + core.NewHash("ad11a89ac6ea6c0e690587e73fdd19fe1b49dfab"): 647890, + core.NewHash("eb70ea7a90ab18828274f30e653c17cbe17d9f2d"): 647937, + core.NewHash("41754e0b7de3458d8e4e702070f54e267eb12a9c"): 647982, + core.NewHash("815c34e5ce7b5dcc952a2d3d94886f417919da87"): 648025, + core.NewHash("38d83776fee0a9c7c9fcac623e4fee7c7412387a"): 648071, + core.NewHash("1e4afea46f068a66e3a477ed9b2910323a16d3d8"): 648115, + core.NewHash("663c615aa83bfd281462f9f8af8f07e5102bb1be"): 648161, + core.NewHash("264e1b0f9adc208b61d4c5307689ed8dfe7c3bb3"): 648206, + core.NewHash("c96c34f2deb3e014a707fcc69c6844eca385fab9"): 648253, + core.NewHash("c8a084fa81de46c9abad1b25f480ede4de88975e"): 648297, + core.NewHash("f8de27e18ab671945bfe7e6b2908c4204a99b268"): 648392, + core.NewHash("9ebefc744b543060a759ab218a28c2cd39b92f25"): 648463, + core.NewHash("2c7b156edcb476708ad79e2887741a6058ba59e3"): 648559, + core.NewHash("153d2672be2de629d154a55fae7360bbe3014ffb"): 648634, + core.NewHash("0e3bb2ec0f41b8f2d4118e80df19fb61adf048bc"): 648679, + core.NewHash("9dc1c7b8290405e11b3f4a2052d994fbaaddbab6"): 648726, + core.NewHash("169ac628c8d70b4726af897f6793d10a0e9a1eae"): 648771, + core.NewHash("5e7d9dc85850df946080183846cfc8c8b441e022"): 648816, + core.NewHash("6d151ea1d153f8ce3bcabdb57e4af1622a8ae2df"): 648861, + core.NewHash("e8199bc0ac368e67e9fd208fbee2656868e94c0f"): 648905, + core.NewHash("ee6003bd7b6020bd4d0b57f31ddd3d12bd5b0cec"): 648949, + core.NewHash("63800af17570031d2424028e2e6e26ed2b85cc08"): 648995, + core.NewHash("c65630952c76e6471b66692e2d3a46f38a3d1f1d"): 649039, + core.NewHash("5483f4b93fbfe07d5cd781bd8385f3efa570c0eb"): 649082, + core.NewHash("74372b85722250f766e3d0b147503b0ce4dae1b4"): 649126, + core.NewHash("efd3dfeb00feb302879a786e1369a51e443c2de1"): 649171, + core.NewHash("4596d25c9c331f691642e71c3c03c57c2b7bab34"): 649217, + core.NewHash("d258e32c50661e3212acd147241ce21ff091086d"): 649313, + core.NewHash("67e0484009fe2ae4d5ec8ac0782982669e45977a"): 649390, + core.NewHash("cf9fde8675c01dc7f11489276432917b9d05d748"): 649436, + core.NewHash("4ab2c1bb989b9859d7e9320908c7772d0322d920"): 649482, + core.NewHash("263e1a56fb875d1bc81f9687997ac1741173307e"): 649528, + core.NewHash("5db67d407dd9c5499541bc1f9860604171c5e433"): 649574, + core.NewHash("8b72625c9f99cc70edd9653aecb9acbe588015b0"): 649618, + core.NewHash("5cb38fd63d96435980bfe4b2ea898d8056ae1e7a"): 649664, + core.NewHash("2305c53cc0dc8b47f635c88d1e3b73710d813b80"): 649716, + core.NewHash("5e08a8c23aeb8300f8eed2a8f3317f91c4fb0481"): 649760, + core.NewHash("350cf5e6e350e718c52af1f32070481a2c69e4d1"): 649783, + core.NewHash("c08b79a207e0f3790d492ebfe3c8da8f0d08b810"): 649828, + core.NewHash("f25e859d9b4069f9d3d6e9192fa8800e96a762fb"): 649875, + core.NewHash("66be14dab40a1cb07aa2da6f2a4938329d898c26"): 649921, + core.NewHash("998443f29233ed8088a37fc6f17415a33f18bca9"): 652386, + core.NewHash("a5b98652e8bf6b18298af6d046374ea59bca530d"): 652436, + core.NewHash("092700901e60298a75b1cc131f7f52f8d262ce13"): 652961, + core.NewHash("af36bcf09e2d87c58a7a5cbf0d6129ef6aa876a0"): 653081, + core.NewHash("e9ca1aec40d032ff3926cbaaeaa421e01a586526"): 653193, + core.NewHash("c439091591e90f5730c8765040f9e338482215df"): 653240, + core.NewHash("7b4f010f01e034e3af416b3e8b64529ecf174add"): 653639, + core.NewHash("d92dfcecc1de725ffe7cbb216f02449965736b59"): 655234, + core.NewHash("a32d07648bbc058721009fa534f8bd7d9fbc4026"): 655259, + core.NewHash("94ab59866f2bc580a755470ab6ce8f7c13ef4ecc"): 655317, + core.NewHash("1570d6c6892a1d1bac98405624c4895701d7bc3d"): 655356, + core.NewHash("296394601f8641b4932feca7e37c0c48b7c5abdb"): 655412, + core.NewHash("82b60fee8157083dfa0886cdf558f5ea7db8c5b4"): 655444, + core.NewHash("fe7cb9a20ef33c624a8c3ee6eb421b7cdb37ada7"): 655476, + core.NewHash("a90d392826f9ac10fa010efa8474637dbc779991"): 655615, + core.NewHash("167e64289c5b8c6cd805c2feb8934304eed65b79"): 655754, + core.NewHash("998db2978a90ac1f250ce2b614e49f0389c28c35"): 655845, + core.NewHash("2dfd74e04dc8b7bad9128be2c131e93feb504c7a"): 655940, + core.NewHash("e880979247ea84c22a0640ef797ae2ddcfcc1fde"): 656035, + core.NewHash("fc360ecb356c0272db3299eb8d6dd5076c0754db"): 656130, + core.NewHash("42f97449bf8f3aa3bcf6be5df7806aae37da901e"): 656225, + core.NewHash("486fe53955157f56c9b8b88f102c04d350a047b3"): 656296, + core.NewHash("aa41c933abdd510a984a6153cc9c8232d4326c0d"): 656339, + core.NewHash("7f06522b58a91e4b92f0a302265c6a11c5789f90"): 656384, + core.NewHash("d15c2421108796cd8108efbc05811b09d8dbda32"): 656429, + core.NewHash("eae00fb056c16b150d0fcd3fac526248e405c081"): 656498, + core.NewHash("c65bbfafabb08720bcf99597e8bb55aff958c729"): 656542, + core.NewHash("6918817c1048f4e5d42f5abac6284fd812d67075"): 656585, + core.NewHash("4b3d0a379bc31908e63c1e7350e91acde8bcc84b"): 656653, + core.NewHash("33bd81c56bbc1dcdf01e7e6ab481222d6baed413"): 656692, + core.NewHash("bb31f584d07d6032fa78727a8b4598d3109411ab"): 656731, + core.NewHash("bf38cb7e82782b94cbd9d8cf554772b91eda8aa1"): 656770, + core.NewHash("6c3e3c022be412d346b4740c4dd9bef67ff16aec"): 656894, + core.NewHash("cce01ac6a05596164da00a28d3e43ce16e4c1348"): 656933, + core.NewHash("d2087036a09f4ceb9372fad325995ea2880d785e"): 656991, + core.NewHash("dcd1755b15d2d687aab4be6b18b6913e63b0784c"): 657123, + core.NewHash("f0bdb36cb7cb14899e185ccd04b61c74e9f754c0"): 657250, + core.NewHash("57b8e6f643d142cdfda419eacbef17664903f7bb"): 657294, + core.NewHash("1e2b25e8b44a5121c40cf89ba8356dba4677656a"): 657339, + core.NewHash("5c9535180a82a49e182eb7f3b01414167d0c56bd"): 657383, + core.NewHash("f2adb91c301b086da2dba28c1ff2f54f73862564"): 657425, + core.NewHash("7ab06752ef4e7a14ce7af4199b918a73595ace3c"): 657561, + core.NewHash("8a154489c9fe8bf828bcbc5f3750444fff12ccad"): 657655, + core.NewHash("72ba3884dadfc42fd1080ef5a48964907f1b33f1"): 657725, + core.NewHash("de1fff5d95f03a380a6a23dc920c251f49f216b5"): 657770, + core.NewHash("fdaa7f6bd947fd2cff9101bc5c4efcf079d99f17"): 657815, + core.NewHash("bafb34dac0b1af391b7d97d817080d70bac1b85f"): 657909, + core.NewHash("5d86f08c460f18ac5cf88ea79235978d9241bd12"): 657953, + core.NewHash("64203ab66cb34dd84d2c9427cb50428c6253b5a2"): 657997, + core.NewHash("b71985b44a842787e619880f4dadac8c8e54fd98"): 658043, + core.NewHash("daf15022fe674b7a7998e9643749fcdef43a2c3b"): 658087, + core.NewHash("02bd9b4ec6aceaa0ccf3574d98b6529b38ff7234"): 658132, + core.NewHash("7e7f7cd62a4f48849eccac2e02076b7fe90290a7"): 658176, + core.NewHash("21c72db12f7d9e6cbff2a7aec85bdebc2c1e3385"): 658220, + core.NewHash("eb797237c426e62ef5fc2332cc55a215ac8d46ab"): 658264, + core.NewHash("70b3ee0e490df1c518f13634feb4c9d750feb360"): 658308, + core.NewHash("7053cc1e2dd97e7d2f6909e08a42768e25416477"): 658377, + core.NewHash("4f43aea594fafb0d186e7ccf26c071a3f9c5d7bf"): 658421, + core.NewHash("4e1812f30ce4011c5ea41b704a0d35259357f334"): 658490, + core.NewHash("211a8d94ec26400c111e6892f4e2d5c40e00cbf5"): 658535, + core.NewHash("50ae5f0ff8d446f01ccfeb740acf9cca1a7f5a78"): 658579, + core.NewHash("527fccb68718a7a1498526b44f2fe9c592e50877"): 658623, + core.NewHash("6546d3e28b8211be46c8d1507179c2f6897709e9"): 658669, + core.NewHash("312dfc162aa4b218977ffb314b49dcc65cb45e1b"): 658715, + core.NewHash("b368d8be506e077a5cbec7ae72e0da73ef7bfc5a"): 658759, + core.NewHash("e77d9946a0ed467a24307f5d45c6698ae25a9e79"): 658802, + core.NewHash("3e9d1677b7e6b25cd4d41d64d394666945ff327f"): 658846, + core.NewHash("7c95cbde3a178e9c55be0fd73496feefd367cdbb"): 658889, + core.NewHash("2af1e190e51db0cd70e7bea7a295abd1afe9a6d7"): 658928, + core.NewHash("e22a267760421d71883d8b6327c8497dc9f10fa5"): 658967, + core.NewHash("ae5355ad198ad7c9715556e839c80e5bd4fe0d88"): 659062, + core.NewHash("5e9f1ae3f1373ed616b6dba4ba5969e6fffc5e09"): 659131, + core.NewHash("d0113b1a695ceb5a7d8e9c8134d625a527e09b4c"): 659174, + core.NewHash("61027a9524ddc3e208a97e309e3574dfc50a369f"): 659436, + core.NewHash("1cf6a738c3716f39cdaae4ffcf14806b005b6f83"): 659480, + core.NewHash("57069a8e53d5d418a45c81491e24b39d550031d4"): 659525, + core.NewHash("13d0880b5ffeb361cdf97ce523f70b72b5c16c44"): 659570, + core.NewHash("9c0759745954ad38f47ebe0e1aaa3d919f16a4a9"): 659613, + core.NewHash("9495845fe007720b563f9dbb11de045e8c44260e"): 659658, + core.NewHash("7bff6f7e701135e8494745b1300fbe5a22c7048b"): 659743, + core.NewHash("08eb1320bee2432466d3fd4e47f1847811f5a8ac"): 659788, + core.NewHash("1804dac68120b355b7bc9bf588376c558b3d0b31"): 659833, + core.NewHash("2ba059a5d3d8e12fd79acbfb2c34919a46c8c5ba"): 659878, + core.NewHash("a2179bc68cd6eeb54bace47a266fbadde22246e6"): 659923, + core.NewHash("445282d1dce9c3deb03234b199adde2423082a40"): 659968, + core.NewHash("ce78942e8fe5dcd9aa784fe9ff4964a348d4c7eb"): 660013, + core.NewHash("0ba9b60452a6766737110bf4a308bbfad21cf664"): 660058, + core.NewHash("ae98f31b39572332ad4c25bff92f4fe591374874"): 660168, + core.NewHash("6f550ff32189ba3d2c2e1eb9a1b827d13068c665"): 660213, + core.NewHash("e368a33f35b9d95bb33bc15e1c20c9185de67c30"): 662910, + core.NewHash("ab713a26bcd7a42a167433881f54d07bec2c338b"): 662976, + core.NewHash("afb2eeec3cc6808b0ad2587f5cc532a57dc6bc63"): 663015, + core.NewHash("6172ed676f684cabb562babf60c27d38871f4b26"): 663054, + core.NewHash("e61a5d7ea6984735db14e6e8813d39edeaf05d8f"): 663149, + core.NewHash("62c52007ad597140dc5562238648e5eca76ef230"): 663220, + core.NewHash("30937f864ec9e1a3398f7d899ff8059c8f2d62f7"): 663266, + core.NewHash("ee8b49ba6b5b50e7f0e46fa9bcb21f111224592d"): 663336, + core.NewHash("6b8df4eca161d39b711644da32c20aed591cf611"): 663404, + core.NewHash("fe7d3ad23608ee95fca848dfd293aeabb3cd90a0"): 663474, + core.NewHash("79643294e5561c1654b7dc5d77e6e82e6e9d6807"): 663544, + core.NewHash("8c33229dc05e45ef544986b2ca3d16d5ca49d8b7"): 663589, + core.NewHash("2df2f9151c51e6fd8d1a9b25542275bc00285136"): 663633, + core.NewHash("b239949f3922812e4943957e8ac063b00bbeb0ce"): 663678, + core.NewHash("f397586fa4d6cba5383900d706805593ce950fd1"): 663723, + core.NewHash("dda2cfc5a8161ce4e91ba4168fed2d8d706de6a7"): 663768, + core.NewHash("520032b13fbd5a1aaae2ac5a16bc83175771f927"): 663813, + core.NewHash("3f7b1dcc4805a84efcc9f561c679f031cc90d59c"): 663857, + core.NewHash("728eedf5cdcacf39353462a919f0ba56873817ef"): 663901, + core.NewHash("de2386c39ef1240d734bc0efcb124e79353bc30f"): 663945, + core.NewHash("d0c39a32d35a5e3b5c63bb0655ddf3b9271c1b7c"): 663989, + core.NewHash("e8b215472c4adab7a12c4061fbb5c63083219b81"): 664034, + core.NewHash("b38093a50d6bb0fc3befd9a3c1931c7bb5ad6ec2"): 664077, + core.NewHash("7e6c1805bc2951e566436fb9a6b33bb6bc7f087c"): 664124, + core.NewHash("c78bfc907c0b46d8db3f94bd01075021bf67a244"): 664163, + core.NewHash("2562d5c8f8e1223a4407461a883c9922a4c942d3"): 664537, + core.NewHash("f4d707b8d754eedac14adfe30527f68efa282405"): 664958, + core.NewHash("838fde4dd0769d486c9a99d5b783d70a7ed360eb"): 665310, + core.NewHash("2feac242d1bd517c94fe2a3c59e0fd713c888f02"): 665498, + core.NewHash("92941728ec816c57dc470f98afb245f98f8e5592"): 665627, + core.NewHash("92988e657db4ed1c9c3c11d353c563488063f6f5"): 665674, + core.NewHash("529a8f74938cad3fc0783d7ba25d9dbcf78a383f"): 665747, + core.NewHash("d307162839c4d684dcfd7d28eb780b0ca3674c28"): 665794, + core.NewHash("e18c84078e97f6166f7a1725a1f318303cd3b0b6"): 665842, + core.NewHash("0cb628e63e8498e5258730f7c8658fcfd4f1af24"): 666006, + core.NewHash("785467b6d1b97919120959c5a3bc89f704435618"): 666170, + core.NewHash("8065d5086796455972c645239550748a3eaa63eb"): 666301, + core.NewHash("5d76161384ab07ff12afcd29c7b6a28c52bbe3dc"): 666378, + core.NewHash("c749c3372a5ff0cbdfd181084cbf1278b5a3ed25"): 666399, + core.NewHash("c79748b74c24e0b1be7a01f3138f3e4c59ab0a15"): 666447, + core.NewHash("7fc59e7667d67161f5b0b07b1f9b82240fe63b67"): 666492, + core.NewHash("6c84ed99939cc626af1f4ffcfc7ceb9666147735"): 666537, + core.NewHash("2bc37da09f90cd49bd666e03ca09076c1c7c613a"): 666581, + core.NewHash("b60f75c839ae38869d89bfdd490b814a2cd74ff8"): 666650, + core.NewHash("6e484fdaa2e8afb86adb15b1e214c3385afc6c49"): 666695, + core.NewHash("aa546fcd2c8dd3dd371ef62cf8d3628a5090083f"): 666717, + core.NewHash("99094f712f17050f9f99d5bf062d340821a171e0"): 666847, + core.NewHash("df8549b3ddbd18f4c1e518e328442e96e8059609"): 666892, + core.NewHash("d9aed9498e95355f13e5637e111c1e1510cd7950"): 666936, + core.NewHash("7bc498f9482597de9061d034f17361ea84bd75da"): 666982, + core.NewHash("cbbdc29a7a91b2c63cd1adc9075ff9f1eb2e7cd7"): 667053, + core.NewHash("9701612bea207c4871603f6a98150a7cc35f2d30"): 667159, + core.NewHash("5cd1b716b1c2b4f67ff8422b1f2049ee3eea4258"): 667207, + core.NewHash("6938251b4f521095fda11a1dc4c38cd89f10b308"): 667286, + core.NewHash("cbff52205da9e2e4d3afdc5d145488e477e78e93"): 667421, + core.NewHash("baf66cb40589f887f73b5e789c7773414d862e78"): 667467, + core.NewHash("f3317bf370ab66c2f50d02f288f59ba3cac38380"): 667515, + core.NewHash("db82be398dbc09e433c8b5564372ee639e7f0cc9"): 667561, + core.NewHash("f5e84411ae2f7eeca5990f136837daac01ba5dae"): 667607, + core.NewHash("eda221b623306b5a3a9e1dbf498c48198c9bafb4"): 667653, + core.NewHash("51fada8298633caf115d61ef60059b6498515faf"): 667701, + core.NewHash("76d5b8464fea5bbf0d7b92eba96f40b9cba48293"): 667749, + core.NewHash("d23030b0cf0e05958fc5d7819d3d841ac5d1d3ab"): 667797, + core.NewHash("2f58ad849e27de2d3357b9ddc94abb5d8aaf4efd"): 667843, + core.NewHash("2904fa0550fe6c5b3008553fde870ed38f9c66bc"): 667887, + core.NewHash("54e23555b02ee9ce87cf2dd36529093ba5e98a0d"): 667995, + core.NewHash("1773e6f8bc1651762616e409f12d26326a3b49de"): 668272, + core.NewHash("1948521b89d291ef92faafa5e84831f4a3fcbd96"): 668316, + core.NewHash("5b4e80a22111064100568ebe7f0c8f3dfe603a95"): 668488, + core.NewHash("c3c6f88a0b642845c75d8eb5403ffdc8f5153b51"): 668660, + core.NewHash("605aef4fc89e2e37c4a91e0fa5e77ef86cf81eec"): 668729, + core.NewHash("05cf1ccbaa2144f9e5b47f020ec976200803e037"): 668762, + core.NewHash("0c6f4a8bf8dfeafe11f826084ee2ac8d4a6d4d50"): 668833, + core.NewHash("b28673e351a066b19261b2156158bad01fdb6f9d"): 668879, + core.NewHash("025fe880b92dccf643e731de136b66967ed11c93"): 668925, + core.NewHash("05a1122f40a448b5342bb1eda4e5e29f227ed659"): 668971, + core.NewHash("87be8ec5b27964fa5c3f91f6d8c52561cefaaa3b"): 669040, + core.NewHash("1233461d2c24c4106454f86f781425d00b18d68b"): 669225, + core.NewHash("062959cb10dbbe8bb734aec71259ba0d34658667"): 672964, + core.NewHash("241958c0f3c23a0b771e9f853b301e0687dfd3c6"): 675380, + core.NewHash("9dc2b05b32034c1c1775fc8dd0cd48cfce01d442"): 677721, + core.NewHash("434f8c3719bbc88ec7805374040d04d87d6688e6"): 679821, + core.NewHash("e9f7f2ad89bf3b9a8a7f24a5fcafe54960a5db0a"): 681096, + core.NewHash("3f8ed4284d2acd7f70c7487dcebd41a5e7895bd3"): 682048, + core.NewHash("7ecedadb121b9f8be2427034c456ed4defce912b"): 682202, + core.NewHash("4e7f23b9ec0af1f2b47d4b346d0319824938f6b5"): 682259, + core.NewHash("9e1445ecf88b478ac874a2559e3031d278f8ff23"): 682303, + core.NewHash("4a38d63e29739d5cb722a21d5d4ade283fba8908"): 682373, + core.NewHash("8a6affbb5ac8c4c026b9ea7201f28e0ca953a320"): 682511, + core.NewHash("a664d25a1b8ad1b1f42f7ff1d56d66ced15c951a"): 683139, + core.NewHash("e4ac0c396e8e17a8c0b993450d5cffa502b5757f"): 683981, + core.NewHash("da15d58c2241988875d295c51d577a5e5b4a9def"): 684092, + core.NewHash("012f53686cf7cb59399d73c095f736852f02aa2b"): 684137, + core.NewHash("0c7c36dcf83cb21184381a062028fe2d75175add"): 848815, + core.NewHash("2c4e0ff99da5b92409e8930a692aad52ac550bc3"): 849458, + core.NewHash("3b42f20f89049b07d9f76309009785e837ffde15"): 849744, + core.NewHash("1d278497f58f2b7c0372f75ffc84d0fce70e339f"): 849912, + core.NewHash("7e1d10c42c55cac55693d91a6f7be267a3ea83c3"): 851309, + core.NewHash("6d867fb4ed06b33d564abe812772d1bacdbee06b"): 851353, + core.NewHash("3b4ac3b287ec2a6decac0772946e44c937880ac8"): 851397, + core.NewHash("d39f691336c599c73f7adc834eb3c14115eec8b0"): 851597, + core.NewHash("323f6424aab4a131a6fb2f44620ef5182e7d3967"): 851798, + core.NewHash("a651be4640da4169bbe8b53397e27f7cb0fb2ab2"): 851988, + core.NewHash("689c514641bab744d37551bd5cdd22f50f5c2c24"): 852034, + core.NewHash("55414ac3c1ad271d89ba06525b3589f2329a8d45"): 852224, + core.NewHash("fd41513552397824c5158c932a17b3b81675599b"): 852465, + core.NewHash("f6c23df070283f22ac9891aa523c25fad6c59daf"): 852511, + core.NewHash("5116e3c6c9be0f387bddcfe824f0370c7722cc16"): 852626, + core.NewHash("d9ffc29e96a0cd877fc8f28d8e3a72c282613c66"): 852672, + core.NewHash("81ffd44fe50c82d655c0e1a9adfe328712842e1d"): 852713, + core.NewHash("f3c2222e079b02a634a9bbdad83b832f38477f31"): 852759, + core.NewHash("e6c999ca662262eac1d9c7e1912490a8208dc0e5"): 852799, + core.NewHash("e760d3426d8ed4ce9e7cca493bbc731d4b9f60af"): 852848, + core.NewHash("a5fa1887a171bc8b4102248f1dfa3b1aeb818017"): 853028, + core.NewHash("6cd6b668cc1e53212d875d69d77c56557f132d06"): 853095, + core.NewHash("ce035d8f94191cf55bf8e4b7ed6dfa9284ffbc80"): 853139, + core.NewHash("2f2ed19e58c495443430546b255fc6ec1dd47dec"): 857018, + core.NewHash("6f5b517e7667612109bfc9ea5176778180c62788"): 857051, + core.NewHash("570b7a5df742ee34832daea38f380ed7bd79b222"): 857158, + core.NewHash("8f1c88295dd20c2ae4dd3d49717736326e9f95ab"): 857202, + core.NewHash("f84e87b72669cc06c61af443725a54633d4cdcc7"): 857245, + core.NewHash("234b1e3f67a1f3ccdc331083a0201cd59fbc260c"): 857289, + core.NewHash("318064ed47da4a0107faf0b8b378c9f819ff971b"): 857334, + core.NewHash("4d57bc04bc0dc422d3185ad43ba3cb31ccc3d19d"): 857378, + core.NewHash("73fab1abdeb364c8be2982c3a7f9be8de7f7d67e"): 857398, + core.NewHash("be25ae0a4ce05f4286e951d716277311549a9e06"): 857641, + core.NewHash("dff500d9dee74a8d055ef8db08286e53b5699658"): 857674, + core.NewHash("d63ac9f689f3400ef6323ac707278139a5eadec8"): 858521, + core.NewHash("6e1121bbdc21c47ecbcfb5d3618bfc057515712d"): 859243, + core.NewHash("ca15fec9eee3f9542ddd35df35c75bceda9c0a1f"): 859281, + core.NewHash("b1a6e1950965d6a9139c7e2623f561c28bae8b31"): 860001, + core.NewHash("3275711b1d133286045645f032bd116ca43688bb"): 860046, + core.NewHash("b2824ee8a2e687997df0343b8896ecd842910233"): 860537, + core.NewHash("425e2c6260f5e304b71e4f73c66fc5e64d83a2c4"): 860567, + core.NewHash("37861a263f9c8a9e9985c34cf2ad75eeca1ecea1"): 860778, + core.NewHash("bde4234d0caf6d3fc2f71025e9cc29d973f57baf"): 860879, + core.NewHash("ec4dd8d0d7b350e8c69ec4f60fa28421a67ef161"): 860900, + core.NewHash("e77dab660b4f3a236be407d343340daf79f0d10e"): 861087, + core.NewHash("11d7461523175d2130dc380268cf440a7be3b1e2"): 861296, + core.NewHash("85dc905b5bdbb215441e3f31b2a40728e7d7418a"): 861370, + core.NewHash("89f3de6a29d90cb25455b9ee3af9c54b4bfe4223"): 861416, + core.NewHash("d32688dfea858452218546436998abae5acef845"): 861440, + core.NewHash("dd4edf862d8bbf01aa650ca4de3488882e63b3a6"): 861606, + core.NewHash("b613a1b900449663b90fae16eb71d7609876a9f8"): 861714, + core.NewHash("399adae297ae698c0c21d847a55c28443878d952"): 862024, + core.NewHash("41d3f71eeff8b9c977ab745fbd19ebe3e425563b"): 862228, + core.NewHash("13b03eea774269d347beb215404483d4bd9f9e1d"): 862264, + core.NewHash("b9a3dc3a73bf91bd6b0007b53c350d2998da4004"): 862453, + core.NewHash("7c785f0657632ee002ee9215bfc26f37b34f09b9"): 862499, + core.NewHash("2ba39bc32fc7950c2cbefbf48c9d589fee4da339"): 862519, + core.NewHash("345b561442828e63df2b0c4570e1bd203f6d24f2"): 862590, + core.NewHash("7b5e8e774c8237726089e8dae65f8386c3c34002"): 862759, + core.NewHash("a0109c68c216bf22d4349cb8a9a492a2d46b0abd"): 862788, + core.NewHash("aec2c2386441d4f2964787f04295604ad1331923"): 862808, + core.NewHash("e0693956e1e6ef484c9983ddb4837f60ab870493"): 862902, + core.NewHash("8edd7be143fc33594e4b2f19808b0382f2b72e9b"): 863156, + core.NewHash("8fb026bd9a9614815cebfc6308d5623217ba6685"): 863309, + core.NewHash("d72d70b89c0a3c46bc764c028373e9da1517a247"): 863355, + core.NewHash("283b893e416e1acd187f926791f8b9f00846e37d"): 863477, + core.NewHash("baee2e7214bb729d32509dcf1cba5433ac596f9d"): 863672, + core.NewHash("c2a8138de891f372e5061e3aa663e15e0db05372"): 863793, + core.NewHash("7346056fbdcd29d929d9858c2cde209086ef102a"): 863889, + core.NewHash("9698d0cd782fb2c13cddb77cc14850ca06a157da"): 863932, + core.NewHash("a187250804316f3728ab2a878c8601779594dfe2"): 863977, + core.NewHash("025df7114563473185a1efa3b6aecf37d71dc279"): 864050, + core.NewHash("f08204255999971858479c9e9faff268ee474ec9"): 864095, + core.NewHash("b413ceba2f11c53031fa0466b26bd2b513590482"): 868374, + core.NewHash("5d32322e3bea9cdb57e113ca8084b6785d7321fa"): 870809, + core.NewHash("516296d5c2ba689335a81f3e79f2e1fa8766e970"): 872897, + core.NewHash("e6fcb5f075501f68f5f118a28b85b47123021a97"): 872959, + core.NewHash("9bf161160673d462c54a22f2cf8f62179f3a2851"): 873015, + core.NewHash("4d68b03f63abc17217bea3fe362b3e18c94d61a4"): 873155, + core.NewHash("72288b356488697f2d181f231d524c8100309476"): 873272, + core.NewHash("8179572ab662ffdc07c6b103bd91075304a2a5b5"): 873304, + core.NewHash("75c2434d9bc00d3ac96229c32d0e5b5b4bfee58d"): 875163, + core.NewHash("af84f09463268f4e1fd25ae9e1e77601aff76144"): 875329, + core.NewHash("074678f90a5b4cb8f11644a8e22867aa3ac8061e"): 875368, + core.NewHash("0d2b4db223355a0f9eecdea194b113a4e99f0ed8"): 876792, + core.NewHash("67aab8b1c18799d9493d2718dae16163ac47ba93"): 882090, + core.NewHash("2f2656ea16b9f76ead916a61de3a154ddb411c42"): 882179, + core.NewHash("32f02909ee45570b4b1c9fddbf04dfe5f4b6b0d6"): 882198, + core.NewHash("92463fd026dce2e0c54f59dc173b51284ffdaa89"): 882811, + core.NewHash("d02967a2c86f0595fad44ef1238415f080b30538"): 883634, + core.NewHash("dd7902c0e8f258577538b903ed8d4fac27da8e58"): 885068, + core.NewHash("b59e15e526a5855bb73735d5099bc6e9b6483422"): 885925, + core.NewHash("d460c5305918848dbe193012939984e75aa8009f"): 885995, + core.NewHash("415461e06a6c87aabdd12f15d06bd424fe578975"): 886953, + core.NewHash("15319c6835652d0b76a4c22ad2eb71e510866ece"): 887811, + core.NewHash("145946f05044c59331c64f27dbe7b18eab6ff541"): 888066, + core.NewHash("a2e858adcc6da92c06db180787b0c6fe7a82fcc8"): 889824, + core.NewHash("1138f7514b3a075d6eb8e6c0e45d652dc76c7478"): 892787, + core.NewHash("419bc54f9a642bfbdd8a8b98375031ae1b5b666a"): 894256, + core.NewHash("1b393db6181795106ae58cb247cc9f26b3790abf"): 895539, + core.NewHash("9580d22150618972b314eb14c84c4c01b8a24674"): 895972, + core.NewHash("1fd851c8d9bfde4d96a527506529f2a99823f3c7"): 896770, + core.NewHash("7e301a2b068a5a2c7356a6cb3b9737be4341e990"): 896807, + core.NewHash("e181803f3f7ba58bc60ff398d1e17528a05c92fb"): 896966, + core.NewHash("d1decd013e4ab65871e7d8b08e039ea6570e54d1"): 897414, + core.NewHash("aa3dc000927820d796220ac3d8334af90e77bc8f"): 897557, + core.NewHash("52969a1b123e3ed5649ac39b092cc3fcf706403f"): 897685, + core.NewHash("63bfbeb88ad13765c9b3a6b041fe21c8689216c2"): 897911, + core.NewHash("cc016d3df809c995874952f2969c4c7df243c82f"): 897952, + core.NewHash("e502f9aeb49f62762d414828c8da21783b9176ac"): 898069, + core.NewHash("649f4ed6493498dbadd68cf0aba5950b1d0592eb"): 898140, + core.NewHash("d5b912b1b5c7c70007c70528ac6386bb4c8fb643"): 898259, + core.NewHash("91d4d88967a5f0338924dbb50d8de97d18714018"): 901362, + core.NewHash("f13a0aa545483a4708ea8faec270779aef49ad45"): 901585, + core.NewHash("eeadc199565619f319aa548522536494ebc0e28a"): 901840, + core.NewHash("df2d62ab2287a91beba9600f2810ecd211fa2ee1"): 902487, + core.NewHash("e1533bfce59a0caa2439aaefb1fce221c7eaa3c2"): 902558, + core.NewHash("460dd0d504656ef26e1157c6703c33cf4686f381"): 903764, + core.NewHash("1c6d33d985f04b8e7b2fdc69bd95dea40a41aebb"): 904867, + core.NewHash("781aaf4c2108c631ae5c451a1caf84b11846b215"): 904915, + core.NewHash("2e8a6d1d7799a1e05794efd57b919d0c9af305f9"): 906294, + core.NewHash("06c589f595123ca9ca0bc072ad6ae160f2ead40b"): 914950, + core.NewHash("91a66c2c36fe0690bc08119183fa9d2f7f4f518f"): 915457, + core.NewHash("2e8e088b6ebe3f5b766653d7f42e34b873996979"): 915590, + core.NewHash("3f804926dbd443f07988859efa6e6c8da3bad55f"): 915623, + core.NewHash("5b9a31acd71ea3f3f75dccb6b01ee8d035efe8d9"): 915731, + core.NewHash("72fccaa9e84b62cb14df5a621f1f5330f83dcfcf"): 915924, + core.NewHash("5d6c14899fe5082c6d181903bbd71d60a476262c"): 916117, + core.NewHash("5c0ec579bd35a58e1f61c04928e40874d04c1bd6"): 916310, + core.NewHash("49e7fd8590720e9a0b2f30378442195f2388e3ca"): 916503, + core.NewHash("5dba293767781a3a5f2f7dd73a74b79400eb2e16"): 916696, + core.NewHash("b87760f2674132c921508f4e0a83d982faecf9f2"): 916889, + core.NewHash("135a9dd3e021950a2fad93c9fa9c8669b9b8bc2e"): 917082, + core.NewHash("f858f2233b54c67b472dd8dfba2090cfb02a2bf9"): 917275, + core.NewHash("3f62439f12b3d21e53c7e97f36fa336102e7f0ae"): 917468, + core.NewHash("461e10f4349816f280f3654d5d4ea85c8dafcee7"): 917605, + core.NewHash("c2b7443e6f9244cc3d527cc4b6da581531fada5a"): 917653, + core.NewHash("5aede5fa98caeb486b084b4514958aca04c4685a"): 917699, + core.NewHash("2fbddd83e4e5abb9e5a5d710c04580803635798d"): 917743, + core.NewHash("b77194be3eb4b40784dd5a9de5a0f30f4c0a5976"): 917787, + core.NewHash("103cc0a636e1acc3e2453e4d8e17704908084c88"): 917833, + core.NewHash("35b38bfa5c149d64e48ab28d82bd866a4e34902b"): 917877, + core.NewHash("2832b863dcf63807c08a6ca14c0dab8892311cf6"): 917923, + core.NewHash("c377dc17b913a8399b810b3f339190addb5becc4"): 917967, + core.NewHash("2153f39848623e34c70011eb86de2e6870da1006"): 918013, + core.NewHash("3052fa7d31c57b9ead106d2953e42f17d7ec7b32"): 918059, + core.NewHash("8ced0cf7aa007789455283fd2a2059c7c5c4c0cb"): 918103, + core.NewHash("5140797fe903815373debf0674181c79557ee479"): 918147, + core.NewHash("ae64b3cd106dae8545b1ec06c21d2a95ac7eca41"): 918193, + core.NewHash("82053110eb6aea52ec1782ddb76bb69847773537"): 918239, + core.NewHash("d03f1b9543e067b8344c140ad6c60ed09bbbe2ef"): 918283, + core.NewHash("99d30ba67e7ddd754daf8d8e8583a279336d2292"): 918326, + core.NewHash("fe3ecad6a2cfdc7b76a92abaa48e7282763cb78d"): 918371, + core.NewHash("a18d7099c62c9c593cee6a19276d62655a21483b"): 918415, + core.NewHash("0e5a769d4e3ceded709231b31894837e14b4cb84"): 918461, + core.NewHash("c38380b0bbc348ab5f0d920de9e89b5efbc043c5"): 918507, + core.NewHash("540cffc5746771d02b197375dfd05547592f6c80"): 918553, + core.NewHash("5d6f169153261525311e5e5143e7ce045c31d8e7"): 918597, + core.NewHash("785071a4ead1ec96071fbb1c6613881a6abb160c"): 918643, + core.NewHash("66853405400dd46c3b47c680521e044072dc9f54"): 918687, + core.NewHash("fab8c4db9f6b8b2de95317b7e55fd6450fb9845c"): 918733, + core.NewHash("edd29f52fe787c38fc4acaeb5b9de3ccf2144178"): 918777, + core.NewHash("986203abb91140a0ba209b8b52f6a95f698a3fd5"): 918821, + core.NewHash("9f31355591200bae412bf7b37f95b4d3d37e7192"): 918867, + core.NewHash("adea64241ad37f3a8f37929faee5dc608608af49"): 918913, + core.NewHash("b5c095019a98824c63da6d75953c0f02a6ea44f5"): 918957, + core.NewHash("697b29f1f4772e0bab5e336b211d343e9df5f689"): 919003, + core.NewHash("4a74516431f815dc2a56155bfb97093a9d8d3b11"): 919047, + core.NewHash("56b374bf8bcd2bb69904d112f9de4e1ef2b4fdb0"): 919091, + core.NewHash("2dd4b8ccc3591cdaa32af78b0759f256604f5da9"): 919134, + core.NewHash("c6bcdf6b1bf277c26181dabf9cd9687b6046762d"): 919180, + core.NewHash("91b43cd6a817bbaaed7b9a93cbea0e6a6726d978"): 919224, + core.NewHash("ff295167da616b4dac5cdd115a50ebaf2015bb47"): 919268, + core.NewHash("93c8add13e047e3336615bbd4651121af722f9a7"): 919314, + core.NewHash("26165e305fdbd4c5039f45586971efc6b1718892"): 919357, + core.NewHash("767bf3f016ac4056c93bcaddf3bc5177ad526173"): 919401, + core.NewHash("24b3c292af7a896b2c014f8b865859b0a29b99fa"): 919447, + core.NewHash("2c4f564c55a1a3badb814358e09b0e4f596fff75"): 919493, + core.NewHash("acc828058640a1d0dfbf8d0ce55833a1843f9ea6"): 919537, + core.NewHash("9d5cd2e07f9c521d9fc75cea121c0b9cfbbeb8fd"): 919582, + core.NewHash("b247ebfc32e97bb27a03ff3d348ed260b1640dd9"): 919628, + core.NewHash("49ae102217e348856f151e8e8dba122ab8cb811f"): 919674, + core.NewHash("db0f2caaf3c2612f75c780746775dc0f6d3754ca"): 919720, + core.NewHash("29b30150454b64c3c4de0f31e11a448c9d3b8e2f"): 919766, + core.NewHash("8bf8dcd8b9e678a280762332d43e5f135a5ed86c"): 919812, + core.NewHash("16d256e1e13ea6d42515c0e1eaabd9b0d4759d46"): 919856, + core.NewHash("864f4b4d077e3579a98e476ac90065de81308e0f"): 919902, + core.NewHash("3e8514e5e52253939e0f5aa92bbc8fd10ee19259"): 919948, + core.NewHash("00838f7b482a7a67303400f6443666314f9e1948"): 919994, + core.NewHash("c373df92e48f2f63a14faa3ac526fe5cdf8ad7e0"): 920040, + core.NewHash("88604cbe0eb6a7dd895b2743070af21ee7bac660"): 920084, + core.NewHash("addc532f0736444bddd9a204e3141bcea16bf19d"): 920130, + core.NewHash("1e5398da9957f0fe972989d91056edcdad34e1f3"): 920176, + core.NewHash("dd18ba349367fa8a6ba434d93b720d509be27487"): 920222, + core.NewHash("4a302a6d6dd87d9593ba141278e6593b4ad8d7a2"): 920268, + core.NewHash("59d32c9a7c446f1b478f19b504478382db66cdc9"): 920312, + core.NewHash("265f24b64272ba98bf98d4ca7e72ab4778762083"): 920356, + core.NewHash("557adef39aa0a5bb9c0d8ae5462bbcc9f2c8ede9"): 920399, + core.NewHash("7caaa9d4aa32d5ccc9d3b50e714c390416bb305b"): 920443, + core.NewHash("da415965c0bf72c1d8a4712a2c45218b77b5b5bc"): 920487, + core.NewHash("e598c85b5ceeb9f700037f09e6c0a093631c2ed5"): 920531, + core.NewHash("1a5af51847143ef3a1e764162d0bd2aa99c137bb"): 920575, + core.NewHash("72900471798e5bfe181ee7ea34b53b6e33c2f61e"): 920619, + core.NewHash("1fd3b0f651fee9ea408c83c01fd2ecfab888d84a"): 920665, + core.NewHash("b1b8961efc01b84b976546b3973c20a46bcbe0da"): 920711, + core.NewHash("eb359f8a0c748f34a1e8b28bfb0a6055bf34609f"): 920757, + core.NewHash("6d2e4643a2d9882b1b0ed4c1f1e22c58a0a5e2e5"): 920801, + core.NewHash("0842584e4a47be3fed7001b8404ea4b622617e19"): 920847, + core.NewHash("9db7902e93feb8f836d2852c53cdaf0665a8ce42"): 920891, + core.NewHash("0a0aff6ab077fca379dd8226fe2a3611b0f4bc8c"): 920937, + core.NewHash("8f8348f409e199d5227e08b1fe9895b849d6414f"): 920981, + core.NewHash("c419b37f9d517cd8ddf2ca81a4405f570dcbdb99"): 921026, + core.NewHash("9a24e85a398c99345e10b8a322cf2650254b719e"): 921069, + core.NewHash("a83073df4c6ac39b3080d4bbc83cbabf0393709a"): 921159, + core.NewHash("3c491958396cb95c84a76984b1e1c9bf76c50840"): 921181, + core.NewHash("b4fb13ddf0f48a2fc0e4285c23d8c1290799cfb6"): 921278, + core.NewHash("7f089728e65ff04c298b8c8146c3ae62ff1fdbd5"): 921351, + core.NewHash("20ed33b044da7dbd502db8bb280e4d9c5232eade"): 921395, + core.NewHash("680830b0ccfe66b543376f5ca2097a3269a6c66c"): 921439, + core.NewHash("8b7338d439c011b86e97e83835bbfb2bc4f09751"): 921481, + core.NewHash("78334238d77220ae6ad79b5f38bfd3cd05ad4407"): 921527, + core.NewHash("8711dc2d2cdf83ddaccf285789159b569a963c26"): 921627, + core.NewHash("6110d0a0d6ba5dc1204cff6ef6b5ee02ac967337"): 921672, + core.NewHash("cebf66fe4b4f7486cd971f8f1f699b8dc9d63ad4"): 921745, + core.NewHash("ea22e0cdb1f801229c8e2e485351e57940ec0eb0"): 921787, + core.NewHash("3013d66113d4387004cf7cbbd17e2b6051326644"): 921831, + core.NewHash("328b839abda0fac3dfac222f502309ddd8685134"): 921874, + core.NewHash("79430489db97be69ec17ff7bb4cc31826e0cf34e"): 921919, + core.NewHash("ec89f0821774b692f388898bf575a660e7451214"): 921962, + core.NewHash("7c89d9866d7439ef9b415ce7299dd1f98c53b6d5"): 922006, + core.NewHash("d219aad77e7829055ffa37a9b0857f6b3172260b"): 922051, + core.NewHash("2fd2a52563c516ca0af5a1040429865f5b893ed1"): 922091, + core.NewHash("151090dec034c02a727ec98359a6ce8b497e0347"): 922136, + core.NewHash("516a8f06b6cf1fc46fc23651b11299ad62d50182"): 922252, + core.NewHash("da41b3d5b107f38c88ee2554968c76af3a671fe1"): 922410, + core.NewHash("ba7a5b789bf664f39028d96809d8b198c89d64d3"): 922456, + core.NewHash("0b707a94a24c487fea9f2f213ac9d695f0bb7ed9"): 922585, + core.NewHash("35c4de6d9235ebe341583f76ca55d847664af93d"): 922714, + core.NewHash("409852169db62ac3129bf8d81ccd543019ca9380"): 922843, + core.NewHash("18b8dd68b789516d27b27e1c280da82cc95155d1"): 922888, + core.NewHash("e14a8c1efa766f9f471cc1c0534da4c7abaa3e02"): 922930, + core.NewHash("b08ceeb08ad660709b9693103c92b940bf2812ba"): 922972, + core.NewHash("51c5d98786eb982aa2275044596c27c31440864c"): 923014, + core.NewHash("eca84698ad3f151050cd44826255c94e1030c4fd"): 923056, + core.NewHash("b6d282d2fef08d5e961257e09d66ef5289baa655"): 923202, + core.NewHash("06cfb544b680bac4bd9beb4fe91fad17c07feac3"): 923347, + core.NewHash("68dafd8f68324e720a4bcaa2121c199b261d7c70"): 923467, + core.NewHash("4806ac0389b60b18f801d07f205f1591942e4146"): 923541, + core.NewHash("9bba87b7860f997071c47b115973d14d59628466"): 923587, + core.NewHash("03a145f351825c7078b1dcbe087d03a4460b67d1"): 923686, + core.NewHash("e5981e320e8cf2d1c8ac45134d37d83e55afc8bf"): 923734, + core.NewHash("fca4933692676ff43ed8d8794f7004431f70abfe"): 923783, + core.NewHash("fb46e5bb6e26f0df8e1803bcb3ef1883411706a1"): 923857, + core.NewHash("f93946071b4280f1d30afbc00def9d950ff4bfeb"): 923904, + core.NewHash("5b6c504846a41b3ab0c8f65ebc6acdc3e320d003"): 923948, + core.NewHash("9f7011ec7a5d41a8f6f0ee487b6b4274847f4ef8"): 924053, + core.NewHash("e21f2835bc434041191b4a324f07258fa412c185"): 924158, + core.NewHash("92bbe656306f2ac62a87168e3f89916148ddefcb"): 924204, + core.NewHash("89eddbd46a3911d66ea42821f8b99c758701d433"): 924325, + core.NewHash("87e31131e8e40e835fddd30b6747d628ba67e401"): 924370, + core.NewHash("15b3065eb847be435f993ab236494b5bc065143e"): 924415, + core.NewHash("627d0912da8a2e62f86ec580f95b4fef9e350ac9"): 924460, + core.NewHash("0a41b78041b3d643fb61e201073b03c796ab002a"): 924537, + core.NewHash("630c4a2eec5fecd7a3dc0af8e56bdb80dea8ae91"): 924615, + core.NewHash("7c48d578eeed9c4385ae1abb4816064f8b7146ea"): 924735, + core.NewHash("75ed5eac2a5bfe391562db01a5e6d487832d519e"): 924778, + core.NewHash("2250fef2596329d5913ad5115701d493250eda47"): 924855, + core.NewHash("95da50152987c4ceed0be7506a78e2fafa3d4675"): 924897, + core.NewHash("99aee1f8e4e33bd6a497bf6e8e8735ee7bf83b8a"): 925226, + core.NewHash("2b9703843780f6bdfbd28a5a6da014a80f4d645d"): 925465, + core.NewHash("a772c27a4ec250aae436cef2dbd4897a4f1f8b9a"): 925559, + core.NewHash("8f9df78bb9e894688b85db58d47c68dd90500024"): 925604, + core.NewHash("2fbf103660138eee37016cc07098f831d4b0fc74"): 925842, + core.NewHash("154edfb6bb50fb49ea44d8376e8bc63f67b59860"): 926006, + core.NewHash("89a59771f672b70f62de8f4ac29c2c2d8a9c634f"): 926078, + core.NewHash("570ef8e362b95839cc91c1b340a49a0a77726c1b"): 926222, + core.NewHash("e3490aa57573cb13dffb44d2881b33498f9b441e"): 926268, + core.NewHash("e84cf70ea77fa248b21a0ef40ad82cdd8cee8d82"): 926314, + core.NewHash("cc677de32e821b0b749b6b49e83fd9b838132208"): 926359, + core.NewHash("cce72f043680cb0a76d2b0e64836f39ec6a6088c"): 926430, + core.NewHash("9d2a6e2f44d63bb40d96334166409939883b03be"): 926501, + core.NewHash("4a319118dad2fc0bfa7eac5a7f19711a0542f6fa"): 926547, + core.NewHash("acc6c2ff69f94154e36c86c030071187c3bd9242"): 926593, + core.NewHash("213a63539ac70dd5ab9a8870a334bcb67445cfee"): 926639, + core.NewHash("510d7a0554550251f86e1a0145b618c98f4e4edd"): 926684, + core.NewHash("7ec7acbae224d0d9275938c74e971a6b76bf1fc6"): 926728, + core.NewHash("1695d7e1d87c4284e9c17ba77bac08a11f3942b7"): 926772, + core.NewHash("2694300a021aaedfe99cb3e24472249d2528855b"): 926815, + core.NewHash("2c71eda42608b6070be70197fa2f3f5555a5f6eb"): 927009, + core.NewHash("60102cd834a836bc8e5dd2a317e32548aed7d9b9"): 927108, + core.NewHash("2d7ca2a4436daa8e19eed31a08e236130dea05ce"): 927155, + core.NewHash("8cd46346f96d00ca3c3ae803deb8a0bc3c214e4e"): 927201, + core.NewHash("2a56f9c003a3cb244410c574c8f6ffbcd60f31e1"): 927247, + core.NewHash("79bd12cf415c6e7d85138b1fdbcc8569df2301ca"): 927290, + core.NewHash("07cc0c732dd4935b4efd5f3a5e60346ef96e79a7"): 927333, + core.NewHash("7fae145ba869164e2fbbc85523c554b0fc925114"): 927378, + core.NewHash("8e0dbe936490fec625dc1d48386a4eb619c749cf"): 927401, + core.NewHash("f75e8362dadc637bf9267d45b4a0bbf5cf82ede1"): 927447, + core.NewHash("2126e6971da2be97eb7bb34a1f5eb928f8d9fd16"): 927565, + core.NewHash("d7f6e6c146f91d2de7571dfcc56e2e8941978bc5"): 927707, + core.NewHash("fea3d0a277c1867ef36fe0c3627e023412f202b9"): 927776, + core.NewHash("4f0b69c31b3610947a1f0c0811e8959136341dae"): 927822, + core.NewHash("b94fbc4f4c70cb482180a68acbff1ee1aa43eb7b"): 927920, + core.NewHash("7f1a1a14d93c59e6bd2b3f6936f7100a4d8f71b6"): 927975, + core.NewHash("64893995d24e59c9eedda285c15c65877ead466c"): 928045, + core.NewHash("6561da22b8f98dc6b8639027956f8ee0c535b759"): 928114, + core.NewHash("9eb0c81a61cc1e5d514f8a918ece34b1469ea67f"): 928159, + core.NewHash("35ae26edd2ce5504c93a3568686b89a5223c13ab"): 928202, + core.NewHash("31123a65e4357e12f8709f245556c8d833f3d518"): 928246, + core.NewHash("155102d41162336e0c6c88240b03587b03e9b7f4"): 928362, + core.NewHash("1d279ce3b1e2572022cc9c0110acfc72770765c5"): 928478, + core.NewHash("171caf7df41da31107d7866ad66715bf6aa68c3e"): 928546, + core.NewHash("c2db767e55dbdf455412f16ff936c3424efec151"): 928591, + core.NewHash("11e28c251781353227a7b9828b8ea93c58ddf366"): 928735, + core.NewHash("d695942ef21fe9243faf924e09c8e0c010f76a8e"): 928781, + core.NewHash("e84341f65a21cc0382d6a0dd2397206c63378778"): 928822, + core.NewHash("2062cfc1b17fd9b5c26cdd2f5f135f5a608aea2c"): 928866, + core.NewHash("528f696c0e27ed3a3ecdd3dff8921ee140923ad7"): 928909, + core.NewHash("f2232ef68dd94914b379b76ad44f1432e151156d"): 929126, + core.NewHash("6fea37ff1da49f32f03d2868bde0a8d39f006dab"): 929255, + core.NewHash("c30e2e4d8fa357dca31689ebc91811e44f5558ac"): 929313, + core.NewHash("587d08be3a95a888f8bdd08593a4a89077f01f53"): 929375, + core.NewHash("f33f5bd1c337edc034a34331c7ff17d915d37f5b"): 929506, + core.NewHash("2ae1714594144b40466e44a3c706f21121960ca9"): 929551, + core.NewHash("f7e498666f23086edb63130ec4db46fe67c9df45"): 929638, + core.NewHash("f55f8fc40fc77dd61d68dde3eea6a1dc7aad9131"): 929729, + core.NewHash("483024582225b8d9c639f19e80c03a341e0c303b"): 929820, + core.NewHash("e2f10abe2f20f402bf697af802aa3334b230b822"): 929911, + core.NewHash("2eaef92254fc7462710c80b538d5b7bc92556cdb"): 929978, + core.NewHash("30fc756404c6b18472448c20faf85d991e5d40f1"): 930021, + core.NewHash("222930f3bc41f9e2203797f99f32b385d4456c58"): 930088, + core.NewHash("2d09b6e86165db36c5ce0adb5a8b24a81f35a9c2"): 930130, + core.NewHash("7f9a7763a65a2037885d70c023967fa032438a05"): 930219, + core.NewHash("18832d727a81068cf554e831eccdf437df4fc58a"): 930285, + core.NewHash("08706d2c53ccfcbd066e613d8719de8841cf5198"): 930354, + core.NewHash("fe0764eccead838daeb0a377e863b5fccb287410"): 930420, + core.NewHash("4a445067f6d676db990f9d821fd155439a6ebf19"): 930526, + core.NewHash("ee9ebb5b04f19f3bfc14e49e31b69de508e51851"): 930572, + core.NewHash("268495ee0b8ffc53249d2ff0fd381009ddb90a38"): 930618, + core.NewHash("347a8cee0c98efa8961f32c335622584e0cdea83"): 930663, + core.NewHash("5402992f28f879f334ab089533af489f41e59e81"): 930706, + core.NewHash("41e7b3dd46d7760218ac6a0ccc5d227f18195201"): 930749, + core.NewHash("93e03fb0c2809b75aecce4b092fec964985988eb"): 930794, + core.NewHash("39fec522ab08150541b4a2ce890073c874f995dc"): 930863, + core.NewHash("ed64b28daf71f6c2dcd73fdb7fa1aefb7dff18f5"): 930908, + core.NewHash("7688f8657195566a7af9cebda4a5fdfee17a95d9"): 930950, + core.NewHash("4668b0d7b9e9cc38fcd4d8d733bcd60e89695db8"): 930993, + core.NewHash("5e48d30f50293ae1e48a80c6e7357ca5f33955c1"): 931126, + core.NewHash("4d04e75fb508a96c99c472cfc6c418cd006d6c6d"): 931258, + core.NewHash("9290431f0277e2c628c0259a82ae489cc39590dd"): 931358, + core.NewHash("3518bbebef543e571ff2f878136f8fb636e40143"): 931402, + core.NewHash("936b9b66f8cb38a2c736fa716dc430304a83559b"): 931502, + core.NewHash("3aa28c47aaf53d40049df3f2bf434bd072fa1562"): 931545, + core.NewHash("f062a0a8df66ca532f3a3887eebc48d12c7d67b9"): 931588, + core.NewHash("990d5aec7818d6d9d6e9fa84eb683b0abed18669"): 931708, + core.NewHash("9ea37b0c68b868a6bd7bb202f986b2f3d090c0bb"): 931753, + core.NewHash("421918e124a56835a53ace1fbb9c1272ec10a4e6"): 931798, + core.NewHash("3d47e262bbe97b2c27a318f11ad32e7886f3ec78"): 931843, + core.NewHash("c178c9913030475b14e87bd1f8e304516e1aad61"): 931933, + core.NewHash("b25d0f8cb8a00878ec87a3aac23adae5835ab067"): 931978, + core.NewHash("7aacc702f06088c9a12f6ebf5d24622cc66d3bc9"): 932021, + core.NewHash("13fa58863c656c57ab7884f6c54456eb8571de76"): 932090, + core.NewHash("db4c3d05d4ac5bd907778d0124994ac0bc7125b7"): 932133, + core.NewHash("07ff2b0b76a2ebf3713f93a5caeb7d818a6c1525"): 932178, + core.NewHash("b72e3aa375b2cc2bf2c64c9d3686acd2cc27115a"): 932221, + core.NewHash("edae33c60c9debca5af82986240b83921f23aef3"): 932266, + core.NewHash("05f14c18008dabd7c2c26789c6a58574c50c681f"): 932407, + core.NewHash("e687e8d7e3526f6d757b9242b09fb7760610a239"): 932451, + core.NewHash("2a13c6060d2b881dced86dae6bb1da48a0b8722c"): 932495, + core.NewHash("ad7d8f0d3cb53a71394360f2ec0b7fb9596635de"): 932541, + core.NewHash("50feabba401d19cbd9211435c4696b385875f449"): 932613, + core.NewHash("96eb107e4c7938aac78a5c56f0026903ba128082"): 932660, + core.NewHash("d40804d2573ecab1f6ab845261ed280009e02be9"): 932705, + core.NewHash("b2e8cd8955e825c83e5116a88169ce8cbcab9be5"): 932751, + core.NewHash("c3091aa3b1bdc2e3c235f43c98c91d2b5cd0b07e"): 932797, + core.NewHash("3bdab2c86819b4bd5b631a2d446d73a1f84ade8d"): 932956, + core.NewHash("7f6823bcc0f5034c8498360ce4ef85a4573af282"): 932993, + core.NewHash("09066feaad58c6f096d43657eca16c33ef310929"): 933017, + core.NewHash("610d0cd33e1bae8e4bd33ff8a90d7069a1d64db8"): 933056, + core.NewHash("88f31ebed30173dbabd37fba064463b23283f2e1"): 933095, + }, +} diff --git a/storage/seekable/storage.go b/storage/seekable/storage.go new file mode 100644 index 0000000..8d75700 --- /dev/null +++ b/storage/seekable/storage.go @@ -0,0 +1,152 @@ +package seekable + +import ( + "fmt" + "os" + + "gopkg.in/src-d/go-git.v3/core" + "gopkg.in/src-d/go-git.v3/formats/packfile" + "gopkg.in/src-d/go-git.v3/storage/seekable/internal/gitdir" + "gopkg.in/src-d/go-git.v3/storage/seekable/internal/index" + "gopkg.in/src-d/go-git.v3/utils/fs" +) + +// ObjectStorage is an implementation of core.ObjectStorage that stores +// data on disk in the standard git format (this is, the .git directory). +// +// Zero values of this type are not safe to use, see the New function below. +// +// Currently only reads are supported, no writting. +// +// Also values from this type are not yet able to track changes on disk, this is, +// Gitdir values will get outdated as soon as repositories change on disk. +type ObjectStorage struct { + dir *gitdir.GitDir + index index.Index +} + +// New returns a new ObjectStorage for the git directory at the specified path. +func New(fs fs.FS, path string) (*ObjectStorage, error) { + s := &ObjectStorage{} + + var err error + s.dir, err = gitdir.New(fs, path) + if err != nil { + return nil, err + } + + s.index, err = buildIndex(s.dir) + + return s, err +} + +func buildIndex(dir *gitdir.GitDir) (index.Index, error) { + fs, idxfile, err := dir.Idxfile() + if err != nil { + if err == gitdir.ErrIdxNotFound { + return buildIndexFromPackfile(dir) + } + return nil, err + } + + return buildIndexFromIdxfile(fs, idxfile) +} + +func buildIndexFromPackfile(dir *gitdir.GitDir) (index.Index, error) { + fs, packfile, err := dir.Packfile() + if err != nil { + return nil, err + } + + f, err := fs.Open(packfile) + if err != nil { + return nil, err + } + + defer func() { + errClose := f.Close() + if err == nil { + err = errClose + } + }() + + return index.NewFromPackfile(f) +} + +func buildIndexFromIdxfile(fs fs.FS, path string) (index.Index, error) { + f, err := fs.Open(path) + if err != nil { + return nil, err + } + + defer func() { + errClose := f.Close() + if err == nil { + err = errClose + } + }() + + return index.NewFromIdx(f) +} + +// Set adds a new object to the storage. As this functionality is not +// yet supported, this method always returns a "not implemented yet" +// error an zero hash. +func (s *ObjectStorage) Set(core.Object) (core.Hash, error) { + return core.ZeroHash, fmt.Errorf("not implemented yet") +} + +// Get returns the object with the given hash, by searching for it in +// the packfile. +func (s *ObjectStorage) Get(h core.Hash) (core.Object, error) { + offset, err := s.index.Get(h) + if err != nil { + return nil, err + } + + fs, path, err := s.dir.Packfile() + if err != nil { + return nil, err + } + + f, err := fs.Open(path) + if err != nil { + return nil, err + } + + defer func() { + errClose := f.Close() + if err == nil { + err = errClose + } + }() + + _, err = f.Seek(offset, os.SEEK_SET) + if err != nil { + return nil, err + } + + r := packfile.NewSeekable(f) + r.HashToOffset = map[core.Hash]int64(s.index) + p := packfile.NewParser(r) + + return p.ReadObject() +} + +// Iter returns an iterator for all the objects in the packfile with the +// given type. +func (s *ObjectStorage) Iter(t core.ObjectType) (core.ObjectIter, error) { + var objects []core.Object + + for hash := range s.index { + object, err := s.Get(hash) + if err != nil { + return nil, err + } + if object.Type() == t { + objects = append(objects, object) + } + } + + return core.NewObjectSliceIter(objects), nil +} diff --git a/storage/seekable/storage_test.go b/storage/seekable/storage_test.go new file mode 100644 index 0000000..bc0ad3d --- /dev/null +++ b/storage/seekable/storage_test.go @@ -0,0 +1,326 @@ +package seekable_test + +import ( + "fmt" + "os" + "reflect" + "sort" + "testing" + + "gopkg.in/src-d/go-git.v3/core" + "gopkg.in/src-d/go-git.v3/formats/packfile" + "gopkg.in/src-d/go-git.v3/storage/memory" + "gopkg.in/src-d/go-git.v3/storage/seekable" + "gopkg.in/src-d/go-git.v3/storage/seekable/internal/gitdir" + "gopkg.in/src-d/go-git.v3/utils/fs" + + "github.com/alcortesm/tgz" + . "gopkg.in/check.v1" +) + +func Test(t *testing.T) { TestingT(t) } + +type FsSuite struct{} + +var _ = Suite(&FsSuite{}) + +var fixtures map[string]string // id to git dir paths (see initFixtures below) + +func fixture(id string, c *C) string { + path, ok := fixtures[id] + c.Assert(ok, Equals, true, Commentf("fixture %q not found", id)) + + return path +} + +var initFixtures = [...]struct { + id string + tgz string +}{ + { + id: "binary-relations", + tgz: "internal/gitdir/fixtures/alcortesm-binary-relations.tgz", + }, { + id: "binary-relations-no-idx", + tgz: "internal/gitdir/fixtures/alcortesm-binary-relations-no-idx.tgz", + }, { + id: "ref-deltas-no-idx", + tgz: "internal/gitdir/fixtures/ref-deltas-no-idx.tgz", + }, +} + +func (s *FsSuite) SetUpSuite(c *C) { + fixtures = make(map[string]string, len(initFixtures)) + for _, init := range initFixtures { + path, err := tgz.Extract(init.tgz) + c.Assert(err, IsNil, Commentf("error extracting %s\n", init.tgz)) + fixtures[init.id] = path + } +} + +func (s *FsSuite) TearDownSuite(c *C) { + for _, v := range fixtures { + err := os.RemoveAll(v) + c.Assert(err, IsNil, Commentf("error removing fixture %q\n", v)) + } +} + +func (s *FsSuite) TestNewErrorNotFound(c *C) { + fs := fs.NewOS() + _, err := seekable.New(fs, "not_found/.git") + c.Assert(err, Equals, gitdir.ErrNotFound) +} + +func (s *FsSuite) TestHashNotFound(c *C) { + path := fixture("binary-relations", c) + + fs := fs.NewOS() + gitPath := fs.Join(path, ".git/") + + sto, err := seekable.New(fs, gitPath) + c.Assert(err, IsNil) + + _, err = sto.Get(core.ZeroHash) + c.Assert(err, Equals, core.ErrObjectNotFound) +} + +func (s *FsSuite) TestGetCompareWithMemoryStorage(c *C) { + for i, fixId := range [...]string{ + "binary-relations", + "binary-relations-no-idx", + "ref-deltas-no-idx", + } { + path := fixture(fixId, c) + com := Commentf("at subtest %d, (fixture id = %q, extracted to %q)", + i, fixId, path) + + fs := fs.NewOS() + gitPath := fs.Join(path, ".git/") + + memSto, err := memStorageFromGitDir(fs, gitPath) + c.Assert(err, IsNil, com) + + seekableSto, err := seekable.New(fs, gitPath) + c.Assert(err, IsNil, com) + + equal, reason, err := equalsStorages(memSto, seekableSto) + c.Assert(err, IsNil, com) + c.Assert(equal, Equals, true, + Commentf("%s - %s\n", com.CheckCommentString(), reason)) + } +} + +func memStorageFromGitDir(fs fs.FS, path string) (*memory.ObjectStorage, error) { + dir, err := gitdir.New(fs, path) + if err != nil { + return nil, err + } + + fs, packfilePath, err := dir.Packfile() + if err != nil { + return nil, err + } + + f, err := fs.Open(packfilePath) + if err != nil { + return nil, err + } + + sto := memory.NewObjectStorage() + r := packfile.NewStream(f) + d := packfile.NewDecoder(r) + err = d.Decode(sto) + if err != nil { + return nil, err + } + + err = f.Close() + if err != nil { + return nil, err + } + + return sto, nil +} + +func equalsStorages(a, b core.ObjectStorage) (bool, string, error) { + for _, typ := range [...]core.ObjectType{ + core.CommitObject, + core.TreeObject, + core.BlobObject, + core.TagObject, + } { + iter, err := a.Iter(typ) + if err != nil { + return false, "", fmt.Errorf("cannot get iterator: %s", err) + } + + for { + ao, err := iter.Next() + if err != nil { + iter.Close() + break + } + + bo, err := b.Get(ao.Hash()) + if err != nil { + return false, "", fmt.Errorf("getting object with hash %s: %s", + ao.Hash(), err) + } + + equal, reason, err := equalsObjects(ao, bo) + if !equal || err != nil { + return equal, reason, fmt.Errorf("comparing objects: %s", err) + } + } + } + + return true, "", nil +} + +func equalsObjects(a, b core.Object) (bool, string, error) { + ah := a.Hash() + bh := b.Hash() + if ah != bh { + return false, fmt.Sprintf("object hashes differ: %s and %s\n", + ah, bh), nil + } + + atyp := a.Type() + btyp := b.Type() + if atyp != btyp { + return false, fmt.Sprintf("object types differ: %d and %d\n", + atyp, btyp), nil + } + + asz := a.Size() + bsz := b.Size() + if asz != bsz { + return false, fmt.Sprintf("object sizes differ: %d and %d\n", + asz, bsz), nil + } + + ac := a.Content() + if ac != nil { + bc := b.Content() + if !reflect.DeepEqual(ac, bc) { + return false, fmt.Sprintf("object contents differ"), nil + } + } + + return true, "", nil +} + +func (s *FsSuite) TestIterCompareWithMemoryStorage(c *C) { + for i, fixId := range [...]string{ + "binary-relations", + "binary-relations-no-idx", + "ref-deltas-no-idx", + } { + + path := fixture(fixId, c) + com := Commentf("at subtest %d, (fixture id = %q, extracted to %q)", + i, fixId, path) + + fs := fs.NewOS() + gitPath := fs.Join(path, ".git/") + + memSto, err := memStorageFromDirPath(fs, gitPath) + c.Assert(err, IsNil, com) + + seekableSto, err := seekable.New(fs, gitPath) + c.Assert(err, IsNil, com) + + for _, typ := range [...]core.ObjectType{ + core.CommitObject, + core.TreeObject, + core.BlobObject, + core.TagObject, + } { + + memObjs, err := iterToSortedSlice(memSto, typ) + c.Assert(err, IsNil, com) + + seekableObjs, err := iterToSortedSlice(seekableSto, typ) + c.Assert(err, IsNil, com) + + for i, o := range memObjs { + c.Assert(seekableObjs[i].Hash(), Equals, o.Hash(), com) + } + } + } +} + +func memStorageFromDirPath(fs fs.FS, path string) (*memory.ObjectStorage, error) { + dir, err := gitdir.New(fs, path) + if err != nil { + return nil, err + } + + fs, packfilePath, err := dir.Packfile() + if err != nil { + return nil, err + } + + sto := memory.NewObjectStorage() + f, err := fs.Open(packfilePath) + if err != nil { + return nil, err + } + + r := packfile.NewStream(f) + d := packfile.NewDecoder(r) + err = d.Decode(sto) + if err != nil { + return nil, err + } + + if err = f.Close(); err != nil { + return nil, err + } + + return sto, nil +} + +func iterToSortedSlice(storage core.ObjectStorage, typ core.ObjectType) ([]core.Object, + error) { + + iter, err := storage.Iter(typ) + if err != nil { + return nil, err + } + + r := make([]core.Object, 0) + for { + obj, err := iter.Next() + if err != nil { + iter.Close() + break + } + r = append(r, obj) + } + + sort.Sort(byHash(r)) + + return r, nil +} + +type byHash []core.Object + +func (a byHash) Len() int { return len(a) } +func (a byHash) Swap(i, j int) { a[i], a[j] = a[j], a[i] } +func (a byHash) Less(i, j int) bool { + return a[i].Hash().String() < a[j].Hash().String() +} + +func (s *FsSuite) TestSet(c *C) { + path := fixture("binary-relations", c) + + fs := fs.NewOS() + gitPath := fs.Join(path, ".git/") + + sto, err := seekable.New(fs, gitPath) + c.Assert(err, IsNil) + + _, err = sto.Set(&memory.Object{}) + c.Assert(err, ErrorMatches, "not implemented yet") +} diff --git a/tag_test.go b/tag_test.go index 46d53fc..239dec3 100644 --- a/tag_test.go +++ b/tag_test.go @@ -60,8 +60,8 @@ func (s *SuiteTag) TestCommit(c *C) { r, ok := s.repos[t.repo] c.Assert(ok, Equals, true) k := 0 - for hash, expected := range t.tags { - if expected.Type != core.CommitObject { + for hash, exp := range t.tags { + if exp.Type != core.CommitObject { continue } tag, err := r.Tag(core.NewHash(hash)) @@ -69,7 +69,7 @@ func (s *SuiteTag) TestCommit(c *C) { commit, err := tag.Commit() c.Assert(err, IsNil) c.Assert(commit.Type(), Equals, core.CommitObject) - c.Assert(commit.Hash.String(), Equals, expected.Object) + c.Assert(commit.Hash.String(), Equals, exp.Object) k++ } } @@ -80,8 +80,8 @@ func (s *SuiteTag) TestTree(c *C) { r, ok := s.repos[t.repo] c.Assert(ok, Equals, true) k := 0 - for hash, expected := range t.tags { - if expected.Type != core.TreeObject { + for hash, exp := range t.tags { + if exp.Type != core.TreeObject { continue } tag, err := r.Tag(core.NewHash(hash)) @@ -89,7 +89,7 @@ func (s *SuiteTag) TestTree(c *C) { tree, err := tag.Tree() c.Assert(err, IsNil) c.Assert(tree.Type(), Equals, core.TreeObject) - c.Assert(tree.Hash.String(), Equals, expected.Object) + c.Assert(tree.Hash.String(), Equals, exp.Object) k++ } } @@ -100,18 +100,18 @@ func (s *SuiteTag) TestBlob(c *C) { r, ok := s.repos[t.repo] c.Assert(ok, Equals, true) k := 0 - for hashString, expected := range t.tags { - if expected.Type != core.BlobObject { + for hashString, exp := range t.tags { + if exp.Type != core.BlobObject { continue } hash := core.NewHash(hashString) tag, err := r.Tag(hash) c.Assert(err, IsNil) - testTagExpected(c, tag, hash, expected, "") + testTagExpected(c, tag, hash, exp, "") blob, err := tag.Blob() c.Assert(err, IsNil) c.Assert(blob.Type(), Equals, core.BlobObject) - c.Assert(blob.Hash.String(), Equals, expected.Object) + c.Assert(blob.Hash.String(), Equals, exp.Object) k++ } } @@ -122,53 +122,53 @@ func (s *SuiteTag) TestObject(c *C) { r, ok := s.repos[t.repo] c.Assert(ok, Equals, true) k := 0 - for hashString, expected := range t.tags { - if expected.Type != core.BlobObject { + for hashString, exp := range t.tags { + if exp.Type != core.BlobObject { continue } hash := core.NewHash(hashString) tag, err := r.Tag(hash) c.Assert(err, IsNil) - testTagExpected(c, tag, hash, expected, "") + testTagExpected(c, tag, hash, exp, "") obj, err := tag.Object() c.Assert(err, IsNil) - c.Assert(obj.Type(), Equals, expected.Type) - c.Assert(obj.ID().String(), Equals, expected.Object) + c.Assert(obj.Type(), Equals, exp.Type) + c.Assert(obj.ID().String(), Equals, exp.Object) k++ } } } -func testTagExpected(c *C, tag *Tag, hash core.Hash, expected expectedTag, comment string) { - when, err := time.Parse(time.RFC3339, expected.When) +func testTagExpected(c *C, tag *Tag, hash core.Hash, exp expectedTag, com string) { + when, err := time.Parse(time.RFC3339, exp.When) c.Assert(err, IsNil) c.Assert(tag, NotNil) c.Assert(tag.Hash.IsZero(), Equals, false) c.Assert(tag.Hash, Equals, tag.ID()) c.Assert(tag.Hash, Equals, hash) c.Assert(tag.Type(), Equals, core.TagObject) - c.Assert(tag.TargetType, Equals, expected.Type, Commentf("%stype=%v, expected=%v", comment, tag.TargetType, expected.Type)) - c.Assert(tag.Target.String(), Equals, expected.Object, Commentf("%sobject=%v, expected=%s", comment, tag.Target, expected.Object)) - c.Assert(tag.Name, Equals, expected.Tag, Commentf("subtest %d, iter %d, tag=%s, expected=%s", comment, tag.Name, expected.Tag)) - c.Assert(tag.Tagger.Name, Equals, expected.TaggerName, Commentf("subtest %d, iter %d, tagger.name=%s, expected=%s", comment, tag.Tagger.Name, expected.TaggerName)) - c.Assert(tag.Tagger.Email, Equals, expected.TaggerEmail, Commentf("subtest %d, iter %d, tagger.email=%s, expected=%s", comment, tag.Tagger.Email, expected.TaggerEmail)) - c.Assert(tag.Tagger.When.Equal(when), Equals, true, Commentf("subtest %d, iter %d, tagger.when=%s, expected=%s", comment, tag.Tagger.When, when)) - c.Assert(tag.Message, Equals, expected.Message, Commentf("subtest %d, iter %d, message=\"%s\", expected=\"%s\"", comment, tag.Message, expected.Message)) + c.Assert(tag.TargetType, Equals, exp.Type, Commentf("%stype=%v, expected=%v", com, tag.TargetType, exp.Type)) + c.Assert(tag.Target.String(), Equals, exp.Object, Commentf("%sobject=%v, expected=%s", com, tag.Target, exp.Object)) + c.Assert(tag.Name, Equals, exp.Tag, Commentf("subtest %d, iter %d, tag=%s, expected=%s", com, tag.Name, exp.Tag)) + c.Assert(tag.Tagger.Name, Equals, exp.TaggerName, Commentf("subtest %d, iter %d, tagger.name=%s, expected=%s", com, tag.Tagger.Name, exp.TaggerName)) + c.Assert(tag.Tagger.Email, Equals, exp.TaggerEmail, Commentf("subtest %d, iter %d, tagger.email=%s, expected=%s", com, tag.Tagger.Email, exp.TaggerEmail)) + c.Assert(tag.Tagger.When.Equal(when), Equals, true, Commentf("subtest %d, iter %d, tagger.when=%s, expected=%s", com, tag.Tagger.When, when)) + c.Assert(tag.Message, Equals, exp.Message, Commentf("subtest %d, iter %d, message=\"%s\", expected=\"%s\"", com, tag.Message, exp.Message)) } -func testTagIter(c *C, iter *TagIter, tags map[string]expectedTag, comment string) { +func testTagIter(c *C, iter *TagIter, tags map[string]expectedTag, com string) { for k := 0; k < len(tags); k++ { - comment = fmt.Sprintf("%siter %d: ", comment, k) + com = fmt.Sprintf("%siter %d: ", com, k) tag, err := iter.Next() c.Assert(err, IsNil) c.Assert(tag, NotNil) c.Assert(tag.Hash.IsZero(), Equals, false) - expected, ok := tags[tag.Hash.String()] - c.Assert(ok, Equals, true, Commentf("%sunexpected tag hash=%v", comment, tag.Hash)) + exp, ok := tags[tag.Hash.String()] + c.Assert(ok, Equals, true, Commentf("%sunexpected tag hash=%v", com, tag.Hash)) - testTagExpected(c, tag, tag.Hash, expected, comment) + testTagExpected(c, tag, tag.Hash, exp, com) } _, err := iter.Next() c.Assert(err, Equals, io.EOF) diff --git a/tree_test.go b/tree_test.go index 2b509f3..4bf42be 100644 --- a/tree_test.go +++ b/tree_test.go @@ -191,16 +191,16 @@ func (s *SuiteTree) TestFile(c *C) { file, err := tree.File(t.path) found := err == nil - comment := Commentf("subtest %d, path=%s, commit=%s", i, t.path, t.commit) - c.Assert(found, Equals, t.found, comment) + com := Commentf("subtest %d, path=%s, commit=%s", i, t.path, t.commit) + c.Assert(found, Equals, t.found, com) if !found { continue } - c.Assert(file.Size, Equals, t.size, comment) - c.Assert(file.Hash.IsZero(), Equals, false, comment) - c.Assert(file.Hash, Equals, file.ID(), comment) - c.Assert(file.Hash.String(), Equals, t.blobHash, comment) + c.Assert(file.Size, Equals, t.size, com) + c.Assert(file.Hash.IsZero(), Equals, false, com) + c.Assert(file.Hash, Equals, file.ID(), com) + c.Assert(file.Hash.String(), Equals, t.blobHash, com) } } diff --git a/utils/fs/fs.go b/utils/fs/fs.go new file mode 100644 index 0000000..df771dd --- /dev/null +++ b/utils/fs/fs.go @@ -0,0 +1,21 @@ +package fs + +import ( + "io" + "os" +) + +// FS interface represent an abstracted filesystem, so you can +// use NewRepositoryFromFS from any medium. +type FS interface { + Stat(path string) (os.FileInfo, error) + Open(path string) (ReadSeekCloser, error) + ReadDir(path string) ([]os.FileInfo, error) + Join(elem ...string) string +} + +// ReadSeekCloser is a Reader, Seeker and Closer. +type ReadSeekCloser interface { + io.ReadCloser + io.Seeker +} diff --git a/utils/fs/os.go b/utils/fs/os.go new file mode 100644 index 0000000..37ad75a --- /dev/null +++ b/utils/fs/os.go @@ -0,0 +1,36 @@ +package fs + +import ( + "io/ioutil" + "os" + "path/filepath" +) + +// OS is a simple FS implementation for the current host filesystem. +type OS struct{} + +// NewOS returns a new OS. +func NewOS() FS { + return &OS{} +} + +// Stat returns the filesystem info for a path. +func (o *OS) Stat(path string) (os.FileInfo, error) { + return os.Stat(path) +} + +// Open returns a ReadSeekCloser for the specified path. +func (o *OS) Open(path string) (ReadSeekCloser, error) { + return os.Open(path) +} + +// ReadDir returns the filesystem info for all the archives under the +// specified path. +func (o *OS) ReadDir(path string) ([]os.FileInfo, error) { + return ioutil.ReadDir(path) +} + +// Join joins the specified elements using the filesystem separator. +func (o *OS) Join(elem ...string) string { + return filepath.Join(elem...) +} diff --git a/utils/fs/os_test.go b/utils/fs/os_test.go new file mode 100644 index 0000000..84fe895 --- /dev/null +++ b/utils/fs/os_test.go @@ -0,0 +1,151 @@ +package fs + +import ( + "io/ioutil" + "os" + "testing" + + "github.com/alcortesm/tgz" + . "gopkg.in/check.v1" +) + +func Test(t *testing.T) { TestingT(t) } + +type FSImplSuite struct { + dir string +} + +var _ = Suite(&FSImplSuite{}) + +func (s *FSImplSuite) SetUpSuite(c *C) { + dir, err := tgz.Extract("../../storage/seekable/internal/gitdir/fixtures/spinnaker-gc.tgz") + c.Assert(err, IsNil) + s.dir = dir +} + +func (s *FSImplSuite) TearDownSuite(c *C) { + err := os.RemoveAll(s.dir) + c.Assert(err, IsNil) +} + +func (s *FSImplSuite) TestJoin(c *C) { + fs := NewOS() + for i, test := range [...]struct { + input []string + expected string + }{ + { + input: []string{}, + expected: "", + }, { + input: []string{"a"}, + expected: "a", + }, { + input: []string{"a", "b"}, + expected: "a/b", + }, { + input: []string{"a", "b", "c"}, + expected: "a/b/c", + }, + } { + obtained := fs.Join(test.input...) + com := Commentf("test %d:\n\tinput = %v", i, test.input) + c.Assert(obtained, Equals, test.expected, com) + } +} + +func (s *FSImplSuite) TestStat(c *C) { + fs := NewOS() + for i, path := range [...]string{ + ".git/index", + ".git/info/refs", + ".git/objects/pack/pack-584416f86235cac0d54bfabbdc399fb2b09a5269.pack", + } { + path := fs.Join(s.dir, path) + com := Commentf("test %d", i) + + real, err := os.Open(path) + c.Assert(err, IsNil, com) + + expected, err := real.Stat() + c.Assert(err, IsNil, com) + + obtained, err := fs.Stat(path) + c.Assert(err, IsNil, com) + + c.Assert(obtained, DeepEquals, expected, com) + + err = real.Close() + c.Assert(err, IsNil, com) + } +} + +func (s *FSImplSuite) TestStatErrors(c *C) { + fs := NewOS() + for i, test := range [...]struct { + input string + errRegExp string + }{ + { + input: "bla", + errRegExp: ".*bla: no such file or directory", + }, { + input: "bla/foo", + errRegExp: ".*bla/foo: no such file or directory", + }, + } { + com := Commentf("test %d", i) + _, err := fs.Stat(test.input) + c.Assert(err, ErrorMatches, test.errRegExp, com) + } +} + +func (s *FSImplSuite) TestOpen(c *C) { + fs := NewOS() + for i, test := range [...]string{ + ".git/index", + ".git/info/refs", + ".git/objects/pack/pack-584416f86235cac0d54bfabbdc399fb2b09a5269.pack", + } { + com := Commentf("test %d", i) + path := fs.Join(s.dir, test) + + real, err := os.Open(path) + c.Assert(err, IsNil, com) + realData, err := ioutil.ReadAll(real) + c.Assert(err, IsNil, com) + err = real.Close() + c.Assert(err, IsNil, com) + + obtained, err := fs.Open(path) + c.Assert(err, IsNil, com) + obtainedData, err := ioutil.ReadAll(obtained) + c.Assert(err, IsNil, com) + err = obtained.Close() + c.Assert(err, IsNil, com) + + c.Assert(obtainedData, DeepEquals, realData, com) + } +} + +func (s *FSImplSuite) TestReadDir(c *C) { + fs := NewOS() + for i, test := range [...]string{ + ".git/info", + ".", + "", + ".git/objects", + ".git/objects/pack", + } { + com := Commentf("test %d", i) + path := fs.Join(s.dir, test) + + expected, err := ioutil.ReadDir(path) + c.Assert(err, IsNil, com) + + obtained, err := fs.ReadDir(path) + c.Assert(err, IsNil, com) + + c.Assert(obtained, DeepEquals, expected, com) + } +} |