Adds support to open local repositories and to use file-based object storage (#55)v3.1.0

* remove some comments

* idx writer/reader

* Shut up ssh tests, they are annoying

* Add file scheme test to clients

* Add dummy file client

* Add test fot file client

* Make tests use fixture endpoint

* add parser for packed-refs format

* add parser for packed-refs format

* WIP adding dir.Refs() tests

* Add test for fixture refs

* refs parser for the refs directory

* Documentation

* Add Capabilities to file client

* tgz.Exatract now accpets a path instead of a Reader

* fix bug in idxfile fanout calculation

* remove dead code

* packfile documentation

* clean packfile parser code

* add core.Object.Content() and returns errors for core.ObjectStorage.Iter()

* add seekable storage

* add dir repos to NewRepository

* clean prints

* Add dir client documentation to README

* Organize the README

* README

* Clean tgz package

* Clean temp dirs after tgz tests

* Gometalinter on gitdir

* Clean pattern function

* metalinter tgz

* metalinter gitdir

* gitdir coverage and remove seekable packfile filedescriptor leak

* gitdir Idxfile tests and remove file descriptor leak

* gitdir Idxfile tests when no idx is found

* clean storage/seekable/internal/index and some formats/idxfile API issues

* clean storage/seekable

* clean formats/idx

* turn packfile/doc.go into packfile/doc.txt

* move formats/packfile/reader to decoder

* fix packfile decoder error names

* improve documentation

* comment packfile decoder errors

* comment public API (format/packfile)

* remve duplicated code in packfile decoder test

* move tracking_reader into an internal package and clean it

* use iota for packfile format

* rename packfile parse.go to packfile object_at.go

* clean packfile deltas

* fix delta header size bug

* improve delta documentation

* clean packfile deltas

* clean packfiles deltas

* clean repository.go

* Remove go 1.5 from Travis CI

Because go 1.5 does not suport internal packages.

* change local repo scheme to local://

* change "local://" to "file://" as the local scheme

* fix broken indentation

* shortens names of variables in short scopes

* more shortening of variable names

* more shortening of variable names

* Rename git dir client to "file", as the scheme used for it

* Fix file format ctor name, now that the package name has change

* Sortcut local repo constructor to not use remotes

The object storage is build directly in the repository ctor, instead
of creating a remote and waiting for the user to pull it.

* update README and fix some errors in it

* remove file scheme client

* Local respositories has now a new ctor

This is, they are no longer identified by the scheme of the URL, but are
created different from inception.

* remove unused URL field form Repository

* move all git dir logic to seekable sotrage ctor

* fix documentation

* Make formats/file/dir an internal package to storage/seekable

* change package storage/seekable to storage/fs

* clean storage/fs

* overall storage/fs clean

* more cleaning

* some metalinter fixes

* upgrade cshared to last changes

* remove dead code

* fix test error info

* remove file scheme check from clients

* fix test error message

* fix test error message

* fix error messages

* style changes

* fix comments everywhere

* style changes

* style changes

* scaffolding and tests for local packfiles without ifx files

* outsource index building from packfile to the packfile decoder

* refactor packfile header reading into a new function

* move code to generate index from packfile back to index package

* add header parsing

* fix documentation errata

* add undeltified and OFS delta support for index building from the packfile

* add tests for packfile with ref-deltas

* support for packfiles with ref-deltas and no idx

* refactor packfile format parser to reuse code

* refactor packfile format parser to reuse code

* refactor packfile format parser to reuse code

* refactor packfile format parser to reuse code

* refactor packfile format parser to reuse code

* WIP refactor packfile format parser to reuse code

* refactor packfile format parser to reuse code

* remove prints from tests

* remove prints from tests

* refactor packfile.core into packfile.parser

* rename packfile reader to something that shows it is a recaller

* rename cannot recall error

* rename packfile.Reader to packfile.ReadRecaller and document

* speed up test by using StreamReader instead of SeekableReader when possible

* clean packfile StreamReader

* stream_reader tests

* refactor packfile.StreamReader into packfile.StreamReadRecaller

* refactor packfile.SeekableReader into packfile.SeekableReadRecaller and document it

* generalize packfile.StreamReadRecaller test to all packfile.ReadRecaller implementations

* speed up storage/fs tests

* speed up tests in . by loading packfiles in memory

* speed up repository tests by using and smaller fixture

* restore doc.go files

* rename packfile.ReadRecaller implementations to shorter names

* update comments to type changes

* packfile.Parser test (WIP)

* packfile.Parser tests and add ForgetAll() to packfile.ReadRecaller

* add test for packfile.ReadRecaller.ForgetAll()

* clarify seekable being able to recallByOffset forgetted objects

* use better names for internal maps

* metalinter packfile package

* speed up some tests

* documentation fixes

* change storage.fs package name to storage.proxy to avoid confusion with new filesystem support

* New fs package and os transparent implementation

Now NewRepositoryFromFS receives a fs and a path and tests are
modified accordingly, but it is still not using for anything.

* add fs to gitdir and proxy.store

* reduce fs interface for easier implementation

* remove garbage dirs from tgz tests

* change file name gitdir/dir.go to gitdir/gitdir.go

* fs.OS tests

* metalinter utils/fs

* add NewRepositoryFromFS documentation to README

* Readability fixes to README

* move tgz to an external dependency

* move filesystem impl. example to example dir

* rename proxy/store.go to proxy/storage.go for coherence with memory/storage.go

* rename proxy package to seekable

1 files changed, 108 insertions, 0 deletions

diff --git a/formats/packfile/seekable.go b/formats/packfile/seekable.go
new file mode 100644
index 0000000..ea1c501
--- /dev/null
+++ b/formats/packfile/seekable.go
@@ -0,0 +1,108 @@
+package packfile
+
+import (
+	"io"
+	"os"
+
+	"gopkg.in/src-d/go-git.v3/core"
+)
+
+// Seekable implements ReadRecaller for the io.ReadSeeker of a packfile.
+// Remembering does not actually stores any reference to the remembered
+// objects; the object offset is remebered instead and the packfile is
+// read again everytime a recall operation is requested. This saves
+// memory buy can be very slow if the associated io.ReadSeeker is slow
+// (like a hard disk).
+type Seekable struct {
+	io.ReadSeeker
+	HashToOffset map[core.Hash]int64
+}
+
+// NewSeekable returns a new Seekable that reads form r.
+func NewSeekable(r io.ReadSeeker) *Seekable {
+	return &Seekable{
+		r,
+		make(map[core.Hash]int64),
+	}
+}
+
+// Read reads up to len(p) bytes into p.
+func (r *Seekable) Read(p []byte) (int, error) {
+	return r.ReadSeeker.Read(p)
+}
+
+// ReadByte reads a byte.
+func (r *Seekable) ReadByte() (byte, error) {
+	var p [1]byte
+	_, err := r.ReadSeeker.Read(p[:])
+	if err != nil {
+		return 0, err
+	}
+
+	return p[0], nil
+}
+
+// Offset returns the offset for the next Read or ReadByte.
+func (r *Seekable) Offset() (int64, error) {
+	return r.Seek(0, os.SEEK_CUR)
+}
+
+// Remember stores the offset of the object and its hash, but not the
+// object itself.  This implementation does not check for already stored
+// offsets, as it is too expensive to build this information from an
+// index every time a get operation is performed on the SeekableReadRecaller.
+func (r *Seekable) Remember(o int64, obj core.Object) error {
+	h := obj.Hash()
+	if _, ok := r.HashToOffset[h]; ok {
+		return ErrDuplicatedObject.AddDetails("with hash %s", h)
+	}
+
+	r.HashToOffset[h] = o
+
+	return nil
+}
+
+// ForgetAll forgets all previously remembered objects.  For efficiency
+// reasons RecallByOffset always find objects, even if they have been
+// forgetted or were never remembered.
+func (r *Seekable) ForgetAll() {
+	r.HashToOffset = make(map[core.Hash]int64)
+}
+
+// RecallByHash returns the object for a given hash by looking for it again in
+// the io.ReadeSeerker.
+func (r *Seekable) RecallByHash(h core.Hash) (core.Object, error) {
+	o, ok := r.HashToOffset[h]
+	if !ok {
+		return nil, ErrCannotRecall.AddDetails("hash not found: %s", h)
+	}
+
+	return r.RecallByOffset(o)
+}
+
+// RecallByOffset returns the object for a given offset by looking for it again in
+// the io.ReadeSeerker. For efficiency reasons, this method always find objects by
+// offset, even if they have not been remembered or if they have been forgetted.
+func (r *Seekable) RecallByOffset(o int64) (obj core.Object, err error) {
+	// remember current offset
+	beforeJump, err := r.Offset()
+	if err != nil {
+		return nil, err
+	}
+
+	defer func() {
+		// jump back
+		_, seekErr := r.Seek(beforeJump, os.SEEK_SET)
+		if err == nil {
+			err = seekErr
+		}
+	}()
+
+	// jump to requested offset
+	_, err = r.Seek(o, os.SEEK_SET)
+	if err != nil {
+		return nil, err
+	}
+
+	return NewParser(r).ReadObject()
+}