Adds support to open local repositories and to use file-based object storage (#55)v3.1.0

* remove some comments

* idx writer/reader

* Shut up ssh tests, they are annoying

* Add file scheme test to clients

* Add dummy file client

* Add test fot file client

* Make tests use fixture endpoint

* add parser for packed-refs format

* add parser for packed-refs format

* WIP adding dir.Refs() tests

* Add test for fixture refs

* refs parser for the refs directory

* Documentation

* Add Capabilities to file client

* tgz.Exatract now accpets a path instead of a Reader

* fix bug in idxfile fanout calculation

* remove dead code

* packfile documentation

* clean packfile parser code

* add core.Object.Content() and returns errors for core.ObjectStorage.Iter()

* add seekable storage

* add dir repos to NewRepository

* clean prints

* Add dir client documentation to README

* Organize the README

* README

* Clean tgz package

* Clean temp dirs after tgz tests

* Gometalinter on gitdir

* Clean pattern function

* metalinter tgz

* metalinter gitdir

* gitdir coverage and remove seekable packfile filedescriptor leak

* gitdir Idxfile tests and remove file descriptor leak

* gitdir Idxfile tests when no idx is found

* clean storage/seekable/internal/index and some formats/idxfile API issues

* clean storage/seekable

* clean formats/idx

* turn packfile/doc.go into packfile/doc.txt

* move formats/packfile/reader to decoder

* fix packfile decoder error names

* improve documentation

* comment packfile decoder errors

* comment public API (format/packfile)

* remve duplicated code in packfile decoder test

* move tracking_reader into an internal package and clean it

* use iota for packfile format

* rename packfile parse.go to packfile object_at.go

* clean packfile deltas

* fix delta header size bug

* improve delta documentation

* clean packfile deltas

* clean packfiles deltas

* clean repository.go

* Remove go 1.5 from Travis CI

Because go 1.5 does not suport internal packages.

* change local repo scheme to local://

* change "local://" to "file://" as the local scheme

* fix broken indentation

* shortens names of variables in short scopes

* more shortening of variable names

* more shortening of variable names

* Rename git dir client to "file", as the scheme used for it

* Fix file format ctor name, now that the package name has change

* Sortcut local repo constructor to not use remotes

The object storage is build directly in the repository ctor, instead
of creating a remote and waiting for the user to pull it.

* update README and fix some errors in it

* remove file scheme client

* Local respositories has now a new ctor

This is, they are no longer identified by the scheme of the URL, but are
created different from inception.

* remove unused URL field form Repository

* move all git dir logic to seekable sotrage ctor

* fix documentation

* Make formats/file/dir an internal package to storage/seekable

* change package storage/seekable to storage/fs

* clean storage/fs

* overall storage/fs clean

* more cleaning

* some metalinter fixes

* upgrade cshared to last changes

* remove dead code

* fix test error info

* remove file scheme check from clients

* fix test error message

* fix test error message

* fix error messages

* style changes

* fix comments everywhere

* style changes

* style changes

* scaffolding and tests for local packfiles without ifx files

* outsource index building from packfile to the packfile decoder

* refactor packfile header reading into a new function

* move code to generate index from packfile back to index package

* add header parsing

* fix documentation errata

* add undeltified and OFS delta support for index building from the packfile

* add tests for packfile with ref-deltas

* support for packfiles with ref-deltas and no idx

* refactor packfile format parser to reuse code

* refactor packfile format parser to reuse code

* refactor packfile format parser to reuse code

* refactor packfile format parser to reuse code

* refactor packfile format parser to reuse code

* WIP refactor packfile format parser to reuse code

* refactor packfile format parser to reuse code

* remove prints from tests

* remove prints from tests

* refactor packfile.core into packfile.parser

* rename packfile reader to something that shows it is a recaller

* rename cannot recall error

* rename packfile.Reader to packfile.ReadRecaller and document

* speed up test by using StreamReader instead of SeekableReader when possible

* clean packfile StreamReader

* stream_reader tests

* refactor packfile.StreamReader into packfile.StreamReadRecaller

* refactor packfile.SeekableReader into packfile.SeekableReadRecaller and document it

* generalize packfile.StreamReadRecaller test to all packfile.ReadRecaller implementations

* speed up storage/fs tests

* speed up tests in . by loading packfiles in memory

* speed up repository tests by using and smaller fixture

* restore doc.go files

* rename packfile.ReadRecaller implementations to shorter names

* update comments to type changes

* packfile.Parser test (WIP)

* packfile.Parser tests and add ForgetAll() to packfile.ReadRecaller

* add test for packfile.ReadRecaller.ForgetAll()

* clarify seekable being able to recallByOffset forgetted objects

* use better names for internal maps

* metalinter packfile package

* speed up some tests

* documentation fixes

* change storage.fs package name to storage.proxy to avoid confusion with new filesystem support

* New fs package and os transparent implementation

Now NewRepositoryFromFS receives a fs and a path and tests are
modified accordingly, but it is still not using for anything.

* add fs to gitdir and proxy.store

* reduce fs interface for easier implementation

* remove garbage dirs from tgz tests

* change file name gitdir/dir.go to gitdir/gitdir.go

* fs.OS tests

* metalinter utils/fs

* add NewRepositoryFromFS documentation to README

* Readability fixes to README

* move tgz to an external dependency

* move filesystem impl. example to example dir

* rename proxy/store.go to proxy/storage.go for coherence with memory/storage.go

* rename proxy package to seekable

2 files changed, 298 insertions, 0 deletions

diff --git a/examples/fs_implementation/main.go b/examples/fs_implementation/main.go
new file mode 100644
index 0000000..0371f7f
--- /dev/null
+++ b/examples/fs_implementation/main.go
@@ -0,0 +1,103 @@
+package main
+
+import (
+	"fmt"
+	"io"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"strings"
+
+	"gopkg.in/src-d/go-git.v3"
+	gogitFS "gopkg.in/src-d/go-git.v3/utils/fs"
+)
+
+func main() {
+	if len(os.Args) != 2 {
+		usage()
+		os.Exit(1)
+	}
+
+	fs := newFS(os.Args[1])
+
+	repo, err := git.NewRepositoryFromFS(fs, ".git")
+	if err != nil {
+		fmt.Fprint(os.Stderr, err)
+		os.Exit(1)
+	}
+
+	iter, err := repo.Commits()
+	if err != nil {
+		fmt.Fprint(os.Stderr, err)
+		os.Exit(1)
+	}
+	defer iter.Close()
+
+	for {
+		commit, err := iter.Next()
+		if err != nil {
+			if err == io.EOF {
+				break
+			}
+
+			fmt.Fprint(os.Stderr, err)
+			os.Exit(1)
+		}
+
+		fmt.Println(commit)
+	}
+}
+
+func usage() {
+	fmt.Fprintf(os.Stderr, "%s <path to .git dir>", os.Args[0])
+}
+
+// A simple proxy filesystem example: It mimics local filesystems, using
+// 'base' as its root and a funny path separator ("--").
+//
+// Example: when constructed with 'newFS("tmp")', a path like 'foo--bar'
+// will represent the local path "/tmp/foo/bar".
+type fs struct {
+	base string
+}
+
+const separator = "--"
+
+func newFS(path string) *fs {
+	return &fs{
+		base: path,
+	}
+}
+
+func (fs *fs) Stat(path string) (info os.FileInfo, err error) {
+	f, err := os.Open(fs.ToReal(path))
+	if err != nil {
+		return nil, err
+	}
+
+	defer func() {
+		errClose := f.Close()
+		if err == nil {
+			err = errClose
+		}
+	}()
+
+	return f.Stat()
+}
+
+func (fs *fs) ToReal(path string) string {
+	parts := strings.Split(path, separator)
+	return filepath.Join(fs.base, filepath.Join(parts...))
+}
+
+func (fs *fs) Open(path string) (gogitFS.ReadSeekCloser, error) {
+	return os.Open(fs.ToReal(path))
+}
+
+func (fs *fs) ReadDir(path string) ([]os.FileInfo, error) {
+	return ioutil.ReadDir(fs.ToReal(path))
+}
+
+func (fs *fs) Join(elem ...string) string {
+	return strings.Join(elem, separator)
+}
diff --git a/examples/fs_implementation/main_test.go b/examples/fs_implementation/main_test.go
new file mode 100644
index 0000000..4f138c2
--- /dev/null
+++ b/examples/fs_implementation/main_test.go
@@ -0,0 +1,195 @@
+package main
+
+import (
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"reflect"
+	"regexp"
+	"testing"
+
+	"github.com/alcortesm/tgz"
+)
+
+func TestMain(m *testing.M) {
+	setUp()
+	rval := m.Run()
+	tearDown()
+	os.Exit(rval)
+}
+
+func setUp() {
+	var err error
+	repo, err = tgz.Extract("../../storage/seekable/internal/gitdir/fixtures/spinnaker-gc.tgz")
+	if err != nil {
+		panic(err)
+	}
+}
+
+var repo string
+
+func tearDown() {
+	err := os.RemoveAll(repo)
+	if err != nil {
+		panic(err)
+	}
+}
+
+func TestJoin(t *testing.T) {
+	fs := newFS("")
+	for i, test := range [...]struct {
+		input    []string
+		expected string
+	}{
+		{
+			input:    []string{},
+			expected: "",
+		}, {
+			input:    []string{"a"},
+			expected: "a",
+		}, {
+			input:    []string{"a", "b"},
+			expected: "a--b",
+		}, {
+			input:    []string{"a", "b", "c"},
+			expected: "a--b--c",
+		},
+	} {
+		obtained := fs.Join(test.input...)
+		if obtained != test.expected {
+			t.Fatalf("test %d:\n\tinput = %v\n\tobtained = %v\n\texpected = %v\n",
+				i, test.input, obtained, test.expected)
+		}
+	}
+}
+
+func TestStat(t *testing.T) {
+	fs := newFS(filepath.Join(repo, ".git/"))
+	for i, path := range [...]string{
+		"index",
+		"info--refs",
+		"objects--pack--pack-584416f86235cac0d54bfabbdc399fb2b09a5269.pack",
+	} {
+		real, err := os.Open(fs.ToReal(path))
+		if err != nil {
+			t.Fatalf("test %d: openning real: %s", err)
+		}
+
+		expected, err := real.Stat()
+		if err != nil {
+			t.Fatalf("test %d: stat on real: %s", err)
+		}
+
+		obtained, err := fs.Stat(path)
+		if err != nil {
+			t.Fatalf("test %d: fs.Stat unexpected error: %s", i, err)
+		}
+
+		if !reflect.DeepEqual(obtained, expected) {
+			t.Fatalf("test %d:\n\tinput = %s\n\tobtained = %v\n\texpected = %v\n",
+				i, path, obtained, expected)
+		}
+
+		err = real.Close()
+		if err != nil {
+			t.Fatalf("test %d: closing real: %s", i, err)
+		}
+	}
+}
+
+func TestStatErrors(t *testing.T) {
+	fs := newFS(filepath.Join(repo, ".git/"))
+	for i, test := range [...]struct {
+		input     string
+		errRegExp string
+	}{
+		{
+			input:     "bla",
+			errRegExp: ".*bla: no such file or directory",
+		}, {
+			input:     "bla--foo",
+			errRegExp: ".*bla/foo: no such file or directory",
+		},
+	} {
+		expected := regexp.MustCompile(test.errRegExp)
+
+		_, err := fs.Stat(test.input)
+		if err == nil {
+			t.Fatalf("test %d: no error returned", i)
+		}
+		if !expected.MatchString(err.Error()) {
+			t.Fatalf("test %d: error missmatch\n\tobtained = %q\n\texpected regexp = %q\n",
+				i, err.Error(), test.errRegExp)
+		}
+	}
+}
+
+func TestOpen(t *testing.T) {
+	fs := newFS(filepath.Join(repo, ".git/"))
+	for i, path := range [...]string{
+		"index",
+		"info--refs",
+		"objects--pack--pack-584416f86235cac0d54bfabbdc399fb2b09a5269.pack",
+	} {
+		real, err := os.Open(fs.ToReal(path))
+		if err != nil {
+			t.Fatalf("test %d: openning real: %s", err)
+		}
+
+		realData, err := ioutil.ReadAll(real)
+		if err != nil {
+			t.Fatal("test %d: ioutil.ReadAll on real: %s", err)
+		}
+
+		err = real.Close()
+		if err != nil {
+			t.Fatal("test %d: closing real: %s", err)
+		}
+
+		obtained, err := fs.Open(path)
+		if err != nil {
+			t.Fatalf("test %d: fs.Open unexpected error: %s", i, err)
+		}
+
+		obtainedData, err := ioutil.ReadAll(obtained)
+		if err != nil {
+			t.Fatal("test %d: ioutil.ReadAll on obtained: %s", err)
+		}
+
+		err = obtained.Close()
+		if err != nil {
+			t.Fatal("test %d: closing obtained: %s", err)
+		}
+
+		if !reflect.DeepEqual(obtainedData, realData) {
+			t.Fatalf("test %d:\n\tinput = %s\n\tobtained = %v\n\texpected = %v\n",
+				i, path, obtainedData, realData)
+		}
+	}
+}
+
+func TestReadDir(t *testing.T) {
+	fs := newFS(filepath.Join(repo, ".git/"))
+	for i, path := range [...]string{
+		"info",
+		".",
+		"",
+		"objects",
+		"objects--pack",
+	} {
+		expected, err := ioutil.ReadDir(fs.ToReal(path))
+		if err != nil {
+			t.Fatalf("test %d: real ReadDir: %s", err)
+		}
+
+		obtained, err := fs.ReadDir(path)
+		if err != nil {
+			t.Fatalf("test %d: fs.ReadDir unexpected error: %s", i, err)
+		}
+
+		if !reflect.DeepEqual(obtained, expected) {
+			t.Fatalf("test %d:\n\tinput = %s\n\tobtained = %v\n\texpected = %v\n",
+				i, path, obtained, expected)
+		}
+	}
+}