path: root/plumbing/format/index/doc.go

// Package index implements encoding and decoding of index format files.
//
//    Git index format
//    ================
//
//    == The Git index file has the following format
//
//      All binary numbers are in network byte order. Version 2 is described
//      here unless stated otherwise.
//
//      - A 12-byte header consisting of
//
//        4-byte signature:
//          The signature is { 'D', 'I', 'R', 'C' } (stands for "dircache")
//
//        4-byte version number:
//          The current supported versions are 2, 3 and 4.
//
//        32-bit number of index entries.
//
//      - A number of sorted index entries (see below).
//
//      - Extensions
//
//        Extensions are identified by signature. Optional extensions can
//        be ignored if Git does not understand them.
//
//        Git currently supports cached tree and resolve undo extensions.
//
//        4-byte extension signature. If the first byte is 'A'..'Z' the
//        extension is optional and can be ignored.
//
//        32-bit size of the extension
//
//        Extension data
//
//      - 160-bit SHA-1 over the content of the index file before this
//        checksum.
//
//    == Index entry
//
//      Index entries are sorted in ascending order on the name field,
//      interpreted as a string of unsigned bytes (i.e. memcmp() order, no
//      localization, no special casing of directory separator '/'). Entries
//      with the same name are sorted by their stage field.
//
//      32-bit ctime seconds, the last time a file's metadata changed
//        this is stat(2) data
//
//      32-bit ctime nanosecond fractions
//        this is stat(2) data
//
//      32-bit mtime seconds, the last time a file's data changed
//        this is stat(2) data
//
//      32-bit mtime nanosecond fractions
//        this is stat(2) data
//
//      32-bit dev
//        this is stat(2) data
//
//      32-bit ino
//        this is stat(2) data
//
//      32-bit mode, split into (high to low bits)
//
//        4-bit object type
//          valid values in binary are 1000 (regular file), 1010 (symbolic link)
//          and 1110 (gitlink)
//
//        3-bit unused
//
//        9-bit unix permission. Only 0755 and 0644 are valid for regular files.
//        Symbolic links and gitlinks have value 0 in this field.
//
//      32-bit uid
//        this is stat(2) data
//
//      32-bit gid
//        this is stat(2) data
//
//      32-bit file size
//        This is the on-disk size from stat(2), truncated to 32-bit.
//
//      160-bit SHA-1 for the represented object
//
//      A 16-bit 'flags' field split into (high to low bits)
//
//        1-bit assume-valid flag
//
//        1-bit extended flag (must be zero in version 2)
//
//        2-bit stage (during merge)
//
//        12-bit name length if the length is less than 0xFFF; otherwise 0xFFF
//        is stored in this field.
//
//      (Version 3 or later) A 16-bit field, only applicable if the
//      "extended flag" above is 1, split into (high to low bits).
//
//        1-bit reserved for future
//
//        1-bit skip-worktree flag (used by sparse checkout)
//
//        1-bit intent-to-add flag (used by "git add -N")
//
//        13-bit unused, must be zero
//
//      Entry path name (variable length) relative to top level directory
//        (without leading slash). '/' is used as path separator. The special
//        path components ".", ".." and ".git" (without quotes) are disallowed.
//        Trailing slash is also disallowed.
//
//        The exact encoding is undefined, but the '.' and '/' characters
//        are encoded in 7-bit ASCII and the encoding cannot contain a NUL
//        byte (iow, this is a UNIX pathname).
//
//      (Version 4) In version 4, the entry path name is prefix-compressed
//        relative to the path name for the previous entry (the very first
//        entry is encoded as if the path name for the previous entry is an
//        empty string).  At the beginning of an entry, an integer N in the
//        variable width encoding (the same encoding as the offset is encoded
//        for OFS_DELTA pack entries; see pack-format.txt) is stored, followed
//        by a NUL-terminated string S.  Removing N bytes from the end of the
//        path name for the previous entry, and replacing it with the string S
//        yields the path name for this entry.
//
//      1-8 nul bytes as necessary to pad the entry to a multiple of eight bytes
//      while keeping the name NUL-terminated.
//
//      (Version 4) In version 4, the padding after the pathname does not
//      exist.
//
//      Interpretation of index entries in split index mode is completely
//      different. See below for details.
//
//    == Extensions
//
//    === Cached tree
//
//      Cached tree extension contains pre-computed hashes for trees that can
//      be derived from the index. It helps speed up tree object generation
//      from index for a new commit.
//
//      When a path is updated in index, the path must be invalidated and
//      removed from tree cache.
//
//      The signature for this extension is { 'T', 'R', 'E', 'E' }.
//
//      A series of entries fill the entire extension; each of which
//      consists of:
//
//      - NUL-terminated path component (relative to its parent directory);
//
//      - ASCII decimal number of entries in the index that is covered by the
//        tree this entry represents (entry_count);
//
//      - A space (ASCII 32);
//
//      - ASCII decimal number that represents the number of subtrees this
//        tree has;
//
//      - A newline (ASCII 10); and
//
//      - 160-bit object name for the object that would result from writing
//        this span of index as a tree.
//
//      An entry can be in an invalidated state and is represented by having
//      a negative number in the entry_count field. In this case, there is no
//      object name and the next entry starts immediately after the newline.
//      When writing an invalid entry, -1 should always be used as entry_count.
//
//      The entries are written out in the top-down, depth-first order.  The
//      first entry represents the root level of the repository, followed by the
//      first subtree--let's call this A--of the root level (with its name
//      relative to the root level), followed by the first subtree of A (with
//      its name relative to A), ...
//
//    === Resolve undo
//
//      A conflict is represented in the index as a set of higher stage entries.
//      When a conflict is resolved (e.g. with "git add path"), these higher
//      stage entries will be removed and a stage-0 entry with proper resolution
//      is added.
//
//      When these higher stage entries are removed, they are saved in the
//      resolve undo extension, so that conflicts can be recreated (e.g. with
//      "git checkout -m"), in case users want to redo a conflict resolution
//      from scratch.
//
//      The signature for this extension is { 'R', 'E', 'U', 'C' }.
//
//      A series of entries fill the entire extension; each of which
//      consists of:
//
//      - NUL-terminated pathname the entry describes (relative to the root of
//        the repository, i.e. full pathname);
//
//      - Three NUL-terminated ASCII octal numbers, entry mode of entries in
//        stage 1 to 3 (a missing stage is represented by "0" in this field);
//        and
//
//      - At most three 160-bit object names of the entry in stages from 1 to 3
//        (nothing is written for a missing stage).
//
//    === Split index
//
//      In split index mode, the majority of index entries could be stored
//      in a separate file. This extension records the changes to be made on
//      top of that to produce the final index.
//
//      The signature for this extension is { 'l', 'i', 'n', 'k' }.
//
//      The extension consists of:
//
//      - 160-bit SHA-1 of the shared index file. The shared index file path
//        is $GIT_DIR/sharedindex.<SHA-1>. If all 160 bits are zero, the
//        index does not require a shared index file.
//
//      - An ewah-encoded delete bitmap, each bit represents an entry in the
//        shared index. If a bit is set, its corresponding entry in the
//        shared index will be removed from the final index.  Note, because
//        a delete operation changes index entry positions, but we do need
//        original positions in replace phase, it's best to just mark
//        entries for removal, then do a mass deletion after replacement.
//
//      - An ewah-encoded replace bitmap, each bit represents an entry in
//        the shared index. If a bit is set, its corresponding entry in the
//        shared index will be replaced with an entry in this index
//        file. All replaced entries are stored in sorted order in this
//        index. The first "1" bit in the replace bitmap corresponds to the
//        first index entry, the second "1" bit to the second entry and so
//        on. Replaced entries may have empty path names to save space.
//
//      The remaining index entries after replaced ones will be added to the
//      final index. These added entries are also sorted by entry name then
//      stage.
//
//    == Untracked cache
//
//      Untracked cache saves the untracked file list and necessary data to
//      verify the cache. The signature for this extension is { 'U', 'N',
//      'T', 'R' }.
//
//      The extension starts with
//
//      - A sequence of NUL-terminated strings, preceded by the size of the
//        sequence in variable width encoding. Each string describes the
//        environment where the cache can be used.
//
//      - Stat data of $GIT_DIR/info/exclude. See "Index entry" section from
//        ctime field until "file size".
//
//      - Stat data of plumbing.excludesfile
//
//      - 32-bit dir_flags (see struct dir_struct)
//
//      - 160-bit SHA-1 of $GIT_DIR/info/exclude. Null SHA-1 means the file
//        does not exist.
//
//      - 160-bit SHA-1 of plumbing.excludesfile. Null SHA-1 means the file does
//        not exist.
//
//      - NUL-terminated string of per-dir exclude file name. This usually
//        is ".gitignore".
//
//      - The number of following directory blocks, variable width
//        encoding. If this number is zero, the extension ends here with a
//        following NUL.
//
//      - A number of directory blocks in depth-first-search order, each
//        consists of
//
//        - The number of untracked entries, variable width encoding.
//
//        - The number of sub-directory blocks, variable width encoding.
//
//        - The directory name terminated by NUL.
//
//        - A number of untracked file/dir names terminated by NUL.
//
//    The remaining data of each directory block is grouped by type:
//
//      - An ewah bitmap, the n-th bit marks whether the n-th directory has
//        valid untracked cache entries.
//
//      - An ewah bitmap, the n-th bit records "check-only" bit of
//        read_directory_recursive() for the n-th directory.
//
//      - An ewah bitmap, the n-th bit indicates whether SHA-1 and stat data
//        is valid for the n-th directory and exists in the next data.
//
//      - An array of stat data. The n-th data corresponds with the n-th
//        "one" bit in the previous ewah bitmap.
//
//      - An array of SHA-1. The n-th SHA-1 corresponds with the n-th "one" bit
//        in the previous ewah bitmap.
//
//      - One NUL.
//
//   == File System Monitor cache
//
//     The file system monitor cache tracks files for which the core.fsmonitor
//     hook has told us about changes.  The signature for this extension is
//     { 'F', 'S', 'M', 'N' }.
//
//     The extension starts with
//
//     - 32-bit version number: the current supported version is 1.
//
//     - 64-bit time: the extension data reflects all changes through the given
//       time which is stored as the nanoseconds elapsed since midnight,
//       January 1, 1970.
//
//    - 32-bit bitmap size: the size of the CE_FSMONITOR_VALID bitmap.
//
//    - An ewah bitmap, the n-th bit indicates whether the n-th index entry
//      is not CE_FSMONITOR_VALID.
//
//  == End of Index Entry
//
//    The End of Index Entry (EOIE) is used to locate the end of the variable
//    length index entries and the begining of the extensions. Code can take
//    advantage of this to quickly locate the index extensions without having
//    to parse through all of the index entries.
//
//    Because it must be able to be loaded before the variable length cache
//    entries and other index extensions, this extension must be written last.
//    The signature for this extension is { 'E', 'O', 'I', 'E' }.
//
//    The extension consists of:
//
//    - 32-bit offset to the end of the index entries
//
//    - 160-bit SHA-1 over the extension types and their sizes (but not
//      their contents).  E.g. if we have "TREE" extension that is N-bytes
//      long, "REUC" extension that is M-bytes long, followed by "EOIE",
//      then the hash would be:
//
//      SHA-1("TREE" + <binary representation of N> +
//        "REUC" + <binary representation of M>)
//
//  == Index Entry Offset Table
//
//    The Index Entry Offset Table (IEOT) is used to help address the CPU
//    cost of loading the index by enabling multi-threading the process of
//    converting cache entries from the on-disk format to the in-memory format.
//    The signature for this extension is { 'I', 'E', 'O', 'T' }.
//
//    The extension consists of:
//
//    - 32-bit version (currently 1)
//
//    - A number of index offset entries each consisting of:
//
//    - 32-bit offset from the begining of the file to the first cache entry
//      in this block of entries.
//
//    - 32-bit count of cache entries in this blockpackage index
package index