aboutsummaryrefslogblamecommitdiffstats
path: root/formats/idxfile/doc.go
blob: 74149abba783768574c5b416463e953180811dd5 (plain) (tree)

































































































































                                                                                   
/*
== Original (version 1) pack-*.idx files have the following format:

  - The header consists of 256 4-byte network byte order
    integers.  N-th entry of this table records the number of
    objects in the corresponding pack, the first byte of whose
    object name is less than or equal to N.  This is called the
    'first-level fan-out' table.

  - The header is followed by sorted 24-byte entries, one entry
    per object in the pack.  Each entry is:

    4-byte network byte order integer, recording where the
    object is stored in the packfile as the offset from the
    beginning.

    20-byte object name.

  - The file is concluded with a trailer:

    A copy of the 20-byte SHA1 checksum at the end of
    corresponding packfile.

    20-byte SHA1-checksum of all of the above.

Pack Idx file:

    --  +--------------------------------+
fanout  | fanout[0] = 2 (for example)    |-.
table   +--------------------------------+ |
        | fanout[1]                      | |
        +--------------------------------+ |
        | fanout[2]                      | |
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
        | fanout[255] = total objects    |---.
    --  +--------------------------------+ | |
main        | offset                         | | |
index       | object name 00XXXXXXXXXXXXXXXX | | |
table       +--------------------------------+ | |
        | offset                         | | |
        | object name 00XXXXXXXXXXXXXXXX | | |
        +--------------------------------+<+ |
      .-| offset                         |   |
      | | object name 01XXXXXXXXXXXXXXXX |   |
      | +--------------------------------+   |
      | | offset                         |   |
      | | object name 01XXXXXXXXXXXXXXXX |   |
      | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~   |
      | | offset                         |   |
      | | object name FFXXXXXXXXXXXXXXXX |   |
    --| +--------------------------------+<--+
trailer   | | packfile checksum              |
      | +--------------------------------+
      | | idxfile checksum               |
      | +--------------------------------+
          .-------.
                  |
Pack file entry: <+

     packed object header:
    1-byte size extension bit (MSB)
           type (next 3 bit)
           size0 (lower 4-bit)
        n-byte sizeN (as long as MSB is set, each 7-bit)
        size0..sizeN form 4+7+7+..+7 bit integer, size0
        is the least significant part, and sizeN is the
        most significant part.
     packed object data:
        If it is not DELTA, then deflated bytes (the size above
        is the size before compression).
    If it is REF_DELTA, then
      20-byte base object name SHA1 (the size above is the
        size of the delta data that follows).
          delta data, deflated.
    If it is OFS_DELTA, then
      n-byte offset (see below) interpreted as a negative
        offset from the type-byte of the header of the
        ofs-delta entry (the size above is the size of
        the delta data that follows).
      delta data, deflated.

     offset encoding:
      n bytes with MSB set in all but the last one.
      The offset is then the number constructed by
      concatenating the lower 7 bit of each byte, and
      for n >= 2 adding 2^7 + 2^14 + ... + 2^(7*(n-1))
      to the result.



== Version 2 pack-*.idx files support packs larger than 4 GiB, and
   have some other reorganizations.  They have the format:

  - A 4-byte magic number '\377tOc' which is an unreasonable
    fanout[0] value.

  - A 4-byte version number (= 2)

  - A 256-entry fan-out table just like v1.

  - A table of sorted 20-byte SHA1 object names.  These are
    packed together without offset values to reduce the cache
    footprint of the binary search for a specific object name.

  - A table of 4-byte CRC32 values of the packed object data.
    This is new in v2 so compressed data can be copied directly
    from pack to pack during repacking without undetected
    data corruption.

  - A table of 4-byte offset values (in network byte order).
    These are usually 31-bit pack file offsets, but large
    offsets are encoded as an index into the next table with
    the msbit set.

  - A table of 8-byte offset entries (empty for pack files less
    than 2 GiB).  Pack files are organized with heavily used
    objects toward the front, so most object references should
    not need to refer to this table.

  - The same trailer as a v1 pack file:

    A copy of the 20-byte SHA1 checksum at the end of
    corresponding packfile.

    20-byte SHA1-checksum of all of the above.

From:
https://www.kernel.org/pub/software/scm/git/docs/v1.7.5/technical/pack-protocol.txt
*/
package idxfile