Re-tooled mandocdb using sqlite3 and ohash.

See the tech@ mailing list entries in June 2012 for details, as well as the
discuss@ mailing list entries from March 2012.
Among other changes, this utility now:
 1.  uses a single sqlite3 database instead of several berkeley dbs
 2.  stores utf-8 encoded strings
 3.  using ohash to aggressively hash its contents
 4.  using fts() instead of manually walking directories

1 files changed, 31 insertions, 177 deletions

diff --git a/mandocdb.8 b/mandocdb.8
index 99469d4a..4cdd8cab 100644
--- a/mandocdb.8
+++ b/mandocdb.8
@@ -1,6 +1,6 @@
 .\"	$Id$
 .\"
-.\" Copyright (c) 2011 Kristaps Dzonsons <kristaps@bsd.lv>
+.\" Copyright (c) 2011, 2012 Kristaps Dzonsons <kristaps@bsd.lv>
 .\"
 .\" Permission to use, copy, modify, and distribute this software for any
 .\" purpose with or without fee is hereby granted, provided that the above
@@ -22,17 +22,17 @@
 .Nd index UNIX manuals
 .Sh SYNOPSIS
 .Nm
-.Op Fl avW
+.Op Fl anvW
 .Op Fl C Ar file
 .Nm
-.Op Fl avW
+.Op Fl anvW
 .Ar dir ...
 .Nm
-.Op Fl vW
+.Op Fl nvW
 .Fl d Ar dir
 .Op Ar
 .Nm
-.Op Fl vW
+.Op Fl nvW
 .Fl u Ar dir
 .Op Ar
 .Nm
@@ -42,21 +42,15 @@ The
 .Nm
 utility extracts keywords from
 .Ux
-manuals and indexes them in a
-.Sx Keyword Database
-and
-.Sx Index Database
-for fast retrieval by
+manuals and indexes them in a database for fast retrieval by
 .Xr apropos 1 ,
 .Xr whatis 1 ,
 and
-.Xr man 1 Ns 's
-.Fl k
-option.
+.Xr man 1 .
 .Pp
 By default,
 .Nm
-creates databases in each
+creates a database in each
 .Ar dir
 using the files
 .Sm off
@@ -70,14 +64,16 @@ and
 .Op Ar arch Li /
 .Ar title . Sy 0
 .Sm on
-in that directory;
-existing databases are truncated.
+in that directory.
+Existing databases are replaced.
 If
 .Ar dir
 is not provided,
 .Nm
 uses the default paths stipulated by
-.Xr man 1 .
+.Xr manpath 1 ,
+or
+.Xr man.conf 5 .
 .Pp
 The arguments are as follows:
 .Bl -tag -width "-C file"
@@ -94,15 +90,17 @@ format.
 Merge (remove and re-add)
 .Ar
 to the database in
-.Ar dir
-without truncating it.
+.Ar dir .
+.It Fl n
+Do not create or modify any database;
+scan and parse only.
 .It Fl t Ar
 Check the given
 .Ar files
 for potential problems.
-No databases are modified.
 Implies
-.Fl a
+.Fl a ,
+.Fl n ,
 and
 .Fl W .
 All diagnostic messages are printed to the standard output;
@@ -111,8 +109,7 @@ the standard error output is not used.
 Remove
 .Ar
 from the database in
-.Ar dir
-without truncating it.
+.Ar dir .
 .It Fl v
 Display all files added or removed to the index.
 .It Fl W
@@ -123,171 +120,28 @@ to the standard error output.
 If fatal parse errors are encountered while parsing, the offending file
 is printed to stderr, omitted from the index, and the parse continues
 with the next input file.
-.Ss Index Database
-The index database,
-.Pa whatis.index ,
-is a
-.Xr recno 3
-database with record values consisting of
-.Pp
-.Bl -enum -compact
-.It
-the character
-.Cm d ,
-.Cm a ,
-or
-.Cm c
-to indicate the file type
-.Po
-.Xr mdoc 7 ,
-.Xr man 7 ,
-and post-formatted, respectively
-.Pc ,
-.It
-the filename relative to the databases' path,
-.It
-the manual section,
-.It
-the manual title,
-.It
-the architecture
-.Pq often empty ,
-.It
-and the description.
-.El
-.Pp
-Each of the above is NUL-terminated.
-.Pp
-If the record value is zero-length, it is unassigned.
-.Ss Keyword Database
-The keyword database,
-.Pa whatis.db ,
-is a
-.Xr btree 3
-database of NUL-terminated keywords (record length is non-zero string
-length plus one) mapping to a 16-byte binary field consisting of the
-64-bit keyword type and the 64-bit
-.Sx Index Database
-record number, both in network-byte order.
-.Pp
-The type bit-mask consists of the following
-values mapping into
-.Xr mdoc 7
-macro identifiers:
-.Pp
-.Bl -column "x0x0000000000000001ULLx" "xLix" -offset indent -compact
-.It Li 0x0000000000000001ULL Ta \&An
-.It Li 0x0000000000000002ULL Ta \&Ar
-.It Li 0x0000000000000004ULL Ta \&At
-.It Li 0x0000000000000008ULL Ta \&Bsx
-.It Li 0x0000000000000010ULL Ta \&Bx
-.It Li 0x0000000000000020ULL Ta \&Cd
-.It Li 0x0000000000000040ULL Ta \&Cm
-.It Li 0x0000000000000080ULL Ta \&Dv
-.It Li 0x0000000000000100ULL Ta \&Dx
-.It Li 0x0000000000000200ULL Ta \&Em
-.It Li 0x0000000000000400ULL Ta \&Er
-.It Li 0x0000000000000800ULL Ta \&Ev
-.It Li 0x0000000000001000ULL Ta \&Fa
-.It Li 0x0000000000002000ULL Ta \&Fl
-.It Li 0x0000000000004000ULL Ta \&Fn
-.It Li 0x0000000000008000ULL Ta \&Ft
-.It Li 0x0000000000010000ULL Ta \&Fx
-.It Li 0x0000000000020000ULL Ta \&Ic
-.It Li 0x0000000000040000ULL Ta \&In
-.It Li 0x0000000000080000ULL Ta \&Lb
-.It Li 0x0000000000100000ULL Ta \&Li
-.It Li 0x0000000000200000ULL Ta \&Lk
-.It Li 0x0000000000400000ULL Ta \&Ms
-.It Li 0x0000000000800000ULL Ta \&Mt
-.It Li 0x0000000001000000ULL Ta \&Nd
-.It Li 0x0000000002000000ULL Ta \&Nm
-.It Li 0x0000000004000000ULL Ta \&Nx
-.It Li 0x0000000008000000ULL Ta \&Ox
-.It Li 0x0000000010000000ULL Ta \&Pa
-.It Li 0x0000000020000000ULL Ta \&Rs
-.It Li 0x0000000040000000ULL Ta \&Sh
-.It Li 0x0000000080000000ULL Ta \&Ss
-.It Li 0x0000000100000000ULL Ta \&St
-.It Li 0x0000000200000000ULL Ta \&Sy
-.It Li 0x0000000400000000ULL Ta \&Tn
-.It Li 0x0000000800000000ULL Ta \&Va
-.It Li 0x0000001000000000ULL Ta \&Vt
-.It Li 0x0000002000000000ULL Ta \&Xr
-.El
-.Sh IMPLEMENTATION NOTES
-The time to construct a new database pair grows linearly with the
-number of keywords in the input files.
-However, removing or updating entries with
-.Fl u
-or
-.Fl d ,
-respectively, grows as a multiple of the index length and input size.
 .Sh FILES
 .Bl -tag -width Ds
-.It Pa whatis.db
-A
-.Xr btree 3
-keyword database mapping keywords to a type and file reference in
-.Pa whatis.index .
-.It Pa whatis.index
-A
-.Xr recno 3
-database of indexed file-names.
-.It Pa /etc/man.conf
-The default
-.Xr man 1
-configuration file.
+.It Pa mandocdb.db
+A database of manpages relative to the directory of the file.
+This file is portable across architectures and systems, so long as the
+manpage hierarchy it indexes does not change.
+.It Pa mandocdb.db~
+A temporary database used during scanning and parsing.
 .El
 .Sh EXIT STATUS
-The
-.Nm
-utility exits with one of the following values:
-.Pp
-.Bl -tag -width Ds -compact
-.It 0
-No errors occurred.
-.It 5
-Invalid command line arguments were specified.
-No input files have been read.
-.It 6
-An operating system error occurred, for example memory exhaustion or an
-error accessing input files.
-Such errors cause
-.Nm
-to exit at once, possibly in the middle of parsing or formatting a file.
-The output databases are corrupt and should be removed.
-.El
-.Sh DIAGNOSTICS
-If the following errors occur, the
-.Nm
-databases should be rebuilt.
-.Bl -diag
-.It "%s: Corrupt database"
-The keyword database file indicated by
-.Pa %s
-is unreadable.
-.It "%s: Corrupt index"
-The index database file indicated by
-.Pa %s
-is unreadable.
-.It "%s: Path too long"
-The file
-.Pa %s
-is too long.
-This usually indicates database corruption or invalid command-line
-arguments.
-.El
+.Ex -std
 .Sh SEE ALSO
 .Xr apropos 1 ,
 .Xr man 1 ,
 .Xr whatis 1 ,
-.Xr btree 3 ,
-.Xr recno 3 ,
 .Xr man.conf 5
 .Sh AUTHORS
 The
 .Nm
 utility was written by
 .An Kristaps Dzonsons ,
-.Mt kristaps@bsd.lv .
+.Mt kristaps@bsd.lv ,
+and
+.An Ingo Schwarze ,
+.Mt schwarze@openbsd.org .