Document the database format and SQL code generation;

suggested by kristaps@ and espie@.

1 files changed, 227 insertions, 0 deletions

diff --git a/mansearch.3 b/mansearch.3
new file mode 100644
index 00000000..bb382db4
--- /dev/null
+++ b/mansearch.3
@@ -0,0 +1,227 @@
+.\"	$Id$
+.\"
+.\" Copyright (c) 2014 Ingo Schwarze <schwarze@openbsd.org>
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate$
+.Dt MANSEARCH 3
+.Os
+.Sh NAME
+.Nm mansearch ,
+.Nm mansearch_setup
+.Nd search manual page databases
+.Sh SYNOPSIS
+.In manpath.h
+.In mansearch.h
+.Ft int
+.Fo mansearch_setup
+.Fa "int start"
+.Fc
+.Ft int
+.Fo mansearch
+.Fa "const struct mansearch *search"
+.Fa "const struct manpaths *paths"
+.Fa "int argc"
+.Fa "char *argv[]"
+.Fa "const char *outkey"
+.Fa "struct manpage **res"
+.Fa "size_t *sz"
+.Fc
+.Sh DESCRIPTION
+The
+.Fn mansearch
+function returns information about manuals matching a search query from a
+.Xr mandoc.db 5
+SQLite3 database.
+.Pp
+The query arguments are as follows:
+.Bl -tag -width Ds
+.It Fa "const struct mansearch *search"
+Search options, defined in
+.In mansearch.h .
+.It Fa "const struct manpaths *paths"
+Directories to be searched, defined in
+.In manpath.h .
+.It Fa "int argc" , "char *argv[]"
+Search criteria, usually taken from the command line.
+.El
+.Pp
+The
+.Fa "const char *outkey"
+selects which data to return in the
+.Va output
+field of the
+.Fa res
+structures.
+It takes any of the macro keys defined in
+.Pa mansearch_const.c
+and described in
+.Xr apropos 1 .
+.Pp
+The output arguments are as follows:
+.Bl -tag -width Ds
+.It Fa "struct manpage **res"
+Returns a pointer to an array of result structures defined in
+.In mansearch.h .
+The user is expected to call
+.Xr free 3
+on the
+.Va file ,
+.Va names ,
+and
+.Va output
+fields of all structures, as well as the
+.Fa res
+array itself.
+.It Fa "size_t *sz"
+Returns the number of result structures contained in
+.Fa res .
+.El
+.Pp
+To speed up searches, the
+.Fn mansearch_setup
+function can optionally be called with a
+.Fa start
+argument of 1 before
+.Fn mansearch
+to set up an SQLite3 pagecache.
+If it was called, it has to be called again with a
+.Fa start
+argument of 0 after the last call to
+.Fn mansearch
+to release the memory used for the pagecache.
+.Sh IMPLEMENTATION NOTES
+For each manual page tree, the search is done in two steps.
+In the first step, a list of pages matching the search criteria is built.
+In the second step, the requested information about these pages is
+retrieved from the database and assembled into the
+.Fa res
+array.
+.Pp
+All function mentioned here are defined in the file
+.Pa mansearch.c .
+No functions except
+.Fn mansearch
+and
+.Fn sql_statement
+build any SQL code, and no functions except
+.Fn mansearch ,
+.Fn buildnames ,
+and
+.Fn buildoutput
+execute it.
+.Ss Finding matches
+The query is built using the following grammar:
+.Bd -literal -offset indent
+<query>     ::=	"SELECT * FROM mpages WHERE" <condition>
+<condition> ::=	"(" <condition> ")" |
+		<condition> "OR" <condition> |
+		<condition> "AND" <condition> |
+		"desc" <operator> "?" |
+		"id IN (SELECT pageid FROM" <subquery> ")"
+<subquery>  ::=	"names WHERE name" <operator> "?" |
+		"keys WHERE key" <operator> "? AND bits & ?"
+<operator>  ::=	"MATCH" | "REGEXP"
+.Ed
+.Pp
+The MATCH and REGEXP operators are implemented by the functions
+.Fn sql_match
+and
+.Fn sql_regexp ,
+respectively.
+This is required because SQLite3 natively neither supports
+case-insensitive substring matching nor regular expression matching,
+but only string identity, shell globbing, and the weird home-brewed
+LIKE operator.
+.Pp
+Command line parsing is done by the function
+.Fn exprcomp
+building a singly linked list of
+.Vt expr
+structures, using the helper functions
+.Fn exprterm
+and
+.Fn exprspec .
+The resulting SQL statement is assembled by the function
+.Fn sql_statement
+and evaluated in the main loop of the
+.Fn mansearch
+function.
+.Ss Assembling the results
+The names, sections, and architectures of the manuals found
+are assembled into the
+.Va names
+field of the result structure by the function
+.Fn buildnames ,
+using the following query:
+.Pp
+.Dl "SELECT * FROM mlinks WHERE pageid=? ORDER BY sec, arch, name"
+.Pp
+If the
+.Fa outkey
+differs from
+.Qq Nd ,
+the requested output data is assembled into the
+.Va output
+field of the result structure by the function
+.Fn buildoutput ,
+using the following query:
+.Pp
+.Dl "SELECT * FROM keys WHERE pageid=? AND bits & ?"
+.Sh FILES
+.Bl -tag -width mandoc.db -compact
+.It Pa mandoc.db
+The manual page database.
+.El
+.Sh EXAMPLES
+The simplest invocation
+.Pp
+.Dl apropos keyword
+.Pp
+results in the following SQL query:
+.Bd -literal
+SELECT * FROM mpages WHERE (
+  id IN (SELECT pageid FROM names WHERE name MATCH 'keyword') OR
+  desc MATCH 'keyword'
+);
+.Ed
+.Pp
+A more complicated request like
+.Pp
+.Dl apropos -s 2 Nm,Xr=getuid
+.Pp
+results in:
+.Bd -literal
+SELECT * FROM mpages WHERE (
+  id IN (SELECT pageid FROM names WHERE name MATCH 'getuid') OR
+  id IN (SELECT pageid FROM keys WHERE key MATCH 'getuid' AND bits & 4)
+) AND id IN (SELECT pageid FROM keys WHERE key REGEXP '^2$' AND bits & 2);
+.Ed
+.Sh SEE ALSO
+.Xr apropos 1 ,
+.Xr mandoc.db 5 ,
+.Xr makewhatis 8
+.Sh HISTORY
+The
+.Fn mansearch
+subsystem first appeared in
+.Ox 5.6 .
+.Sh AUTHORS
+.An -nosplit
+A module to search manual page databases was first written by
+.An Kristaps Dzonsons Aq Mt kristaps@bsd.lv
+in 2011, at first using the Berkeley DB;
+he rewrote it for SQLite3 in 2012.
+The current version received major changes from
+.An Ingo Schwarze Aq Mt schwarze@openbsd.org .