summaryrefslogtreecommitdiffstats
path: root/mandocdb.c
Commit message (Collapse)AuthorAgeFilesLines
...
* More tweaking of set_basedir().Ingo Schwarze2014-06-201-23/+39
| | | | | | | | | | | | 1) Do not error out when getcwd(3) fails, only fail when inaccessibility of the cwd prevents processing of relative paths given on the command line. 2) Do not uselessly call set_basedir() twice in a row. While fts_read(3) in treescan() does cause the cwd to jump around, fts_close(3) is always called at the end, putting us back where we came from. The -d/-u fallback code already relied on this. 3) Fix the man-root-dir indicator in say().
* Some simple set_basedir() cleanup; more to come.Ingo Schwarze2014-06-191-16/+20
| | | | | | | | | | | | | | 1) Refrain from calling set_basedir() in the -t case, and do not attempt to strip anything from the file names in that case. Testing individual files cannot reasonably have any notion of a base dir. 2) Remove the possibility of passing NULL to set_basedir(). It was dangerous because it was not idempotent, and it served no purpose except closing a file descriptor right before exit(), which is pointless. Besides, the file descriptor is likely to be removed completely, soon. 3) Make sure that /foobar isn't treated as a subdirectory of /foo; this fixes a bug reported by espie@.
* Merge OpenBSD rev. 1.108 by sthen@; original commit message:Ingo Schwarze2014-06-181-1/+2
| | | | | | Don't display "unable to open mandoc.db" error messages (SQLITE_CANTOPEN) in the code which opens mandocdb's sqlite database when updating/deleting individual files (as used and only really useful for pkg_add/pkg_delete).
* Render roff escape sequences contained in manual page descriptionsIngo Schwarze2014-05-071-1/+10
| | | | | before putting them into the mpages table. Issue found by bentley@ in OpenBSD::Getopt(3p).
* Fix two memory leaks in makewhatis -n:Ingo Schwarze2014-05-071-1/+13
| | | | | 1. As found by nigel@, names_check() requires database access. 2. Do not leak names and strings in -n mode.
* Improve error handling in dbopen(). If PRAGMA SQL statements fail,Ingo Schwarze2014-04-271-6/+21
| | | | | | | | | | | | | report the error, close the database, and return failure from dbopen(), such that the main program can recover and rebuild the database. As noticed by stsp@, this can happen when database files are accessible, but corrupt or in the wrong format, which will now automatically be repaired. Besides, use a safer idiom after sqlite3_open*() failure that also handles out-of-memory situations correctly, and do not forget to close the database after CREATE TABLE failure.
* Reduce the verbosity of makewhatis -t:Ingo Schwarze2014-04-251-0/+2
| | | | | | | | In the past, it always showed the title lines of the files processed. Now, it only shows them when called with -D. That is better because pkg_create calls makewhatis -t. It is also more consistent with -D behaviour in non- -t modes. Issue reported by ajacoutot@; ok espie@ ajacoutot@ jasper@.
* Audit malloc(3)/calloc(3)/realloc(3) usage.Ingo Schwarze2014-04-231-4/+4
| | | | | | | * Change eight reallocs to reallocarray to be safe from overflows. * Change one malloc to reallocarray to be safe from overflows. * Change one calloc to reallocarray, no zeroing needed. * Change the order of arguments of three callocs (aesthetical).
* Audit strlcpy(3)/strlcat(3) usage:Ingo Schwarze2014-04-231-22/+22
| | | | | | * Add missing truncation checks to three calls. * In four cases where we know that the distination buffer is large enough, cast the return vailue to (void).
* improve SQL style: avoid "SELECT *", be explicit in what columns we want;Ingo Schwarze2014-04-231-1/+2
| | | | suggested by espie@.
* KNF: case (FOO): -> case FOO:, remove /* LINTED */ and /* ARGSUSED */,Ingo Schwarze2014-04-201-65/+66
| | | | | remove trailing whitespace and blanks before tabs, improve some indenting; no functional change
* Two minor tweaks regarding the fallback from -u/-d to default mode:Ingo Schwarze2014-04-191-6/+8
| | | | | | | (1) Use all files found on the command line, but do *not* use all stray files found during fallback tree recursion. (2) If the fallback works, call that success, i.e. exit(0). As pointed out by naddy@, the latter is required for ports' happiness.
* Properly handle symlinks (hardlinks and .so only files were already ok):Ingo Schwarze2014-04-191-16/+75
| | | | | | | | Use the file name of the symlink but the inode number of the file pointed to, such that we get multiple mlinks records but not multiple mpages records. Also make sure they do not point outside the tree we are processing. Issue found by kili@ in desktop-file-edit(1), thanks!
* In update mode, when opening the database fails, probably because it isIngo Schwarze2014-04-181-7/+17
| | | | | | | | missing or corrupt, just rebuild it from scratch. This also helps when installing the very first port on a freshly installed machine and is similar to what espie@'s classical makewhatis(8) did. Issue reported by naddy@ via kili@.
* Rename the mpages.id column to mpages.pageid. There is no good reasonIngo Schwarze2014-04-161-11/+11
| | | | to call this kid by a different name here than in all other tables.
* Give the mlinks and keys tables a pageid index,Ingo Schwarze2014-04-161-1/+3
| | | | | | | | | | | | | | | | as suggested by jeremy@ and espie@. The mlinks index speeds up basic apropos(1) searches by around 30% because it speeds up the final SELECT FROM mlinks query by about 95%. For large result sets, the overall speedup gets even larger, in the extreme case of "apropos Nd~." bymore than 90%. The keys index finally makes the apropos(1) -O option usable: It no longer incurs relevant extra cost, while in the past it was embarrassingly slow. This comes at a cost: Total database build times grow by about 5%, and each index adds about 10% database size with -Q. I consider that acceptable in view of the huge apropos(1) performance gains. The -Q database for /usr/share/man still remains below 1 MB.
* Unify description handling across all document types (mdoc, man, cat).Ingo Schwarze2014-04-131-8/+3
| | | | | | | | Assert that the description is unset right before calling the parse_* handler, and assign a default if it's still unset right afterwards. Remove all stray asserts and default assignments found elsewhere. This fixes SQL_STEP failures for man(7) pages lacking descriptions.
* better error reporting in case of SQL errors: mention dir and fileIngo Schwarze2014-04-131-8/+8
|
* Next speed optimization step for the new apropos(1).Ingo Schwarze2014-04-101-26/+64
| | | | | | | | | | | | | | Split manual names out of the common "keys" table into their own "names" table. This reduces standard apropos(1) search times (i.e. searching for names and descriptions only) by typically about 70% for the full /usr/share/man database. (Yes, that multiplies with the previous optimization step, so both together have reduced search times by a factor of more than six. I'm not done yet, expect more to come.) Even with the minimal databases built with makewhatis(8) -Q, this step still reduces search times by 15-20%. For both cases, database sizes and build times hardly change (+/-2%).
* After careful gprof(1)ing of the new apropos(1), move the descriptionsIngo Schwarze2014-04-091-17/+8
| | | | | | | | | | | | back from the keys table to the mpages table: I found a good way to still use them in searches, without complication of the code. On my notebook, this reduces typical apropos(1) search times by about 40%, it reduces /usr/share/man database size by 6% in makewhatis(8) -Q mode and by 2% in standard mode (less overhead storing pointers to mpages), and it doesn't measurably change database build times (may even be going down by a percent or so because less data is being copied around in ohashes).
* In -p (picky) mode, warn unless each filename (aka mlink)Ingo Schwarze2014-04-041-24/+29
| | | | | appears as a name in the NAME section. While here, garbage collect two unused variables, both called "match".
* Warn about missing mlinks.Ingo Schwarze2014-04-041-0/+40
| | | | | | This is really expensive, more than tripling database build times, so only do it when the -p (picky) option was given, but none of the following options were given: -Q (quick), -d, -u, or -t.
* Remember which names are in the NAME section.Ingo Schwarze2014-04-041-5/+8
| | | | | | | This helps to find missing MLINKS. Database build times do not change and database growth is minimal (1.2% with -Q, 0.7% without -Q in /usr/share/man), so making this optional would be pointless.
* When the -n or -t flag is given to makewhatis(8),Ingo Schwarze2014-04-041-5/+39
| | | | | | write names and decriptions to stdout, in a format similar to apropos(1) output. Inspired by espie@'s makewhatis.
* Instead of silently doing nothing at all,Ingo Schwarze2014-04-031-0/+5
| | | | | | | | | | warn and return non-zero when the manpath is empty, that is, when /etc/man.conf is non-existent or unreadable AND the environment variable MANPATH is unset or empty AND no directories were given on the command line. Inspired by the error handling in espie@'s makewhatis(8), except that one doesn't know about MANPATH.
* Rename the -W option to -p (mnemonics: picky, print to stderr):Ingo Schwarze2014-04-031-7/+10
| | | | | | | That letter was already chosen by espie@ for OpenBSD 2.7, so avoid being gratuitiously different more than a decade later. Accept -v for backward compatibility with espie@'s makewhatis, even though it does nothing right now.
* The -v option of mandocdb(8) clashes with the -v option of espie@'sIngo Schwarze2014-04-031-12/+12
| | | | | makewhatis(8), which traditionally does something different, so rename it to -D (mnemonics: Debug, Dump, Display).
* Without bloating mandoc(1) itself, let mandocdb(8) support filesIngo Schwarze2014-03-261-16/+80
| | | | | called manN/X.N.gz and catN/X.0.gz, reading them through a pipe(2) from gunzip(1) -c. Asked for by various people in the past.
* Improve error reporting.Ingo Schwarze2014-03-261-33/+51
| | | | | | | | Simplify combining a custom format string with perror(), avoiding many manual calls to strerror(errno). For low-level failures, report attempted function calls. Do not abuse the say() filename argument for files outside the basedir, and even less for other text.
* If an .Nd block contains macros, avoid fragmented entries in mandocdb(8),Ingo Schwarze2014-03-231-25/+3
| | | | instead use the .Nd content recursively.
* If a man(7) NAME section contains macros, avoid truncated or emptyIngo Schwarze2014-03-231-46/+7
| | | | | | entries for .Nd in mandocdb(8), instead use the macro content recursively. This improves indexing of more than 200 manuals in Xenocara, i.e. more than 15%, in particular GL and some Xkb.
* avoid repetitive code for asprintf error handlingIngo Schwarze2014-03-231-4/+1
|
* The files mandoc.c and mandoc.h contained both specialised low-levelIngo Schwarze2014-03-231-0/+1
| | | | | | | functions used for multiple languages (mdoc, man, roff), for example mandoc_escape(), mandoc_getarg(), mandoc_eos(), and generic auxiliary functions. Split the auxiliaries out into their own file and header. While here, do some #include cleanup.
* Register pure .so pages as mlinks, not as mpages.Ingo Schwarze2014-03-191-19/+67
| | | | | | | This doesn't affect /usr/share/man, but improves /usr/X11R6/man: * Eliminates multiple apropos(1) output for such pages. * Reduces X11R6 database size from 450 kB to 240 kB (-47%). * Reduces X11R6 database build time from 1.68s to 1.00s (-40%).
* Without the MPARSE_SO option, if the file contains nothing but aIngo Schwarze2014-03-191-1/+1
| | | | | | single .so request, do not read the file pointed to, but instead let mparse_result() provide the file name pointed to as a return value. To be used by makewhatis(8) in the future.
* Generalize the mparse_alloc() and roff_alloc() functions by givingIngo Schwarze2014-03-191-5/+5
| | | | | | | | them an "options" argument, replacing the existing "inttype" and "quick" arguments, preparing for a future MPARSE_SO option. Store this argument in struct mparse and struct roff, replacing the existing "inttype", "parsetype", and "quick" members. No functional change except one tiny cosmetic fix in roff_TH().
* Allow checking that databases are up to date even when you have no writeIngo Schwarze2014-03-181-20/+94
| | | | | | permission on the databases, as requested by espie@ quite some time ago. But make sure to not slow database generation down when you do have write permission, and to not delay error reporting in -Q mode.
* Implement the \: (optional line break) escape sequence,Ingo Schwarze2014-01-221-5/+14
| | | | | | | documented in the Ossanna-Kernighan-Ritter troff manual and also supported by groff. Missing feature reported by Steffen Nurpmeso <sdaoden at gmail dot com>.
* Avoid the risk of trying to modify a literal string.Ingo Schwarze2014-01-191-2/+3
|
* Always store the arch in lower-case only.Ingo Schwarze2014-01-191-5/+9
| | | | Reduces database size by ~0.5%, and by ~1.5% with -Q.
* Support a second -v on mandocdb(8) to show keys while they are being added;Ingo Schwarze2014-01-191-1/+16
| | | | | i need that for debugging, in particular to be used with -t. To be able to do so, provide a global table of key names, for reuse.
* Cope with slightly broken NAME sections in man(7) pagesIngo Schwarze2014-01-181-0/+9
| | | | | | having a trailing comma after the last name, like this: ASN1_OBJECT_new, ASN1_OBJECT_free, - object allocation functions
* Drop the AUTOINCREMENT PRIMARY KEYs from the mlinks and keys tables.Ingo Schwarze2014-01-181-4/+2
| | | | | | | | | | They are completely unused, and i cannot imagine what they *could* ever be used for; but apparently, they are expensive to generate. Standard DB build time goes down by 10%, now at 1.9x of makewhatis. Standard DB size goes down by 4%, now at 11x of makewhatis. DB build time with -Q goes down by 15%, now at 0.28x of makewhatis. DB size with -Q goes down by 3%, now at 3.35x of makewhatis.
* Despite some experimenting, i'm unable to find any relevant effect ofIngo Schwarze2014-01-181-3/+1
| | | | | | | | | | | | creating an index for the keys table on apropos(1) search times; apparently, adding that index was premature optimization in the first place; so, stop adding that index. Its root gone, the following evil is reduced (/usr/share/man on my notebook) - DB build time with -Q goes down by 15%, now at 1/3 of makewhatis - DB size with -Q goes down by 35%, now at 3.5x of makewhatis - full DB build time goes down by 12%, now at 2.1x of makewhatis - full DB size goes down by 42%, now at 11.5x of makewhatis
* Do not sync to disk after each individual manual page (duh!),Ingo Schwarze2014-01-061-4/+6
| | | | | | | | | | | | only sync to disk one single time when all data is ready. Rebuild times for /usr/share/man/mandoc.db shrink on my notebook: In standard mode from 45 seconds to 11 seconds (75% reduction) In -Q mode from 25 seconds to 3.1 seconds (87% reduction) For comparison: makewhatis(8): 4.2 seconds That is, in -Q mode, we are now *faster* than the existing makewhatis(8), and careful profiling shows there is still a lot of room for improval.
* Fix mandocdb(8) -d and -u.Ingo Schwarze2014-01-061-14/+35
| | | | It was broken by recent optimizations.
* Rename dbindex() to dbadd() to be less confusing.Ingo Schwarze2014-01-061-5/+5
| | | | | The concept of an index file is gone since the switch to SQLite. No functional change.
* Remove the redundant "file" column from the "mlinks" table.Ingo Schwarze2014-01-061-3/+1
| | | | | The contents can easily be reconstructed from sec, arch, name, form. Shrinks the database by another 3% in standard mode and 9% in -Q mode.
* Drop Nd from the mpages table, it is still in the keys table.Ingo Schwarze2014-01-061-14/+1
| | | | | This shrinks the database in standard mode by 3%, in -Q mode by 9%, without loss of functionality.
* Add an option -Q (quick) to mandocdb(8)Ingo Schwarze2014-01-051-7/+11
| | | | | | | | | | | | | | | for accelerated generation of reduced-size databases. Implement this by allowing the parsers to optionally abort the parse sequence after the NAME section. While here, garbage collect the unused void *arg attribute of struct mparse and mparse_alloc() and fix some errors in mandoc(3). This reduces the processing time of mandocdb(8) on /usr/share/man by a factor of 2 and the database size by a factor of 4. However, it still takes 5 times the time and 6 times the space of makewhatis(8), so more work is clearly needed.
generated by cgit (git 2.34.1) at 2024-11-18 06:01:55 +0000