Overhaul installation documentation

11 files changed, 282 insertions, 188 deletions

diff --git a/builds.sr.ht/installation.md b/builds.sr.ht/installation.md
index 54d3e2e..b8bc3ce 100644
--- a/builds.sr.ht/installation.md
+++ b/builds.sr.ht/installation.md
@@ -15,13 +15,8 @@ such](/installation.md). However, it is important that you configure *two* Redis
 servers - one that the runners should have access to, and one that they should
 not. Insert connection details for the former into build.sr.ht's configuration
 file under the `redis` key. Each build runner will also need a local redis
-instance running. In an insecure deployment (all services on the same server)
-you can get away with a single Redis instance.
-
-We suggest using an SSH tunnel to share the slave Redis instance between job
-runners and the master server, but you can use any method you prefer. If you use
-an SSH tunnel, you will likely want to use a reverse tunnel initiated from the
-master server, so the slaves are unable to SSH into the master server.
+instance running. In an insecure deployment (all services on the same server,
+running only trusted builds), you can get away with a single Redis instance.
 
 # Security model
 
@@ -95,17 +90,16 @@ necessary. It's recommended that you set up cron jobs to build fresh images
 frequently - a script at `contrib/submit_image_build` is provided for this
 purpose.
 
-Note: You will need nested virtualization enabled in order to build images
-from within a pre-existing build image (i.e. via the `build.yml` file). If you
-run into issues with `modprobe kvm_intel` within the genimg script, you can
-fix this by removing the module and then re-inserting it with `insmod
-kvm_intel.ko nested=1` in the directory containing the kernel module.
-
-### Image-specific notes
+**Note**: it is recommended that you modify our `build.yml` files to suit your
+instance's needs, then run it on *our* hosted builds.sr.ht instance to bootstrap
+your images. This is the fastest and most convenient way to bootstrap the images
+you need.
 
-* NixOS can be bootstrapped from any distribution. The provided build.yml does
-  it from Alpine, but it can be easily switched to eg. Archlinux just by
-  changing the host image and adjusting the packages.
+**Note**: You will need nested virtualization enabled in order to build images
+from within a pre-existing build image (i.e. via the `build.yml` file). If you
+run into issues with `modprobe kvm_intel` within the genimg script, you can fix
+this by removing the module and then re-inserting it with `insmod kvm_intel.ko
+nested=1` in the directory containing the kernel module.
 
 ## Creating new images
 
diff --git a/dispatch.sr.ht/installation.md b/dispatch.sr.ht/installation.md
index eaf9dda..478286f 100644
--- a/dispatch.sr.ht/installation.md
+++ b/dispatch.sr.ht/installation.md
@@ -8,11 +8,34 @@ dispatch.sr.ht is a standard sr.ht web service and can be installed through the
 [standard procedure](/installation.md). However, there is an additional step
 required.
 
+## Daemons
+
+- `dispatch.sr.ht`: the web service
+
 ## GitHub OAuth client
 
-You need to register your dispatch.sr.ht installation under [GitHub OAuth
-Apps](https://github.com/settings/developers). For the `Authorization callback
-URL` use the URL of your service instance with `/github/callback` appended (for
-example `https://dispatch.sr.ht/github/callback`). Put the `Client ID` and
-`Client Secret` you get from GitHub into your sourcehut config in the
+Register under [GitHub OAuth Apps](https://github.com/settings/developers). For
+the `Authorization callback URL` use the URL of your service instance with
+`/github/callback` appended (for example
+`https://dispatch.sr.ht/github/callback`). Put the `Client ID` and `Client
+Secret` you get from GitHub into your sourcehut config in the
 `[dispatch.sr.ht::github]` section.
+
+## GitLab OAuth Client
+
+Register under [GitLab Applications](https://gitlab.com/profile/applications).
+For the scopes required, select "api". For the callback, specify the URL of your
+service with `/gitlab/callback/<gitlab instance>` appended, e.g.
+`https://dispatch.sr.ht/gitlab/callback/gitlab.com`. Add the instance to your
+config file under `[dispatch.sr.ht::gitlab]` as a single option, whose key is
+the FQDN of the GitLab instance in question, and whose value is the instance
+name, "Application ID" and "Secret", separated by a colon. For example:
+
+```
+[dispatch.sr.ht::gitlab]
+# ...
+gitlab.com=GitLab:application id...:secret...
+gitlab.freedesktop.org=gitlab.freedesktop.org:application id...:secret...
+```
+
+Repeat this process for any additional instances you wish to support.
diff --git a/git.sr.ht/installation.md b/git.sr.ht/installation.md
index d306038..2631f9d 100644
--- a/git.sr.ht/installation.md
+++ b/git.sr.ht/installation.md
@@ -10,6 +10,16 @@ git.sr.ht is a standard sr.ht web service and can be installed through the
 [standard procedure](/installation.md). However, there are several additional
 steps required.
 
+## Daemons
+
+- `git.sr.ht`: the web service
+- `git.sr.ht-webhooks`: webhook delivery service
+
+## Cronjobs
+
+- `gitsrht-periodic`: various maintenance tasks. Recommended configuration is
+  `*/20 * * * * gitsrht-periodic`
+
 ## Repository storage
 
 You will need to set up a directory for repositories to be stored in - we
@@ -46,13 +56,6 @@ for additional services (e.g. man.sr.ht).
 Authorization logs are written to `/var/log/gitsrht-dispatch` and
 `gitsrht-shell`.
 
-## Cronjobs
-
-You must also configure `gitsrht-periodic` to run periodically with your
-favorite cron daemon. We recommend the following crontab:
-
-    */20 * * * * gitsrht-periodic
-
 ## HTTP(s) Cloning
 
 git.sr.ht does not do this for you - you need to wire it up in nginx. Here's an
@@ -85,4 +88,3 @@ of private repositories.
 If you don't have `/run/fcgiwrap.sock` on your system, you'll need to install
 the `fcgiwrap` package (for instance: `apt-get install fcgiwrap`). On some
 systems, the script might be `/run/fcgiwrap.socket`.
-
diff --git a/hg.sr.ht/installation.md b/hg.sr.ht/installation.md
index a8ac218..113e926 100644
--- a/hg.sr.ht/installation.md
+++ b/hg.sr.ht/installation.md
@@ -10,6 +10,23 @@ hg.sr.ht is a standard sr.ht web service and can be installed through the
 [standard procedure](/installation.md). However, there are several additional
 steps required.
 
+## Daemons
+
+- `hg.sr.ht`: the web service
+
+## Cronjobs
+
+- `hgsrht-periodic`: various maintenance tasks. Recommended configuration is
+  `*/20 * * * * hgsrht-periodic`
+- `hgsrht-clonebundles`: optional cronjob to generate [clone
+  bundles][clonebundles]. Recommended to run daily, e.g.
+  `0 * * * * hgsrht-clonebundles`
+
+[clonebundles]: https://www.mercurial-scm.org/wiki/ClonebundlesExtension
+
+Clone bundles are computationally expensive to create, and use up more disk
+space, but may dramatically speed up clone operations on large repositories.
+
 ## Repository storage
 
 You will need to set up a directory for repositories to be stored in - we
@@ -31,20 +48,6 @@ works.
 
 Authorization logs are written to `hg-srht-shell`.
 
-## Cronjobs
-
-You must also configure `hgsrht-periodic` to run periodically with your
-favorite cron daemon. We recommend the following crontab:
-
-    */20 * * * * hgsrht-periodic
-
-If you'd like to generate
-[clonebundles](https://www.mercurial-scm.org/wiki/ClonebundlesExtension) (which
-require more disk space but dramatically speed up cloning large repositories),
-also add a nightly task for running the script:
-
-    0 * * * * hgsrht-clonebundles
-
 ## HTTP(s) Cloning
 
 hg.sr.ht does not do this for you - you need to wire it up in nginx. Here's an
diff --git a/installation.md b/installation.md
index 8ff64b4..d8b9201 100644
--- a/installation.md
+++ b/installation.md
@@ -2,11 +2,20 @@
 title: Installation
 ---
 
-Installation of sr.ht web services uses a mostly consistent procedure across the
-network.
+sr.ht is a distributed system. Each service, such as git.sr.ht and builds.sr.ht,
+runs independently, with its own database and resources. They communicate with
+each other using mainly their respective APIs and webhooks, and are mostly
+tolerant to the temporary or permanent absence of their peers. The system is
+fairly complex, and is designed more for deployments at-scale than for
+small-scale installations. However, you can choose a subset of these services to
+install as your needs demand, and it will work correctly.
 
-Specific details to each service are available on each service's installation
-page:
+Developers wishing to hack on or contribute to sr.ht, see
+[hacking.md](/hacking.md).
+
+This page documents details common to the installation of all packages, but many
+services have specific needs. Specific details to each service are available on
+each service's installation page:
 
 - [builds.sr.ht](/builds.sr.ht/installation.md)
 - [dispatch.sr.ht](/dispatch.sr.ht/installation.md)
@@ -23,87 +32,140 @@ All sr.ht sysadmins are encouraged to subscribe to the
 low-volume list of sysadmin-oriented announcements regarding breaking changes,
 security vulnerabilities, and so on.
 
-General installation instructions follow.
-
 # Packages
 
-sr.ht provides a number of Linux distribution package repositories for your
-use. For details specific to your distribution, see [packages.md](/packages.md).
-Even if you wish to install sr.ht services from source, your distribution may
-not include some of our dependencies and using our package repositories is
-recommended.
-
-Installation from packages will also give you distro-specific daemon services
-and will handle database migrations automatically during system updates.
+The only supported installation is Alpine Linux hosts using the SourceHut
+[Alpine Linux package repository](/packages.md). There are also
+community-maintained SourceHut distributions for [Debian and Arch
+Linux](/packages.md), which are preferred before installing SourceHut from
+source.
 
-# Development
+SourceHut is still in alpha, and has no stable releases. If you are interested
+in packaging it for your distribution, please reach out to us for help in
+automating the maintenance of a third-party distribution which is automatically
+updated following our (frequent) upstream rolling releases. Please refrain from
+packaging SourceHut for your upstream distribution repositories until we have
+shipped stable versions of our software.
 
-Please send patches to [sr.ht-dev](https://lists.sr.ht/~sircmpwn/sr.ht-dev).
-This is also the right place to ask questions about the code. Also check out
-[hacking on sr.ht](#hacking-on-srht) for an amended installation procedure for
-local hacking.
+Packages also ship with the correct users and groups rigged up, and with
+whichever daemon configurations are particular to your distribution's init
+system (e.g. OpenRC scripts or systemd units).
 
 # Prerequisites
 
-You will need at least:
+Generally speaking, most services require the following services:
+
+- A PostgreSQL server: persistent storage
+- A Redis server: ephemeral storage, caching, work distribution
+- A mail server: incoming and outgoing mail
+- A cron daemon: running scheduled tasks
+
+Additionally, many services are able to integrate with some optional tools:
+
+- [Prometheus](https://prometheus.io/) for monitoring
+- Any S3-compatible object storage, such as [Minio](https://min.io/) or
+  [Ceph](https://ceph.io/)
+
+# Installing SourceHut services
+
+These instructions apply generally to all services. Consult service-specific
+documentation for amendments to these procedures.
+
+## sr.ht core
+
+[core.sr.ht](https://git.sr.ht/~sircmpwn/core.sr.ht) is a Python package which
+provides common functionality for all services. For users are installing from
+packages, it is not necessary to concern yourself with this, as it is
+automatically pulled in as a dependency. For users building from source, you
+must build this package first. Be aware that the dependencies in `setup.py` are
+not generally kept up-to-date, consult the latest
+[Alpine package][alpine package] for an up-to-date and comprehensive list.
+
+[alpine package]: https://git.sr.ht/~sircmpwn/sr.ht-apkbuilds/tree/master/sr.ht/py3-srht/APKBUILD
+
+## meta.sr.ht
 
-- A PostgreSQL server
-- A Redis server
-- A mail server
-- A cron daemon
+meta.sr.ht is the only service which is required. It is responsible for storing
+your account details and profile, handling logins, storing SSH and PGP keys,
+keeping a secure audit log, and providing some management interfaces for
+administrators.
 
-In order to use most sr.ht services.
+Consult the [meta.sr.ht installation guide](/meta.sr.ht/installation.md) for
+details on the installation procedure.
 
-# sr.ht core
+## Service configuration
 
-sr.ht core is a Python package that provides shared functionality across all
-sr.ht services. It also contains the default templates and stylesheets that give
-sr.ht a consistent look and feel.
+All services use a shared configuration file at `/etc/sr.ht/config.ini`. Each
+site provides an example configuration in `config.example.ini` in their
+respective source code repositories. It is the administrator's responsibility to
+consult these examples to produce a unified configuration file which is
+applicable to all of the services which they intend to operate.
 
-The core package is listed as a dependency of the official `*.sr.ht` packages,
-so installing it explicitly is not necessary if you are using our package
-repositories. If you are not using our packages, obtain the [source
-code](https://git.sr.ht/~sircmpwn/srht), initialize submodules and install it
-like a typical Python package (`./setup.py install --prefix=/usr`). You will
-need to install its dependencies beforehand, for an up-to-date list see [the
-package](https://git.sr.ht/~sircmpwn/sr.ht-pkgbuilds/tree/master/python-srht/PKGBUILD).
+Some configuration options are applicable to all services.
 
-# Package Installation
+### Service keys
 
-Packages are named as you would expect: meta.sr.ht is called `meta.sr.ht`. On
-package managers where dots are not permitted in package names, dashes are used.
-Underscores are used on systems where dashes are not permitted.
+Encryption is used throughout SourceHut to encrypt and validate communications.
+There are two main keys which you need to generate: service keys, the network
+key, and the webhook key.
 
-If installing from source, see [the
-packages](https://git.sr.ht/~sircmpwn/sr.ht-pkgbuilds) for an up-to-date list of
-dependencies and install it like any other Python package: `./setup.py install
---prefix=/usr`. When installing from source, daemon service files are not
-provided; you must write one appropriate to your system.
+Service keys are used to encrypt session cookies, and if you configure
+load-balancing, it must be consistent between all nodes of that service. They
+can be generated with `srht-keygen service`.
 
-# Site Configuration
+The network key needs to be consistent throughout all services and nodes in your
+sr.ht installation. It is used to encrypt and sign internal communications
+between services. This key is generated with `srht-keygen network`.
 
-The config file for all sr.ht sites is located at `/etc/sr.ht/config.ini`. Each
-site provides an example configuration in `config.example.ini`, which includes
-the global (shared) config options, and any options specific to its operation.
-You should merge the configs of each sr.ht site you want to run.
+The webhook key is also consistent throughout all services and nodes, but is an
+asymmetric key. It is used to sign webhook payloads to validate the authenticity
+of the request. You can generate this with `srht-keygen webhook`, store the
+private key in your config file and distribute to the public key to any parties
+interested in authenticating webhook payloads from your services. This is also
+used to validate webhooks used internally; it is not optional.
 
-# Database Configuration
+### Mail configuration
 
-Start by setting your config.ini's connection string to a superuser, then run
-the following commands to create the initial schema:
+Outgoing emails from sr.ht are configured in the `[mail]` section. If you fill
+out the `error-to` address, the services will send exceptions to this address
+for diagnostics. You must also generate a PGP key (without a password) for sr.ht
+services to sign outgoing emails with (and optionally encrypt emails to each
+recipient).
 
-    $ python3
-    >>> from [module].app import db
-    >>> db.create()
+## Database Configuration
 
-Substitute `[module]` for the specific application's module, such as `metasrht`
-or `buildsrht`. When complete, you may update your connection string to use a
-less privileged user.
+Each service requires its own database, though they can co-exist on the same
+server. It is also recommended to give each service its own database login, with
+full access rights to that database so that it may manage its own schema
+migrations.
 
-## Schema Upgrades
+After you populate your `config.ini`'s connection string, you may use the
+`[module]-initdb` script to populate the schema and stamp the latest revision
+for migrations, for example `metasrht-initdb` to set up meta.sr.ht's database.
+
+### Schema Upgrades
 
 We use [alembic](http://alembic.zzzcomputing.com/en/latest/) to manage schema
-migrations. We source your alembic config from your main sr.ht `config.ini` -
+migrations. We use custom scripts to automatically retrieve database credentials
+from your main sr.ht config file. If you have installed from distribution
+packages, set `[service] migrate-on-upgrade=yes` (where service is e.g.
+`[meta.sr.ht]`) to have migrations automatically performed during normal service
+upgrades.
+
+Otherwise, you may use `srht-migrate <service> upgrade head` to run updates for
+core.sr.ht migrations, and `<service>-migrate upgrade head` to run
+service-specific upgrades. For example, to upgrade the database schema for
+git.sr.ht, run `srht-migrate git.sr.ht upgrade head`, then `gitsrht-migrate
+upgrade head`. Other alembic commands are available, use `gitsrht-migrate
+--help` for more information.
+
+## Upgrade procedure
+
+1. Stop all services
+2. Run your distro's update commands (e.g. `apk update && apk upgrade`)
+3. Resume all services
+
+We source your alembic config from your main sr.ht `config.ini` -
 no need to write an `alembic.ini` file. Run `srht-migrate <service> stamp head
 && <service>-migrate stamp head` once to tell alembic the schema is up-to-date
 (e.g. `srht-migrate man.sr.ht stamp head && mansrht-migrate stamp head`).
@@ -111,35 +173,16 @@ Future upgrades will be managed automatically by the package, or if you're
 using the source code you can run `srht-migrate <service> upgrade head &&
 <service>-migrate upgrade head` when you pull the latest version down.
 
-## Becoming admin
-
-After setting up meta.sr.ht and registering yourself a user account, you can
-give that account admin permissions:
-
-    $ python3
-    >>> from metasrht.app import db, User, UserType
-    >>> u = User.query.filter_by(username='[your username]').one()
-    >>> u.user_type = UserType.admin
-    >>> User.query.session.commit()
-
-# Compile static assets
-
-This step is only necessary for users configuring sr.ht from source. Run `make`
-in the root of the repository to compile static assets. `setup.py` will usually
-do this for you.
-
-# Start the service
+## Start the daemons
 
-A service file is included in the packages and uses the same name as the
-package. On Arch Linux, for example, you can run `systemctl enable --now
-meta.sr.ht` to start up meta.sr.ht, or `rc-update add meta.sr.ht && service
-meta.sr.ht start` on Alpine. The service will be running at the port specified
-in its config file, and you must now prepare to proxy to it.
+Daemon configuration is provided in the distribution packages. Use the service
+manager appropriate to your distro to start the daemons, e.g. `service git.sr.ht
+start` or `systemctl start git.sr.ht`.
 
-# Proxy configuration
+## Proxy configuration
 
-The exact nginx configuration you use will vary depending on your needs. Here is
-an example for meta.sr.ht:
+The exact configuration you use will vary depending on your needs. Here is an
+example nginx configuration for meta.sr.ht:
 
     server {
         listen 80;
@@ -155,13 +198,12 @@ an example for meta.sr.ht:
     }
 
     server {
-        listen 443 ssl;
-        listen [::]:443 ssl;
+        listen 443 ssl http2;
+        listen [::]:443 ssl http2;
         server_name meta.sr.ht;
         client_max_body_size 100M;
-        ssl_certificate /var/lib/acme/live/meta.sr.ht/fullchain;
-        ssl_certificate_key /var/lib/acme/live/meta.sr.ht/privkey;
-        ssl_trusted_certificate /var/lib/acme/live/meta.sr.ht/fullchain;
+        ssl_certificate /etc/ssl/uacme/meta.sr.ht/cert.pem;
+        ssl_certificate_key /etc/ssl/uacme/private/meta.sr.ht/key.pem;
 
         location / {
             proxy_pass http://127.0.0.1:5002;
@@ -170,23 +212,19 @@ an example for meta.sr.ht:
         location /static {
             root /usr/lib/python3.6/site-packages/metasrht;
         }
-
-        location ^~ /.well-known {
-            root /var/www;
-        }
     }
 
 Once the proxy is configured, you should be able to access your new service.
 
-# OAuth configuration
+## OAuth configuration
 
 For services other than meta.sr.ht, you have to create and configure an OAuth
-client before users can log into your service.  To do that, visit your profile
+client before users can log into your service. To do this, visit your profile
 on your meta.sr.ht instance, select the OAuth tab and register a new client.
 Append the path `/oauth/callback` to the URL of your service instance and
 choose this value as the base redirect URI (for example
-`https://git.sr.ht/oauth/callback`). Update your configuration file with the
-client ID and secret.
+`https://git.sr.ht/oauth/callback`). Update your service configuration file with
+the generated client ID and secret.
 
 You then need to configure that client as "preauthorized", i.e. first-party.
 This skips the OAuth consent screen that third-party applications are subject to
@@ -199,16 +237,8 @@ update oauthclient set preauthorized = true where client_id = 'your client ID';
 
 Now you should be able to log into your new service.
 
-# Hacking on sr.ht
-
-If you just want to get the codebase running to hack on it, follow these steps:
-
-1. [Prerequisites](#prerequisites)
-2. [sr.ht core](#srht-core), however, you can simply clone it somewhere and add
-   it to your Python path. Also export `SRHT_PATH=/path/to/sr.ht-core` to use
-   your development repository for generating static assets.
-3. [Site configuration](#site-configuration), but you can just put `config.ini`
-   in the working directory and sr.ht will read it there.
-4. [Database configuration](#database-configuration)
-5. [Compile static assets](#compile-static-assets)
-6. `./run.py` will start the development server.
+Alternatively, you can generally run most SourceHut services by pointing them at
+*our* meta.sr.ht instance and registering them as an OAuth client. For example,
+you could run a custom builds.sr.ht instance which logs in with hosted
+meta.sr.ht accounts by registering for an OAuth client there and skipping the
+preauthorized step. YMMV.
diff --git a/lists.sr.ht/installation.md b/lists.sr.ht/installation.md
index 66f83fc..da123ed 100644
--- a/lists.sr.ht/installation.md
+++ b/lists.sr.ht/installation.md
@@ -2,23 +2,27 @@
 title: lists.sr.ht installation
 ---
 
-lists.sr.ht is the mailing list service for lists.sr.ht. There are a number of
-important components:
+lists.sr.ht is the mailing list service for lists.sr.ht.
 
-- The **LMTP daemon**, which receives incoming email from a separate mail
-  server (e.g. postfix), does minimal processing &amp; validation, and
-  dispatches it to the **celery worker**.
-- The **celery worker** is distributed on as many cores and/or servers as you
-  like and can take its time processing emails. It's responsible for forwarwding
-  and archiving emails.
-- The **web interface** displays the archive, manages subscriptions, etc, on the
-  web.
+# Installation
 
-# LMTP daemon installation
+lists.sr.ht is a standard sr.ht web service and can be installed through the
+[standard procedure](/installation.md). However, there are several additional
+steps required.
+
+## Daemons
+
+- `lists.sr.ht`: the web service
+- `lists.sr.ht-webhooks`: webhook delivery service
+- `lists.sr.ht-lmtp`: incoming mail delivery service
+- `lists.sr.ht-process`: mail processing service
+
+## LMTP daemon installation
 
 The LMTP daemon is available in the `lists.sr.ht` package, at the binary
 `lists-srht-lmtp`. It needs to run on your mail server, and you need to
-configure your MTA to forward emails to it.
+configure your MTA to forward emails to it. Alternatively, it may be configured
+to accept SMTP and run on another server. See `config.ini` for details.
 
 The LMTP daemon uses the same config file as the others, and there are some
 options there specifically catered to it. The most important is the Unix socket
@@ -47,18 +51,13 @@ This forwards mail to @lists.sr.ht to the LMTP socket, and processes mail to
 remainder of your postfix (or other mail server) configuration is left as an
 exercise to the reader.
 
-# Celery worker installation
+## Celery worker installation
 
 On servers which should process forwarding and archival, install the
-`lists.sr.ht` package and enable the `lists.sr.ht-worker` service. Make sure you
-review the config file for worker-specific options as well. The Redis instance
-you use here should be shared with the LMTP daemon and other worker nodes to
-ensure that messages can be distributed correctly.
-
-# Web service installation
-
-The lists.sr.ht web service is a standard sr.ht web service and can be installed
-through the [standard procedure](/installation.md).
+`lists.sr.ht` package and enable the `lists.sr.ht-process` service. Make sure
+you review the config file for worker-specific options as well. The Redis
+instance you use here should be shared with the LMTP daemon and other worker
+nodes to ensure that messages can be distributed correctly.
 
 # Hacking on lists.sr.ht
 
diff --git a/man.sr.ht/installation.md b/man.sr.ht/installation.md
index 1313085..a8c41c8 100644
--- a/man.sr.ht/installation.md
+++ b/man.sr.ht/installation.md
@@ -2,7 +2,14 @@
 title: man.sr.ht installation
 ---
 
+man.sr.ht is the wiki service for sr.ht.
+
+# Installation
+
 man.sr.ht is a standard sr.ht web service and can be installed through the
-[standard procedure](/installation.md).
+[standard procedure](/installation.md). man.sr.ht requires a git.sr.ht instance
+to be available, as it uses git.sr.ht as the backing storage for wikis.
+
+## Daemons
 
-**TODO**: Document man.sr.ht-specific installation steps
+- `man.sr.ht`: the web service
diff --git a/meta.sr.ht/installation.md b/meta.sr.ht/installation.md
index 377112f..60b007e 100644
--- a/meta.sr.ht/installation.md
+++ b/meta.sr.ht/installation.md
@@ -2,12 +2,33 @@
 title: meta.sr.ht installation
 ---
 
-meta.sr.ht is the central autentication service and service bus for the sr.ht
-network, and a working meta.sr.ht is a basic requirement of running any other
-sr.ht software.
+meta.sr.ht is the central authentication and account service for the sr.ht
+network, and a working meta.sr.ht installation is a basic requirement of running
+any other sr.ht software.
 
 # Installation
 
 meta.sr.ht is a standard sr.ht web service and can be installed through the
-[standard procedure](/installation.md). Additionally, you will need to set up a
-cron job to run `metasrht-daily` daily.
+[standard procedure](/installation.md).
+
+## Daemons
+
+- `meta.sr.ht`: the web service
+- `meta.sr.ht-webhooks`: webhook delivery service
+
+## Cronjobs
+
+- `metasrht-daily`: purges old audit log entries and runs billing; run daily.
+
+## Billing configuration
+
+To enable billing, consult the `meta.sr.ht::billing` config section. You will
+need a working Stripe account for payment processing, and may need to install
+the additional `meta.sr.ht-billing` package to obtain additional dependencies.
+
+# Post-installation tasks
+
+## Creating users & admins
+
+Use the `metasrht-createuser` script. `metasrht-createuser -t admin <username>
+<email>` will create an administrator.
diff --git a/packages.md b/packages.md
index 1b1d200..c22032e 100644
--- a/packages.md
+++ b/packages.md
@@ -11,7 +11,8 @@ can find packages for sr.ht software in various distributions at
 ## Alpine Linux
 
 This is the recommended distribution of SourceHut, and the one we use for the
-hosted instance upstream.
+hosted instance upstream. This must be the first line in `/etc/apk/repositories`
+&mdash; we shadow some upstream packages with patched versions.
 
 **Mirror URL**: `https://mirror.sr.ht/alpine/v3.11/sr.ht`
 
diff --git a/paste.sr.ht/installation.md b/paste.sr.ht/installation.md
index c6224e6..7dd0f49 100644
--- a/paste.sr.ht/installation.md
+++ b/paste.sr.ht/installation.md
@@ -4,3 +4,7 @@ title: paste.sr.ht installation
 
 paste.sr.ht is a standard sr.ht web service and can be installed through the
 [standard procedure](/installation.md).
+
+## Daemons
+
+- `paste.sr.ht`: the web service
diff --git a/todo.sr.ht/installation.md b/todo.sr.ht/installation.md
index 8db1784..08d5f52 100644
--- a/todo.sr.ht/installation.md
+++ b/todo.sr.ht/installation.md
@@ -5,4 +5,14 @@ title: todo.sr.ht installation
 todo.sr.ht is a standard sr.ht web service and can be installed through the
 [standard procedure](/installation.md).
 
-**TODO**: Document todo.sr.ht-specific installation steps
+## Daemons
+
+- `todo.sr.ht`: the web service
+- `todo.sr.ht-lmtp`: incoming mail service
+- `todo.sr.ht-webhooks`: webhook delivery service
+
+## Mail support
+
+See [lists.sr.ht](/lists.sr.ht/installation.md) for information on mail server
+configuration. Incoming emails for todo.sr.ht should be directed to the
+`todo.sr.ht-lmtp` service in a manner similar to the `lists.sr.ht-lmtp` service.