From e62d312b36d5816b6ccfe7e23d04bf55d9425d38 Mon Sep 17 00:00:00 2001 From: Drew DeVault Date: Thu, 26 Mar 2020 10:21:05 -0400 Subject: Overhaul installation documentation --- installation.md | 256 +++++++++++++++++++++++++++++++------------------------- 1 file changed, 143 insertions(+), 113 deletions(-) (limited to 'installation.md') diff --git a/installation.md b/installation.md index 8ff64b4..d8b9201 100644 --- a/installation.md +++ b/installation.md @@ -2,11 +2,20 @@ title: Installation --- -Installation of sr.ht web services uses a mostly consistent procedure across the -network. +sr.ht is a distributed system. Each service, such as git.sr.ht and builds.sr.ht, +runs independently, with its own database and resources. They communicate with +each other using mainly their respective APIs and webhooks, and are mostly +tolerant to the temporary or permanent absence of their peers. The system is +fairly complex, and is designed more for deployments at-scale than for +small-scale installations. However, you can choose a subset of these services to +install as your needs demand, and it will work correctly. -Specific details to each service are available on each service's installation -page: +Developers wishing to hack on or contribute to sr.ht, see +[hacking.md](/hacking.md). + +This page documents details common to the installation of all packages, but many +services have specific needs. Specific details to each service are available on +each service's installation page: - [builds.sr.ht](/builds.sr.ht/installation.md) - [dispatch.sr.ht](/dispatch.sr.ht/installation.md) @@ -23,87 +32,140 @@ All sr.ht sysadmins are encouraged to subscribe to the low-volume list of sysadmin-oriented announcements regarding breaking changes, security vulnerabilities, and so on. -General installation instructions follow. - # Packages -sr.ht provides a number of Linux distribution package repositories for your -use. For details specific to your distribution, see [packages.md](/packages.md). -Even if you wish to install sr.ht services from source, your distribution may -not include some of our dependencies and using our package repositories is -recommended. - -Installation from packages will also give you distro-specific daemon services -and will handle database migrations automatically during system updates. +The only supported installation is Alpine Linux hosts using the SourceHut +[Alpine Linux package repository](/packages.md). There are also +community-maintained SourceHut distributions for [Debian and Arch +Linux](/packages.md), which are preferred before installing SourceHut from +source. -# Development +SourceHut is still in alpha, and has no stable releases. If you are interested +in packaging it for your distribution, please reach out to us for help in +automating the maintenance of a third-party distribution which is automatically +updated following our (frequent) upstream rolling releases. Please refrain from +packaging SourceHut for your upstream distribution repositories until we have +shipped stable versions of our software. -Please send patches to [sr.ht-dev](https://lists.sr.ht/~sircmpwn/sr.ht-dev). -This is also the right place to ask questions about the code. Also check out -[hacking on sr.ht](#hacking-on-srht) for an amended installation procedure for -local hacking. +Packages also ship with the correct users and groups rigged up, and with +whichever daemon configurations are particular to your distribution's init +system (e.g. OpenRC scripts or systemd units). # Prerequisites -You will need at least: +Generally speaking, most services require the following services: + +- A PostgreSQL server: persistent storage +- A Redis server: ephemeral storage, caching, work distribution +- A mail server: incoming and outgoing mail +- A cron daemon: running scheduled tasks + +Additionally, many services are able to integrate with some optional tools: + +- [Prometheus](https://prometheus.io/) for monitoring +- Any S3-compatible object storage, such as [Minio](https://min.io/) or + [Ceph](https://ceph.io/) + +# Installing SourceHut services + +These instructions apply generally to all services. Consult service-specific +documentation for amendments to these procedures. + +## sr.ht core + +[core.sr.ht](https://git.sr.ht/~sircmpwn/core.sr.ht) is a Python package which +provides common functionality for all services. For users are installing from +packages, it is not necessary to concern yourself with this, as it is +automatically pulled in as a dependency. For users building from source, you +must build this package first. Be aware that the dependencies in `setup.py` are +not generally kept up-to-date, consult the latest +[Alpine package][alpine package] for an up-to-date and comprehensive list. + +[alpine package]: https://git.sr.ht/~sircmpwn/sr.ht-apkbuilds/tree/master/sr.ht/py3-srht/APKBUILD + +## meta.sr.ht -- A PostgreSQL server -- A Redis server -- A mail server -- A cron daemon +meta.sr.ht is the only service which is required. It is responsible for storing +your account details and profile, handling logins, storing SSH and PGP keys, +keeping a secure audit log, and providing some management interfaces for +administrators. -In order to use most sr.ht services. +Consult the [meta.sr.ht installation guide](/meta.sr.ht/installation.md) for +details on the installation procedure. -# sr.ht core +## Service configuration -sr.ht core is a Python package that provides shared functionality across all -sr.ht services. It also contains the default templates and stylesheets that give -sr.ht a consistent look and feel. +All services use a shared configuration file at `/etc/sr.ht/config.ini`. Each +site provides an example configuration in `config.example.ini` in their +respective source code repositories. It is the administrator's responsibility to +consult these examples to produce a unified configuration file which is +applicable to all of the services which they intend to operate. -The core package is listed as a dependency of the official `*.sr.ht` packages, -so installing it explicitly is not necessary if you are using our package -repositories. If you are not using our packages, obtain the [source -code](https://git.sr.ht/~sircmpwn/srht), initialize submodules and install it -like a typical Python package (`./setup.py install --prefix=/usr`). You will -need to install its dependencies beforehand, for an up-to-date list see [the -package](https://git.sr.ht/~sircmpwn/sr.ht-pkgbuilds/tree/master/python-srht/PKGBUILD). +Some configuration options are applicable to all services. -# Package Installation +### Service keys -Packages are named as you would expect: meta.sr.ht is called `meta.sr.ht`. On -package managers where dots are not permitted in package names, dashes are used. -Underscores are used on systems where dashes are not permitted. +Encryption is used throughout SourceHut to encrypt and validate communications. +There are two main keys which you need to generate: service keys, the network +key, and the webhook key. -If installing from source, see [the -packages](https://git.sr.ht/~sircmpwn/sr.ht-pkgbuilds) for an up-to-date list of -dependencies and install it like any other Python package: `./setup.py install ---prefix=/usr`. When installing from source, daemon service files are not -provided; you must write one appropriate to your system. +Service keys are used to encrypt session cookies, and if you configure +load-balancing, it must be consistent between all nodes of that service. They +can be generated with `srht-keygen service`. -# Site Configuration +The network key needs to be consistent throughout all services and nodes in your +sr.ht installation. It is used to encrypt and sign internal communications +between services. This key is generated with `srht-keygen network`. -The config file for all sr.ht sites is located at `/etc/sr.ht/config.ini`. Each -site provides an example configuration in `config.example.ini`, which includes -the global (shared) config options, and any options specific to its operation. -You should merge the configs of each sr.ht site you want to run. +The webhook key is also consistent throughout all services and nodes, but is an +asymmetric key. It is used to sign webhook payloads to validate the authenticity +of the request. You can generate this with `srht-keygen webhook`, store the +private key in your config file and distribute to the public key to any parties +interested in authenticating webhook payloads from your services. This is also +used to validate webhooks used internally; it is not optional. -# Database Configuration +### Mail configuration -Start by setting your config.ini's connection string to a superuser, then run -the following commands to create the initial schema: +Outgoing emails from sr.ht are configured in the `[mail]` section. If you fill +out the `error-to` address, the services will send exceptions to this address +for diagnostics. You must also generate a PGP key (without a password) for sr.ht +services to sign outgoing emails with (and optionally encrypt emails to each +recipient). - $ python3 - >>> from [module].app import db - >>> db.create() +## Database Configuration -Substitute `[module]` for the specific application's module, such as `metasrht` -or `buildsrht`. When complete, you may update your connection string to use a -less privileged user. +Each service requires its own database, though they can co-exist on the same +server. It is also recommended to give each service its own database login, with +full access rights to that database so that it may manage its own schema +migrations. -## Schema Upgrades +After you populate your `config.ini`'s connection string, you may use the +`[module]-initdb` script to populate the schema and stamp the latest revision +for migrations, for example `metasrht-initdb` to set up meta.sr.ht's database. + +### Schema Upgrades We use [alembic](http://alembic.zzzcomputing.com/en/latest/) to manage schema -migrations. We source your alembic config from your main sr.ht `config.ini` - +migrations. We use custom scripts to automatically retrieve database credentials +from your main sr.ht config file. If you have installed from distribution +packages, set `[service] migrate-on-upgrade=yes` (where service is e.g. +`[meta.sr.ht]`) to have migrations automatically performed during normal service +upgrades. + +Otherwise, you may use `srht-migrate upgrade head` to run updates for +core.sr.ht migrations, and `-migrate upgrade head` to run +service-specific upgrades. For example, to upgrade the database schema for +git.sr.ht, run `srht-migrate git.sr.ht upgrade head`, then `gitsrht-migrate +upgrade head`. Other alembic commands are available, use `gitsrht-migrate +--help` for more information. + +## Upgrade procedure + +1. Stop all services +2. Run your distro's update commands (e.g. `apk update && apk upgrade`) +3. Resume all services + +We source your alembic config from your main sr.ht `config.ini` - no need to write an `alembic.ini` file. Run `srht-migrate stamp head && -migrate stamp head` once to tell alembic the schema is up-to-date (e.g. `srht-migrate man.sr.ht stamp head && mansrht-migrate stamp head`). @@ -111,35 +173,16 @@ Future upgrades will be managed automatically by the package, or if you're using the source code you can run `srht-migrate upgrade head && -migrate upgrade head` when you pull the latest version down. -## Becoming admin - -After setting up meta.sr.ht and registering yourself a user account, you can -give that account admin permissions: - - $ python3 - >>> from metasrht.app import db, User, UserType - >>> u = User.query.filter_by(username='[your username]').one() - >>> u.user_type = UserType.admin - >>> User.query.session.commit() - -# Compile static assets - -This step is only necessary for users configuring sr.ht from source. Run `make` -in the root of the repository to compile static assets. `setup.py` will usually -do this for you. - -# Start the service +## Start the daemons -A service file is included in the packages and uses the same name as the -package. On Arch Linux, for example, you can run `systemctl enable --now -meta.sr.ht` to start up meta.sr.ht, or `rc-update add meta.sr.ht && service -meta.sr.ht start` on Alpine. The service will be running at the port specified -in its config file, and you must now prepare to proxy to it. +Daemon configuration is provided in the distribution packages. Use the service +manager appropriate to your distro to start the daemons, e.g. `service git.sr.ht +start` or `systemctl start git.sr.ht`. -# Proxy configuration +## Proxy configuration -The exact nginx configuration you use will vary depending on your needs. Here is -an example for meta.sr.ht: +The exact configuration you use will vary depending on your needs. Here is an +example nginx configuration for meta.sr.ht: server { listen 80; @@ -155,13 +198,12 @@ an example for meta.sr.ht: } server { - listen 443 ssl; - listen [::]:443 ssl; + listen 443 ssl http2; + listen [::]:443 ssl http2; server_name meta.sr.ht; client_max_body_size 100M; - ssl_certificate /var/lib/acme/live/meta.sr.ht/fullchain; - ssl_certificate_key /var/lib/acme/live/meta.sr.ht/privkey; - ssl_trusted_certificate /var/lib/acme/live/meta.sr.ht/fullchain; + ssl_certificate /etc/ssl/uacme/meta.sr.ht/cert.pem; + ssl_certificate_key /etc/ssl/uacme/private/meta.sr.ht/key.pem; location / { proxy_pass http://127.0.0.1:5002; @@ -170,23 +212,19 @@ an example for meta.sr.ht: location /static { root /usr/lib/python3.6/site-packages/metasrht; } - - location ^~ /.well-known { - root /var/www; - } } Once the proxy is configured, you should be able to access your new service. -# OAuth configuration +## OAuth configuration For services other than meta.sr.ht, you have to create and configure an OAuth -client before users can log into your service. To do that, visit your profile +client before users can log into your service. To do this, visit your profile on your meta.sr.ht instance, select the OAuth tab and register a new client. Append the path `/oauth/callback` to the URL of your service instance and choose this value as the base redirect URI (for example -`https://git.sr.ht/oauth/callback`). Update your configuration file with the -client ID and secret. +`https://git.sr.ht/oauth/callback`). Update your service configuration file with +the generated client ID and secret. You then need to configure that client as "preauthorized", i.e. first-party. This skips the OAuth consent screen that third-party applications are subject to @@ -199,16 +237,8 @@ update oauthclient set preauthorized = true where client_id = 'your client ID'; Now you should be able to log into your new service. -# Hacking on sr.ht - -If you just want to get the codebase running to hack on it, follow these steps: - -1. [Prerequisites](#prerequisites) -2. [sr.ht core](#srht-core), however, you can simply clone it somewhere and add - it to your Python path. Also export `SRHT_PATH=/path/to/sr.ht-core` to use - your development repository for generating static assets. -3. [Site configuration](#site-configuration), but you can just put `config.ini` - in the working directory and sr.ht will read it there. -4. [Database configuration](#database-configuration) -5. [Compile static assets](#compile-static-assets) -6. `./run.py` will start the development server. +Alternatively, you can generally run most SourceHut services by pointing them at +*our* meta.sr.ht instance and registering them as an OAuth client. For example, +you could run a custom builds.sr.ht instance which logs in with hosted +meta.sr.ht accounts by registering for an OAuth client there and skipping the +preauthorized step. YMMV. -- cgit