diff options
-rw-r--r-- | configuration.md | 137 | ||||
-rw-r--r-- | hacking.md | 27 | ||||
-rw-r--r-- | installation.md | 261 | ||||
-rw-r--r-- | packages.md | 104 |
4 files changed, 265 insertions, 264 deletions
diff --git a/configuration.md b/configuration.md new file mode 100644 index 0000000..bfbcf2e --- /dev/null +++ b/configuration.md @@ -0,0 +1,137 @@ +--- +title: Configuration +--- + +This document provides general information about the configuration process of +sr.ht services. + +Similar to the installation process, configuring sr.ht services typically +involves steps that are specific each service. That being said, they all share +certain aspects which are described below. Be sure to take a look at each +service's configuration page for more details: + +- [builds.sr.ht](/builds.sr.ht/configuration.md) +- [dispatch.sr.ht](/dispatch.sr.ht/configuration.md) +- [git.sr.ht](/git.sr.ht/configuration.md) +- [hg.sr.ht](/hg.sr.ht/configuration.md) +- [lists.sr.ht](/lists.sr.ht/configuration.md) +- [man.sr.ht](/man.sr.ht/configuration.md) +- [meta.sr.ht](/meta.sr.ht/configuration.md) +- [paste.sr.ht](/paste.sr.ht/configuration.md) +- [todo.sr.ht](/todo.sr.ht/configuration.md) + +# Prerequisites + +Before configuring any sr.ht service, ensure that you have the following +services installed. Every sr.ht service makes use of them, so your deployment +will likely not work as expected if they are missing. + +- [core.sr.ht][core] - Provides functionality used by all sr.ht services. + + - For users installing from packages, this package is automatically pulled + in as a dependency. + + - For users installing from source, this package must be built *first*. + + <div class="alert alert-info"> + <strong>Note:</strong> The dependencies in <code>setup.py</code> may + not be up-to-date. For an up-to-date list of dependencies, consult + the latest + <a href="https://git.sr.ht/~sircmpwn/sr.ht-apkbuilds/tree/master/sr.ht/py3-srht/APKBUILD" + class="alert-link">Alpine package</a>. + </div> + +- [meta.sr.ht][meta] - Performs various administrative tasks (e.g., storing + account details, handling logins, SSH and PGP key storage). + +[meta]: https://git.sr.ht/~sircmpwn/meta.sr.ht +[core]: https://git.sr.ht/~sircmpwn/core.sr.ht + +# Files + +## `config.ini` + +sr.ht services all use a shared configuration file, located at +`/etc/sr.ht/config.ini`. The specific options you will configure will depend on +the subset of and configuration of sr.ht services you choose to run. + +Each service provides an example configuration file, called +`config.example.ini`, in their source code repository. You should consult these +when creating your unified `config.ini`. Additionally, you may refer to each +service's configuration reference pages for a full list of configuration +options: + +- [sr.ht](/sr.ht/configuration_reference.md) +- [builds.sr.ht](/builds.sr.ht/configuration_reference.md) +- [dispatch.sr.ht](/dispatch.sr.ht/configuration_reference.md) +- [git.sr.ht](/git.sr.ht/configuration_reference.md) +- [hg.sr.ht](/hg.sr.ht/configuration_reference.md) +- [lists.sr.ht](/lists.sr.ht/configuration_reference.md) +- [man.sr.ht](/man.sr.ht/configuration_reference.md) +- [meta.sr.ht](/meta.sr.ht/configuration_reference.md) +- [paste.sr.ht](/paste.sr.ht/configuration_reference.md) +- [todo.sr.ht](/todo.sr.ht/configuration_reference.md) + +## Service Files + +Service files for daemons are pre-configured by their distribution packages. + +Thus, after setting up your `config.ini` appropriately, you may start them +using your distribution's service manager (e.g., `service git.sr.ht start`). + +# Databases + +Each service requires its own database. It is recommended to give each service +its own database login, with full permissions for its database so that it may +manage its own schema migrations. + +Once you populate the `connection-string` field in your `config.ini`, you may +use the `<module>-initdb` script to populate the schema and stamp the latest +revision for migrations (e.g., `metasrht-initdb` to set up meta.sr.ht's +database). Note that if you have not configured SSL for your PostgreSQL server, +you may have to append `?sslmode=disable` to your connection string for some +services to work. + +## Schema Upgrades + +We use [alembic](https://alembic.zzzcomputing.com/en/latest/) to manage schema +migrations. We use custom scripts to automatically retrieve database +credentials from your `config.ini`. If you have installed from distribution +packages, set `[<service>] migrate-on-upgrade=yes` to have migrations +automatically performed during normal service upgrades (e.g., `[meta.sr.ht] +migrate-on-upgrade=yes`). + +Otherwise, you may use `srht-migrate <service> upgrade head` to run updates for +core.sr.ht migrations, and `<service>-migrate upgrade head` to run +service-specific upgrades. For example, to upgrade the database schema for +git.sr.ht, run `srht-migrate git.sr.ht upgrade head`, then `gitsrht-migrate +upgrade head`. Other alembic commands are available, use `gitsrht-migrate +--help` for more information. + +# Upgrade Procedure + +1. Stop all services. +2. Update and upgrade your packages (e.g. `apk update && apk upgrade`). +3. Resume all services. + +If you have high availability requirements, the web services may be +load-balanced to avoid an outage. + +Database migrations will be performed automatically during upgrades. We source +your alembic config from your main `config.ini`, so there's no need to write an +`alembic.ini` file. + +Run `srht-migrate <service> stamp head && <service>-migrate stamp head` once to +tell alembic the schema is up-to-date (e.g. `srht-migrate man.sr.ht stamp head +&& mansrht-migrate stamp head`). Future upgrades will be managed automatically +by the package, or if you're using the source code you can run `srht-migrate +<service> upgrade head && <service>-migrate upgrade head` when you pull the +latest version down. + +# Integrations + +Many sr.ht services are able to integrate with third-party tools, such as: + +- [Prometheus](https://prometheus.io/) for monitoring. +- Any S3-compatible object storage (e.g., [Minio](https://min.io/), + [Ceph](https://ceph.io/)). @@ -3,23 +3,22 @@ title: Hacking on SourceHut toc: false --- -For developers interested in hacking on SourceHut, an expedited installation -procedure is available. The bare minimum is a PostgreSQL database and a Redis -server, and you can generally skip the webhooks daemons unless you need them. -It is recommended that you still use our distribution packages to obtain the -dependencies, though. +This document provides information for those interested in hacking on or +contributing to SourceHut. -If you don't intend to work on core.sr.ht, install the package to skip this -step. If you don't intend to work on meta.sr.ht, it may be easier to install the -package and configure it normally on localhost than to try and get it up and -running from source. +# Installation -You can also *usually* get away without configuring the outgoing mail settings, -but you may still have to generate a PGP key to get things started up. You can -also skip the reverse proxy and use our development servers via `python3 -run.py`. +Here are some tips to expedite the installation procedure: -# Sending patches +* A PostgreSQL database and Redis server are always required. +* Generally, you can skip the webhooks daemons unless you need them. +* Always prefer package installations for obtaining dependencies. +* You can *usually* get away without configuring the outgoing mail settings, + but you may still have to generate a PGP key to get things started up. +* You can skip setting up a reverse proxy and use our development servers via + `python3 run.py` instead. + +# Sending Patches We use [git send-email](https://git-send-email.io) and discuss patches (and other development topics) on the diff --git a/installation.md b/installation.md index d025907..bd3c4f0 100644 --- a/installation.md +++ b/installation.md @@ -2,20 +2,30 @@ title: Installation --- -sr.ht is a distributed system. Each service, such as git.sr.ht and builds.sr.ht, -runs independently, with its own database and resources. They communicate with -each other using mainly their respective APIs and webhooks, and are mostly -tolerant to the temporary or permanent absence of their peers. The system is -fairly complex, and is designed more for deployment at scale than for -small-scale installations. However, you can choose a subset of these services to -install as your needs demand, and it will work correctly. - -Developers wishing to hack on or contribute to sr.ht, see -[hacking.md](/hacking.md). - -This page documents details common to the installation of all packages, but many -services have specific needs. Specific details to each service are available on -each service's installation page: +This document covers the installation steps common to all sr.ht services. + +<div class="alert alert-success"> + If you would like to hack on or contribute sr.ht, see the + <a href="/hacking.md" class="alert-link">Hacking</a> page. +</div> + +<div class="alert alert-warning"> + <strong>Warning:</strong> Currently, the only officially supported method for + installing sr.ht software is through packages on Alpine Linux hosts. +</div> + +sr.ht is a distributed system in which each service (e.g., git.sr.ht, +builds.sr.ht) runs independently with its own database and resources, +communicates with other services through their respective APIs and webhooks, +and is highly tolerant to the temporary or permanent absence of its peers. +This type of design allows you to choose and deploy any subset of services +depending on your needs, making it ideal for both large-scale deployments and +small-scale installations. + +However, being distributed also means that the installation and deployment +process of can become a bit more involved. In particular, many sr.ht services +have their own, unique requirements that necessitate extra installation steps. +Consult each service's installation page for more details: - [builds.sr.ht](/builds.sr.ht/installation.md) - [dispatch.sr.ht](/dispatch.sr.ht/installation.md) @@ -27,218 +37,41 @@ each service's installation page: - [paste.sr.ht](/paste.sr.ht/installation.md) - [todo.sr.ht](/todo.sr.ht/installation.md) -All sr.ht sysadmins are encouraged to subscribe to the -[sr.ht-admins](https://lists.sr.ht/~sircmpwn/sr.ht-admins) mailing list, a +# sr.ht-admins + +If you are a sr.ht system administrator, we encourage you to to subscribe to +the [sr.ht-admins](https://lists.sr.ht/~sircmpwn/sr.ht-admins) mailing list, a low-volume list of sysadmin-oriented announcements regarding breaking changes, security vulnerabilities, and so on. -# Packages - -The only supported installation is Alpine Linux hosts using the SourceHut -[Alpine Linux package repository](/packages.md). There are also -community-maintained SourceHut distributions for [Debian and Arch -Linux](/packages.md), which are preferred before installing SourceHut from -source. - -SourceHut is still in alpha, and has no stable releases. If you are interested -in packaging it for your distribution, please reach out to us for help in -automating the maintenance of a third-party distribution which is automatically -updated following our (frequent) upstream rolling releases. Please refrain from -packaging SourceHut for your upstream distribution repositories until we have -shipped stable versions of our software. - -Packages also ship with the correct users and groups rigged up, and with -whichever daemon configurations are particular to your distribution's init -system (e.g. OpenRC scripts or systemd units). - # Prerequisites -Generally speaking, most services require the following services: - -- A PostgreSQL server: persistent storage -- A Redis server: ephemeral storage, caching, work distribution -- A mail server: incoming and outgoing mail -- A cron daemon: running scheduled tasks - -Additionally, many services are able to integrate with some optional tools: - -- [Prometheus](https://prometheus.io/) for monitoring -- Any S3-compatible object storage, such as [Minio](https://min.io/) or - [Ceph](https://ceph.io/) - -# Installing SourceHut services - -These instructions apply generally to all services. Consult service-specific -documentation for amendments to these procedures. - -## sr.ht core - -[core.sr.ht](https://git.sr.ht/~sircmpwn/core.sr.ht) is a Python package which -provides common functionality for all services. For users installing from -packages, it is not necessary to concern yourself with this, as it is -automatically pulled in as a dependency. For users building from source, you -must build this package first. Be aware that the dependencies in `setup.py` are -not generally kept up-to-date, consult the latest -[Alpine package][alpine package] for an up-to-date and comprehensive list. - -[alpine package]: https://git.sr.ht/~sircmpwn/sr.ht-apkbuilds/tree/master/sr.ht/py3-srht/APKBUILD - -## meta.sr.ht - -meta.sr.ht is the only service which is required. It is responsible for storing -your account details and profile, handling logins, storing SSH and PGP keys, -keeping a secure audit log, and providing some management interfaces for -administrators. - -Consult the [meta.sr.ht installation guide](/meta.sr.ht/installation.md) for -details on the installation procedure. - -## Service configuration - -All services use a shared configuration file at `/etc/sr.ht/config.ini`. Each -site provides an example configuration in `config.example.ini` in their -respective source code repositories. It is the administrator's responsibility to -consult these examples to produce a unified configuration file which is -applicable to all of the services which they intend to operate. - -Some configuration options are applicable to all services. - -### Service keys - -Encryption is used throughout SourceHut to encrypt and validate communications. -There are three main keys which you need to generate: service keys, the network -key, and the webhook key. - -Service keys are used to encrypt session cookies, and if you configure -load-balancing, it must be consistent between all nodes of that service. They -can be generated with `srht-keygen service`. - -The network key needs to be consistent throughout all services and nodes in your -sr.ht installation. It is used to encrypt and sign internal communications -between services. This key is generated with `srht-keygen network`. - -The webhook key is also consistent throughout all services and nodes, but is an -asymmetric key. It is used to sign webhook payloads to validate the authenticity -of the request. You can generate this with `srht-keygen webhook`, store the -private key in your config file and distribute to the public key to any parties -interested in authenticating webhook payloads from your services. This is also -used to validate webhooks used internally; it is not optional. - -### Mail configuration - -Outgoing emails from sr.ht are configured in the `[mail]` section. If you fill -out the `error-to` address, the services will send exceptions to this address -for diagnostics. You must also generate a PGP key (without a password) for sr.ht -services to sign outgoing emails with (and optionally encrypt emails to each -recipient). +Most sr.ht services require the following: -## Database Configuration +- A [PostgreSQL](https://www.postgresql.org/) server - persistent storage -Each service requires its own database, though they can co-exist on the same -server. It is also recommended to give each service its own database login, with -full access rights to that database so that it may manage its own schema -migrations. +- A [Redis](https://redis.io) server - ephemeral storage, caching, work + distribution -After you populate your `config.ini`'s connection string, you may use the -`[module]-initdb` script to populate the schema and stamp the latest revision -for migrations, for example `metasrht-initdb` to set up meta.sr.ht's database. +- A mail server - incoming/outgoing mail -### Schema Upgrades +- A cron daemon - scheduled tasks -We use [alembic](https://alembic.zzzcomputing.com/en/latest/) to manage schema -migrations. We use custom scripts to automatically retrieve database credentials -from your main sr.ht config file. If you have installed from distribution -packages, set `[service] migrate-on-upgrade=yes` (where service is e.g. -`[meta.sr.ht]`) to have migrations automatically performed during normal service -upgrades. +# Installing From Packages -Otherwise, you may use `srht-migrate <service> upgrade head` to run updates for -core.sr.ht migrations, and `<service>-migrate upgrade head` to run -service-specific upgrades. For example, to upgrade the database schema for -git.sr.ht, run `srht-migrate git.sr.ht upgrade head`, then `gitsrht-migrate -upgrade head`. Other alembic commands are available, use `gitsrht-migrate ---help` for more information. +1. Add the appropriate [SourceHut package repository](/packages.md) to your + package manager. -## Upgrade procedure +2. Install your sr.ht services! -1. Stop all services -2. Run your distro's update commands (e.g. `apk update && apk upgrade`) -3. Resume all services +<div class="alert alert-info"> + <strong>Note:</strong> Each package repository has a dedicated maintainer, + listed on the packages page, who should be your first stop for any + distribution-related issues. Alpine Linux is officially maintained by + SourceHut; the other repositories are community-maintained. +</div> -We source your alembic config from your main sr.ht `config.ini` - -no need to write an `alembic.ini` file. Run `srht-migrate <service> stamp head -&& <service>-migrate stamp head` once to tell alembic the schema is up-to-date -(e.g. `srht-migrate man.sr.ht stamp head && mansrht-migrate stamp head`). -Future upgrades will be managed automatically by the package, or if you're -using the source code you can run `srht-migrate <service> upgrade head && -<service>-migrate upgrade head` when you pull the latest version down. - -## Start the daemons - -Daemon configuration is provided in the distribution packages. Use the service -manager appropriate to your distro to start the daemons, e.g. `service git.sr.ht -start` or `systemctl start git.sr.ht`. - -## Proxy configuration +# Next Steps -The exact configuration you use will vary depending on your needs. Here is an -example nginx configuration for meta.sr.ht: - - server { - listen 80; - server_name meta.sr.ht; - - location / { - return 302 https://$server_name$request_uri; - } - - location ^~ /.well-known { - root /var/www; - } - } - - server { - listen 443 ssl http2; - listen [::]:443 ssl http2; - server_name meta.sr.ht; - client_max_body_size 100M; - ssl_certificate /etc/ssl/uacme/meta.sr.ht/cert.pem; - ssl_certificate_key /etc/ssl/uacme/private/meta.sr.ht/key.pem; - - location / { - proxy_pass http://127.0.0.1:5002; - } - - location /static { - root /usr/lib/python3.6/site-packages/metasrht; - } - } - -Once the proxy is configured, you should be able to access your new service. - -## OAuth configuration - -For services other than meta.sr.ht, you have to create and configure an OAuth -client before users can log into your service. To do this, visit your profile -on your meta.sr.ht instance, select the OAuth tab and register a new client. -Append the path `/oauth/callback` to the URL of your service instance and -choose this value as the base redirect URI (for example -`https://git.sr.ht/oauth/callback`). Update your service configuration file with -the generated client ID and secret. - -You then need to configure that client as "preauthorized", i.e. first-party. -This skips the OAuth consent screen that third-party applications are subject to -and enables some extra API features. Log into the meta.sr.ht database and run -the following query (adjusting the client ID as appropriate): - -```sql -update oauthclient set preauthorized = true where client_id = 'your client ID'; -``` - -Now you should be able to log into your new service. - -Alternatively, you can generally run most SourceHut services by pointing them at -*our* meta.sr.ht instance and registering them as an OAuth client. For example, -you could run a custom builds.sr.ht instance which logs in with hosted -meta.sr.ht accounts by registering for an OAuth client there and skipping the -preauthorized step. YMMV. +After installing your sr.ht services, you'll need to configure them. For more +information, see the [Configuration](/configuration.md) page. diff --git a/packages.md b/packages.md index 80b6550..28a3826 100644 --- a/packages.md +++ b/packages.md @@ -1,70 +1,102 @@ --- -title: Package mirrors +title: Package Repositories --- -sr.ht operates a number of automatically-updated package repositories where you -can find packages for sr.ht software in various distributions at +SourceHut operates a number of package repositories for sr.ht software. These +repositories, which support various distributions, are available at [mirror.sr.ht](https://mirror.sr.ht). -All of our package repositories accept patches as necessary on -[sr.ht-packages](https://lists.sr.ht/~sircmpwn/sr.ht-packages). +If you would like to contribute to the package repositories, we accept +patches on the [sr.ht-packages](https://lists.sr.ht/~sircmpwn/sr.ht-packages) +mailing list. -# Package Repositories +<div class="alert alert-warning"> + <p> + <strong>Warning:</strong> SourceHut is still in alpha, and has no stable + releases. As such, we do not recommend packaging SourceHut for your upstream + distribution repositories until we have shipped stable versions of our + software. + </p> + <p> + If you are still interested in packaging it for your distribution, please + reach out to us for help in automating the maintenance of a third-party + distribution which is automatically updated following our (frequent) + upstream rolling releases. + </p> +</div> -## Alpine Linux +# Alpine Linux -This is the recommended distribution of SourceHut, and the one we use for the -hosted instance upstream. This must be the first line in `/etc/apk/repositories` -— we shadow some upstream packages with patched versions. You will need -the community repository enabled. +**Mirror**: https://mirror.sr.ht/alpine/v3.12/sr.ht -**Mirror URL**: `https://mirror.sr.ht/alpine/v3.12/sr.ht` - -**Signing Key**: [alpine@sr.ht.pub](https://mirror.sr.ht/alpine/alpine%40sr.ht.rsa.pub) - -``` -$ wget -q -O /etc/apk/keys/alpine@sr.ht.rsa.pub https://mirror.sr.ht/alpine/alpine@sr.ht.rsa.pub -``` +**Signing key**: [alpine@sr.ht.pub](https://mirror.sr.ht/alpine/alpine%40sr.ht.rsa.pub) **Maintainer**: Drew DeVault <sir@cmpwn.com> **Source**: https://git.sr.ht/~sircmpwn/sr.ht-apkbuilds -## Arch Linux +```sh +# Add the following line to /etc/apk/repositories +https://mirror.sr.ht/alpine/v3.12/sr.ht -**Mirror URL**: `https://mirror.sr.ht/archlinux/sr.ht` +# Install signing key +wget -q -O /etc/apk/keys/alpine@sr.ht.rsa.pub https://mirror.sr.ht/alpine/alpine@sr.ht.rsa.pub -``` -[sr.ht] -Server = https://mirror.sr.ht/archlinux/sr.ht +# Update package index +apk update ``` -**Signing Key**: `C0AAFC1676BD998617C94C42DC59670F1EB0A189` +<div class="alert alert-warning"> + <strong>Warning:</strong> SourceHut shadows some upstream packages with + patched versions. The SourceHut mirror must be the <em>first</em> repository + in <code>/etc/apk/repositories</code>. +</div> -``` -$ pacman-key --recv-keys C0AAFC1676BD998617C94C42DC59670F1EB0A189 -$ pacman-key --lsign-key C0AAFC1676BD998617C94C42DC59670F1EB0A189 -``` +<div class="alert alert-warning"> + <strong>Warning:</strong> The Alpine Linux community package repository must + be enabled. +</div> + +# Arch Linux + +**Mirror**: https://mirror.sr.ht/archlinux/sr.ht + +**Signing key**: `C0AAFC1676BD998617C94C42DC59670F1EB0A189` **Maintainer**: Eli Schwartz <eschwartz@archlinux.org> **Source**: https://git.sr.ht/~sircmpwn/sr.ht-pkgbuilds -## Debian +```sh +# Add the following two lines to /etc/pacman.conf +[sr.ht] +Server = https://mirror.sr.ht/archlinux/sr.ht -**Mirror URL**: `https://mirror.sr.ht/debian` +# Install signing key +sudo pacman-key --recv-keys C0AAFC1676BD998617C94C42DC59670F1EB0A189 +sudo pacman-key --lsign-key C0AAFC1676BD998617C94C42DC59670F1EB0A189 -``` -deb https://mirror.sr.ht/debian sid main +# Update package index +sudo pacman -Syu ``` -**Signing Key**: [key.asc](https://mirror.sr.ht/debian/key.asc) `6B1296C65B24472674E7B6520585B50AC6A4914D` +# Debian +**Mirror**: https://mirror.sr.ht/debian -``` -$ wget -q https://mirror.sr.ht/debian/key.asc -O - | sudo apt-key --keyring /etc/apt/trusted.gpg.d/debian.sr.ht.gpg add - -``` +**Signing key**: [key.asc](https://mirror.sr.ht/debian/key.asc) `6B1296C65B24472674E7B6520585B50AC6A4914D` **Maintainer**: Denis Laxalde <denis@laxalde.org> **Source**: https://git.sr.ht/~dlax/sr.ht-debbuilds + +```sh +# Add the following line to /etc/apt/sources.list +deb https://mirror.sr.ht/debian sid main + +# Install signing key +wget -q https://mirror.sr.ht/debian/key.asc -O - | sudo apt-key --keyring /etc/apt/trusted.gpg.d/debian.sr.ht.gpg add - + +# Update package index +sudo apt update +``` |