diff options
| -rw-r--r-- | builds.sr.ht/installation.md | 110 | ||||
| -rw-r--r-- | installation.md | 162 | ||||
| -rw-r--r-- | meta.sr.ht/installation.md | 11 |
3 files changed, 198 insertions, 85 deletions
diff --git a/builds.sr.ht/installation.md b/builds.sr.ht/installation.md index b7201f6..7eb0fe0 100644 --- a/builds.sr.ht/installation.md +++ b/builds.sr.ht/installation.md @@ -4,14 +4,22 @@ many servers, but both can be installed on the same server for small installations (though [not without risk](#security-model)). We'll start by setting up the master server. -# Prerequisites +# Web service -- [meta.sr.ht](/meta.sr.ht/installation.md) -- [An OAuth client ID and secret for meta.sr.ht](https://meta.sr.ht/oauth) -- PostgreSQL server -- Two redis servers +The master server is a standard sr.ht web service and can be [installed as +such](/installation.md). However, it is important that you configure *two* Redis +servers - one that the runners should have access to, and one that they should +not. Insert connection details for the former into build.sr.ht's configuration +file under the `redis` key. Each build runner will also need a local redis +instance running. In an insecure deployment (all services on the same server) +you can get away with a single Redis instance. -## Security model +We suggest using an SSH tunnel to share the slave Redis instance between job +runners and the master server, but you can use any method you prefer. If you use +an SSH tunnel, you will likely want to use a reverse tunnel initiated from the +master server, so the slaves are unable to SSH into the master server. + +# Security model Let's start with a brief overview of the security model of builds.sr.ht. Because builds.sr.ht runs arbitrary user code (and allows users to utilize @@ -31,91 +39,26 @@ untrusted builds. Remember that if you build only your own software, integration with other services could end up running untrusted builds (for example, automatic testing of patches via lists.sr.ht). -## PostgreSQL configuration +# Package installation -Start by setting your config.ini's connection string to a superuser, then run -the following commands to create the initial schema: +On each runner, install the builds.sr.ht and builds.sr.ht-images packages. - $ python3 - >>> from buildsrht.app import db - >>> db.create() +# Database configuration -Then create two users, one for the master and one for the runners (or one for -each runner if you prefer). They need the following permissions: +Create two users, one for the master and one for the runners (or one for each +runner if you prefer). They need the following permissions: - **master** should have ownership over the database and full read/write/alter access to all table - **runner** should have read/write access to the jobs table If you are running the master and runners on the same server, you will only be -able to use one user. +able to use one user - the master user. Configure both the web service and build +runner with this account. Otherwise, two separate accounts is recommended. Note: in the future runners will not have database access. -## Redis configuration - -The master server should have two redis servers, one for its own use (as part of -the standard sr.ht configuration) and one for communication with the runners. -Runners should be provided access to only their redis instance - if distributed, -we recommend setting up an SSH tunnel from the master to the runner to offer -access to the appropriate Redis instance. - -## Package installation - -On the master server, install builds.sr.ht either [from -source](https://git.sr.ht/~sircmpwn/builds.sr.ht) or [from -packages](/packages.md). On the runners, install builds.sr.ht and -builds.sr.ht-images, the latter containing templates for our upstream build -images and glue code for running them. - -Place a config file in `/etc/sr.ht/builds.ini` based on -[config.ini.example](https://git.sr.ht/~sircmpwn/builds.sr.ht/tree/config.ini.example). -[Register an OAuth client](https://meta.sr.ht/oauth) and include its client ID -and secret in the config file. Under the `[builds.sr.ht]` section, configure -`redis` on the master to point to the *runner* redis - it will use this to -submit jobs to the runners. The other options are relevant to the build runner, -which we will set up later. - -Start builds.sr.ht (`systemctl start builds.sr.ht` or `service builds.sr.ht -start`) and configure a reverse proxy - here's our nginx configuration for -builds.sr.ht upstream: - - server { - listen 80; - server_name builds.sr.ht; - - location / { - return 302 https://$server_name$request_uri; - } - - location ^~ /.well-known { - root /var/www; - } - } - - server { - listen 443 ssl; - listen [::]:443 ssl; - server_name builds.sr.ht; - client_max_body_size 100M; - ssl_certificate /var/lib/acme/live/builds.sr.ht/fullchain; - ssl_certificate_key /var/lib/acme/live/builds.sr.ht/privkey; - ssl_trusted_certificate /var/lib/acme/live/builds.sr.ht/fullchain; - - location / { - proxy_pass http://127.0.0.1:5002; - } - - location /static { - root /usr/lib/python3.6/site-packages/buildsrht; - } - - location ^~ /.well-known { - root /var/www; - } - } - -## Runner setup +# Install images On the runner, install the `builds.sr.ht-images` package (if building from source, this package is simply the `images` directory copied to @@ -136,9 +79,10 @@ build of qemu and nothing else. Next, build any of the images you need: ... and so on ... Note: some images may not build on your host system without the installation of -extra tools, which may not be availalbe. However, all images can be built on -themselves, and a build.yml file is typically provided which can do the job if -run in a similar environment as the target image. +extra tools, which may or may not be available for your host system. However, +all images can be built on themselves, and a build.yml file is typically +provided which can do the job if run in a similar environment as the target +image. ## Creating new images @@ -150,7 +94,7 @@ should be possible to make virtually any kind of image (including non-Linux operating systems - the only hard requirement is an ssh daemon) with enough effort. -## Configuring the runner +# Additional configuration Write an `/etc/sr.ht/builds.ini` configuration file similar to the one you wrote on the master server. Only the `[sr.ht]` and `[builds.sr.ht]` sections are diff --git a/installation.md b/installation.md new file mode 100644 index 0000000..f0b37bd --- /dev/null +++ b/installation.md @@ -0,0 +1,162 @@ +Installation of sr.ht web services uses a mostly consistent procedure across the +network. Specific details to each service are available on each service's +installation page: + +- [build.sr.ht](/builds.sr.ht/installation.md) +- [git.sr.ht](/git.sr.ht/installation.md) +- [man.sr.ht](/man.sr.ht/installation.md) +- [meta.sr.ht](/meta.sr.ht/installation.md) +- [todo.sr.ht](/todo.sr.ht/installation.md) + +General installation instructions follow. + +# Packages + +sr.ht provides a number of Linux distribution package repositories for your +use. For details specific to your distribution, see [packages.md](/packages.md). +Even if you wish to install sr.ht services from source, your distribution may +not include some of our dependencies and using our package repositories is +recommended. + +Installation from packages will also give you distro-specific daemon services +and will handle database migrations automatically during system updates. + +# Prerequisites + +You will need at least: + +- A PostgreSQL server +- A Redis server +- A mail server +- A cron daemon + +In order to use most sr.ht services. + +# sr.ht core + +sr.ht core is a Python package that provides shared functionality across all +sr.ht services. It also contains the default templates and stylesheets that give +sr.ht a consistent look and feel. + +The core package is listed as a dependency of the official `*.sr.ht` packages, +so installing it explicitly is not necessary if you are using our package +repositories. If you are not using our packages, obtain the [source +code](https://git.sr.ht/~sircmpwn/srht) and install it like a typical Python +package (`./setup.py install --prefix=/usr`). You will need to install its +dependencies beforehand, for an up-to-date list see [the +package](https://git.sr.ht/~sircmpwn/sr.ht-pkgbuilds/tree/python-srht/PKGBUILD). + +# Package Installation + +Packages are named as you would expect: meta.sr.ht is called `meta.sr.ht`. On +package managers where dots are not permitted in package names, dashes are used. +Underscores are used on systems where dashes are not permitted. + +If installing from source, see [the +packages](https://git.sr.ht/~sircmpwn/sr.ht-pkgbuilds) for an up-to-date list of +dependencies and install it like any other Python package: `./setup.py install +--prefix=/usr`. When installing from source, daemon service files are not +provided; you must write one appropriate to your system. + +# Site Configuration + +Each site will search for its config file in two locations: `./config.ini` and +`/etc/sr.ht/subdomain.ini` (for example, meta.sr.ht looks for +`/etc/sr.ht/meta.ini`). The exact nature of this config file varies depending on +which service you're setting up - from source, see `config.ini.example` and from +packages see `/etc/sr.ht/subdomain.ini.example`. + +# Database Configuration + +Start by setting your config.ini's connection string to a superuser, then run +the following commands to create the initial schema: + + $ python3 + >>> from [module].app import db + >>> db.create() + +Substitute `[module]` for the specific application's module, such as `metasrht` +or `buildsrht`. When complete, you may update your connection string to use a +less privileged user. + +## Schema Upgrades + +We use [alembic](http://alembic.zzzcomputing.com/en/latest/) to manage schema +migrations. A sample `alembic.ini` is provided (`alembic.ini.example` from +source and `/etc/sr.ht/alembic/subdomain.ini.example` from packages), which you +should modify to suit your needs - you will probably only need to set the +connection string. + +Run `alembic stamp head` once to tell alembic the schema is up-to-date. Future +upgrades will be managed automatically by the package, or if you're using the +source code you can run `alembic upgrade head` when you pull the latest version +down. + +# Compile static assets + +This step is only necessary for users configuring sr.ht from source. Run `make` +in the root of the repository to compile static assets. + +# Start the service + +A service file is included in the packages and uses the same name as the +package. On Arch Linux, for example, you can run `systemctl enable --now +meta.sr.ht` to start up meta.sr.ht, or `rc-update add meta.sr.ht && service +meta.sr.ht start` on Alpine. The service will be running at the port specified +in its config file, and you must now prepare to proxy to it. + +# Proxy configuration + +The exact nginx configuration you use will vary depending on your needs. Here is +an example for meta.sr.ht: + + server { + listen 80; + server_name meta.sr.ht; + + location / { + return 302 https://$server_name$request_uri; + } + + location ^~ /.well-known { + root /var/www; + } + } + + server { + listen 443 ssl; + listen [::]:443 ssl; + server_name meta.sr.ht; + client_max_body_size 100M; + ssl_certificate /var/lib/acme/live/meta.sr.ht/fullchain; + ssl_certificate_key /var/lib/acme/live/meta.sr.ht/privkey; + ssl_trusted_certificate /var/lib/acme/live/meta.sr.ht/fullchain; + + location / { + proxy_pass http://127.0.0.1:5002; + } + + location /static { + root /usr/lib/python3.6/site-packages/metasrht; + } + + location ^~ /.well-known { + root /var/www; + } + } + +Once the proxy is configured, you should be able to log into your new service. + +# Hacking on sr.ht + +If you just want to get the codebase running to hack on it, follow these steps: + +1. [Prerequisites](#prerequisites) +2. [sr.ht core](#sr.ht-core), however, you can simply clone it somewhere and add + it to your Python path. Also export `SRHT_PATH=/path/to/sr.ht-core` to use + your development repository for generating static assets. +3. [Site configuration](#site-configuration), but you can just put `config.ini` + in the working directory and sr.ht will read it there. +4. [Database configuration](#database-configuration) +5. [Compile static assets](#compile-static-assets) +6. `./run.py` will start the development server. diff --git a/meta.sr.ht/installation.md b/meta.sr.ht/installation.md index 90ade71..1ec1224 100644 --- a/meta.sr.ht/installation.md +++ b/meta.sr.ht/installation.md @@ -1,2 +1,9 @@ -meta.sr.ht is the only hard requirement of an sr.ht network, which is why it'd -be great if someone actually wrote installation instructions here. +meta.sr.ht is the central autentication service and service bus for the sr.ht +network, and a working meta.sr.ht is a basic requirement of running any other +sr.ht software. + +# Installation + +meta.sr.ht is a standard sr.ht web service and can be installed through the +[standard procedure](/installation.md). Additionally, you will need to set up a +cron job to run `meta-audit-cleanup` daily. |