diff options
| -rw-r--r-- | builds.sr.ht/api.md | 9 | ||||
| -rw-r--r-- | builds.sr.ht/index.md | 5 | ||||
| -rw-r--r-- | builds.sr.ht/installation.md | 166 | ||||
| -rw-r--r-- | meta.sr.ht/installation.md | 2 | ||||
| -rw-r--r-- | packages.md | 25 |
5 files changed, 207 insertions, 0 deletions
diff --git a/builds.sr.ht/api.md b/builds.sr.ht/api.md index b46c575..9fec4e6 100644 --- a/builds.sr.ht/api.md +++ b/builds.sr.ht/api.md @@ -75,3 +75,12 @@ Gets information about a job by its ID. - **running**: task is in-progress - **success**: task completed without errors - **failed**: task completed with errors + +## GET /api/jobs/:id/manifest + +Returns the original job's build manifest as plain text. + +## POST /api/jobs/:id/start + +Starts a job that was created with `execute=false`. Returns an empty JSON +object when successful. diff --git a/builds.sr.ht/index.md b/builds.sr.ht/index.md index ef06b5d..d199abe 100644 --- a/builds.sr.ht/index.md +++ b/builds.sr.ht/index.md @@ -4,6 +4,11 @@ your specifications and run your scripts in it. This is generally used to compile and test patches, deploy websites, build and publish packages, and so on. +**See also**: + +- [API reference](api.md) +- [Installation guide](installation.md) + # How jobs are submitted Unlike some other build systems, builds.sr.ht does not let you configure builds diff --git a/builds.sr.ht/installation.md b/builds.sr.ht/installation.md new file mode 100644 index 0000000..b7201f6 --- /dev/null +++ b/builds.sr.ht/installation.md @@ -0,0 +1,166 @@ +There are two components to builds.sr.ht: the job runner and the master server. +Typically installations will have one master and many runners distributed on +many servers, but both can be installed on the same server for small +installations (though [not without risk](#security-model)). We'll start by +setting up the master server. + +# Prerequisites + +- [meta.sr.ht](/meta.sr.ht/installation.md) +- [An OAuth client ID and secret for meta.sr.ht](https://meta.sr.ht/oauth) +- PostgreSQL server +- Two redis servers + +## Security model + +Let's start with a brief overview of the security model of builds.sr.ht. +Because builds.sr.ht runs arbitrary user code (and allows users to utilize +root), it's important to carefully secure the build environments. To this end, +builds run in a sandbox which consists of: + +- A KVM virtual machine via qemu +- Inside of an otherwise empty docker image +- Running as an unprivledged user +- On a server which is isolated by: + - Being on physically separate server from anything important + - Using its own isolated redis instance + - Having minimal database access + +We suggest you take similar precautions if your servers could be running +untrusted builds. Remember that if you build only your own software, integration +with other services could end up running untrusted builds (for example, +automatic testing of patches via lists.sr.ht). + +## PostgreSQL configuration + +Start by setting your config.ini's connection string to a superuser, then run +the following commands to create the initial schema: + + $ python3 + >>> from buildsrht.app import db + >>> db.create() + +Then create two users, one for the master and one for the runners (or one for +each runner if you prefer). They need the following permissions: + +- **master** should have ownership over the database and full read/write/alter + access to all table +- **runner** should have read/write access to the jobs table + +If you are running the master and runners on the same server, you will only be +able to use one user. + +Note: in the future runners will not have database access. + +## Redis configuration + +The master server should have two redis servers, one for its own use (as part of +the standard sr.ht configuration) and one for communication with the runners. +Runners should be provided access to only their redis instance - if distributed, +we recommend setting up an SSH tunnel from the master to the runner to offer +access to the appropriate Redis instance. + +## Package installation + +On the master server, install builds.sr.ht either [from +source](https://git.sr.ht/~sircmpwn/builds.sr.ht) or [from +packages](/packages.md). On the runners, install builds.sr.ht and +builds.sr.ht-images, the latter containing templates for our upstream build +images and glue code for running them. + +Place a config file in `/etc/sr.ht/builds.ini` based on +[config.ini.example](https://git.sr.ht/~sircmpwn/builds.sr.ht/tree/config.ini.example). +[Register an OAuth client](https://meta.sr.ht/oauth) and include its client ID +and secret in the config file. Under the `[builds.sr.ht]` section, configure +`redis` on the master to point to the *runner* redis - it will use this to +submit jobs to the runners. The other options are relevant to the build runner, +which we will set up later. + +Start builds.sr.ht (`systemctl start builds.sr.ht` or `service builds.sr.ht +start`) and configure a reverse proxy - here's our nginx configuration for +builds.sr.ht upstream: + + server { + listen 80; + server_name builds.sr.ht; + + location / { + return 302 https://$server_name$request_uri; + } + + location ^~ /.well-known { + root /var/www; + } + } + + server { + listen 443 ssl; + listen [::]:443 ssl; + server_name builds.sr.ht; + client_max_body_size 100M; + ssl_certificate /var/lib/acme/live/builds.sr.ht/fullchain; + ssl_certificate_key /var/lib/acme/live/builds.sr.ht/privkey; + ssl_trusted_certificate /var/lib/acme/live/builds.sr.ht/fullchain; + + location / { + proxy_pass http://127.0.0.1:5002; + } + + location /static { + root /usr/lib/python3.6/site-packages/buildsrht; + } + + location ^~ /.well-known { + root /var/www; + } + } + +## Runner setup + +On the runner, install the `builds.sr.ht-images` package (if building from +source, this package is simply the `images` directory copied to +`/var/lib/images`), as well as docker. Build the docker image like so: + + $ cd /var/lib/images + $ docker build -t qemu -f qemu/Dockerfile . + +This will build a docker image named `qemu` which contains a statically linked +build of qemu and nothing else. Next, build any of the images you need: + + $ cd archlinux + $ sudo ./genimg + $ ../control archlinux sanity-check + $ cd ../debian/jessie + $ sudo ./genimg + $ ../../control debian/jessie sanity-check + ... and so on ... + +Note: some images may not build on your host system without the installation of +extra tools, which may not be availalbe. However, all images can be built on +themselves, and a build.yml file is typically provided which can do the job if +run in a similar environment as the target image. + +## Creating new images + +If you require any additional images, you should write a `genimg` script which +produces `root.img.qcow2`, `initrd`, and `kernel` files. You should also write a +`functions` script, which is sourced by the control script for booting your +image, installing packages, and so on. Review existing images for guidance, it +should be possible to make virtually any kind of image (including non-Linux +operating systems - the only hard requirement is an ssh daemon) with enough +effort. + +## Configuring the runner + +Write an `/etc/sr.ht/builds.ini` configuration file similar to the one you wrote +on the master server. Only the `[sr.ht]` and `[builds.sr.ht]` sections are +required for the runners. `images` should be set to the installation path of +your images (`/var/lib/images`) and `buildlogs` should be set to the path where +the runner should write its build logs (the runner user should be able to create +files and directories here). Set `runner` to the hostname of the build runner. +You will need to configure nginx to serve the build logs directory in order for +build logs to appear correctly on the website. + +Once all of this is done, start the `builds.sr.ht-runner` service and it's off +to the races. Submit builds on the master server and they should run correctly +at this point. diff --git a/meta.sr.ht/installation.md b/meta.sr.ht/installation.md new file mode 100644 index 0000000..90ade71 --- /dev/null +++ b/meta.sr.ht/installation.md @@ -0,0 +1,2 @@ +meta.sr.ht is the only hard requirement of an sr.ht network, which is why it'd +be great if someone actually wrote installation instructions here. diff --git a/packages.md b/packages.md new file mode 100644 index 0000000..f8f8cad --- /dev/null +++ b/packages.md @@ -0,0 +1,25 @@ +sr.ht operates a number of automatically-updated package repositories where you +can find packages for sr.ht software in various distributions at +[mirror.sr.ht](https://mirror.sr.ht). + +# Package Repositories + +## Alpine Linux + +Note: this repository is a work in progress. + +**Mirror URL**: `https://mirror.sr.ht/alpine/sr.ht` + +**Signing Key**: [alpine@sr.ht.pub](https://mirror.sr.ht/alpine/sr.ht/alpine%40sr.ht.rsa.pub) + +## Arch Linux + +**Mirror URL**: `https://mirror.sr.ht/archlinux/sr.ht` + +**Signing Key**: `1ED4E14494433AE3FD302CAB74B4BCCEA60D0437` + +# Database Updates + +If present at `/etc/sr.ht/alembic/:name.ini` (where :name is "builds.sr.ht", +"git.sr.ht", etc), the packages will automatically run database migrations when +upgraded. |