Security incident report 2021-11-16

3 files changed, 121 insertions, 0 deletions

diff --git a/ops/index.md b/ops/index.md
index 9a6a711..8c6a9e2 100644
--- a/ops/index.md
+++ b/ops/index.md
@@ -20,6 +20,7 @@ Additional resources:
 - [Provisioning & allocation](/ops/provisioning.md)
 - [PostgreSQL robustness planning](/ops/robust-psql.md)
 - [SourceHut scalability plans](/ops/scale.md)
+- [Security incident reports](/ops/security-incidents)
 
 # Operational Resources
 
diff --git a/ops/security-incidents/2021-11-16-hg.sr.ht.md b/ops/security-incidents/2021-11-16-hg.sr.ht.md
new file mode 100644
index 0000000..a42f121
--- /dev/null
+++ b/ops/security-incidents/2021-11-16-hg.sr.ht.md
@@ -0,0 +1,115 @@
+# hg.sr.ht security incident 2021-11-16
+
+On November 16th, we were alerted to a security vulnerability in hg.sr.ht.
+Unauthenticated users could clone private hg repositories via https, which is
+normally not permitted, provided they knew the URL in advance.
+
+During the period in which hg.sr.ht was vulnerable, we identified 24 private Hg
+repositories which had been cloned in this way, representing a total of 5 users,
+from 7 IP addresses. We believe that this activity was benign, but have notified
+the affected users nevertheless.
+
+Third-party instances of hg.sr.ht are not expected to be affected by this issue.
+
+## Full details
+
+git.sr.ht serves git repositories over HTTPs via nginx for improved performance.
+To secure private repositories, we use the nginx "auth_request" feature to make
+an HTTP request to our web service to determine if the clone should be
+permitted. The configuration looks like this:
+
+```
+location = /authorize {
+  proxy_pass http://127.0.0.1:5001;
+  proxy_pass_request_body off;
+  proxy_set_header Content-Length "";
+  proxy_set_header X-Original-URI $request_uri;
+}
+
+location ~ ^/([^/]+)/([^/]+)/(HEAD|info/refs|objects/info/.*|git-upload-pack).*$ {
+  auth_request /authorize;
+  # ...
+```
+
+However, hg.sr.ht uses a somewhat different approach. HTTP requests from the
+Mercurial client are routed directly to the web application, which runs a shim
+over hgweb to service the request. This shim also calls the authorize endpoint,
+which is implemented in code shared between git.sr.ht and hg.sr.ht, to determine
+if it should allow the clone to proceed. The relevant hg.sr.ht code is:
+
+```
+def _get_hgweb_config(self, req):
+    # Check that this repository is authorized for public https
+    # cloning. The authorize endpoint is implemented in scmsrht.
+    auth_endpoint = req.url_root + 'authorize'
+    headers = {'X-Original-URI': req.url}
+    auth_resp = requests.get(auth_endpoint, headers=headers)
+    if auth_resp.status_code != 200:
+        return None
+```
+
+On October 27th, as part of a broader initiative to bring our nginx
+configurations under version control, the configuration details for git.sr.ht's
+authorize endpoint were copied into hg.sr.ht's nginx configuration. Because we
+do not use the "auth_request" feature in hg.sr.ht, this was not necessary.
+Moreover, it introduced a vulnerability by causing nginx to overwrite the
+X-Original-URI header set by this Python code to "/authorize" -- the value of
+$request_uri. The authorize endpoint is designed to permit access for any routes
+other than a repository, which caused all requests to be permitted regardless of
+the URI.
+
+## Mitigations
+
+The issue was corrected by removing the erroneously copied authorize route from
+the hg.sr.ht nginx configuration.
+
+To catch similar issues sooner, we have also added a cronjob which periodically
+attempts to clone private repositories without authorization. Should it succeed,
+operators will be notified.
+
+To identify potentially affected users, we consulted HTTP logs from the
+vulnerable time period for hg clone requests. We identified all Hg repositories
+which had been cloned over HTTP in this period, narrowed the list down to
+private repositories, and sent the following email to the owners of these
+repositories:
+
+```
+To: $user
+From: Drew DeVault <sir@cmpwn.com>
+Subject: Notice of potential disclosure of private repository data on hg.sr.ht
+
+It has come to our attention that your private repositories on hg.sr.ht
+may have been affected by a recent vulnerability in hg.sr.ht. Assuming
+they knew the URL in advance, this vulnerability allowed arbitrary users
+to clone your private hg.sr.ht repositories.
+
+During the time period when this vulnerability was present, we recorded
+HTTPs clone activity for the following private repositories on your
+account:
+
+$repos
+
+The following IP addresses were implicated in this activity:
+
+$addresses
+
+If one or more of these addresses is not associated with your own
+activity, it is possible that someone has obtained a copy of your
+private repository.
+
+I apologise for the oversight. If you have any questions, please reply
+to this email and I will answer them to the best of my ability.
+```
+
+## Timeline
+
+**2021-10-27**: Updates to our nginx configuration introduce the vulnerability.
+
+**2021-11-16 9:04 PM**: User reaches out to us to responsibly disclose the issue.
+
+**2021-11-16 9:26 PM**: SourceHut confirms the bug, starts investigation.
+
+**2021-11-16 9:48 PM**: A fix is identified and applied. SourceHut begins to
+identify potentially affected users.
+
+**2021-11-16 10:54 PM**: Potentially affected users are notified.
diff --git a/ops/security-incidents/index.md b/ops/security-incidents/index.md
new file mode 100644
index 0000000..ba44c91
--- /dev/null
+++ b/ops/security-incidents/index.md
@@ -0,0 +1,5 @@
+# Security incident reports
+
+May this list never grow.
+
+- [hg.sr.ht security incident 2021-11-16](/ops/security-incidents/2021-11-16-hg.sr.ht.md)