http://code.secondbit.org/infra/codestorage/hg-ssh/ en-us infra/codestorage/hg-ssh Changelog http://code.secondbit.org/infra/codestorage/hg-ssh/rev/4c6afe37e83a http://code.secondbit.org/infra/codestorage/hg-ssh/rev/4c6afe37e83a changeset 4c6afe37e83a branch default bookmark tag tip user Paddy <paddy@secondbit.org> description Pull hostkeys when pulling SSH keys.

Rather than relying on Kubernetes secrets and baking public keys right in, which
was bound to get fraught, we now have some graceful degradation. It defaults to
automatically-generated random keys, but will try to download some keys from
Google Cloud Storage for the host. If it can find some, it'll try to use those,
instead. files Dockerfile
hostkeys/ssh_host_dsa_key.pub
hostkeys/ssh_host_ecdsa_key.pub
hostkeys/ssh_host_ed25519_key.pub
hostkeys/ssh_host_key.pub
hostkeys/ssh_host_rsa_key.pub
pullkeys.sh
run.sh
]]> Paddy <paddy@secondbit.org> Thu, 15 Oct 2015 01:32:28 -0700 http://code.secondbit.org/infra/codestorage/hg-ssh/rev/bf7b66df555f http://code.secondbit.org/infra/codestorage/hg-ssh/rev/bf7b66df555f changeset bf7b66df555f branch bookmark tag user Paddy <paddy@secondbit.org> description This time with bonus host key pinning.

You know what's _super obnoxious_? Getting a MITM warning just because a Docker
container restarted.

So, basically, every time you set up an SSH server, it generated its own
public/private keypair. And the first time you connect, you'd get that public
key from it, and store it locally. Then if you tried connecting later, and a
different public key was used, OpenSSH would vomit an error in your face and
stop the connection, because it thought you were running into a
man-in-the-middle attack. Which is generally a Good Thing.

But when we apply this to Docker, it becomes problematic. I'm not subject to a
MITM attack, my container just restarted, you silly machine.

The correct way around this problem was to reuse the public/private key pair as
part of the image. So I put the public keys in the image itself, and used
Kubernetes secrets to mount the private keys at /data/ssh/, and then a script to
copy them all over into the /etc/ssh directory, where they'd override the
generated ones. This means that every instance of this container has the same
public/private key combo, meaning no more warning. That solves the problem in a
reasonably secure (I think) way when the image is only used by us, but if anyone
else wants to use the image, suddenly they need to edit it. They need to change
the public keys, because they don't have our private keys. files Dockerfile
hostkeys/ssh_host_dsa_key.pub
hostkeys/ssh_host_ecdsa_key.pub
hostkeys/ssh_host_ed25519_key.pub
hostkeys/ssh_host_key.pub
hostkeys/ssh_host_rsa_key.pub
run.sh
]]> Paddy <paddy@secondbit.org> Thu, 15 Oct 2015 00:14:32 -0700 http://code.secondbit.org/infra/codestorage/hg-ssh/rev/1b1ca7817a10 http://code.secondbit.org/infra/codestorage/hg-ssh/rev/1b1ca7817a10 changeset 1b1ca7817a10 branch bookmark tag user Paddy <paddy@secondbit.org> description Set trust settings to avoid annoying message.

Update the hgrc to trust the .hgrc files of everyone in the committers group,
because I got tired of seeing the "not trusting file
/mounted/repos/blah-blah/.hg/hgrc from untrusted user root, group committers"
message every time I pushed. files hgrc
]]> Paddy <paddy@secondbit.org> Wed, 14 Oct 2015 23:17:12 -0700 http://code.secondbit.org/infra/codestorage/hg-ssh/rev/bc0c83d5015d http://code.secondbit.org/infra/codestorage/hg-ssh/rev/bc0c83d5015d changeset bc0c83d5015d branch bookmark tag user Paddy <paddy@secondbit.org> description Use the relative DNS address.

Use the relative DNS address when digging for web frontends, which now works
thanks to +search on dig. This allows us to transfer between namespaces without
needing to change anything. Also makes this, in general, more flexible. files post-commit-broadcast.sh
]]> Paddy <paddy@secondbit.org> Wed, 14 Oct 2015 23:01:04 -0700 http://code.secondbit.org/infra/codestorage/hg-ssh/rev/121585c71fd7 http://code.secondbit.org/infra/codestorage/hg-ssh/rev/121585c71fd7 changeset 121585c71fd7 branch bookmark tag user Paddy <paddy@secondbit.org> description Update to broadcast pushes to all our web frontends.

We'll need dig, so install that when we apt-get.

Fix a typo in the hashbang line for create_user.sh

Add an hgrc file that goes in /etc/mercurial/hgrc to add the
changegroup.fe_publish hook to all our repos. Basically, any time we get a
change on disk, that automatically gets propagated out to all the frontend using
our post-commit-broadcast.sh script.

Write the aforementioned post-commit-broadcast.sh script. This basically finds
the repo we're in (by stripping known prefixes), then uses dig to compile a list
of web frontends. Finally, for each web frontend, we do an hg push over http.

Not so hard, but it means all our web frontends are kept recent.

This has a few shortcomings. I don't think it will work when starting a new
repo; I think we need to wait for hg-repo-sync to back that up, then the web
frontend to pull from backups. Another possibility is that we push, then stand
up a new front end before the push makes it into the backups. The frontend then
won't have that push until it pulls again from backups. These are rare,
minutes-long windows where we're out of sync, so I'm really ok with these
failure modes. files Dockerfile
create_user.sh
hgrc
post-commit-broadcast.sh
]]> Paddy <paddy@secondbit.org> Wed, 14 Oct 2015 20:55:18 -0700 http://code.secondbit.org/infra/codestorage/hg-ssh/rev/2f4a2a20ad6d http://code.secondbit.org/infra/codestorage/hg-ssh/rev/2f4a2a20ad6d changeset 2f4a2a20ad6d branch bookmark tag user Paddy <paddy@secondbit.org> description Update to be more modular.

We updated our Mercurial serving architecture to be a bit more modular. The main
difference is that we now are based off the secondbit/hg-repo-sync image, and
don't need to do as much setup to get the basics (Mercurial, folders, etc.)

We now have a pullkeys.ssh script, which pulls down the SSH keys stored in a
specified GCS bucket, and creates users for them. This allows us to update who
has push access, without modifying the docker image.

We also have a custom run.sh script now, instead of starting sshd directly,
because we need to do a few things when starting this up:

1. Modify the permissions on the mounted directories while we're root so SSH
users can write and read the committed files. We have to do this at start time
instead of at image build time because Kubernetes' volumes don't respect the
permissions set at build time.
2. Pull all the repos backed up to GCS to local disk, which means that startup
automatically picks up at the last known state. This script is built into the
image by secondbit/hg-repo-sync.
3. Pull all the SSH keys from GCS, using the new script. This creates the new
users and lets us SSH into the server, while keeping the user definitions
separate from the image itself.
4. Finally, start the SSH daemon. files Dockerfile
pullkeys.sh
run.sh
]]> Paddy <paddy@secondbit.org> Sun, 11 Oct 2015 17:06:15 -0700 http://code.secondbit.org/infra/codestorage/hg-ssh/rev/eeaf3e97ed44 http://code.secondbit.org/infra/codestorage/hg-ssh/rev/eeaf3e97ed44 changeset eeaf3e97ed44 branch bookmark tag user Paddy <paddy@secondbit.org> description Initial commit.

This hardcodes more than it should and syncs with Google Cloud Storage more than
is really necessary, but it A) works and B) is currently on code.secondbit.org,
so that should probably be recorded for posterity. files .hgignore
Dockerfile
create_user.sh
]]> Paddy <paddy@secondbit.org> Mon, 17 Aug 2015 19:07:28 -0400