infra/codestorage/hg-repo-sync: cc08c8ed2128 pull.sh

Update to be more modular. We now have a more modular approach to running our mercurial servers, and the approach to the individual containers is a bit more modular as well. The hg-repo-sync container now has a few responsibilities: 1. Install Mercurial 2. Create the group that’s going to be writing all our files 3. Create the folders we’ll store files in, and make the new group the owner 4. Define a script that pulls repos from GCS to local disk 5. Define a script that pushes repos to GCS from local disk 6. Define a script that causes a delay between runs This mainly involved breaking the previous run script into a few separate scripts, so they could be invoked individually. This docker container _can_ be used as a standalone image that will keep the local disk in sync with GCS, but it’s more useful as a base image for our other containers to work off of to achieve specialised functionality.

1 #!/bin/bash

2 today=$(date +%F)

4 yesterday=$(date --date yesterday +%F)

6 DOMAIN=${DOMAIN:-code.secondbit.org}

7 BACKUP_BUCKET=${BACKUP_BUCKET:-backups.$DOMAIN}

9 echo "Fixing permissions..."

10 chgrp -R 2000 /mounted/repos

12 echo "Cleaning up..."

13 rm -rf /tmp/repos/*

15 echo "Downloading bundles from gs://${BACKUP_BUCKET}/${today}"

17 output=$(gsutil cp -R gs://$BACKUP_BUCKET/$today/\* /tmp/repos 2>&1)

18 echo $output

20 bundles=$(find /tmp/repos -name '*.bundle')

22 # basically, if we get an error downloading the bundles, try for yesterday's

23 # this could happen if the pod restarts between the last upload of day A and the first upload of day B

24 if [[ $output == *"No URLs matched:"* ]]

25 then

26 echo "Downloading yesterday's bundles from gs://${BACKUP_BUCKET}/${yesterday}"

27 gsutil cp -R gs://$BACKUP_BUCKET/$yesterday/\* /tmp/repos

28 mv /tmp/repos/$yesterday /tmp/repos/$today

29 bundles=$(find /tmp/repos -name '*.bundle')

30 fi

32 for bundle in $bundles

33 do

34 dir=$(dirname $bundle)

35 stripped=${dir#.}

36 stripped=${stripped#/tmp/repos}

37 target=${bundle#/tmp/repos}

38 target=${target%.bundle}

39 if [ -d "/mounted/repos${target}" ]

40 then

41 echo "Pulling changes from $bundle to /mounted/repos$target"

42 hg --cwd /mounted/repos${target} pull $bundle

43 else

44 echo "Creating /mounted/repos$target repo from $bundle"

45 hg clone $bundle /mounted/repos${target}

46 chgrp -R 2000 /mounted/repos${target}

47 fi

48 done

50 echo "Cleaning up..."

51 rm -rf /tmp/repos/*

53 echo "Pull complete."

infra/codestorage/hg-repo-sync

infra/codestorage/hg-repo-sync/pull.sh