infra/codestorage/hg-repo-sync
infra/codestorage/hg-repo-sync/run.sh
Update to be more modular. We now have a more modular approach to running our mercurial servers, and the approach to the individual containers is a bit more modular as well. The hg-repo-sync container now has a few responsibilities: 1. Install Mercurial 2. Create the group that’s going to be writing all our files 3. Create the folders we’ll store files in, and make the new group the owner 4. Define a script that pulls repos from GCS to local disk 5. Define a script that pushes repos to GCS from local disk 6. Define a script that causes a delay between runs This mainly involved breaking the previous run script into a few separate scripts, so they could be invoked individually. This docker container _can_ be used as a standalone image that will keep the local disk in sync with GCS, but it’s more useful as a base image for our other containers to work off of to achieve specialised functionality.
1.1 --- a/run.sh Mon Aug 17 18:56:19 2015 -0400 1.2 +++ b/run.sh Sun Oct 11 03:48:40 2015 -0700 1.3 @@ -1,75 +1,17 @@ 1.4 #!/bin/bash 1.5 -today=$(date +%F) 1.6 1.7 -# added trying to be clever 1.8 -yesterday=$(date --date yesterday +%F) 1.9 +DOPUSH=${PUSH:-"1"} 1.10 +DOPULL=${PULL:-"1"} 1.11 +DOSLEEP=${DOSLEEP:-"1"} 1.12 1.13 -echo "Fixing permissions..." 1.14 -chgrp -R 2000 /current 1.15 -echo "Cleaning up..." 1.16 -rm -rf /scratch/* 1.17 -echo "Downloading bundles from Google Cloud Storage..." 1.18 -output=$(gsutil cp -R gs://backups.code.secondbit.org/$today/\* /scratch 2>&1) 1.19 -echo $output 1.20 - 1.21 -bundles=$(find /scratch -name '*.bundle') 1.22 - 1.23 -# added trying to be clever 1.24 -# basically, if we get an error downloading the bundles, try for yesterday's 1.25 -# this could happen if the pod restarts between the last upload of day A and the first upload of day B 1.26 -if [ $output == *"No URLs matched:"* ] 1.27 - echo "Downloading yesterday's bundles from Google Cloud Storage..." 1.28 - gsutil cp -R gs://backups.code.secondbit.org/$yesterday/\* /scratch 1.29 - mv /scratch/$yesterday /scratch/$today 1.30 - bundles=$(find /scratch -name '*.bundle' | wc -l) 1.31 +if [ $DOPUSH -eq "1" ]; then 1.32 + /bin/bash /usr/local/bin/helpers/push.sh 1.33 fi 1.34 1.35 -for bundle in $bundles 1.36 -do 1.37 - dir=$(dirname $bundle) 1.38 - echo "dir=$dir" 1.39 - stripped=${dir#.} 1.40 - echo "stripped=$stripped" 1.41 - stripped=${stripped#/scratch} 1.42 - echo "stripped=$stripped" 1.43 - target=${bundle#/scratch} 1.44 - echo "target=$target" 1.45 - target=${target%.bundle} 1.46 - echo "target=$target" 1.47 - if [ -d "/current${target}" ] 1.48 - then 1.49 - echo "Pulling changes from $bundle to /current$target" 1.50 - hg --cwd /current${target} pull $bundle 1.51 - else 1.52 - echo "Creating /current$target repo from $bundle" 1.53 - hg clone $bundle /current${target} 1.54 - chgrp -R 2000 /current${target} 1.55 - fi 1.56 -done 1.57 -echo "Cleaning up..." 1.58 -rm -rf /scratch/* 1.59 +if [ $DOPULL -eq "1" ]; then 1.60 + /bin/bash /usr/local/bin/helpers/pull.sh 1.61 +fi 1.62 1.63 -repos=$(find /current -name .hg -type d) 1.64 -for repo in $repos 1.65 -do 1.66 - dir=$(dirname $repo) 1.67 - echo "dir=$dir" 1.68 - stripped=${dir#.} 1.69 - echo "stripped=$stripped" 1.70 - stripped=${stripped#/current} 1.71 - echo "stripped=$stripped" 1.72 - target=/scratch/$today$stripped.bundle 1.73 - echo "target=$target" 1.74 - mkdir -p $(dirname $target) 1.75 - echo "Bundling $dir to $target" 1.76 - hg --cwd $dir bundle --all $target 1.77 -done 1.78 -echo "Pushing bundles to Google Cloud Storage..." 1.79 -gsutil cp -R /scratch/* gs://backups.code.secondbit.org/ 1.80 -echo "Cleaning up..." 1.81 -rm -rf /scratch/* 1.82 -# sleep between 5 and 10 minutes 1.83 -# randomized to prevent all our servers running this at the same time 1.84 -sleepfor="$[($RANDOM % 5) + 5]m" 1.85 -echo "Sleeping for $sleepfor..." 1.86 -sleep $sleepfor 1.87 +if [ $DOSLEEP -eq "1" ]; then 1.88 + /bin/bash /usr/local/bin/helpers/sleep.sh 1.89 +fi