Rsync and Cron

I was approached to build a web application that extracts student applications from a database and sends the results over to another server in a .csv file at regular intervals.

I needed to build a light-weight application that functioned in the background, was reliable and required little maintenance or intervention. Since creating a web service was off the table for this client, it was a perfect opportunity to use both rsync and cron in any meaningful way. I’m a fan of both. But first, cron.

CRON

Who doesn’t love a good cron job? I do. For me it reinforces that machines are good at doing things I’d rather not do; repetition.

1 0-23/2 * * * /usr/bin/php /var/www/app/cronjob.php
  • every minute, every two hours, tell php to hit the script at that location

RSYNC

rsync -azvv --delete --log-file=/var/www/html/app/storage/logs/rsync_log -e ssh /var/www/html/app/storage/payload/ user@111.11.11.111:/cygdrive/c/Shared/app/data/
  • –delete  — any files that are deleted on the server that generated the rsync command, will also get deleted on the other server.
  • –log-file  — gotta write that down.
  • – e  — choose your shell, in this case ssh

Love it…better than SCP. Built into the behaviour of rsync is a fallback measure. Should one or more of the rsync cron jobs fail, whatever .csv wasn’t transferred at the time of failure will be transferred at the next successful attempt.

Brad

A few of my favourite things: Agile software development with the potential for significant social impact combined with responsible and appropriate use of data, machine learning algorithms and systems that support research and evidence based decision making.

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *

*