Server backup to AWS S3
One thing is your data to be stolen. But to loose your data for good, that’s a different story.
Two decades ago, the biggest risk was hardware failure. Today it is ransom, industrial espionage, business competition wars.
Your data may have low value to others, but have huge value to you or your clients.
So even if you hardened security of your server to some level, it’s good to be ready for the worst and have a disaster recovery plan.
Another goal is to have a way to restore your data in some time backward, in case of your or your users/customers error — mistakenly deleted or overwritten files/data. Usual approach is to have backup of last 7 days, last 4 weeks and monthly backup of 12 or more months.
From the security standpoint: your backups should not be available from your server, the one you’re backing up. In case it will be taken control of, intruder will not be able to delete your backups or overwrite them.
My server is hosted at Digital Ocean. They offer full snapshot backup 4x weekly for 20% of the price of your droplet and recently they added 7x daily incremental snapshot for 30%, although this is not available in all datacenters (July 2024).
Good thing is, that you cannot delete these snapshot backups. So even in case someone gets to your web console, you’re safe. Yet he could change your password, delete 2FA, delete droplet etc.
Digital Ocean also offers object storage with S3 capabilities, called Spaces, yet not all capabilities as AWS itself. One of them is IAM users policies. In Digital Ocean, you have one user who has full access and that’s it. So Spaces is good product for CDN, but not for paranoid backup.
In AWS you can set up detailed policies for users, so our user for automatic backup from our server can have permission to upload file (PUT + multipart upload) and that’s it. So this way you’re safu even against intruder with root privileges. Point is, to have not one point of failure. Intruder can be anyone, not only sophisticated hacker. Session cookies can be stolen from your browser. You may not close your browser or terminal in some place where you should have, etc.
Now let’s come to technicalities.
As for the mysql, you can do mysqldump or mysqlpump and pipe it straight to your S3 bucket. You can use the same filename of backup every day, set up versioning bucket, and set up Lifecycle rules to delete old versions. For example for daily backup prefix:
This config will retain 1 latest backup + 6 last overwritten versions of your daily backup and delete all older. So even your server stops sending backups and 7 days pass, these files will retain.
(Another task would be to check integrity of those mysql backups. To avoid your golden bricks to be replaced by tungsten/wolfram bricks [Die Hard 3] or your Monet painting by nice copy [The Thomas Crown Affair])
AWS S3 for files backup
Now the game is gonna get interesting. I have 200K of user files to back up. Usually I would use tar incremental or differential backup for daily/weekly backups. Incremental backups are not very convenient to restore. You have to extract one by one. If you made full backup on 1st day of month and then incremental backups, and you want snapshot of your files of 25th day of that month, you have to extract full backup and then all of daily incremental backups from 2nd to 25th day sequentially. Differential backups are easy to restore, but they are storage-size consuming, especially in the end of the month.
So here is the great use of AWS S3 versioning bucket. You can backup daily, only new files as of yesterday’s backup. And this way you have snapshot backup of all your files, one by one. It’s the least storage-consuming and also you can restore snapshot of all your files or just particular file or subfolder (even recursively), for any particular day in the past, not just 30 days backwards. For this task, download this beautiful Python script from Angelo Compagnucci, called S3-pit-restore.
When you install AWS CLI to your machine, you’ll get useful command ‘sync’, thus you can easily sync files between your server filesystem and S3 bucket. So all done, right? Not exactly.
When you want to upload new files from your server to S3, sync command firstly downloads a list of all (prefixed) files from your bucket, then compares it with those within your filesystem and then uploads new files, creating new versions of them. That’s fine. But there are two buts. You pay for the LIST requests. If you have 200K files, one request per 1000 files max, so you pay 200 requests, monthly 6000 requests, that is $0.0324 per month. If your machine is not at AWS (EC2), you also pay for outbound data. One 200K list of files has 60MB. That is 1.8GB per month, $0.18/month. Also this process takes about 3+ minutes for 200K objects. Considering that those 200K files is 15GB, you could pay $0.37 for storage, but instead you would pay +$0.214 just for daily syncing.
So the question arises: if you’re the solely uploader to that S3 bucket, why should you even download that list? You don’t have to, just note somewhere the state of your files when you uploaded last time.
Get Samuelson78x’s stories in your inbox
Join Medium for free to get updates from this writer.
So I took inspiration from tar with it’s ‘snar’ file. I made php script, which makes list of all files and their timestamps of last modified. Next day it just compares actual filesystem with this snar file and then uploads new files to S3 and deletes deleted files from S3. It takes 3 seconds. It’s 60x+ faster, no LIST requests from S3, no outbound data. (upload to AWS is free)
It also deletes “directories” in S3 in recursive way, so deleting is fast.
First I tried to use ‘cp’ and ‘rm’ commands from aws cli, I called them from php using exec. It works fine, but it’s slow and cpu expensive. Uploading 100 of new files takes 82s, i.e. 0.708s per file. (same datacenter!)
So I made second version of the script, utilizing AWS SDK. This utilizes concurrent uploads (default 5), recursive deletion and also concurrent deletion in case of list of files. Now uploading 100 files takes 2.35s.
With this SDK version, it’s so fast, that I used it even for initial upload of all files and it has the same speed as bulk AWS CLI sync or cp.
You can download it here:
https://gist.github.com/SamuelGee/e1fe324937c442e196b800dad52a8855
If you have bigger files inventory (millions), you should probably add some memory optimization of arrays. I run it on php5.6 on my machine. On php7 it should be 3x better in memory use. It utilizes multipart uploads — default chunk is set to 20MB. It should be able to upload files bigger than 5GB (haven’t tested).
Available command params:
--dryrun if you just want to see difference between your s3sync.snar file
and current filesystem
--makesnar together with --dryrun it will create current snar file,
without any operations on S3. For example if you updated your filesystem
from S3 with AWS CLI sync or copy, you will need new snar.
--silent don't output progress bar. E.g. if you want to save output
from cron to logfile >> output.logYou can also set array of strings, files or directories, to exclude from backup, see examples in the code. Same for ignore symlinks or not, default yes.
It creates log file s3sync.log, in which you can find record of every file which has been uploaded to S3 or deleted from S3 — even files within recursive delete.
It also creates zero-length objects — for every directory of your filesystem. The reason is, to have a nice mirror - including empty directories. If you use ‘aws cli sync’, once you delete all files in particular directory, that directory will disappear in S3 (it’s just prefix). I don’t prefer this behavior, I will rather pay for PUT for empty objects. Of my 200K objects, 27K of them are directories, so keeping structure is important.
As for S3 bucket class, use S3 standard. In cheaper classes, there is a minimum file size of 128KB. Try
find . -type f -size -128k | wc -lin my case it was 90% of all files. So even I may pay less per GB in other classes, payable GB would be 10x of the real one.
In best case, use “Intelligent Tiering” on S3.
You can also set up Lifecycle Rules to permanently delete deleted/overwritten versions of files, e.g. older than 1 year etc. My users’ files don’t change dramatically, so I pay minimum overhead for deleted files, while having snapshot backup for every day of any years backwards — and all of it for minimal price. Thanks Jeff!
For info about Amazon S3 tiers and calculations, see my other article:
Thanks for watching. Bye!
