My strategy for storing and archiving massive amounts of digital content.
ROCHESTER, New York- It almost feels like a ritual. Every time I return to my parent’s house from a year or so abroad I pile up all of the external hard drives that I’ve accumulated in the intervening time and dump everything onto a massive drive. I just sit there whenever I do this and think about all the travels that I had, all the people I’ve met, all the projects I completed.
I sit there and it sinks in that I’m home. I made it back. Again.
In my first ten years of traveling around the world collecting content I accumulated around 350 gigs of data. Over the past couple of years I collected more than this each month. Seriously, I collect over 500 gigs of data every thirty days. I now have to travel with a stack of external hard drives — I don’t even think about using my laptop for storage.
Fortunately, the cost of portable external hard drives have come down significantly and their storage capacities have exploded.
Today I picked up a 4 TB Western Digital external hard drive at Best Buy for $100. This will last me six months.
The rules of storage:
1) Always have everything on no less than two drives. Hard drives should not be trusted. Back them up immediately, every time you transfer content from SD cards or other sources.
2) Ultimately aim to have at least three to five hard copies stored at various locations and one copy stored in the cloud or on virtual servers.
3) Regularly update your hard copy archives. Hard drives don’t last forever, so periodically make copies of your archived drives.
The drive on the left is my 4 TB A drive. The two in the middle are 2 TB B drives which backup the content on the A drive. The tower on the right is an archive drive. It’s only 8 TB so I will have to upgrade it the next time I do a big content dump.
I have three types of physical drives: A drives, where I store new content; B drives, where I immediately back up this content; and massive archive drives, where I store everything I’ve done yet.
I also archive my A drives when they get full (i.e. stick them on a shelf).
I erase and reuse my B drives after I unload the content (taken from the A drives) onto an giant archive drive.
When my archive drives get full I transfer all the content onto a larger archive drive. I then retire the previous one onto a shelf or in a safe at my parents’ house in Rochester and / or my in-laws’ house in Maine, where it just sits, hopefully remaining in working order for a span of years.
The idea is that as time goes on I will have all of my content copied to an ever growing number of archived hard drives. As it stands now, my early content is probably sitting on five or so different drives divided between Rochester and Maine, and when I get a new archive drive — which is going to happen the next time I return from a jaunt abroad with a massive multi-TB content dump.
As far as online storage goes, this is a little tedious, as the file sizes of videos in the codecs that I use tend to be huge (Prores 422 or RAW DNG). I try to upload them when my connection is fast enough, but I don’t get too obsessed with this. It will all eventually get up there.
The online drive gives me not only an additional backup but also the ability to access my video and image content from anywhere in the world with a fast enough internet connection. Online access is essential, as without it I would need to carry my entire collection of content at all times, which would be untenable. The downsize is that I’m often not able to upload the files before needing to archive a drive, so there are many glaring holes in my collection with some countries, cities, projects, etc that I don’t have access to when away from my large archive drives. But like I said, someday it will all get up there.
I’m currently using Mega for online storage. My older content is still virtualized via Dropbox, but I’m now only using this for routine syncing of relatively smaller files due to the platform’s storage restrictions.
As far as the type of portable drives I use, I tend to go for Western Digital MyPassports. They are cheap and relatively fast. I used to pay more for the armored shock and waterproof drives, but then I had one break and I concluded it wasn’t worth the cost increase.
Ideally, I would like to use super high end solid state drives as my portable A drives, but at their current costs I could get five WD MyPassports.
Currently, I’m mostly going for 4 TB portable drives, but I’m basically shooting for the largest size possible at an acceptable price point.
As I mentioned previously, 4 TB will get me around six months of content collection, so, ideally, I will only need to spend $200 to $300 per year for portable storage (I reformat and reused the portable B drives).
The big archive drives though, that’s a different story. Eventually, I would like to get a massive RAID server and hook it up to either my parent’s or in-law’s modem so I can have access to it anywhere. A big investment, but probably worth it.