Clojure is just fantastic for this kind of thing. You can really tell Rich Hickey and the wider community has spent a lot of time thinking about developer problems and actually fixing them with sensible tooling.
I've done something similar. It started with creating a frontend for more easily viewing my albums on flickr (https://ash.code27.co.za).
Then when flickr was acquired by smugmug I migrated all my photos to DropBox, only to later find out that a small number of photos have been randomly truncated.
In the end I just started to fork out for Flickr, although I have similar feelings each year when I need to renew.
good stuff you've written there, mate. it shows the tight feedback loop of REPL driven development. but, the second part of this series reminds me how hard frontend development is.
It's the standard developer story. 50 hours into development, you're like, wait, how long would it have taken to go to each album and click Select All->Download?
The hardest I had to do was to change the docker compose file and update my postgres instance. As long as you don't blindly update its fine. There's the occasional bug, but so far never had issues with the data
I'd also recommend S3 intelligent tiering for backups. Storage price goes down to near-glacier levels if you don't access the files, but is available instantly if you do need them. Perfect for important backups.
Couple things to note depending on the traffic and what you're storing:
- Intelligent tiering won't auto-tier <128KB files, will bill them as if they were 128KB in size and at Frequent Access prices
- Glacier and Infrequent Access tiers will bill you a good amount for reading objects, so if they're accessed more than a handful times a year, it might be better to keep them in a higher tier
- Standard Infrequent Access, Glacier Flexible Retrieval and Glacier Deep Archive bill a minimum of 30, 90 and 180 days per object, so better not using it for short lived objects.
I was working in Goldman Sachs's 'Fixed income, currency and commodities' division for a while. Commonly referred to as FICC internally. Which gave me some pause as a German.