Personally I wouldn't trust a microSD card for any production role beyond being a bootloader. You'll be paged one day only to find you had to make a time-consuming visit to your co-lo, and that all your data was irretrievably gone.
The solution to this is to have one "real" PC acting as a NAS and serving iSCSI. You would have to divide the extra cost of this PC across all the ARM servers you have.
Micro SD are notoriously bad for server usage, they are designed to store infrequently changed files for a while, like mp3 collection, cameraphone snapshots, apps etc... Going with a brand name might help.
In order to extend their lifespan, you might want to minimize the writes by disabling some logs and the "last access" updates on files (`noatime` for "no access times").
Instead of disabling the logs, you can put them on a tmpfs. You'll lose them if the machine crashes or reboots, but it's still better than not having them at all.
I run a Dreamplug pretty much 24/7, as my "homebrew" backup and NAS solution; the SD is used to boot and then mount an external RAID enclosure connected via eSATA, which holds the real data. Works quite well, in a fraction of the space that a "real" pc (or even a MacMini) would need.
You could also get an "industrial grade" SD card; they claim a lifetime of years instead of months. They're usually using SLC Flash and (possibly) more advanced wear leveling compared to consumer cards.
i wouldn't count on that, google's study of hard drive lifetime/durability indicated that "enterprise" hard drives had almost identical lifetimes and performance characteristics to consumer gear (baring of course stuff like 15k RPM drives).
In other words, "enterprise" is more or less exactly the same stuff, but at a premium. Price discrimination strikes again.
In the Flash case there's a big physical difference between MLC and SLC. You're probably about as likely to get struck down by firmware failure in both cases, though.
You're probably right, it's certainly the component I'd be least surprised about if it failed.
I do nightly backups off the box, and if the machine does die, I'll post down a newly configured micro-SD card for insertion before doing much investigation.
SD is my main concern with these hacker boards, I mentioned it in a previous item. They shouldn't really be relying on this sort of basic flash memory, but I know why... it's accessible and cheap - plus many uses won't be 24/7. Time will tell, but it feels like a time bomb that will affect people who don't know about it (i.e. those that don't know about configuring these devices for minimal writes).
The solution to this is to have one "real" PC acting as a NAS and serving iSCSI. You would have to divide the extra cost of this PC across all the ARM servers you have.