> When a value is retrieved via its key, IronDB...
> Looks up that key in every store.
> Counts each unique, returned value.
> Determines the most commonly returned value as the 'correct' value.
> Returns this most common correct value
> Then IronDB self heals: if any store(s) returned a value different than the determined correct value, or no value at all, the correct value is rewritten to that store. In this way, consensus, reliability, and redundancy is maintained.
I'm not sure any of these properties are ensured the way anyone would want. Why do this? What are the failure modes this protects against? If only one of the backing stores is active when a result is written, and then the others later become active with no data, is blank data returned for the prior result? It seems like recording timestamps would fix this problem nicely on a single system and make this thing overall quite reliable.
Cookies are frequently cleared by users, and the other datastores -- IndexedDB, LocalStorage, and SessionStorage -- can be unceremoniously purged by the browser under storage pressure.
Thanks for the response! FWIW I'm overall really interested in this, as I maintain an application that uses localStorage to keep very important data while the app is offline until it can be uploaded.
I recently tested localForage [0] to see if it could be more reliable and get around storage limitations of localStorage. Unfortunately, LocalForage has a very annoying problem where it picks a single storage backend to use, but the one it chooses can switch on page reloads (yes, even on the same device/browser.) and this switching causes data loss as keys stored in the other backends are unaccessible. I'm very interested to see if IronDB can help here! Thanks for working on this.
> Why do this? What are the failure modes this protects against?
I have a use case for this: I've written what is basically a crud web app for a non-profit. That's an offline app (appcache) and right now uses localstorage for keeping data offline in the browser while in the field. That is pretty brittle as is, but dedicated application installations didn't fit the budget. The app is used in the field by volunteers all over the country. With volunteers fluctuating and many people being privacy conscious, it happens regularly that they change Firefox preferences to delete cookies etc. on shutdown, and now newer Firefoxes deprecating localstorage, potentially resulting in data loss.
Maybe this is useful tool, but several things strike as weird to me:
- save all values everywhere: why not select the one that would last the longest (that people delete least often)? Or maybe two, if you want a cookie but also want to survive over "delete all cookies". But then again, if a user wants to delete cookies then you really shouldn't try to keep it.
- "Data is stored resiliently but can also be voluntarily purged if the user designedly clears cookies and application storage." So you delete data when the user wants to delete data. Which database doesn't have this feature?
- doesn't use Flash, Silverlight, or Java -- which do?
- "healing" is usually used in the context of distributed computing, but here it means that it gets around user wanting to delete stuff from it's browser, but perhaps isn't in-the-know enough to delete everything but just the cookies.
No offence to the author but all this sound malicious. Perhaps this is why evercookie isn't maintained?
> save all values everywhere: why not select the one that would last the longest (that people delete least often)?
For improved resiliency.
You can also configure IronDB to only use any two datastores of your choice, if
you so desire: IronStorage's constructor takes an Array of storage
implementations of your choice. See https://github.com/gruns/irondb#api.
> So you delete data when the user wants to delete data. Which database doesn't have this feature?
Evercookie's goal is to store data indelibly, regardless of user intention.
IronDB's goal is to store data reliably, e.g. in the face of storage eviction,
but not against user intention. If the user clears all browsing data, that
willful action is respected.
Thank you for your feedback. I'll clarify this in the docs.
Why do we need a library to store data reliably? Aren't browsers API already reliable? Does the LocalStorage API for example sometimes fails to store its data properly?
If anyone could point to an up to date docs around storage eviction in browsers it would be greatly appreciated. It's been a big hit to our team's confidence in the "use the platform" mentality that this is not accurately documented.
That's the answer I was looking for, thanks. The intuition I have is that if someone deletes something from localStorage (or clears localStorage altogether), they want that data gone.
Those docs are dated unfortunately. For example, in my tests on multiple android devices and Mac OS the storage limit on Chrome is closer to 20% of total disk size and you don't need to prompt the user for persistent storage like you prompt for access to gps.
The usecase I've seen more recently is that paywall news sites now store the number of page reads in multiple places. You end up having to clear more than just your cookies to 'reset' the read count.
localForage only uses a single datastore. E.g. LocalStorage or SessionStorage or IndexedDB. Thus, stored data is lost forever if the browser unceremoniously purges that datastore due to storage pressure.
IronDB redundantly stores data in multiple datastores -- cookies, IndexedDB, LocalStorage, and SessionStorage -- and self-heals if the data in any thereof is deleted or corrupted. For example, stored data is safeguarded in the event the user clears their cookies or IndexedDB is purged due to storage pressure.
tl;dr: IronDB is resilient in the face of data deletion. localForage isn't.
It's kind of ridiculous that a library needs to exist to guard against this browser behavior, although I empathize with the browser vendors' argument, which is that browser storage should essentially be used for caching and that browsers should somewhat aggressively defend against disk space consumption that most users don't even know about.
It would be nice to have an in-between where if a user authorized an app to permanently consume up to X amount of disk space the browser would allow it. But pulling off that user interaction well (i.e., getting the user to make an informed decision rather than blindly clicking yes or no) seems quite hard.
I'm not sure any of these properties are ensured the way anyone would want. Why do this? What are the failure modes this protects against? If only one of the backing stores is active when a result is written, and then the others later become active with no data, is blank data returned for the prior result? It seems like recording timestamps would fix this problem nicely on a single system and make this thing overall quite reliable.