> I'm running an elastic search for a personal project on google-cloud and I use as a search index for my application.
He very clearly says it’s a personal project. Trying to learn new topics outside of your direct responsibilities, while employed, is very common in the software industry. Not everyone that works at a company is involved in databases at that company.
Free? It would have required more effort, but they could have encrypted all the data, and then sent the key to a well-known white-hat security researcher, or someone who could be trusted to administrate important cases (they'd of course be free to ignore it). The encryption could be done on the compromised server with a forEach, so it'd be a single request.
I think some people in this thread want to be a bit too "absolutist" about it. Everyone's servers were exposed to heartbleed, spectre, meltdown, etc so the absolutists would apparently want the whole internet deleted.
Edit: It would be helpful if down-voter could explain (I might learn something).
> and then sent the key to a well-known white-hat security researcher
Would you like it if someone involved you in adjudicating potentially illegal (under CFAA & others) without your consent?
This is clearly not a white hat hacker looking to teach people lessons about security. If it were, they could have furnished a list to the major cloud providers of broken instances and given them time to notify and remediate.
Sure even the best hardened Service can have vulnerabilities, but that's how life is, better have a door with a key than one without, even when someone is capable to open your door with a Lock-pick.
> don't expose services that are not meant to face the Inet directly
I did not (in the slightest) suggest that people should do this. I was commenting on the "free-ness" of the lesson (read the comment I was replying to). It could have been more "free" with a little more effort. Straight-up deletion wasn't the only option.
>It could have been more "free" with a little more effort.
Even White-Hats work not for free (for companys).
Don't build Cars if you don't know how a break work, don't build IT-Services if you have no the slightest idea how to secure them.
Except of course they probably already knew that, they just accepted to risk to their toy database as a trade-off for the convenience of being able to directly access it over the internet.
Yeah, imagine if someone started stalking you on the Internet and making up random assumptions about your employer based on the questions you asked on a question-and-answer site. Oh and wait, they are doing this slanderous gossip under a pseudonym themselves, so you can't even call them out personally!
Stuff like this is what drives away underrepresented groups from engaging on the internet. Maybe everyone who upvoted and participated in the uninformed speculation from 'user5994461 should reconsider.
I expect if the author learns about this exchange he would never ask a question that has even a remote possibility of inviting speculation about levels of his knowledge or practices at a company where he happens to work at the time (in other words, pretty much any question at all) using his name.
That kind of public ridicule and possible resulting flak from management is why more and more developers participate in knowledge exchange by asking questions from under a throwaway pseudonym (much like user5994461), and only answer or edit other questions from accounts that connect to their identity in any way.
This attack uses public write access, which is how they can delete stuff. I think we can agree that this is not good, and I also think we can agree that a database shouldn't be exposed as-is without an application layer or API on top
Ultimately, companies like MongoDB and Elasticsearch are culpable for selling database technology that is insecure by default, presumably because that's the easiest way to boost their metrics for the VC overlords.
online databases that can be written and deleted by anyone on the internet are no good at all. The data can't be trusted. Of course no welfare for you! All I do is to replace all the names with my name and I can take all the welfare in the whole country! Or for example, doing a search for names and replacing all female names with male names ... how can you trust a database like that?
Making decisions based on a writable database (to the world, and not just from data sources like census, etc) is utterly useless.
Facebook, Twitter, or even Mediawiki, don't permit any random IP address full database access. (Or had better not.)
Rather, for the first two, large numbers of agents may request access limited to a specific account, with limited capabilities granted.
Even Mediawiki, with an extraordinarily open access model (painfully so in most cases) has checks on extensive abuse, and gradations of permissions.
Suggesting that any of these are comparable to full DBA access as the Meow attack (with considerable merits0 targets suggests an exceeding poor grasp of distinctions or misreading of GP's comment.
> I’d much rather have my data deleted until it’s secured than have it stolen by someone else
There are multiple logical fallacies in this sentence. First is the use of the world 'until' which is ambiguous here; it suggests that your data can be 'undeleted' after the DB has been secured or you would rather not have any data stored anywhere that is not secured. Either option to me seems like an incorrect read of your comment but I'm not sure. And "than have it stolen by someone else" seems to imply that you know that this data was never copied and cannot be stolen still. I think that seems incorrect, unless there is something I missed that assures everyone that the data could not have been stolen during these hacks.
Lastly, your personally preferred outcome for your personal data is not a measure for all of society, but you grant it that "public service" label as if your preference matters above everyone else's. You don't know what other people think about their data. You don't know what the data even is. What if some of it was just a hobby project for someone, with no financial implications of unsecured data or of data loss, but with emotional attachment to their data? Do they not matter to you?
A blind deletion of unknown data belonging to unknown people is not a public service.
I assume the comment was partially in jest. But this would actually work well if it was consistent and fast. If databases get wiped before you have time to put anything important in them then noone gets hurt.
Yeah, it's bad for the industry right now, but this is just a transition period! Once we get through the pain of losing a few databases, the new steady state where nobody's data is stored in world-writable databases will be better for everyone, and that will be worth the cost.
Consider if this happened five years ago, it would have had a smaller cost than happening today. And it was probably going to happen at some point, so better that it happened five years ago than today. By the same argument, better that it happened now than at any point in the future.
I'm not sure how serious I am about this argument but...at least a little bit? I guess the alternative argument is that any day now software vendors would have all moved to secure-by-default platforms where deploying a world-writable Redis in production would have been so difficult that it rarely happened.
Docker uses it's own iptables rules which have priority over the system ones. Therefore, even if you have an iptables-based firewall blocking all ports, a docker service will still be reachable, unless configured not to be in docker itself.
I do not understand what you mean by "priority over the system ones"
A docker container can have internal ports exposed explicitly, or use host networking. In any case these are ports exposed by the docker-proxy executable - an executable like any other on the system.
Then come the iptables rules of the system (which open or not data flow to the ports exposed by docker-proxy).
Taking precedence was not the ideal word - it uses the same ip tables, but it inserts its own table as the first one. Therefore it 'ignores' system rules, which might come at a surprise.
> There are multiple logical fallacies in this sentence.
No, there aren't any fallacies in that sentence and can't be.
The statement expresses a personal preference; to be fallacious there must be some logic that can be unsound. That is, it must start from some premises and then derive a conclusion. To find a fallacy, you have to show that at some point the conclusion does not follow from the premises.
Since it's a simple assertion, it is implicitly sound. (The graph of premises to conclusions is just a single node.) And since the author knows with certainty what his preferences are, we can take it as true. It's fruitless to argue with people about what their preferences are.
> First is the use of the world 'until' which is ambiguous here
Virtually all "fallacies" you see online are just people typing their thoughts in a hurry. Take advantage of interaction and ask them to clarify.
> Lastly, your personally preferred outcome for your personal data is not a measure for all of society, but you grant it that "public service" label as if your preference matters above everyone else's.
And as a member of the public, if it serves my interest, it is a public service to some extent.
Now, fair enough, you're trying to attack it as not being some broader notion of a public service. You have that broader notion in mind, but you don't explain what it is.
Instead you apply your internal definition through "as if..." which puts you in the territory of inventing a claim they simply never made. That's not even fallacious, it's pure fiction.
> A blind deletion of unknown data belonging to unknown people is not a public service.
You do make some claims, mostly coached as questions, that might lead to this conclusion. You never plainly state your premises, nor do you connect them to this conclusion.
So after all that, your conclusion is a non sequitur!
It can be, imagine I saw a fire alarm and pressed the button because I thought a fire started, it didn't and I learnt that the fire alarm only looked like it was working, knowing that this would not be fixed for 24 hrs I choose to smash the alarm so it's visibly broken. Is that vandalism?
I completely missed the poor consistency from the "I would rather" comment above. I would also prefer my data deleted and not stolen, but had to read your comment to realize there is no evidence to suggest that. It is funny how much I assume being at least partially aware of my ignorance of the topic.
It is, better than to steal the data, you know what a really bad service is? Let your Database wide open, and expose your customers data (maybe?) for everyone to read.
First thing I always do on any new VPS is to sort out SSH (disable root login, disable password login), set up fail2ban, install and configure ufw... and if I need to set up something like redis or similar, make sure it only listens to internal connections and also that it is decently auth'd. For deployment and other things I make users that can only write to certain directories; no sudo. It's nothing new or special but it gets lost in distributed systems.
It's a lot more work when doing it in the cloud and spinning up these things from docker containers in K8S...but you're entirely to blame if you don't know what you're deploying and don't understand any of the potential threats.
Do you know of any good resources for learning this stuff? I'm interested in being able to do this sort of thing on a small scale, but there seems to be an awful lot that I don't know I don't know.
Recommend to setup two subnets in your project. One public and one private. This prevents this sort of issues, instances in the private subnet simply don't get a public IP, they can't be reached over the internet.
For reference, the standard practice in a company is to have a (third) separate subnet for databases, with zero internet access (no NAT gateway). Connection must be explicitly opened from/to database clients. It's a nightmare to manage on premise but it works really well in the cloud with firewalls allowing traffic based on instance tags.
> Recommend to setup two subnets in your project. One public and one private.
This is very good advice. We recently had a uni project where we had to use a MongoDB database. Somebody just apt-get installed a mongodb onto a DO droplet called it a day. Two days later the only remaining records prompted us to transfer x amount of BTC to a adress that was store in our DB. It just contained dummy data, but it is worrying that something like this apparently happens to lots of companies as well.
The only thing I find weird is that ElasticSearch itself does not offer a way to handle authentication, it was just enabled by a plugin that was paid (it seems like its free now).
> The only thing I find weird is that ElasticSearch itself does not offer a way to handle authentication, it was just enabled by a plugin that was paid (it seems like its free now).
"Wierd" is an interesting euphemism for "irresponsible." Defaults are very important. Insecure by default is insecure for 90+% of deployments.
I have _some_ sympathy for ElasticSearch and Redis, having designed/built their software under the assumption it isn't ever intended to be publicly accessible over the internet.
I have a bunch of fairly important personal documents in a filing cabinet with no lock. And I'm perfectly fine with that. I wouldn't keep it in my front yard, because that's obviously stupid, but keeping it inside behind my locked door and upstairs in my office? A perfectly acceptable risk (for me and my files).
I do agree that ElasticSearch do a quite poor/irresponsible job of pointing out their cabinet has no lock. I think Redis do a better job, but are seriously let down by all the internet tutorials that just say "sudo yum install redis" as a minor intermediate step in getting example-todo-list-de-jour working - without even a footnote explaining that anybody who actually visited the redis site now has instructions on how to p0wn your box. ( http://antirez.com/news/96 ) I do think the "Securing Redis" section of this page - https://redis.io/topics/quickstart - deserves to be much closer to the top - I'd have put it before the how to download/install/start instructions myself (though I _think_ recent versions of redis only bind to localhost in the default config, maybe?)
Personally, I reckon that applies at least as much (if not more) to the devs installing random software packages onto internet connected and un-firewalled servers - as it does to database developers who document clearly that their software is not intended and is actively unsafe to install on directly internet connected servers...
> Somebody just apt-get installed a mongodb onto a DO droplet called it a day. Two days later the only remaining records prompted us to transfer x amount of BTC to a adress that was store in our DB.
If the default install does this, then I'd blame the package /distro maintainers. It should definitely at least only listen on localhost by default, with stern warnings what is going to happen if you change that without setting up proper security.
MongoDB only binds to localhost for at least the last four versions (4+ years). Someone would have had to install a really old version or intentionally configure it to listen to public IP.
Issue is with AWS this setup instantaneously bumps the bill up from a few dollars a month to a few tens of dollars a month. Deal-breaker for personal projects. But, you can still secure the database with whitelisted IP addresses, which is what I do.
You can cheese infinite loop detection by setting a max number of edge traversals (that way you don't just loop longer when someone speeds up the happy path).
For real code, you wouldn't generate a web page with 5 million entries in it, so you can be pretty sure that the data is bad even if it's not cyclical (but it probably is)
These are not emacs commands. They aren't even unix shell commands. They are TTY commands, some of them dating back to the dot matrix teletype terminals.
My favorite is ^U, which 90% of the time lets you start over on a password prompt when you are sure you just fat fingered but not sure how badly.
Thank you, that one is really useful! The one I had always remembered was ^H for whenever no other way to delete characters works. I need it in some SQL REPLs which aren't configurable.
I think you might be conflating these (or maybe I should say the gp is). Nevertheless.
Do you have a reference to the history of key combos like ctrl f, b, n, p and a and e? Those are typically referred to as emacs style navigation and I am genuinely unaware of history of those as common tty control codes outside of emacs for cursor movement. They weren’t dec vt control codes. Ctrl-U was though and even has ASCII assignment as “NAK”. Ctrl-H and C are similar.. but people don’t typically refer to those as “emacs” keys.
> Do you have a reference to the history of key combos like ctrl f, b, n, p and a and e? Those are typically referred to as emacs style navigation and I am genuinely unaware of history of those as common tty control codes outside of emacs for cursor movement.
I've always heard of it as an ASCII control character and gets its history from Unix interpretations of really old IBM keyboards which got its history from typewriters. I've literally never used emacs for anything other than ^X to exit; I'd rather use cat than emacs. I use vim. But ^H has worked for me as intended on a serial console and on telnet.
Some diving through wikipedia:
[0] says: Pressing the backspace key on a computer terminal would generate the ASCII code 08, BS or Backspace, a control code which would delete the preceding character. That control code could also be accessed by pressing Control-H, as H is the eighth letter of the Latin alphabet.
[1] says: In some typewriters, a typist would, for example, type a lowercase letter A with acute accent (á) by typing a lowercase letter A, backspace, and then the acute accent key. This technique (also known as overstrike) is the basis for such spacing modifiers in computer character sets such as the ASCII caret (^, for the circumflex accent).
[2] says: Unix (command line and programs using readline): Ctrl+H = Delete previous character
[3] supports [0] and says: Caret notation is a notation for control characters in ASCII. The notation assigns ^A to control-code 1, sequentially through the alphabet to ^Z assigned to control-code 26 (0x1A). Often a control character can be typed on a keyboard by holding down the Ctrl and typing the character shown after the caret.
However, it's worth noting that it also says The meaning or interpretation of, or response to the individual control-codes is not prescribed by the caret notation.
But, despite that, ASCII describes control character 8 as backspace [4] [5].
According to this Wikipedia article, work on ASCII began in 1960 and its first release in 1963 [6].
Emacs' first release was in 1985 [7].
I suggest that perhaps in your bubble control sequences are referred to as emacs style navigation even if it's not necessarily the most historically accurate. I'm glad you're working with *nix enough to be familiar with emacs and there's always new old things to learn. There's a lot of history to learn and understand.
I read your post and thought, "I've misremembered the story."
But the Wikipedia page for the Teletype-33 claims that it 1) had control characters, and 2) was inspiration for some of the ASCII character set, which was defined later in the same year:
1) If it‘s a tree, it ain‘t got no loops
2) The stack isn‘t to deal with loops, the „visited“ flag at each edge is there for that. The stack (for DFS, BFS would be a queue) is there to keep track of which nodes have been visited such that you can construct a path from the starting node to the one you‘re looking for.
Obviously there are variants to this, depending on what you‘re actually trying to achieve with it. My point is that a stack would be a very inefficient way to deal with loops.
1) you're right, I edited my message to reflect that I meant a graph traversal algorithm.
2) a visited flag on an edge? That won't support simultaneous traversals. Keeping a stack is a lot more efficient than permitting only one traversal at a time.
I‘m not sure why you‘re bringing concurrency to the table.
My point still is that looking something up in a stack (did I visit this node?) costs O(n) time, so the BFS will degrade from O(m+n) to O(m*n+n).
To come back to the concurrency, if you can index your edges in some way, you can also store the visited flag in a separate datastracture to support concurrent access (one „flag store“ for each access).
If meant as a public service, it would have been much less destructive to use the change passwords API [0] to set random passwords for all of the users.
Given that "unsecured" means "data are accessible and modifiable by anyone", creating tremendous externalities for all referenced in the data, , I'm happy with deletion.
FTA:
One of the first publicly known examples of a Meow attack is an Elasticsearch database belonging to a VPN provider that claimed not to keep any logs.
In some cases, I might be tempted to agree with you, but this is blindly being applied by an automated attack. What if some of that deleted data is volunteer-canvassed anonymized survey data of homeless people, and its loss sets back a homeless relief program by months, resulting in several people freezing to death this winter?
But if they had used the password changes API to assign random passwords to all accounts, as suggested, then the data couldn't be modified. Am I missing something?
Parent's point is that any conclusion one could make from the data is worthless because, being public and unsecured, it could have been modified by any Internet user at any time before a password was set.
Both the DB admins and the attackers should both feel guilt. However, if the attackers simply assigned randomly-generated passwords to all of the accounts, then no data would be lost and the DB admins would still have their DBs temporarily become inaccessible while they figured out how to force-reset their passwords. If you're going to go for disruption, I think the suggested lockout gives a much better ratio of good being done to potential damage being done.
If the databases in question (Elastic, MongoDB, others) make it too easy to set up unsecured access, possibly because they default to an unsecured state on installation, then some good may come of this: The reputation hit to the database vendors should encourage them to mend their ways.
If that happens, then the attack can arguably be justified despite the damage — consider all the future database installations which wouldn't otherwise have been secured which are now spared from not only destruction but theft.
I've said it on here before, but the way in which Elasticsearch used to lock away critical security functionality (like TLS support and RBAC) behind a paid subscription whilst making just enough functionality available for free such that users could shoot their foot off is disgusting. This only ever changed after Open Distro for Elasticsearch came onto the scene and forced Elastic's hand.
I entirely agree the vendors are (partially) to blame here.
Well, it's just another attempt at monetizing the product. Nowadays, companies and developers expect everything to be OSS (and I love it) yet it's incredibly expensive to develop SW (and very few people do OSS just because of passion--I tried and failed miserably).
Locking RBAC and TLS behind a paid subscription is a sure way to force companies with security teams to pay for it (or not to use it).
This particular lesson comes relatively cheap--dropping your data is not the worst an attacker could do with it. Hopefully, more people will research what I'd call "security 101"...
It _can_ be. At the same time some of the most used software in the world manages just fine without a company running paid subscription plans and locking free users out of critical security components.
ElasticSearch/MongoDB/Redis et al are trying a new model for how to create OSS with a company behind it funding all/most of the development. That's OK, and I'm super interested to see how it works out long term. But there are many many counter examples of similar sized or way bigger software projects that never needed to do this. Pretty much everything that those three databases depend on to be used in applications is OSS that never had a "paid subscription" locking access up. How useful would any of them be without Linux, or Apache/Nginx, or Ruby/Python/PHP/Perl.
My fear is that half-assed-OSS that does a _great_ job of "capturing developer mindshare" but a lousy job of securing free use of their software - is one day going to be the root cause of some _spectacularly expensive_ data breach, after which pointy haired bosses and less technical C suite suits are going to feel the full power of Oracle's golf-course-and-expensive-restaurant marketing army, and nobody in a company bigger than 10 or 12 people will ever be able to use any database with less that a half million a year license because "due diligence!" and "risk mitigation!" (and "Waygu steak with expensive whiskey" and "dirty free software hippies exposing you to data breaches!!!")
The thing is that no one cares. It's a cultural issue. If Elastic makes it "harder" to get off the starting line by setting secure defaults, they'll get their lunch eaten by a fork called EasyElastic that people perceive as "easier".
This probably isn't an excuse for Elastic anymore, but it's how Elastic was born ("easier than Solr!" -- they didn't invent full-text search, after all) and it's how whatever supplants Elastic will be born. Why is MySQL dominant over Postgres even though "referential integrity" didn't come to the game until v5? Why Docker over jails or OpenVZ? etc. People adopt technology because it's fashionable and it becomes fashionable in part because it's perceived as easy to use. Security and ease-of-use are not quite true opposites, but there's definitely some intrinsic tension.
We have a lot of hapless practitioners in the space and the root problems here won't go away until we get some standards. Better solutions usually lose because crappy stuff focuses more effort on marketing and an "easy" onboarding process, where better stuff focuses on the operational complications of the real world.
I use jsonb heavily. While it is amazing, I definitely wouldn’t rely on it as a general purpose replacement for NoSQL/schemaless data storage.
An example of an issue I am dealing with currently: while you can create a gin index to speed up containment queries, Postgres doesn’t keep any statistics about jsonb columns. This means the query planner will sometimes do stupid things, like using the index even for very non-selective overlap conditions, which is a lot slower than just doing a sequential scan.
Less of an issue for me but worth considering: the size of the gin index in my use case seems to be about 5x bigger than the size of the unindexed data. I was surprised by the size increase. I only use the containment operator so I could make a smaller/faster index using the jsonb_path_ops operator class. This is on my todo list :)
Like all non-btree indexes in Postgres, the index is unordered. That means sorting by values in the jsonb column will always be slow. This doesn’t matter for selective queries, but exacerbates my already slow non-selective queries that return large result sets.
That said, if your queries are selective, jsonb + gin indexes are surprisingly performant (in the 0.5-10ms range for small result sets). My use case is a mix of structured relational data with jsonb for user-defined values (which of course they want to use for querying/sorting and I was dumb enough to say “sure, why not?”)
In terms of the magnitude of data, there’s roughly 10 million rows. Each team using this service has the query scoped to about 500k-1 million records, and then additional filters (on the jsonb column) will scope that down to anywhere between 60k-0 results.
Thanks for the detailed response. I'm curious, is the NoSQL store a canonical store of its own data? If not, how do you replicate from postgres to the secondary store?
I ask because where I work we sync postgres to a secondary store for search, but the way it's done in a piecemeal, application-specific way gives me the heebie jeebies. It almost certainly will result in that secondary store drifting. Unfortunately we can't use something like zombodb [1] as we're on amazon RDS. It seems like you know your stuff, and seeing non-deterministic consistency irritates the heck out of me!
Thank you for raising this question. Well, for my things I use MongoDB because of very convenient integration with programming languages (Go, C++ via Mongo C++) and zero hassle with schema
But I'll be happy to replace it by something else, my load is extremely small and only single requirement is to have DB as network daemon, not as embedded storage as it will be used by 2 applications (main daemon an API).
If you have ever used Mongo Atlas, you would understand why people use mongo. The administration ease that the platform provides is unmatched by any other DBaaS I am aware of.
Surprisingly, I feel Mongo has been kind of surging back into developers' minds. I thought Mongo was utterly dead, and I also don't know why would one use Mongo instead of JSONB. The last I heard (3-4 years ago?), there were some fundamental problems with Mongo.
That said, I've incidentally heard a lot about Mongo in the last half a year. Might be my bubble. Might be MongoDB actually maturing and getting really good. I hope it's the latter.
Seriously this is the most annoying thing ever, especially if someone on your team things you need to expose the ports to redis in a docker compose. I’ve come back from a weekend where my redis instance was being used for crypto mining.
Anything that is insecure by default in 2020 should be killed off IMO.
My guess would be that they went with "ports" instead of "expose" which makes it public. But even with careful composition of your docker-compose you might want to be careful, Docker can interact with iptables in surprising ways and long iptables rulesets are almost comically difficult to validate sometimes. The result is that if you're using both Docker (and even more if you use Compose, Kubernetes, some other orchestrator) and doing anything else with iptables (including using it as a host firewall) you need to take a lot of caution to make sure you don't mess anything up. These tools usually confine their changes to custom chains to make this stuff easier to sort out but at the same time it can make it more difficult to understand what's happening, particularly with the use of tagging and etc.
In general I would recommend putting some kind of protection between machines running Docker (and especially orchestrators) and the internet. This could be cloud provider mechanisms (security groups, ACLs, etc), a firewall appliance, NAT gateway configuration, etc. depending on the situation. It's not necessary, but it makes the situation easier to audit/validate, and more layers of protection seldom hurt. If nothing else it means that much of the time you'll need to make a mistake in two places instead of one, in order to have an unintended exposure.
Friendly FYI, "EXPOSE" is a no-op that is only meant as visual documentation to end-users.
"The EXPOSE instruction does not actually publish the port. It functions as a type of documentation between the person who builds the image and the person who runs the container, about which ports are intended to be published."
Secure by default is super onerous though. What if I just want to try out something before committing to it, do I really need to jump through a bunch of security hoops?
Of course you don't need to waste time setting up security properly. And it's such a pain locking and unlocking your car and your house every time you go in and out too. Tell me, where do you live again?
I'm 58 and bald, so channeling yoda is easy: a valuable lesson you need to learn, and learn it the cheap way or the expensive way you can.
The cheap way is to invest in a cheap VPS for a month, fire up sshd and a webserver, and then check the logs when the month is up.
The expensive way is to carry on treating security as an afterthought. It'll cost you your pride, your reputation, and possibly your career.
Absolutely. Laziness isn't an excuse for not caring about security. Security should be the top of your mind any time you connect anything to the internet.
The fact it's not, is why we're seeing major attacks/leaks/etc almost every single day now.
Right, well now we know why secure by default isn't a thing. If your product is super onerous to use, people will switch to a competitor that is easier to work with.
A secure by default product is a dead on arrival product.
Then you just pass the `--disable-all-security` flag. (Or whatever similar method the project you're using exposes to allow that.) Secure-by-default doesn't have to be complicated; it's just a way to ensure people don't shoot themselves in the foot without comprehending what they're doing.
Thank you for this reply; it's my favorite post in the entire discussion. It illustrates perfectly why insecure defaults are a terrible way to implement "tutorial mode".
I'm not. I'm just using flask/mongodb/whatever for some internal app. But if it's secure by default I need to jump through a bunch of hoops to secure something that won't even be on the public internet.
Yes, this is terrible, and when I discovered this, I was appalled how it seemed no one was taking this seriously.
When you use ufw with a default DENY policy, you tend to assume that whatever isn't explicitly listed gets DENIED. This is not the case with Docker, and I think it's just a matter of time until someone loses big because of this issue.
Only 50%? I'd put it much closer to 100% tbh. Why people are so eager to funnel all their traffic through some of the shadiest businesses on the Internet is beyond me...
That's just victim blaming. The same logic applies to every crime: "lock your doors if you don't want your TV to be stolen!".
It also works at any level of security: "Lock your doors and hire guards if you don't want clever thieves breaking a window..."
But if you require everyone to take adequate measures to physically secure their houses, you don't even need laws and morality!
And while this may provide the sort of negative reinforcement you are hoping for, actual damages in each case are going to be all over the place. That makes this one of the worst possible punishments. Imagine the penalty for speeding is a random outcome somewhere between a stern reminder and the death penality. There would be nothing fair and little useful about that, even though it does follow the same basic principle of do bad -> be harmed.
You're not thinking of the right victim. You're considering the organizations who compiled the databases in question. If the databases are composed entirely of their own information that they created themselves, then sure they've suffered a loss. In most cases, however, these databases include privacy-sensitive personal information about the customers of the organization. Those customers have been victims of bad security practices ever since MongoDb was first installed, and this "meow" hack has ended their victimization.
> But if you require everyone to take adequate measures to physically secure their houses, you don't even need laws and morality!
If laws and morality were sufficient protection against malicious actors, I might agree with you.
However, in a world where cyber vandals are often beyond any accessible jurisdiction (and may even be supported by their local authorities), laws and morality are clearly not effective at keeping unauthorized users out of private systems. As such, the responsibility for keeping private information secure naturally falls on the people running the systems.
Putting up a network firewall (or at least requiring authentication) would have prevented the damage described in the OP, and is a rudimentary security measure that has been common practice for decades. The people who suffered significant damage from this attack should strongly consider outsourcing system administration to someone who knows what they're doing.
the truth is in the middle you have to take some responsibility, but yes of course some blame goes to those who also make it possible. If people just throw their hands up and say they're victims and have no responsibility to secure their systems then they will always be victims.
My website once started failing because someone had connected to redis and set a password on it. That was the day I found out the redis docker container accepts everything that can connect to it unauthenticated by default.
IDK, if someone kept changing the table names in my DB every week I'd probably throw a password on it, even if I were really lazy. Most of these people probably didn't realize their DBs were unsecured, and that gets the point across quickly (particularly if the new table names are chosen instructively).
That sounds reasonable, but you'd think most people would also be concerned about their databases being publicly accessible in the first place, yet here we are.
I don't think this is the case of people not being concerned, but simply the ignorance on their part about the setup. People just presume that the defaults are safe, and never bother getting into the details.
At the risk of arguing "no true Scotsman," someone who is concerned about security likely wouldn't make assumptions about defaults. Or rather, someone appropriately paranoid about security concerns would not trust defaults without at least reviewing them.
Not everyone feels comfortable and/or knowledgeable enough to review the settings. What they really should do is to hire an admin/devops consultant, but for so many projects it's just not a realistic expectation. That's why I put all the blame on developers who chose to publish the software that's unsafe by the default. It's done on purpose for marketing/sales reasons, to make onboarding faster and easier and get as many users as possible with "simple to start using" service, at the cost of putting users in danger later on.
Strawman. And a weak one at that. Fuck off with conflating the minimum amount of competency needed to secure the data you needlessly collect with that kind of naivete.
This reminds me of "crackit"[0] from a few years ago with Redis. A lot of folks kept their Redis server bound to 0.0.0.0 with no firewall or published port 6379 by "accident" with Docker and by default Redis uses no password. It was a lot worse than meow because with some Redis configuration magic anyone could inject their own SSH keys onto the server.
This article says Redis is affected but I would be curious to see which version of Redis was being used because they changed their default configuration after crackit was wide spread.
Yeah one thing though with Docker is that in some cases it injects its rules into iptables before the firewall application's.
I was using arno-iptables-firewall and this suffered from that, docker containers would be world accessible. In general I only bind them to localhost anyway, but I figured this out when testing. It doesn't seem to happen with UFW.
But I can imagine some people know how to set up a firewall but then just assume it works and don't check. This is the kind I do feel sorry for, at least they tried to protect it.
This happened to me. Was just getting started with docker and got everything working and a few months later someone had set a password on my redis database. Who knows what else happened before that.
This is what happens when you lay off all of your sysadmins because "the cloud", move that role to devops and then downsize that to a subduty of a developer.
I’ve seen just as many sysadmins do this as developers. It’s not a question of job title as a psychological pitfall (people who are looking for things to succeed don’t ask when they should fail) and companies not specifically retaining people with security experience because they cost more.
It can be though. Downsizing and getting rid of specialists certainly hurts companies. There are only so many hours in the day and that desperate guy working 14-16 hours a day because of covid downsizing is eventually going to eff up no matter how talented she is.
I’m not sure exactly what you’re disagreeing with. My point was just that it’s not useful to direct criticism at a job title when there are so many examples of failures by people with any given title. I’ve seen people who are ostensibly pen-testers or auditors blithely telling others to click through important warnings or have a root-on-all-machines password to make their work easier.
Downsizing and other false economies are definitely a contributing factor. Security and reliability are easy to dismiss as expensive overhead until they suddenly aren’t.
i'm sure it's not that. most likely they are databases that development setup for testing or developing a quick server and just forgot to hand it over to sysops or dbas. happened all the time where i use to work.
Somehow I feel good about this. The article claims nothing good can come of deleting exposed databases, but I strongly disagree - I'd by far rather my data be deleted than stolen and shared. If the owner doesn't have proper backups AND can't secure a database, they have no business hosting such data, period. IMHO.
I think this is a little simplistic. Depending on what data is being deleted, it may have real life economic consequences for individual people. What if one of the databases has a record of credits you've purchased at your local spin studio? Hopefully they have a back up, but if they don't, you and/or the owners stand to make significant losses. Are there databases that could be lost without consequence except to their owner? Sure. But that is far from all of them.
I also just think it is a little uncharitable to wish harm on people simply because whoever did their IT was inexpert at their job? Like, how does the local mom and pop correctly evaluate a person's IT chops? The nephew says they can set up their website for cheap, and they want to be nice, so they give him the job. Turns out he's a newb and later their database gets deleted and you are on here saying that's a good thing? Hrm. I don't agree.
It can definitely have real world consequences, but couldn't the same be said for somebody being a whistleblower for a company that doesn't following building codes? The company could take a huge financial hit and people might lose their jobs because of their practices being exposed.
Sometimes the best path forward does harm, sure. It's just hard for me to agree that deleting these databases is the harm-minimizing path. One example of a less harmful path that comes to mind immediately is installing a random password on the unsecured database and emailing the domain owner the password. That would cause downtime but it would limit the irreversible damage. You could even say that you will delete the database if it is found again with an unsecured password, if you wanted to add some stick to your carrot. It does not seem like this attack has harm-minimization in mind.
What you propose is illegal in most 1st/2nd world countries. In mine, the company could thank you and then put you straight to jail for 30 years.
Unfortunately very few small businesses run sade reporting programs and often react with attack.
The problem is with the e-mailing part. A mom&pop is unlikely to track you down if you lock out their DB, but they'll likely report you to police if you contact them about it.
Unfortunately it's rarely that simple. If you look at the currently exposed MongoDB instances you'll see that most of them are in the cloud without any obvious attribution. You could email the cloud providers and see if they will reach out to the end-user but chances are they already know about it. Here's an article I wrote on that subject, although it was related to industrial control systems:
It's easy to say "you could have just emailed them" when you are not the one doing this for years without things getting better. Often admins flat out ignore you. Even if not they usually do nothing. And if they do something it takes ages.
In the article one provider was notified that their database was without a password an publicly accessible.
They secured it, and somehow managed to make it publicly accessible again without password, this time it got hit by this attack.
Honestly this is like if a company decides to keep their paper records with my information on a public side walk, and somebody saw that and decided to bring them to the landfill.
Is it legal or fair? In a perfect world no, but at this point the company is not blameless.
I certainly think the most legal approach is to do nothing except notify, or maybe nothing at all. But if you must modify the database, locking it reversibly is more defensible morally.
Your comparison isn't fair - blowing the whistle is supposed to be a last resort. Internal disclosure and attempting to fix the issue collaboratively is always the first step.
This attack is indiscriminate and is without warning, so it eliminates the possibility for database owners to fix the problem in good faith.
Finally, after years and years and years of being told over and over and the bottom line never improving.
I got doxxed with the Equifax breach. How many other companies in the world will take someone's word that they are me based on that data, and what potential is there for my life amongst millions of others to go completely sideways because of companies who won't addmit the systems are broken?
I say the house is already burning, but maybe throwing some fireworks into the blaze will convince the right parties to finally put the damn fire out.
Beyond sick and tired of breach after breach after breach after "oh, there were millions of voter records showing publicly" "no, <insert product name> defaults to no security"
Just because it's understandable for Mom & Pop to not care about the privacy of the people whose data they've collected doesn't make it a good idea. They should care. If the buck doesn't stop with business owners, with whom does it stop?
IMO we specialize too much and miss important things that we mentally outsource to other people. I'm equally disturbed how often I meet people in tech who are unable to do basic home repairs or cook for themselves. Something, like say, a few weeks of quarantine, and people's anxiety goes through the roof because they've relied on other people for basic life skills.
I don't find your example very convincing. Any database storing personal data needs to be properly secured, and if that gym also has ID credit card or other more sensitive data, that data might better be destroyed than stolen.
If it's a publicly accessible wiki with no sensitive data whatsoever, and that's meant to be publicly accessible, then there's a reasonable excuse for the poor security and it's not helping anyone to destroy it.
The point is that, if my credit card info is staying in a web-exposed, insecure DB, it is safer for me that it be destroyed than left alone.
I have no idea of that is the intention of the attackers, or if they are maybe even stealing the info before deleting it. But assuming they were good Samaritans and just deleting it, that is the best outcome for me as a user, better than if it stayed up for another day.
What happens when mom & pop are storing your name and credit card # in plain text and then your identity gets stolen and credit ruined? Should we still be "charitable" to them and their d-bag nephew?
Your credit card number being stolen is a problem for your bank, not a problem for you. You can't steal someone's identity with a credit card number.
The concern in this case is when there is some social problem with being in Mom & Pop Inc's customer database. There are probably some people that buy some things that they don't want other people to know about. When the database gets hacked and you are linked to being their customer, that is the unfortunate and potentially damaging information leak. A credit card just gets reissued and the bank reverses the transaction. No big deal.
It's a problem for the vendors, not the banks. They get hit with chargebacks for fraud that's no fault of their own, hurting the whole ecosystem of vendors and their customers.
The problem could be easily solved by Visa/MC/Discover/Amex implementing chip and pin, or at least 2FA SMS authorization. Bestbuy.com has it working somehow.
Like, how does the local mom and pop correctly evaluate a person's IT chops?
Usually by price and unfortunately both mom and pop like a bargain - I've seen this play out more times than I would like.
Also how do you evaluate, say, a landscaper's chops? Or any other kind of contractor's for that matter? By doing research beforehand, checking what kind of reputation that person has etc.
Low-effort or lack of research gives you bad services, for which you pay in losses like these.
In construction and landscaping work those companies are usually licensed, bonded and insured. If they fuck up the work there's obvious financial recourse. Also, the measure of them fucking up is generally a lot clearer for physical labor and for mom and pop businesses, getting construction work inspected by a 3rd party is usually more straightforward and cheaper.
In software, financial recourse generally means you have to jump straight to lawsuits. There's no licensing for who's qualified to build a website, developers don't have to escrow funds or carry malpractice insurance in case they make a mistake, a development business should have insurance in place but there's not always easy or affordable ways to assign fault in most IT situations if you want to pursue them. Software and IT forensics are prohibitively costly and usually mean a lot of money has to be on the line which rule out mom and pop businesses entirely. IT and software mistakes also usually take longer to rear their heads, and people in IT and software also aren't known for sticking around for decades. How do you sue an LLC that dissolved 5 years ago?
If someone does a shitty job in home improvement stuff it's usually not visible for years down the line. And good luck with your recourse by then. I've never heard of a homeowner getting recourse unless it's insanely obviously bad right away. The vast majority end up just living with the defects or hiring someone else to do the job again.
I don't how it is in the US, but in my country an audit that would reveal such an obvious lack of security costs no more than the equivalent of a single minimum salary - usually much less. On top of that several companies that offer such services are widely known because their media presence is mostly articles about vulnerabilities in routers, phones, operating systems etc.
I hail from a post-communist country so I assumed the culture in the US is more developed in this regard.
I'm not sure what your country's going rates are but for the US, a small business might budget a few thousand dollars total for their website. A minimal security audit that would catch missing or stupid security would basically double their expenses. Unless they truly needed some custom feature they'd take one look at a security auditor's quotes and then go sign up for Wix or Squarespace immediately. It's not that the services don't exist, its just that they're expensive and typical non-technology business websites don't really need that much put into them.
> Are there databases that could be lost without consequence except to their owner
I would certainly hope the owner of the insecure database would face massive consequences. IMO there's not _nearly_ enough of that. This sort of breach should be financially ruinous for _any_ company.
What if the database only contains my blog posts? Or my own personal health information I'm tracking? Or all the Hearthstone cards? Or any other of the infinite data sets that are totally insignificant to anyone beyond the owner of the database?
Not everything is about you, and also you totally misunderstood the post you quoted.
Then you'll learn real quick to secure your database, make backups, or not post personal stuff online.
And if you've configured your database that poorly, am I supposed to assume you've properly configured your server against becoming part of a DDoS attack? Or a bot net? The base level of negligence you're defending enables a multitude of attacks. Losing your DB would be an immediate sign you don't know what you're doing, and thus shouldn't be doing it.
Fine, what about every poor person who gets their data stolen from an unsecured database, and gets their identity stolen?
Yours is also a rich person statement.
Unsecured databases are a huge loss for everyone.
Anything that forces a shift in this naive behavior of vendors, implementors and executives is not just fine by me, I welcome it. If my life suffers because of data of mine that's lost from companies' databases, I now know who to cease doing business with.
But what about the companies making millions in profit each year that are carelessly exposing sensitive data because they don't want to spend a little more on quality IT work?
No one wants to see their local pizza shop lose their pizza-credit database, but if that's the price to pay for data security then so be it.
> What if one of the databases has a record of credits you've purchased at your local spin studio?
Usually when you buy something, you get an email receipt. So you print out your email receipt and go to the mom and pop store.
Given they are a local mom and pop store, you likely have a long term relationship with them and they may even remember you buying credits. So it will be a hassle, but likely ok.
It is the huge corporate stores that don’t have long term relationships that would be hurt by this thing the most.
Depending on what data is being deleted, not deleting it could also have significantly worse consequences when it later falls into the wrong hands.
In the case of the spin studio, you can prove what you paid, the owner just lost his evidence of what he no longer owes you, and will hopefully in the future stop exposing your personal data on the Internet.
> Like, how does the local mom and pop correctly evaluate a person's IT chops?
Not my problem.
> The nephew says they can set up their website for cheap, and they want to be nice, so they give him the job. Turns out he's a newb and later their database gets deleted and you are on here saying that's a good thing? Hrm. I don't agree.
Mom and pop prefer nepotism over skill, credentials and reputation, without even a second opinion. There is a reason that this is frowned upon (and has been for at least some 2000 years before mom and pop were born), regardless of the domain.
On the off chance that their database doesn't contain any personally identifying information on their customers, this is an idiot tax. In any other case, their loss is completely justified when compared to the potential losses, abuse and manipulation of their customers that come with exposing their PII to the public.
No other bad actors can get it, but we don't know if it's already been found, and now that it's gone we have no idea what data is out in the wild. And as you note, we can't trust the companies to accurately report it themselves.
I think you make an important point though - deleted or not, there is no real way to know what's been exposed, and no guarantee that they'll ever admit it; so torch all the data expeditiously, and we'll just have to comb through 'successful' leaks just as always.
Another side is that with their database blanked, that will force more companies to explain their downtime or complete loss of data, rather than quietly secure it again and pretend nothing happened
Maybe this actor A downloaded the data, then deleted the database, preventing others from accessing and selling the same data? Only A can sell this data now?
maybe the authors of meow should "improve" it with a feature that reports every instance to HIBP before deleting it. that is if their intention with this malware was a benevolent one :) but I guess feature iteration in malware that is "supposed to be good" would be tricky
Oh, they'll have to say something when they suddenly stop doing business until a backup can get pulled, and the new db instances actually secured before putting them up again.
Even a bland 'we lost parts of our data and we will have to start recovery processes. please stand by' is a signal.
Ideally they'd report it so that password managers could warn everyone, but with just the database URI there isn't necessarily any obvious way to know what domain or business its associated with.
For me, it's not exactly "good." But I am more upset with the database owners than I am with the kitties. Don't leave the barn door open, or this (or worse) will happen to you, and happen again. If they were instead exfiltrating and selling the data, the equation would change. I'm not saying the cats are doing good, but I do say that the "responsible adults" did the greatest harm by not cat-proofing their databases that contain PII.
I'm ambivalent about this action in most cases, but in some specific cases there can be a clear reason to have an exposed database: namely when the database is a guest-accessible, read-only repository of public data, i.e. the self-hosted equivalent of publishing a Google BigQuery dataset.
As someone who runs such a "public-access data library" myself, I would be slightly annoyed if someone came along and burned it down, just because it has an unpatched vulnerability.
...but if it got deleted because I left default admin creds on it, though, that'd be my own fault.
Read-only in practice, not inherently read-only in the way that e.g. CD-ROM is. Such systems still need to have their otherwise-static dataset updated "online" by an ETL pipeline agent-user. Which often means, in the DBMSes with less fine-grained security models, that such users need to have full DML (and even DDL) capabilities, rather than only insert capability.
Who says it's consumer data? It can be a personal project, a blog, etc.
Just because it's not secure doesn't mean you should delete the data, because where does such reasoning end?
Reminds me of the super meat boy web version with database creds in client. Dev knew, but just did a quick implementation. Hacker wanted to prove his point and ruined it for everybody. Making a secure version was not worth the effort, so now because of this prick nobody could enjoy it.
Yeah. I don't care if some big business loses their Elasticsearch data and their site stops working until they get it secured and re-hydrated with data from their relational database. Good, they learned a lesson.
But I would feel bad if someone's small business had to shut down or lose a bunch of money because they lost all their customer data. I'd feel bad if someone lost all the data they'd been using for a personal project. If they didn't have backups and proper security, shame on them, but ideally they would be contacted and given advice. Ideally, their data would only be deleted if the effect would be minimal.
On the other hand, if this is something that happens consistently -- all unsecured databases get deleted immediately -- maybe the data would be stolen less and everyone would have to learn their lesson early...
> But I would feel bad if someone's small business had to shut down or lose a bunch of money because they lost all their customer data.
Don't. When businesses of any size cut corners and provide services they aren't qualified to provide, it gives them an advantage compared to businesses that try to do it properly. They make more money or charge less and can often out compete competent owners. They'll also be the first ones to brag about how brilliant they are at business (in my experience).
Small businesses are no exception. Delete away IMO.
Said small businesses might have no idea their data wasn’t secure. That would be on whoever developed their technology, not necessarily the business. Not all small businesses with customer or sales data is a technology company.
Good point. You're right. I was really only thinking about tech companies that are doing their own deployments, but I guess there's plenty of room for collateral damage where people don't deserve it.
Seems like a lost opportunity to have a more nuanced position here. And when you can't think of a single case that could make you feel bad for someone, it's usually a sign that you don't have one.
I’d feel bad if someone’s hobby project was deleted. Small businesses losing customer data is only slightly more sympathetic than people getting sick because they didn’t think the health code applied to them.
If you collect it, you need to be responsible for keeping it safe. Anything affected by this is already exposed and has to be assumed to have been breached.
What if it was a small business's inventory data rather than customer data?
Seems to me, there are a lot of things businesses could store in a database which don't necessarily need to be private, or which at worst won't harm anyone other than the database creator if exposed.
That’s why I was specific about customer data: it’s basically a question of who’s harmed - if the cost is borne by the person cutting corners it’s more of a self-correcting problem.
It's not a matter of "cutting corners." Think of all of the small businesses that recently moved online due to store closures. These businesses simply do not have the budget required to create something comparable to, say, Best Buy's e-commerce. Sure, Shopify might come close, but how do you think Mom and Pop will find and create an e-commerce solution?
That’s the very definition of cutting corners. If they aren’t confident of their ability to operate safely they need to either hire a professional or go without - just as not wanting to pay a plumber doesn’t exempt you from meeting the health code or saving on accountants will be a get out of jail free card when you get audited.
If it sounds like I’m unsympathetic, yes, that’s true. Playing around with building your own database is a good learning experience but that changes once you expose other people to your mistakes. I’ve also dealt with a few small businesses and the people trying to run a business like this are always trying to save a buck - they’re the same ones who stiff contractors, avoid paying overtime, do their own taxes creatively, etc. If you have a successful business, you’ll drop a few bucks on Shopify, Wix, etc. to focus on the business rather than a distraction.
Think how easy it is to take advantage of grandma for "tech support," whether it be from India or at a seedy computer repair shop. It wouldn't surprise me to hear that the majority of small business owners have never heard of Shopify or Wix, as they are more likely to turn to someone they trust, whether that be a "tech-literate" cousin or a local service. Keep in mind that many of these businesses didn't even have websites a few months ago, let alone e-commerce solutions. Not everyone lives in SV or NYC and is perennially exposed to their ads.
I agree that these businesses shouldn't be doing this by themselves, but the tech industry shares some culpability. It should be ingrained in people's heads to think of security first. Most people outside the SV bubble are still using [SO or pet's name]123 as their password. I go to a university in NYC. I've tried to convince several (college) friends to use password managers to no avail. This isn't just a problem endemic in old people. Someone needs to "mainstream" good security. A good start would come from, say, Apple, by including security keys with new iPhones, as much of a pipe dream as that might be.
> Yeah, I don't care if a big chain restaurant is closed down for having too poor hygiene. But I would feel bad if someone's small restaurant had to shut down because the cook doesn't bother to wash his hands at work.
If you are holding other people's data for them, you have a responsibility to do your best to keep the data safe. If you don't know how to do that and don't have time to learn, you can hire someone who is more knowledgeable.
And what about the responsibility to not destroy someone's property?
Do you have the same opinion about shoplifters walking away with merchandise? Would your argument be that there should be armed guards and searches in every retail store? Isn't it reasonable that a thief be criticized and penalized for their actions even if the theft was "easy" to commit and is it OK to blame the victim for not being prepared?
Agree with nkrisc -- this is like, if I contracted with a storage company, and then my stuff was vandalized because the company's "secure storage location" was an unlocked box out on the sidewalk. Obviously the vandal is directly to blame, but it's also absolutely negligence from the company.
I'm sympathetic for the people who were only storing their own data, but not for companies that failed to safeguard their customers' data. If I borrow stuff from friends, I take better care of it than if it were mine. I hold companies to the same standard.
I'm not trying to disregard the negligence concern but I am trying to ensure that the perpetrator is also called out. Several of the top rated comments on this article are praising the perpetrator.
You will notice that in my post I'm not defending the people behind the meow attacks. I agree, they are very much in the wrong.
But being careless with other people's data is also wrong and we should not feel bad for companies whose negligent practices backfire on them. That is what I was trying to highlight with my analogy.
Like bacteria, there will always be bad actors trying to exploit poor security. If not these attackers, then someone else. That is why we have security measures.
The people we should feel bad for are the individual customers affected.
Ps also, there is a big difference between being careless with your own property (your analogy) and someone else's property/data/wellbeing (my analogy and the case at hand).
I agree but I don't think that changes my point that the person who destroys the data has more culpability than the storage service in the destruction of the data.
There tends to be a pass given to people destroying data and I don't think that is right.
As engineers we have to assume that there is always someone out there looking to break into our systems. We don't get to blame them for our failure to secure our systems.
For us to be angry at the hackers is as fruitless as it would be for the unhygienic cook to be angry at the bacteria.
Why are you making the assumption that I'm making the point "as an engineer" as opposed to just a citizen who thinks it is reasonable to expect people not do destroy something that doesn't belong to them?
Your analogy about bacteria doesn't make any sense, we don't expect the bacteria to be actively seeking out unhygienic cooks. If you want to use your analogy it would be like having someone shake the cook's hand in order to put a mild irritant on their hands so that when they prepare food without washing or gloving their hands the irritant is spread to the food, thus highlighting the fact that the chef wasn't following good hygiene. Would you expect that behavior to be excused? Would you be OK with that if you were the one throwing up?
You're presumably an engineer, I'm an engineer, this is an engineering forum. That is why I may have "assumed you made the point as an engineer".
It sounds as if you think I'm defending the attackers. I'm not. I'm pointing out that the presence of malicious actors is a fact of life on the Internet, like it or not.
I'm not going to take my analogy further. I think it's reasonably clear what I meant.
I'm not sure wiping out data from these unsecured databases is the answer, but even for amateur installations which have data of no relevance, the database could be used for nefarious things or the machine itself it runs on could be taken over and used for things such as DDOS attacks.
In that sense, receiving a strong notification that your compute is available to anyone and you should secure it is a good thing.
This is more akin to a person knowing the basics of driving a car but not which side of the road to use or what to do at a traffic light. They are a danger to themselves and others, the others in this case being the users of whatever services the unsecured databases provide.
My sympathy for people learning the basics of our field and missing a few points stops when others are harmed.
Although the parent's analogy is arguably flawed, there's a very good point in the fact that there are users who are not involved in the implementation of the service - "People with IoT apps for their home". They're not drivers, to follow the driving analogy.
It's unrealistic to expect that the population at large starts to pay a significant attention, in particular because the services/gadgets are a black box. How does one know if a device is safe? A layman surely can't; even somebody who's "just a dev" likely can't.
Given the large-scale nature, probably some form of regulation would be the most realistic mitigation. Following the analogy, such users are taxi clients, and for similar reasons, taxis are regulated.
With that in mind, certainly the engineering side of the equation should be held accountable. But it seems that the market is not punishing it at all.
Yup. My point is some people might not even know that their database is accessible from the web lol. It’s pretty easy to follow a tutorial or get something OOTB that’s not secure, so we shouldn’t be saying we’re glad this happened. Even if it’s big businesses, what if said businesses were storing important data such as health records?
I think the learn by failing is a good mentality but was hoping we can be mindful of the fact that this harms more than just the “big bad man”
Edit: Addendum for a more thoughtful discussion, it would be great if these databases and tools provided some default security OOTB requiring no configuration whatsoever. Example: rather than creating user and password with root, is rather have some CMS site generate a random one!
Legislation is just so far behind. User data is useful, but there should be requirements before you can just accumulate it. Cars are useful too but require a licence and insurance to drive.
> Given the large-scale nature, probably some form of regulation would be the most realistic mitigation.
Rather than regulation, how about trademark-protected certification? I.e., similar to what Underwriter Laboratories ("UL") does for consumer electrical products in the U.S.?
Except rather than the government requiring certification by UL or similar, organizations could simply decide for themselves whether or not to use uncertified products. And perhaps insurance companies could price certification status into relevant policies.
This is my thought - why would an IoT device expose a database publicly? And if so, shouldn't the companies producing those devices not be following such bad practices? Maybe the consumer who bought such a device should go to the manufacturer and complain about being sold an inherently insecure device.
Just because its offering some useful service doesn't indemnify the ownership from the bad methods they use to deliver the service.
Exposing your database to the internet with default creds is not "standards and best practices" - its highly negligent, and if you are taking people's money for such a service, I have no pity for you.
If you can't or don't know how to secure it, it shouldn't be online.
My argument is more akin to a child learning not to leave their bike unattended on a city street corner overnight. I can come by pick up the bike, and tell you the dangers, but there's only one real way to learn.
And clearly my opinion isn't even close to comparison with somebody being killed in a robbery.
Just putting a password on a database is more than a 'standard', it's truly common sense. Really, even a beginner should think of this. Otherwise they shouldn't be messing with this stuff.
And if they do get 'meowed', lesson well deserved.
Would you feel the same way if someone burned your house down if you left the door unlocked? Would you support the idea of people walking through a neighborhood and checking every door in a similar way? Does your opinion change if it happened in a business district?
I think it is fine to argue that doors should be locked but that doesn't mean that a crime hasn't been committed when someone takes advantage of a situation.
Not sure why you are focusing on the PII scenario. The original report seems to say it is just "unsecured databases" and not databases that have PII information posted.
You are also making an subtle assumption that the service is being administered by a 3rd party. Could be that the service is being administered by the owner of the data.
The folks hit didn't have backups (if they did, well anyone can restore them), nor did they secure their db. One is forgivable. Both together get no sympathy from me. Rather disgust that some of them had PII from customers that trusted them.
More justification that this is an appropriate way to teach people a lesson. You don't seem to care at all what the impact might be on those affected. You are in affect encouraging the criminal behavior because the ends justify the means, apparently.
While I understand the concern that organizations aren't taking proper care to protect their data I think legitimizing vigilante punishment for those mistakes is a very problematic stance.
It's literally any unsecured database, not just databases containing PII.
What if you just have a toy database for a toy project on the public internet and some jerk just deletes it to "teach you a lesson"? It's like, gee, thanks...
Except that I didn’t leave the doors open. Someone I trusted with the keys, left them in their safe. Unlocked. So I rather have their whole place burned down and MY keys melted at the same time.
You are suggesting that it was just "keys" but that isn't the case here. You don't know what type of data is being destroyed.
A better example might be a storage unit service that left the front gate unlocked. If someone torches the place to illustrate that they need better security would you be comfortable with that? Isn't there a better approach that we should encourage or is OK to encourage people to destroy things that aren't protected to teach people "lessons"?
Maybe. However, if this was my diary or photoalbum, bank statements, medical reports or anything of that sort I’d rather have them destroyed than knowing that they may have been copied...
The lesson is not for the people, it’s for the companies that take shortcuts to save money. It for the MBA’s that think things don’t have to be properly engineered.
So you are rationalizing a crime because it teaches companies a lesson? That is a crazy rejection of the rule of law. Would you be comfortable in applying that logic to all crimes?
I live in a small (~20k population) rural technically-city (but... it's a town) where crime is not much of an issue; my car sits unlocked in my driveway and I seldom lock my house -- when I do, it's almost always when I'm at home (alone) -- it's about a feeling of security, not any real risk of a break-in. I've lived this way in this town for many years and have never experienced a robbery. I think it's pretty low risk, not irresponsible.
I've also lived in a larger city. There, I absolutely locked my doors. And I've traveled to tourist-y locations known for pickpockets, and kept an even closer watch on my belongings.
The internet is two orders of magnitude larger than the world's largest city and policing it is exceptionally difficult, since it spans national borders. There will be crime.
I wish this weren't the case. I much prefer living in a safe place where I don't have to worry about locking my doors when I go out. But when crime is common, it's negligent not to protect against it.
I agree with all that and I also don't want to let the perpetrator off the hook. In your example, doesn't matter if the crime happens in a place where it is uncommon or not, it is still a crime and we shouldn't excuse the perpetrator by suggesting that the victim needed to be taught a lesson.
I think the major difference is that losing my house along with all my belongings is much worse than losing just some of my belongings.
Also data being exposed publicly can be used nefariously by multiple parties, so is likely worse in most scenarios compared to just outright deletion of the data
Ok, but losing "just some of your belongings" is bad also, right? When thinking about the culpability of the person deleting the data or storing the data, we have to start from the assumption that the owner of the data values it.
Whatever analysis you want to put on the situation I don't think it is reasonable to start with the idea that some of the data might not be that valuable.
That obviously isn't true. Some data shouldn't exist: CP. Some data can exist, but it's backed-up so well that deletion is never a problem. For example, I'm not going to forget my birth date any time soon! In fact, very little of the information that businesses have about me needs preservation. I remember it all, and if I decide the business still deserves it I can give it to them again.
It is of course possible that some of this data didn't need to be kept private, and therefore shouldn't have been deleted. Maybe some medical researchers had compiled the data they needed to formulate the ultimate cure to COVID-19? (I hope they had anonymized all the patient data!) Until those researchers come forward to lament humanity's loss, I'll just assume that all the "victims" who don't want to go into too much detail about the "lost" data were playing fast and loose with their customers' private information.
So your argument is that any random person is in a position to evaluate whether someone else's "data shouldn't exist" and take unilateral action to delete it? And are you suggesting that in this particular case the person launching this attack is taking time to evaluate the nature of the data before taking action?
> I'll just assume that all the "victims" who don't want to go into too much detail about the "lost" data were playing fast and loose with their customers' private information.
Why the scare quotes? It takes some serious amount of chutzpah to advocate that it is a reasonable assumption to assume the data wasn't important and to use the lack of public complaint as evidence that the data wasn't really important. Why do you even think it was "customer" data?
I didn't just invent this idea that businesses are careless with data their customers would prefer to be kept private. Basically every breach we ever hear about features this prominently. Somehow we've created an economy in which there exists a vast asymmetry between corporations who pad their books a few percentage points by abusing their position and the humans who suffer such abuses. The fact that the publicity of small bits of data about a human can cause that human massive harms is itself a contingent creation of our screwed-up system, which benefits the giant companies whose lobbyists write the laws. It's as if someone decided we should all live under the "protect your True Name at all costs" system from the Earthsea novels, without giving any of us any way to do that.
There's very little a customer can do to determine how or even whether her confidential data is protected. Even if she had this knowledge, in many cases she can't just decide to do business elsewhere. In many cases she was never a customer in the first place! In this context, an open database is like a shoddily constructed tall building that will collapse at the first stiff breeze. It shouldn't exist, and anyone who destroys it upon discovering it is doing humanity a service. Even if the building's owners had somehow kept the general public out (which you'd like us to assume), those owners themselves increased their danger with every bit of data added. Now, since the building has been destroyed, its owners and occupants are no longer in steadily increasing danger.
You seem entirely focused on PII concerns and arguing as if the only organizations affected by this incident are "giant companies". That doesn't seem to be the case. I haven't seen any suggestion that this incident is focused on that type of data.
As much as I agree with all the concerns posted here about how data should be protected better I don't think it is necessary to excuse and legitimize the unauthorized access along the way.
Wealthy interests built the system, but they're not the only abusive actors within it. It would not surprise if smaller firms completely failed to protect the data of other parties more often than larger firms did so. The best way for database operators to prevent unauthorized access and deletion is to secure their databases in some way. The best way for anyone else to prevent abusive access is to delete unsecured databases. Working together, this problem will be solved eventually.
I think I'm having a somewhat uncharacteristic relativist thought... I feel that the indiscriminate and nearly unregulated collection of any data any company feels like grabbing hold of _should_ be countered.
I feel like the wrongness of deleting unsecured data is a pittance compared to the crapload of other wrongs that have been visited upon us 'products' by failures of diligence or desire or consequence.
I would very much like to hear, if it turns out to be so, that the operators of 'meow' are selectively targeting more likely corporate data, especially with customer/user data, but in the end I'm still ok with the idea of burn it all and let the DB vendors and IT staff who let their asses hang out explain why security was so low on their priority lists.
And yes, I'll take some potential difficulties in my own life due to unexpected deletion complications in the process. I'm not asking anyone else to accept anything I'm not going to be okay with myself.
Better off? The idea that victims deserve to be victimized because they didn't take enough care is trotted out every time a security issue comes up on HN.
Thr victims are the unaware customers who's data has been stolen because the company wasn't providing security.
If a business left the store open with the customers credit cards details on display. Anyone passing by can go in and copy that info. Someone sees this and burns the exposed records. Perhaps they helped the victim.
Remember no one burned the store down or the table holding the records. They burned only the exposed records.
You don't know who was the storage vendor and who's data was being deleted and you have no idea what that data represents or what the consequences are of having to restore it.
I think it is entirely reasonable to start with the presumption that people have a right to their data and to their property, that it is valuable to them.
If a site/service has a right to allow a person to delete data. The machine can be setup however they like. These are not hacked databases. The system said welcome what do you want to do? You can read everything or delete everything or add anything.
Yes they are. The method of the hack was 'simple' to you, but that doesn't mean it's just magically not a hack any more. These are hacked databases.
> The system said welcome what do you want to do? You can read everything or delete everything or add anything.
I don't understand this. Are you suggesting that the attackers were greeted by the database with an English-language legal directive of what legal permissions they had in the database? What do you mean by this statement? Surely the databases said no such thing.
If you're implying that a private door with no lock is not private but actually shared property that can be destroyed or added to in any way, then I think you're wrong. None of this comment makes sense to me. An unsecured database holding private data, or an unlocked door to a private business or building, is not an open legal invitation for vandalism.
If you attempt to connect to a database the database will greet you. It can be configured to ask for a login. It can be configured to have no login. If it doesn't have a login and gives you a welcome prompt it's not called hacking. In order to hack something it needed to be secured to start with.
Databases do greet users in mostly english. Need help? Type help. On the list of things the system allows deleting data appeared to be one.
If you knock on a door and the door opens and says welcome what do you want to do (delete data, read data) and you pick delete it doesn't mean it's illegal vandalism.
> It can be configured to have no login. If it doesn't have a login and gives you a welcome prompt it's not called hacking. In order to hack something it needed to be secured to start with.
I very strongly disagree with this. Can you cite somewhere that unauthorized database access is not hacking if there is no password? To me and I think the law that is definitely illegal and hacking.
> If you knock on a door and the door opens and says welcome what do you want to do (delete data, read data) and you pick delete it doesn't mean it's illegal vandalism.
Yes it does! If you don’t have authorized access then that database isn’t yours and entering it is illegal.
I don't have a feeling one way or the other, but I see where you're coming from and I think there's an interesting aspect that many of the folks here seem to be missing.
Were this sort of attack to become part of the "noise" of the internet (much as the continual bombarding of my SSH ports) then peoples databases would get deleted _before_ they contain any meaningful amount of data.
So in practice this sort of gross vandalism is limited to the appearance of such an attack, but not ongoing.
I had this the other day building OS images, which accidentally left the system a passwordless login. Within less than a few hours it was (presumably) spewing mail or doing awful things -- long before anything went anywhere near production data or any kind of trust.
> [Article] They could be the work of a vigilante trying to give administrators a hard lesson in security by raining destruction on unsecured data.
It could be a person attempting to prevent the data from falling into the wrong hands. Problem is: once it's deleted, you have no idea whether your data was stolen and shared. A better option would be to first send a copy to Have I Been Pwned.
I wholeheartedly agree. I cannot think of a scenario where a company exposes my data and I would not want it to be deleted ASAP. The only thing is that such companies might not be able anymore (if data was not backed up) to email me about a “breach”.
It's stuff like this that reminds me that the internet is in many ways still in a loosely regulated, "Wild West" state. This is pretty clearly willful destruction (I.e. vandalism; https://legal-dictionary.thefreedictionary.com/Willful+damag...). It's illegal in the real world, and should be illegal in the digital world. A lot of people are saying that organizations that had these DBs in public "had it coming", or "now they'll learn." What's a real world parallel for this? If an organisation is putting its customers at risk, you can report them. In these cases, companies with insecure data stores are putting their customers at risk by exposing their data. Is there anyone you can report them to? Is there any organisation that will hold them accountable to actually make changes?
I'd also note that not all DBs contain other people's data. Those have no moral concerns with bring public. There is a risk that someone will destroy it, but I'd say that's the same risk taken with public art or something. Yes it's public; yes someone can destroy it; yes it's illegal for someone to destroy it (even though it's public); no, the fact that it's public is not illegal.
> It's illegal in the real world, and should be illegal in the digital world
It is illegal in the USA. And is easily an arguable civil case as well. The problem is identifying the perportrator.
Organizations are only way to hold poor actors accountable.
Bad PR and going out of business cause data critical to operations has all been deleted are strong incentives. Unfortunately businesses typically lobby for harsh laws, tyrancial surveillance rather than the expensive and difficult process of improving their operational security.
Bad PR works to an extent, but bad PR can be combatted with good PR, not necessarily with changes. I think an independent regulator would be useful; one that can testify that a company is following best practices for data, etc. Although if the regulators don't have any teeth, or if they don't have a clear benefit for companies to allow a regular, comprehensive, internal audit, it definitely won't gain any adoption.
Yeah, I think enforcement is probably a big part of what keeps the internet a "Wild West" for these cases; and improvements to enforcementability could fundamentally alter what the internet is :/
Yeah, I don't think the internet can be fundamentally altered to the extent that it could be regulated. Unless it somehow becomes a major priority for the world leaders - in which case I'll bet we can say goodbye to any privacy we still had.
It's been said before plenty of times, but I think the solution has to be sought in increasing awareness of the dangers of the internet. Compare it to the electrical grid. 99% of people would not mess with it without consulting/hiring an expert. Not just because it's illegal, but because of the obvious danger. But for the informational grid, "nobody" (exaggerating) cares because they cannot oversee the consequences.
Problem then is, how do you educate people about this... I don't know.
Actually, a good physical example is restaurant health inspections. They're responsible for testing the safety of an organisation, and making sure that data is publicly visible, and in extreme cases, shutting the organisation down for negligence.
They are based on disasters, but at least we try not to repeat the same ones.
One of the things that drew me to software was the idea that you could fix a problem once and for all and then just keep reusing the solution for ever and ever. I don't think I'd have the heart to go back and tell myself what a stupid notion that turns out to be.
For people who pride themselves on going 'faster' than previous eras of innovation, we seem to ignore the fact that we have now been the 'Wild West' for longer than the actual Wild West...
2) Even if it were, and my twenty-something assistant left my shop door open at night consistently, to me, the question of the legality of the resulting damage would be rather secondary.
What if your shop door was locked but was laughably easy to open with lockpicks? I get to victim blame you for choosing an insecure lock and browbeat you into purchasing more expensive and onerous security.
I don't know :( But this feels like some sort of terrorist tactic, and I don't think this should be the way things on the internet are regulated either.
Actually, that's one of things that makes this frustrating. Because there is no real regulation, a group/individual decided to "become" the law. The became lawmaker, judge, and executioner. They decided what was illegal, collected the guilty, and punished them for it. That's what makes this feel like a "Wild West" situation. The made themselves regulators of the internet.
The part that confuses me here is that everybody seems to take in stride that all these public-facing databases are already tracked and indexed. Like, how does https://www.shodan.io/search?query=meow+indices know all this? What am I missing here?
Is this attack literally "attempt access each database listed on shodan.io and destroy it if that works"?
I might be missing some major aspect (I certainly hope so), but isn't this like wondering why all those fireworks that people keep storing on the streets were eventually set off by some kid with a mask? Why isn't the question "why didn't this happen sooner"?
It has already happened in the past. Repeatedly. There's news coverage about this at least once a year. And it doesn't require using Shodan as there are plenty of open-source tools for scanning the Internet nowadays.
For example, this was from the same news website a few years ago:
Jeez, that's a pretty impressive dumpster fire. And it's been going for half a decade. Kudos for keeping track of it and periodically doing your part in reminding the world.
Is there an inexpensive service out there that does “mock” attacks if you give it a bunch of host names and ports? I know it’s something you could create yourself but would be nice to have a third party try to connect to your databases and immediately alert you if it was able to gain access.
Would especially be useful if you were tinkering with firewall/security settings and accidentally opened something up.
Shodan Monitor will do it and if you're only keeping track of <= 16 IPs then you would just need the membership which is a one-time payment of $49 (i.e. no subscription cost) for a lifetime account upgrade (https://www.shodan.io/store/member). You just provide an IP/ network/ domain and we'll notify you if anything changes or becomes vulnerable. It's basically Google Alerts but for network ports:
We don't share revenue information but we've seen steady growth for the past 10+ years. All I can say is that Shodan is a profitable, bootstrapped business with >3.5 million users, 80%+ of Fortune 100 and thousands of universities (we let them setup monitoring for free for up to 120k IPs).
I'm using this and really love it, but it's annoying if you want to use a domain name instead of IPs, because if the DNS records change the old IPs stay in with the new ones.
That shouldn't be happening, can you confirm? It should flush out the old ones based on the current DNS information. The whole point is to automatically update the alerts based on the latest DNS information so you don't need to manage that yourself.
Sorry, that obviously shouldn't happen (both domain not updating and support falling through). We use Shodan Monitor for our own infrastructure and haven't seen that issue (or heard about it from our other customers) but I'll keep an eye out.
If this keeps happening weekly, they'll fix it. Maybe there should be a government Agency for Deleting Publicly Exposed Databases that's likely to hit any that you stand up within a week. Also, make having had a publicly exposed database deleted something that is in the public record, and highly prejudicial evidence in civil liability cases.
A Department of Botnet-Suseptible IoT Device Bricking would be useful in the same way.
If a business or org is careless enough to leave their data exposed, they deserve to be punished like this. I'd rather have my sensitive data deleted by hackers than exposed by hackers.
Way back in the early 2000s I was a young mid level developer and we had a SQL Server backed solution. There was a wide spread attack on Sql Server installations that didn’t change the default blank SA password. We were one of the companies that didn’t.
But, even then I knew not to have a publicly accessible database server. We just didn’t give the server a public IP address. Nothing fancy. We weren’t affected but I immediately changed the password.
Fast forward to 2018. The company I worked for was just starting to ramp up an in-house development staff led by a new CTO. Everything had been outsourced to a foreign agency. They had a publicly accessible ElasticSearch cluster. I wasn’t on the team responsible for it, but I know that the architect on the team knew better. Even though he didn’t setup the original cluster, he knew it was insecure and just didn’t prioritize recreating it inside our VPC.
Of course we got hacked and someone deleted everything in it. Then they decided to go ahead and recreate it inside the VPC. Luckily we didn’t use ES as a primary store and we were just offline all day while we ran the process to repopulate the cluster from our Mysql database.
Why do people keep making the same mistake? I was definitely not any world class software architect at 25 years old, but even I knew to be cautious about giving servers public IP addresses unnecessarily.
Because you're completely wrong; what you're advocating is nothing more than security by obscurity. An address is just an address; it tells you where something is. If you don't have a firewall in place, then any attacker who cares enough to actually route a packet to your internal servers can access them. If you do have a firewall in place, then an attacker gains nothing from knowing the address of a server they can't send any packets to. Private networks add a huge amount of complexity with its own security holes (they're easy to scan once you're inside them, a lot of operating systems treat them as somehow "safer" than the public internet...). They do more harm than good.
> You realize you can’t get to a private IP address over the Internet right?
Of course you can - most end users have private IP addresses these days, yet they somehow manage to communicate with people on the internet.
A private IP address is of course not routable directly on the Internet. But routers route, and can certainly route a packet from the public internet to an RFC 1918 private address. If the routers on the edge of your RFC1918 network do not have firewalls in place, then they will happily route packets to your internal servers and you're no better off (against a serious attacker; a script kiddie might ignore you as too much effort, sure) than if you used public addresses. If those routers do have firewalls in place, then you're also no better off than if you used public addresses.
In the case of the attack in question - how would they have initiated a command to erase the ElasticSearch cluster from the Internet?
Any NAT would be stateful and the communication would have had to be initiated from the cluster.
Not having a public IP address is not about “obscuring” the IP address. It’s not like so said why didn’t they have ES listening on a non standard port.
> Any NAT would be stateful and the communication would have had to be initiated from the cluster.
Only if the router is configured that way - and you don't need NAT to configure it like that. If the router is open then you just set the destination address to the address of the server you want it to go to. (As for getting it to the router you either find a way onto the same segment, explicitly specify it as a routing hop, use ip-in-ip...).
NAT is not a security mechanism. Often the same device does both, but they are separate functions.
> Not having a public IP address is not about “obscuring” the IP address. It’s not like so said why didn’t they have ES listening on a non standard port.
Only if the router is configured that way - and you don't need NAT to configure it like that. If the router is open then you just set the destination address to the address of the server you want it to go to. (As for getting it to the router you either find a way onto the same segment, explicitly specify it as a routing hop, use ip-in-ip...).
So if someone purposefully for some reason goes in and configures their router to map a port to a specific internal IP address to allow internet traffic to their ES cluster it isn’t secure? Isn’t that kind of stretch? If I posted the root credentials to my AWS account and turned off MFA that would be insecure also. Does that also mean I’m depending on “security through obscurity”?
It's exactly like that!
So am I also “obscuring” my IP address when I am testing an API locally and I configure it to only listen on 127.0.0.1:3000? Oh no, I just told you my IP address!
Everything you mentioned could only be done if an inside malicious actor purposefully made it insecure.
> So if someone purposefully for some reason goes in and configures their router to map a port to a specific internal IP address to allow internet traffic to their ES cluster it isn’t secure?
That's not what I'm talking about. Suppose your router receives a packet whose destination is that internal IP address. Then it's going to send it there, unless it's configured to block that traffic.
> So am I also “obscuring” my IP address when I am testing an API locally and I configure it to only listen on 127.0.0.1:3000?
No, quite the opposite. Listening only on the loopback interface is a real security mechanism. Binding to 127.0.0.1 achieves that (though there have been bugs in the past; binding explicitly to the loopback device is better).
> Oh no, I just told you my IP address!
That's exactly my point! The address doesn't matter, the actual network routing is what matters. Same thing if you're running an actual airgapped network: what makes it safe is the fact that it's physically disconnected, not that you used particular addresses on it.
Real-life example: the UK military's internal servers have real IP addresses in the 25.0.0.0 range. Probably some of them are insecure. But it doesn't matter because, even though they have public IP addresses, you have no way to send packets to those IP addresses, because they've got proper firewalling in place. It's no different from if they were running a private network - except that it means they don't have to mess around with NAT, if someone connects from a VPN they don't have to worry about address collisions.... IPv6 works the same way.
That's not what I'm talking about. Suppose your router receives a packet whose destination is that internal IP address. Then it's going to send it there, unless it's configured to block that traffic.
What router is set up by default to route traffic from the internet to a private IP address unless the traffic initiated from the private IP address? Someone would have to purposefully configure their router to do it.
Of course someone can purposefully configure an insecure system.
Any router designed for professional environments won't second-guess your networking setup. Either it will route everything by default or it will deny everything by default and route only those routes you've specifically added, but either way it's not going to do anything differently just because one side or the other is an RFC1918 private address.
I understand that. But, we are talking about defaults. The reason the exploits that this submission is talking about took place is because too many people didn’t change the defaults.
Assuming people took the sensible leap that private resources equal private IP address, they are not going to then go out of their way to configure their router to route their private resources.
As far as “route everything” how is it going to know how to route from a public IP address to your ES server unless you specifically tell it?
I’m sure no one who got hacked went out of their way to configure their router to make their ES cluster publicly accessible.
> Assuming people took the sensible leap that private resources equal private IP address, they are not going to then go out of their way to configure their router to route their private resources.
That works until you have a legitimate need to expose a few of those private IP addresses publicly (which is bound to happen sooner or later). It's a bad idea to rely on the same thing to carry two subtly different meanings - especially when one of them is security-critical.
> As far as “route everything” how is it going to know how to route from a public IP address to your ES server unless you specifically tell it?
The router knows how to reach the ES server (it has to be able to send packets to that server if it's providing that server's connection to the internet). So if someone on the outside does `route add <ES server> via <router>` (or, these days, slightly more sophisticated alternatives) then they have access to the ES server same as if it was just on the internet. A few years back this was a big source of vulnerabilities - organisations assumed that their servers were safe because they didn't have public addresses, but nothing was actually stopping people sending packets to those servers if they figured out (or guessed) what the addresses were.
I don't agree with the GP about "routers route" (at least in the context of a cloud VPC), but a more serious problem that IS seen all the time in the wild is SSRF, and as I understand it elastisearch presents an HTTP REST interface, which makes it a fine target for such attacks (where as, say, Postgres, will not accept HTTP connections).
Relying solely on routability for authorization means that you're authorizing everything that can route to it in any context. Maybe that's fine.
Because there are always new developers showing up that haven't been taught. (Eternal September if you will)
The real solution would be:
1. We are past the point were our practice needs professional licensure. We need standards, a governing body, and ethics.
2. Those above items need to be taught to new developers. How long has security been an after thought to CS degree programs? I know I never touched it in an academic setting. We didn't even have a class that covered it. You wanted to learn about it you had to seek it out.
3. Vendors to do the right thing and stop offering default passwords. That isn't going to happen, so we have to force them to, either trough legislation or through other means.
I can't believe people are victim blaming the db admins for not knowing about vulnerability. What good comes of destroying the db instead of talking about the vulnerability to the open source projects? Coincidentally shodan; that I've never heard of.
There's a difference between a vulnerability, and a common misconfiguration that usually comes from a "make it work first, security later" mindset.
The good that comes from destroying the DB is:
a) the data is no longer exposed to the Internet, where more malicious actors could take it, affecting the customers of the incompetent company
b) ignoring it stops being a viable option - leaking your customer's data all over the place often doesn't have sufficiently obvious and severe consequences for the company doing the leaking to discourage it. Disruption that breaks production will get their attention, and they likely will secure their database in the future.
(No moral or legal judgement regarding this action, just answering the "what good comes of it" question.)
Edit: Also, someone commented further below on the difficulty of doing it the right way - it's hard to contact the companies, and it's even harder to get them to actually listen and fix it instead of ignoring it or trying to "shoot the messenger". This approach may be wrong and/or illegal, but it it much likely to actually draw the attention of the right people, and prevent them from simply ignoring the problem.
The companies running those open databases aren't just victims; they're also perpetrators of privacy violations. In many cases, they're even collecting data for a purpose that the data subject receives no benefit from.
You don't need to be a carpenter to know that you should install a lock on the front door of your house. How does anyone get to the point of standing up a production db and is allowing writes from unauthenticated connections?
I am pretty salty since as a sysadmin, I have been getting 'just pipe it to su bash' and 'i need allow any any' and 'bro I need chmod 777 on this directory and all its children' and 'bro this service account has to be a domain admin' from developers my entire professional career. Everything that there is to say has already been said and I am not really sure what to do about it. Nobody is out there peddling these cool fixes as truth and yet they seem to have a cult all the same.
What can we do to make this common knowledge? This needs to be on the same level as washing your hands and not accepting candy from strangers, yet every week we see a new data breach that boils down to 'somebody used the rights as they were designed'.
I think that this is an instance of ethical hacking. They hurry up to remove any and all instances of PII before some hostile actor scrapes them while at the same time punishing the people who leave their servers open without caring for their costumers.
To whoever is doing that: you are doing god's work, please keep it up.
What I don't get about Shodan: Why aren't all unsecured databases found instantly (at the moment Shodan went online), but recurring attacks/dumps like this one that rely on it? Do they update their crawl data in waves?
One real limitation is getting data out of Shodan. Having done a few different projects that involve large-scale use of Shodan results (e.g. several hundred thousand records), this kind of thing usually ends up costing $300 for either export credits or a service plan. Sure, $300 isn't really that much to cause millions in damage, but I think it's a big factor in why we don't often see Shodan used for huge-scale malfeasance. You both have to put up the money and in paying you probably give up some identification info, and I don't know if Shodan has complied with law enforcement in the past but I can sure see them getting a warrant for "the person who just spend hundreds to export and/or query every unsecured MongoDB."
Also, as a bit of an aside, the relationship between "export credits" and "query credits," and the export system and API of Shodan, are extremely confusing and just a bad bit of product design. Each one seems to be capable of things the other isn't, but they're priced on totally different systems.
But really it's mostly just a matter of motivation, I think. Pulling even just thousands of entries from Shodan, writing some software to use them, and then running it in a reasonably deniable way, takes effort and is pretty slow (why we see this going for multiple days). It's not a huge amount of effort but it's enough that "script kiddie" types don't really seem to do it, you need to be motivated and spend the time on it.
Contrary to security urban legend it seems like the number of people who are highly motivated to purely cause damage is not actually that large, people only put in the time if they can figure out a way to gain from it... and just deleting data doesn't really achieve that. You've got to figure out a way to hold it for ransom and/or collect and leverage sensitive data. We've seen both happening on various scales with this kind of unsecured database and we'll probably see more of both as we go forward... but keep in mind that in the ransomware game, encrypting computers is both easier (established off-the-shelf ransomware can be purchased) and probably shows higher returns, so the "professionals" aren't spending a lot of time messing around with exposed databases.
We're actually getting rid of export credits because it's caused confusion over the years. We now just have query credits to download data/ do searches, and scan credits for users that want to request on-demand scans. We announced this change in the most recent Shodan Update newsletter. You can already use our new website (https://beta.shodan.io) to download data using your query credits.
Export credits were the first way I tried to monetize Shodan and it became a legacy system that lots of companies used so I was hesitant to get rid of it until something better was in place.
I'll also add that the API was purposely not designed for downloading lots of search results. The API is designed for security operations center (SOC) use cases. Companies that need large-scale, bulk access to our data would need to check out our enterprise platform (https://enterprise.shodan.io).
This is what I've assumed, but it's in a pretty uncomfortable place right now as e.g. the documentation often refers to export credits with a broken link.
The API is somewhat unsuitable for exporting large volumes because it seems remarkably unstable as to ordering, it suggests that you can do paginated requests but the second page tends to have 30% overlap with the first page.
I 100% understand the product motive to move large exports to an "Enterprise" feature, but it's rather disappointing because as a small-scale independent operation I don't expect to be able to afford it, and that would go for a lot of productive people in security research. But then, that's capitalism.
I decided that a broken link is better than having people spend money on something that will be deprecated. We're obviously working on cleaning up those broken links but it's an easy thing to explain if anybody emails support@shodan.io
The ordering is based on timestamp and it can happen that new results were indexed in between your 1st request and 2nd request which creates an overlapping result. A 30% overlap is unusual and sounds like it's for a query with many results.
Finally, most researchers don't actually need to download data. They could just use our free API and facet queries to get the information without downloading the actual data. This entire website is powered by a free API key that uses facets:
I think a lot of researchers go into the default mode of "I want to have the data" but using facets is way easier, faster and doesn't cost any money at all. And you can navigate the available facets using our new beta website (another area we're trying to make things a bit clearer). For example:
Note that we provide free upgrades to universities/ students/ professors as well as routinely work together with researchers so we're not trying to push them into the enterprise product. We also let universities monitor up to ~120k IPs for free using Shodan Monitor (https://monitor.shodan.io). But the use cases for researchers are few and we figure that if you need lots of data then you can send us an email.
A story like this pops up every year and preventing ransomware/ mass-deletion of publicly exposed databases has proven to be very challenging to stop from happening. I mean, I wrote about this issue 5 years ago:
We've also sent the raw data to various database vendors for free but even for them it's difficult to reach out to customers to get it fixed. And then there's always the worry that you'll get shot as the messenger of bad news. We've had a lot more success in getting things taken offline when we already have some relationship with the organization or at least a mutual customer.
In the past, older versions of MongoDB were more public than newer versions but that isn't the case anymore based on what we're seeing right now:
And in terms of Shodan, we crawl 24/7 (i.e. not waves) and update the search engine as the data is collected with a small delay (<1 hour) so anybody that gets real-time notifications (https://monitor.shodan.io) for their networks will see it before it shows up on the search index.
Why is mongodb seem to show up alot with this. Does their default set up hide some unsecured users? Its been a while but I dont remember that being in there.
No, their default sets up no authentication at all IIRC. Combined with Dockerized installations punching through some firewall setups (as discussed elsewhere), you'll get meowed.
By default though, MongoDB will only listen on localhost and I believe it'll show you a big warning on boot up if you don't have authentication configured. They used to listen on 0.0.0.0 by default but that was fixed many years ago.
And this issue doesn't just affect MongoDB - imo since the "webscale" days it's been a favorite to knock on but the public exposure of data happens across many technologies. Here's a comparison with a few others:
I wonder if you can find a remote code execution vulnerability in client libraries for one of these databases, and set up a few honeypot databases... you might be able to catch who's doing it and retaliate. The legalities of doing that are certainly questionable, but then again, so is this.
If this turns out to be an effective lesson on security, systems should implement their own meow to protect their users.
E.g. A database That intentionally removes itself if the default password/an insecure password is used, with an easy-to-follow guide in error log on how to properly configure it.
You are allowed to judge people for taking far, far too long to do the right thing.
It indicates a pattern of poor judgement, which speaks to trust. You know they are going to let you down each time a new issue comes up.
Faulting people for being cautious around such bad actors (which I'm not saying you're doing, but the response will) speaks to your judgement, not the vendor's.
Seem like this could be easily reconfigured to do the opposite. Leave the databases but overwrite all the existing data then write dummy data until disk is exhausted. Does that cross another ethical line?
It took way too long. When you have a company that builds a great product with no security whatsoever and makes their business model based on selling the security module on it and changes their behavior only when threatened by a competing open source product, you have to wonder when the catastrophe comes.
Note I don't blame anyone, it was just bound to happen.
Bad comparison. This is akin to someone random walking into a restaurant and looking under the fryer and finding a dead rat and removing it with the fryer.
After having read a number of articles talking about data leaks on a gigantic scale, I decided to check it out. Because shodan is not free to use/behind a paywall, I wrote a simple windows console tool which scans all known Azure subnets for unsecured elasticsearch instances and logs the results. I was baffled by the amount of instances this tool found within the first few hours :( To say that security in IT is getting out of control would be an understatement...
Doing aggregate queries is free - only if you want to download actual data do we start charging money. You could just do the following via the CLI using a free account:
People are talking about more responsible disclosure. Is it feasible to even track down the owners of 4,000 different unsecured databases, much less go through the whole process with them to ensure the database is properly secured?
Some of these attacks leave a calling card sort of thing, e.g. a database with a document that says "hey, lock your stuff up".
That's more or less the best you can do without unreasonable effort, and there's no guarantee the database's owners will ever see the extra database unless you also destroy stuff on the way to make them pay attention...
When that SQL Server worm was going around I had three or four different machines spamming my firewall trying to search for more victims, but I was only able to track one of the IP addresses back to contact information.
That person was very grateful for the heads up, but the other three were SOL.
This is bad when it comes to personal data, like the VPN provider that claimed not to be logging. Companies should spend every effort to secure people's data, and can face large fines in the event of a leak. In erasing the data, Meow is also erasing the evidence of their crimes. Instead, Meow should ransom the data and set a fine proportional to the company's size, revenue, the sensitivity of the personal data, whether that personal data should have been collected in the first place, whether that data should be public-facing, etc. Then Meow can be made a public service, perhaps paid with taxpayer money, and bring about justice.
Short answer: cloud images with poor defaults. I've written about this a few times before and the problem hasn't really changed since the article was posted:
Because the open-source version of Elastic does not contain any security (not even a basic auth) and requires at least a reverse-proxy in front of it which adds difficulty of connecting two things together. And Elastic-licensed Elastic with Security needs to be configured by chaning its config file. That is apparently too complicated for most "IT specialists". :) `sudo apt-get install elasticsearch && sudo systemctl start elasticsearch` and they are done.
Docker overriding iptables rules, in my case. I was using somebody else's project distributed via docker-compose config, which made the port for elasticsearch public, which I was not aware of (I don't normally use docker or elasticsearch). Luckily I was able to regenerate the data stored in elasticsearch, though I had to do it twice because it got wiped again after regeneration and then I had to google what the hell is going on.
> “In this server, all the collected information is anonymous and only be used for analyzing the user’s network performance & problems to improve service quality. So far, no information has been leaked.”
I can't see how you can keep enough info to analyse an individual users service, without keeping logs on their access details (source/target IPs). What BS.
And the Hippocratic oath applies to online vigilantes how, exactly? Right now, I think these actions are causing more good than twenty years of lacklustre legislation have done, worldwide.
That was aimed at everyone on this thread that feels comfortable with deleting databases that they don't own.
Vigilantes make their own choices, but professionals are judged by their reputation. As I've said elsewhere, I challenge anyone, especially professionals, to tweet "I will delete your data if you don't secure it" and to add it to your resume/CV as a strength.
It’s 2020, where security should be at the forefront of almost any tech endeavour. It’s not something to think about afterwards, not any more.
If, given the frequent and public attention to hacked data, you aren’t thinking about how to make sure your data is safe, there really can’t be any sympathy when you get hit by an attack that relies on zero security applied to your database.
C’mon, do we as an industry really learn NOTHING from all the hacks we’ve lived through ?
If you really believe this action is warranted, as I've read on this thread, I challenge you to tweet under your real name "I will delete your data if you don't secure it", and add it to your resume/CV as one of your strengths.
Also as a rule of thumb never ever expose anything but port 80 and 443 if hosting a webapp.
If you must expose services other than http/s then be sure to not leak its version, have it secured properly and _always_ up to date. The user running such services should also be a non privileged user, the daemon chrooted, and the OS should have appropriate process and filesystem permissions in place.
Okay, and I prefer being waterboarded over being drawn & quartered. But it doesn't mean I support the practice of waterboarding, since there's another option, namely just not waterboarding in the first place.
This contrarianism is so strange to me. Data not being deleted by a bad actor is preferable to having it deleted, and I would think that would be the main takeaway here, rather than this weird descent into counterfactuals. Where does the impulse come from to bypass the normal answer, treat it like a trick question and go into contrarian mode by measuring it against counterfactuals? I think when you do that you lose sight of the most important thing here, which is the fact that data is being wantonly deleted and that it is bad that this is happening.
Why is the person doing the deleting so low, relatively speaking, in your ranking of people's responsibility for them doing the deleting?
Also, do you think that this person or persons would refrain from deleting the data if they had the opportunity, but it qualified as a "good idea" to keep online? I.e. they might review, say, medical records, spend some time thinking to themselves whether it was 'necessary' to be online, and then decide to delete or not delete depending on their judgment?
For me, it's because the odds of this person showing up quickly approach 1 as time approaches infinity, and that person's effect would be nil if it weren't for necessary causes 1) through 19).
Blaming the person that hacked you is like blaming the individual rock that sinks your boat when you navigate too close to a rocky shore. The rock may have done 100% of the damage to your boat, but if it hadn't been that rock, it would have been another one.
I was hacked using what at the time was novel (but is now a known exploit): someone claiming to be me transferred my SIM card to a new phone, over the phone, which then enabled them to defeat my 2FA (which of course I've now removed ALL cellphone recovery from).
Are you saying that it's my fault for either picking an insecure provider (T-Mobile, who I absolutely bitched out and told them to put a note on my account to not permit any SIM transfer without me physically being in a store under a camera), or for not staying abreast of the very latest in social-engineering exploits that assholes were using to try to steal bitcoins, and manage my security accordingly?
Rocks don't have moral agency. And the comment they replied to I think was clearly about the blameworthiness of the bad actor.
So I guess problem I'm having is with the equivocation between cause-and-effect responsibility and moral responsibility, which I think was exploited here to indulge in a fun little switcheroo by talking about something they didn't mean.
Mostly because it's a very effective way to ensure things get fixed, while gaining the "attacker" nothing. It's harmful, but so is finding 4000 insecure databases, sending 4000 notification emails, and having 3950 of them ignored (and that approach is probably more risky, so far as inviting legal trouble and expenses). It also neatly removes anyone else's ability to take the data.
Don't expose MySQL databases to the internet. Just don't. Stick an API layer in at the very least with key based auth, and only the bare minimum capabilities allowed for the user.
That said, if you'd read the article you'd see that so far only unsecured MongoDB, Elasticsearch and Redis installations are being attacked so far.
Most SAAS db providers provide their database over the internet, but secured with a login/pass. Eg all DB's on Heroku elements marketplace work like this.
Yes, or they provide an internal subnet only accessible to servers from the same tenant. Usually it's double useful because usually this traffic is not charged.
I'm doing my first web project (self-taught), which is the prototype for an offering me and a partner are developing to become a startup. I was about to start deployment (for the first time in my life) this week, but now I'm afraid. It's a flask app. We serve users forms (POST), then I use this input to run calculations on the server through a python script which makes queries to a MySQL db, then I return results to browser my listing a result-array dynamically using a Jinja template. Is this unsecure? Any tip or accessible reading material to help me understand this matter? We don't store credit card info or other sensitive information for now, because we didn't reach any clients yet, but final version should have at least login functionality. Any light/knowledge/consideration will be much appreciated!
PS: I don't know exactly what you mean by "Stick an API layer in at the very least with key based auth" because I never used an API before and didn't know I would need something like this.
>PS: I don't know exactly what you mean by "Stick an API layer in at the very least with key based auth" because I never used an API before and didn't know I would need something like this
Don't allow access to `mysql -h hostname` from WWW. Instead, keep your MySQL server accessible only by machines you control in the same security group/network/etc. To get access to the database from the public, create an API that allows predefined queries only to users you've authorized. This API would also be hosted within the same security group/network/etc. The public never gets direct access to the database, yet gets the predefined access they need.
It sounds like you are probably fine. Your python flask script is the API layer. Just make sure it can only run sql that you want it to run (ideally, read only until you get it authenticated), and that it escapes user input, and that no one except admins have access to the actual db underneath, and you are totally fine.
Also, for that matter, its a good idea to run backups too, just in case something goes wrong and you need to restore.
This is literally people allowing users to run arbitrary commands on their fully exposed db, and then being surprised when someone runs a delete.
Your API, in this case, is basically your Flask application so you're pretty much done.
If you want to run a database securely, you configure a decent password (should be asked during installation), create new users for each application/database (easy with tools like MySQL workbench) and make all applications running on the server (Flask, in this instance) connect to localhost/127.0.0.1. Don't expose services like databases to the outside world if you cannot in any way help it.
Then enable a firewall on your server (Ubuntu server comes with UFW by default) and lastly use a port scanning tool like nmap/zenmap to check if you have left any services other than HTTP(S) exposed.
I don't know much myself about Flask deployments specifically, but you'll probably use something like nginx or Apache to handle the actual web requests. There's an article here [0] on how to do that. After you've done that, you can secure your connections for free through Let's Encrypt clients like certbot [1] which handle most of the configuration for you.
TL;DR:
1. Configure your application to connect to localhost with a secure password
2. Enable a firewall, run "nmap <your domain>" to check if it works (should only expose HTTP(S) and probably SSH if you did it right).
3. Install nginx or Apache to handle requests for you and follow the guide at [1]
That makes any number of assumptions about the API layer, what "key based auth" is used and what data the database has.
For example I'd probably trust MySQL or PostgreSQL key validation more than what some random dev has coded in a private repo (more eyes on the code and probably better developers looking at it).
Auth is something that a lot of devs still do not implement properly and even popular libs for it like passport for node and others (including default configs for most JWT libs) have had very bad security issues.
If your minimum permissions map well onto the databases permission model then it's better to not have a layer in between. Proper db permissions and a using TLS/SSH as a transport is probably better.
Trouble is modern web-development is so dumb they just treat databases as dumb-blackboxes. So hence they end up in situations like this where security is thrown out the window in the name of minimising any obstacles to get data from the database (and no doubt dump it all into a stupid array in the frontend).
This creates a self-perpetuating lore amongst devs that databases are "slow".
But this is largely because they can't be bothered to use the database in the correct manner (correct schema design, sprocs etc. etc.)
And don't get me started on the "portable query" junk ! Sure you can write dumb queries that run on everything from SQLite to Oracle, but its far from being a remotely sensible thing to do.
https://stackoverflow.com/questions/63067062/elastic-search-...
Edit: The person raising that question is working for Atlassian (Jira), looks like Atlassian got their database deleted lol