Jesus, HN goes from zero to lynch mob faster than reddit these days.
Guy drops a zero day on a major service provider, guy gets his account suspended (temporarily, it turns out). In what possible world is disabling an account that has recently exploited your live product in a very visible way not ok? Remember, you don't have a chance to call a meeting with the C level guys and your community manager - you're one or two guys responding on a weekend.
The rest of the "oh my god the sky is falling" drivel about how terrible a bug it could have been and how they should never have had such a vulnerable bug in the first place is even worse. Security bugs are fuckups by nature - nobody sat and said well shit I was going to code this wrong but since it might allow a lot of access I won't. In terms of OH SHIT bugs this is actually rather small - I'm sure github's live infrastructure has been open to lower level remote execution vulnerabilities over the years - newsflash: we all have been. Getting user or superuser or db admin is going to almost certainly be a lot worse than an application authentication level vulnerability.
You say none of that matters because it's such an obvious bug and people have known not to do that kind of thing for years? Say hello to our old friends "buffer overflow" & "use after free" - still grabbing msft aapl & goog after all these years.
> and how they should never have had such a vulnerable bug in the first place is even worse.
Bugs happen. Even stupid oh-my-god-i-can't-believe-i-did-that bugs happen. And they happen to the best of us.
However, when someone reports a vulnerability about my code to me or I discover a problem myself, the very first thing I do is break out the grep. I grep the shit out of my code. Because I am a human being. I am a creature of habit. And if I screwed up in one place, I promise you, I did it in other places too.
The problem? Github didn't do that. At least, that's the impression from the information coming out. The guy reported the issue on Friday, they fixed that specific instance of the issue ... and it remained a problem in other places. That is unacceptable and unprofessional. They should have burned the midnight oil and made sure the same problem wasn't prevalent in other parts of the code.
Having said that, I am a loyal Github client and will remain so. Every service provider I use gets a once-a-year-screw-up credit. Github just used theirs. Switching because of one incident is premature and will be sure to cause regrets.
They should have burned the midnight oil and made sure the same problem wasn't prevalent in other parts of the code.
I understand that's the feeling here, but it's unrealistic. I've reported dozens of bugs to shops that ranged in size from 1 to borg. You simply never see a whole set of bugs fixed and pushed live over a weekend. Not even close. Exactly what company have you seen set this standard for professional? The smallest amount of time between notification and public disclosure I've seen get called responsible disclosure is a month. Many, many, many enterprises take six months or longer to push fixes (which is far too long but that's another whole discussion)
However, if such a problem was reported to LedgerSMB here is how we handle it:
1) Scope out the problem. What's affected? Are other related open source projects affected? How bad is it? This itself can take a bit of time. We do not rush this because we don't want a full disclosure when it happens to bring other problems into the fore.
2) Within a few days we let the reporter know what we have found and give them a chance to offer feedback.
3) Then we get everyone on the core team together and talk solutions. Once we implement it, we test and release a patch. Two weeks after the patch is released, we release a full disclosure along with a hat tip to the one who discovered it.
The whole process takes time. But the fact is that it's generally better to get a full fix out in two weeks than a partial fix out tomorrow.
So no, don't burn the midnight oil. More speed, less haste.
And this whole thing really doesn't look good for Github.
We didn't have the luxury of time. We were going back and forth with him over the weekend, and he pushed the commit 2 days after the original report. I admit, I was having some issues understanding his vague reports and went down a different rabbit hole early Sunday morning.
I definitely wanted to do a more careful audit and investigation, but my hands were tied.
I don't think anything will happen to Github or even Rails users. There user base will not likely decrease.It will increase infact.
More importantly people reading this forum are very scanty in the larger picture of people writing software around the world. Software and tools get used in 99% of cases for average programmers because of jobs,popularity and usability reasons.
I bet of all the Rails programmers, and Git hub users very few would have heard the news. And of those few who have heard it very few would understand the seriousness of the issue.
The only benefit that has come out of this, is a serious problem has been averted. If this vulnerability had been discovered by some body who makes his bread and butter cracking systems. He could have done a great deal of damage to a lot of systems.
Lets hope that Rails finds and fixes these sort of vulnerabilities. To avoid bigger problems in the future.
"The guy reported the issue on Friday, they fixed that specific instance of the issue ... and it remained a problem in other places."
If I have correctly understood the issue, Igor at first informed Rails, and this was the right thing to do. He was ignored by Rails and then he wanted to proof his view point by applying it to GitHub. How could GitHub inform other places? By contacting Rails, but they already downplayed the issue...
"What I want you to see in that thread I mentioned is the way the core team perceives this. You are not discovering anything unknown, we already know this stuff and we like attr protection to work the way it is."
After reading for how long he tried to bring attention to this and only got a top guy to say that kind of stuff. The guy who hacked is not right in any way but I don't even have words to describe the person who wrote the above line.
I'm not condemning anyone, but it seems that when someone points out that your framework ships Insecure By Default code with absolutely no warning in the generated code, you'd take that seriously.
Instead, the thread is full of "We've discussed this before, we like it the way it is" and "Rails is not responsible here."
Man, I've just read through this whole incident and this is the exact same conclusion I got. Fair enough, he got his account suspended, they investigated and they reinstated his account. But the comments from the devs and admin is just disgusting, acting like this is a non-issue and calling the guy a troll. This is not the kind of attitude to have. I understand the homakov probably could have acted a bit more "professional", but not surprised he acted this way after the way he was belittled. He honestly comes across very shocked he was able to do what he did, and from the comments its obvious this was made a bigger deal by the users than it was by him.
I am not a rails developer, but I am pretty sure this would have woken up a lot of people for a lot of similar issues, there will be a mad flurry of devs rechecking a whole bunch of code. Had he reported this disacreetly what are the chances of this much publicity have been generated?!?
I thought there were two different issues, though not being a rails jockey I could easily be mistaken. Even if it was disclosed four days before it was exploited I stand by my take if not my terminology.
Nope, this bug is exactly what he used for his demonstration (and there are warnings about attr_accessible going back 3 or 4 years, so it's not a "0-day" by most accounts, more of a "3-years"" vuln)
You are not getting the point about this situation.
The rants that are coming in are not about what has happened, but what could have happened.
Imagine a situation if somebody had deleted all the data or worse committed malicious code to important repos. And then used it create a bigger mess later it would have been disastrous.
If this can't be taken seriously I don't know what can be.
What exactly would you require to have happened to demonstrate that github was taking the problem "seriously" enough?
Should they have shut down github.com entirely. Tweeted in all caps every 15 minutes until they fixed the problem? Called a televised press conference? What?
I'm just pointing out the seriousness of the situation.
Punishing Egor Homakov in this case is a classic example of 'No good deed goes unpunished'. Had this vulnerability been found by somebody with evil on his mind. We would be having a very different debate now.
We all make mistakes. And I don't really blame Github for this. But we must at no cost downplay this incident.
I was on Egor's end of a similar incident back in my more impulsive years. I wasn't treated quite as well as he was but even so I eventually came around to the realization that being overly confrontational is never the right way to gain attention for security issues (even though it did result in the issue being fixed rather quickly).
The LedgerSMB project started in a similar shitstorm. I found an ability to forge credentials in SQL-Ledger. I submitted it. I went back and tried a month later on a new version (no communication from the SL author) and it was a little harder but not too hard. I exploited again, sent another email, was told to bug off.....
I tried to get the issue fixed for six months. I finally gave up and forked. When we forked we issued a security advisory publicly and stated we would offer a full disclosure soon. That's when the shitstorm started in earnest. I was accused of fearmongering. I was told I didn't understand security, that the software was plenty secure, and many choice lines that out of professionalism I will refrain from reposting to this forum.
Dozens of emails.
The end result was that Dieter fixed SQL-Ledger shortly after the fork, because those who stayed behind made him. It would not have been fixed without the fork.
After reading all this including your story,Personally I will never report any security or other incident to anybody(If I happen to find one).
Because no good deed goes unpunished. No one appreciates what good caused by your help.
It just bruises peoples egos and they violently lynch you for 'How dare you point a mistake at a genius like me, you should have tried whispering in my year'.
Its for incidents like this people refuse to help in not just security situations but also in emergency situations because you get entangled in unnecessary mess. People just let the world burn.
"Houses aren't very secure, here's a video of me picking the lock on my own front door."
"I demonstrated how insecure your house is by picking the front door lock and leaving a note on your bed."
Sometimes it can be difficult to have the empathy and perspective to see how frightening and unconscionable the 2nd action can be, but it very much is.
"But thats exactly why I left you that note. Because it frightens me just how insecure your house is. I care about you and don't want to see you hurt. I did it as a last resort, I tried to inform you but you clearly didn't take me seriously.
Empathy was casusing me pain everytime I saw you 'lock' your door with that elastic band. Attention seeking or malicious behaviour would have been to break into all the insecure doors on the street.
I broke into yours, so you would take security seriously, because I care about you and your wellbeing."
How is that so very different from, say, kidnapping someone's children and holding them hostage until they fix whatever you want fixed?
The problem here is that when you violate someone's trust you change the landscape. People get scared, they question your motives, they go into a fight or flight response. Yes, this sometimes results in the problem being fixed faster because they are very much more motivated now, but the same is true if you kidnap their family, right?
If you think someone is letting down their customers by not responding fast enough, then you go public. But violating trust is a quick way to end a professional relationship.
Surely you and every other commenter can see that breaking into houses and kidnapping children are in a completely different universe from posting comments sent from 1000 years in the future on Github right?
> Guy drops a zero day on a major service provider
This was anything but a zero-day, and that's the whole reason people are mad. Homakov very clearly made an effort to get this patched before exploiting it before taking action himself. They're not mad that it wasn't fixed immediately on Sunday; they're mad that, on Sunday, there was still a problem in the first place. Big difference.
Give me an F'in break. I understand security is not something to take lightly, but no system is infallible. There was an oversight, plain and simple. It is debatable whether the Github/Rails Core Team was too lax, but I for one am tired of hearing developers whine and make a witch trial out of groups of developers that have moved the development community forward several huge steps just to make themselves sound smart or feel fulfilled. If you're such a hero, why didn't you discover this loophole? Please stop writing provocative statements and behaving as if the sky is falling on top of your head and the very fiber of our being is at stake. An open source language and a website written in that language were shown to have a flaw. Which has since been fixed. I hate developers who like to sound smart at the expense of somebody else. Get over yourself.
I also don't see how you can blast Github for its oversight of this issue but then defend the "hundreds of thousands" of sites that use Rails. Aren't they as culpable? Oh, I suppose Github is held to a higher standard than the rest of the dependent apps. If this is Github's fault, then it is also every other developer's fault who doesn't by default disable mass-assignment of attributes.
You're conflating two issues here. He's arguing that the Rails team was ignoring an important issue by noting it was an easy end-user fix, and GitHub overreacted by suspending him after he tried several times to bring it to their attention, and then grossly mislead their user base as to the extent of the issue (which sounds like a really fundamental security issue that any professional Rails developer should know how to avoid) and how they discovered it (i.e. they didn't discover it, they had to be shown it). You're responding as if GitHub was simply being attacked for not fixing the issue.
Did he notify github directly, or did he notify the rails team in a github-issue in the rails project?
If he didn't actually contact github directly and just assumed they would see the rails issue on a weekend, then I wouldn't exactly call it 'notice'.
If he did submit it to github via the proper channels before taking action, and nothing was done within a reasonable timeframe (eg. not just an hour or two on a sunday), then what he did would make a bit more sense.
It's sort of a sticky situation. My take on it wasn't that this was so much to screw with GitHub as it was to show the Rails team the true severity of an issue they downplayed. It gets a bit circuitous because Rails is hosted on GitHub and GitHub is a Rails app. Had he reported it to GitHub and they patched it, which is arguably the proper thing to have done, nothing would have happened on the Rails side of the equation. All told, I think he did the right thing.
The reason for exploiting github was to get the rails issue noticed, not to notify github of a vuln. If he quietly notifies github then there are still thousands of github sites out there that remain vulnerable. By exploiting GitHub and getting this huge response he allows the news of the vulnerability to get tons more coverage than it would have otherwise.
Fair enough. Ignoring the issue and not fixing it after it was brought to their attention to me is where they screwed up.
As for the account suspension, I'm not sure I agree that the account should not have been suspended. Github is a code repository first, and I don't think they have an obligation to keep people around who are exposing security flaws by notifying the entire community. As the author of the post points out, hundreds of thousands of apps rely on Github, so to an extent it is their responsibility to block people who may jeopardize their users. Let's say that they left his account active, and then two months from now he exposed a larger security flaw by greatly damaging a users' app or business. I'll bet there would be more posts like this one blasting Github for not suspending his account. Personally, I think they should offer him a job.
As the author of the post points out, hundreds of thousands of apps rely on Github, so to an extent it is their responsibility to block people who may jeopardize their users.
But they haven't blocked him. They blocked his account, so all he has to do is create another.
Github has put their users in far more danger by being dicks to a guy to gain nothing.
In what way do they have to prevent him from ever accessing the site from any account ever again? The best they can do is suspend his account per policy while they are investigating.
There are two issues with any exploit: (1) prevent future exploits and (2) making sure that whoever discovered the exploit hasn't retained any unauthorized access.
Fixing the bug addresses (1) and suspending his account gives them time to address (2).
But the thing here is that if he genuinely wanted to retain unauthorized access, he had at the very least several days to create a ton of alternative accounts to make use of this exploit with.
Suspending his account wouldn't have affected him if he was being black hat about this. A suspension in this case serves pretty much no purpose other than to make Github feel better about themselves.
Aye, that's where it gets hairy – I (or anyone else) could write equally impassioned blog posts for making him their chief security officer, or for suing him. The article starts going down the road of playing Monday morning quarterback with that issue, but it's short enough that it's acceptable. Not looking forward to the avalanche of posts on either side of the debate, though.
I agree, and quote: "Please stop writing provocative statements and behaving as if the sky is falling on top of your head and the very fiber of our being is at stake. An open source language and a website written in that language were shown to have a flaw. Which has since been fixed. I hate developers who like to sound smart at the expense of somebody else. Get over yourself."
I have lost all trust in GitHub, and not because of the vulnerability, but because of their response. With their suspension of hamakov's account and deceptive blog post about the extent of the hole, GitHub has guaranteed that they won't be the first to know about the next vulnerability (and there's always another).
I've downgraded my paid account to a free account, and won't keep any non-public data on GitHub in the future. I had a similar response with my (non-paid) DropBox account. I guess I didn't rationally evaluate cloud resources, and have trusted far too many people.
The way GitHub reacted (blocking @homakov) is just wrong and destroyed all my confidence in them. Even more so when it was pointed out that @zedshaw crashed GitHub and didn't get blocked. http://sheddingbikes.com/posts/1306816425.html
Edit: Given that they have now stated that suspending @homakov was only temporary I no longer bear any ill will towards them.
I'm still disturbed by their security practices though. I expected better by the github guys and I don't like what this implies about the rest of their App.
(And yes I am using attr_accessible and not attr_protected since its inception)
Oh wow, that nickmartini story is really something. Trolling through github ? Just sad.
And github forgetting the block user functionality makes me think they don't want to listen to user needs. Sure, it may not be a very popular request, but I bet for a minority it's the most important.
It is rails guys, not ruby guys. I am still wondering why people can't distinguish two different communities. You don't mistake python and django or php and kohana or java and strut, right?
It doesn't work that way. Rails is a major reason for the kind of push Ruby got.
There was a time when CGI and Perl were synonymous. You won't believe how many people have a similar opinion about JQuery and Javascript these days.
A few days back I wanted to use a object oriented language for a big project. Generally I straight away go and use Perl for all my experiments. But since this time I wanted Java programmers to be working with my project later I thought let me use Python as its more syntactically closer to Java. When I started coding, my manager peeked over my shoulder and asked if it was Python in which I was coding I replied yes. He immediately asked me to stop writing in it, as he thinks writing in 2.x is waste of time as it is going to go away, 3.x is not yet having all the libraries. And writing 2.x will force a huge rewrite effort later. I tried and reasoned enough to convince other wise. But alas, it didn't fly.
That is how it works with pointy haired managers. They read something some where and then hold strong opinions about a particular technology.
As programmers we can try and educate people in forums like these.
But managers don't read these forums. They are likely to read magazines from IBM and Oracle, where XML's are glorified and eclipse is presented as the biggest productivity booster ever. Unless we get a forum on such magazines, we won't be able to make much difference.
You're right of course. I've honestly been trying to avoid the Rails community for as long as possible (I'd like to avoid being associated with anyone who would call himself a "Rockstar Programmer"), and with Rails being as big as it is, I've pretty much avoided Ruby entirely.
Ruby community is quite reasonable. I usually avoid discussions in framework communities. At the other hand, programming language communities tend to be more reasonable and nerdy. I feel bad that you missed the chance to know ruby language due to anti rails.
I've coded in Rails since forever (0.5) and I honestly don't see how that dongml thing is funny. I'm not even able to understand how it is supposed to be funny, is it the dicks?
For sure I think that a language shouldn't be avoided for something different than technical reasons, and nothing else.
I code a lot in Ruby and have never touched Rails... Initially I got into Ruby because metasploit is written in it. Ruby might be popular because of Rails but it is not relevant to everyone who codes in Ruby.
I think this Depends on the country or language somewhat too - many ruby guys in Japan are another breed entirely (using it as a playground for esoteric languages, quines..)
They suspended the account, they didn't cancel it. On a Sunday morning when you're fixing a security hole and you know who penetrated it you suspend that person's account. On Monday morning you figure out what to do with them.
"On a Sunday morning when you're fixing a security hole and you know who penetrated it you suspend that person's account."
Haha what. Do you maintain any sites? Tell us what ones. I want to warn all of your users that the admin is someone who will disable an account of someone who committed to master on a project that is not theirs and feel he has accomplished something.
We suspended it after fixing the bug to make sure he didn't retain access to something he shouldn't. We rarely do this, but he wasn't upfront with everything he was doing on the site like people that disclose vulnerabilities responsibly.
Oddly, I don't feel that GitHub is being upfront with everything that's going on here. It appears to be a very complicated story, but it was presented in an entirely different manner. I recognize that it's difficult to present a coherent story when everything is playing out in real time on the web, but the blog posts seem to have shared just the wrong amount of information.
> he wasn't upfront with everything he was doing on the site like people that disclose vulnerabilities responsibly
If he was, he wouldn't have really made his point, would he? There are a lot of other Ruby sites which have this sort of bug.
I actually do understand suspending his account, and I even kind of get the intentionally misleading communications (you have to keep the corporate customers happy, right?), but the new "white hat" policy thing is kind of silly, because that's clearly not what this was about.
Well, you should have just stated that in the first place instead of name calling and acting immature. A simple "we have suspended his account while we carry out further investigations, this is practice" would have gone a long way to ease people's minds and kind of reactions you got. Instead, Github just came across as massive douches.
EDIT: On second thoughts, I think I am being overly critical on all parties. Instead, I think I hands up "we messed up, but we are learning" approach would be better, and we see how both sides act if/when this happens again, lessons learnt and all that!
I'm just going to latch on to this comment to make mention of a GitHub alternative for private repositories: http://repositoryhosting.com/
I've been a happy customer for a while now, and have seen them recommended on HN many times. You get unlimited repositories with unlimited users for less than the cost of GitHub's cheapest 5 repo plan.
My open source code is on GH, but it's all also pushed to RH, along with all my private code.
IMO the best location for private repositories is your own equipment. Or rented equipment but with the private code and data on encrypted block devices or filesystems.
I know it can be a faf to setup proper reliable secure backups and so forth (though with git it shouldn't be too hard give the whole thing is designed with wide but efficient distribution in mind), but if you stuff is sensitive enough (in a business sense, some other financial sense, or for more personal reasons) to care about keeping private then I would think twice before trusting a third party with the data. No matter how trustworthy, reliable, and secure they try to be, every one makes mistakes.
Maybe I'm just paranoid. Or just plain old fashioned. But "everything in the cloud" just scares me. Keep public stuff on public services by all means, but keep your private stuff under greater control.
"No matter how trustworthy, reliable, and secure they try to be, every one makes mistakes." Which can include you. I get that it'd be pretty hard for anyone else to access code that is just on your own laptop plus encrypted off-site backup storage, but once you get to the point where you need to collaborate with other people and need some sort of a service available on the web, I'd put more trust in the security chops of a trustworthy third party than I would in myself.
What makes you think they are saints? About a year ago I discovered that they didn't protect attachments to tickets in private repositories (since fixed). Anyone who could guess the URL could access the content. (It looked like the cause was keeping the attachments in S3 without front-ending them.)
On contacting them I was told it would be fixed in a day or two, and that it was no big deal since you had to guess the URL. The values you had to guess in the URL were a ticket number (they start from 1), a repository name, a date (YMD) and a filename. Sure there is some variety in there but it is not in the billions of possibilities, just hundreds and that won't make any computer break into a sweat and in my words at the time "easy". To make matters worse you can't delete the attachments to tickets.
This may not affect you. It certainly affected me. For example we had some keyfiles in one ticket. Coredumps in others.
Two weeks later the issue still hadn't been fixed and I don't know when it was. I've never seen disclosure of the issue. There wasn't even any way of knowing if attachments had been accessed in an unauthorized way since there was no checking in the first place.
I prefer to pay someone to host the private repos. No free tier means every user is treated like a paying customer, and every user's data is considered valuable. It also means that as long as the pricing is sane, the company isn't going to shut its doors for lack of revenue. Before RH I was paying Springloops.
It's $6/month to take care of my most important assets. That doesn't even buy a meal at McDonalds anymore; it's worth it.
> as long as the pricing is sane, the company isn't going to shut its doors for lack of revenue.
BitBucket is owned by Atlassian for 2 years now and they don't look like they're about to go out of business ($60mil revenue in 2011, 400employees, worldwide offices). When they bought it, it was them who set private repos to unlimited (when @jespern was running it alone, it was set to 5), so I believe they know what they're doing and that it fits their business model. Over 5 collaborators need a paid plan and that's just how they set up the pricing scheme.
As far as I can see as an active user and lurking through their blog and bugtracker, BB is actively developped, so I assume you don't have to worry about them running out of business, looks like it's a valuable asset in Atlassian's software portfolio (at least valuable enough to work on it).
Except when they don't handle security well, and you end up with anybody that feels like it capable of reading your private source code... Like, say, NOW.
I used to use them years ago and it certainly wasn't issue free. When they took away their free private accounts it was a pretty big pain only to have them re-implement them less than a year later.
This is a perfect summary and exactly the sentiments that I couldn't express in the article. I've quoted you with attribution if you are okay with that?
This is probably the perfect time to switch to http://bitbucket.org/ . They offer free unlimited public and private repositories for both Git and Mercurial. However, the free private repositories are limited to five collaborators.
^ this, I just moved my prive repo to bitbucket hoping that Atlassian would have handled this issue better. At least they seem to have more experience looking at their long history in software. I will also look into setting up my own git(orious) server.
I personally know some people that work for Atlassian and they are awesome. Moreover, they really do care for their customers. I have no experience using Bitbucket, but given the people I know I would definitely trust them.
I have to agree that suspending the account was dumb as if he wanted to be malicious, he'd have created dummy accounts and would have committed malicious code to popular projects.
But then, going so far to say losing all trust in github.. there was a bug, it's fixed and they suspended the "attacker". I wonder what will happen in the following days with script kiddies trying to "hack" all rails website. In fact, isn't the overloading of parameters something as old as the earth that security experts would check first as a vulnerability?
Not to downplay the exploit at all, but for everyone raging that this occurred and are worrying about the integrity of their codebase, we are talking about an SCM for god sakes aren't we? Git is distributed and all most users would need to do who are freaking out right now about evil commits is to go back and audit your git commit history no?
> When the large portion of the technical world all depends on a single service, and that service is vulnerable to a variety of attacks, that makes anyone who consumes these services also vulnerable.
I don't mean to diminish the severity of this exploit, and the impact it has/could have had if left unchecked.
BUT, isn't one of the biggest perks of Git the fact that it's a distributed SCM? It's not a service where you must trust all of your data with the one provider, who might go belly up at any point and take it with them.
Yes, GitHub provides some fantastic social features and helps with community involvement through these features, but if you are dependant on a "single service" and you're using a DSCM/DVCS, you should probably look at a few alternatives to reduce that dependancy.
I agree wholeheartedly. I always facepalm when people aren't able to work because the internet is down (read: no github access). Just start an SSH server, netcat some keys around (or use an USB stick), and get back to work again :)
Transmitting files via Google's email servers from two computers on the same network in the same office in the same room, especially if you're in europe (ridicilously far away from GMail's servers) instead of directly between the computers on the local network is one of my main pet peeves..
While GitHub's uptime has improved immensely, it was a serious problem. It's not as simple as pushing your repo someplace else when you use submodules. If anyone has a fix for that I'd love to hear it. I've ceased using submodules in most cases precisely because GitHub used to go down so frequently, but now I have the same problem with anything that's a :git entry in my Gemfile.
One workaround for submodules would be local DNS entries, i.e. edit /etc/hosts to make github.com the temporary local github.com. Also, I generally mirror submodule repos and use those. But neither of these "solutions" fixes the actual issue of course. Not using submodules sounds like the best solution to me :)
LANs are broken. Why is it that we have usable tools for connecting halfway round the world, but find it massively hard to coordinate ourselves across a small office when the internet is down. It is total madness.
I guess it's just about familiarity. We do have the tools (mdns/avahi/bonjour, tcp/netcat, ...). But the internet is usually up, so we don't need to learn how to set up our own local servers, and there's no punishment for transmitting a file across the atlantic when it could have been transferred to another room in the same building and back again.
The fact that there is not persistent caching of mission dependent code and data as a fundamental commercial standard shows how badly we as a species in general can judge risk, even when the stakes are really high.
We...don't? I mean, if you're reliant on cloud services, yeah, you'll have problems--that's obvious. But if you self-host your important stuff (I run my own redmine instance off a local machine) and know how to use SSH, you should be fine in the majority of cases where the LAN is fine but has no Internet access.
It seems so. I freelance, so have to jump into various different offices. And on the whole, a hell of a lot of small companies that shouldn't need to be that net-dependent seem crippled these days if their net connection is being flaky.
99.9% of the time, tools which work for communicating over the internet work very well for communicating locally. It's not worthwhile for most people to devote the time to building, maintaining, or learning a second tool for the LAN side for that 0.1% of the time when your internet connection is down.
True, but they are rarely set up properly. Also, when they are set up they are often only used for things that are thought of as 'local tasks' such as a media store or printer server and not for replication of the online system that you have for some insane reason decided should be hosting your accounts or whatever.
I don't think anyone is dreadfully concerned about service availability - although that is certainly a concern.
The problem is that source code is incredibly valuable and often represents many man-years of effort. A competitor could use this to bypass a lot of the early architectural fumbling around that we all do. Also, source code often contains passwords and shared secrets like Facebook secret keys. Disclosure wouldn't be the end of the world, but it could certainly be disastrous for a lot of companies.
Do people actually host confidential code on Github, vs. open source projects? I'm worried about the integrity of open source projects, but not really their confidentiality.
According to https://github.com/about, they have 59 employees and have never taken VC. Since the opensource projects don't pay, the math suggests that yes, quite a lot of organizations are paying to host confidential code.
People use services like GitHub because they provide security, reliability, uptime, protection against hw failure, centralized issues and patches, and so on. If we have to worry about each of these on our pwn, what's the point?
Akin's law #2: To design a spacecraft right takes an infinite amount of effort. This is why it's a good idea to design them to operate when some things are wrong.
> If we have to worry about each of these on our pwn, what's the point?
You should always have to worry about these things, regardless of who's hosting things. It just so happens that Github, to date, has been checking the boxes in these areas and has established a reputation for doing so.
But, if GitHub losing your repo/having it trashed beyond repair will kill your project (or severely hamper it), the cost of setting up a second Git server and a couple of hooks to mirror your push is very low, in both dollars and man-hours.
That's not how git works. I have the entire repo locally. If I didn't, I wouldn't be able to commit anything. So if Github goes poof, I'd be pissed, but I wouldn't lose a single line of code. I would obviously lose things like issues/wiki. Actually, it would be pretty cool if Github turned issues/wiki into some sort of git repo of markdown files and allow me to pull them and commit to them. Something for them to think about =)
I fail to see what GitHub did wrong here. They were attacked, they suspended the account doing the hacking, and they fixed the problem. Then, they blogged about it, explaining in detail what happened. Apparently they weren't quite reverent enough for the person who wrote this article.
Egor is a human whose motivations were totally obvious and whose actions were transparently harmless (and, in fact, net helpful.) The Github team are behaving foolishly if they can't or won't distinguish that from an "attack."
Organizations should be expected to make decisions using reasoning that amounts to more than word association, e.g. he "attacked" us, so we'll "suspend" him.
I would disagree with this, quite a lot. He brought up an issue with the Rails team, they pointed him at the canonical, "here is where we talked about this before, sorry." Still not satisfied, he found the same exploit in Github to prove a point. Rather than do the sensible thing by creating a dummy account and contacting Github showing how he messed things up, he barged into the Rails organization and left a silly commit. Github is first and foremost a business organization where lots of companies pay them lots of money to "take this shit seriously" and protect their data, so they did the right thing by shutting him down.
I honestly don't see the meaningful difference between contacting Github and leaving a silly commit, except that the former would probably get the bug fixed quietly; in contrast, now everybody is aware that the bug existed in Github and is aware of the potential for it to exist everywhere. He successfully proved his point, which apparently was a pretty good point. Isn't that a better outcome?
As for Github's responsibility: Github failed to protect people's data the minute the bug went live. That data was open to Egor since the moment he discovered the bug until the moment they fixed it. Making a silly commit did not make anyone's data more or less vulnerable, so I don't believe that "taking this shit seriously" implies flipping out over it.
> I honestly don't see the meaningful difference between contacting Github and leaving a silly commit, except that the former would probably get the bug fixed quietly; in contrast, now everybody is aware that the bug existed in Github and is aware of the potential for it to exist everywhere. He successfully proved his point, which apparently was a pretty good point. Isn't that a better outcome?
I'm not denying that by doing what he did it certainly got the word out and made everyone understand how serious of a problem this is. It was a very good point and I think the outcome is the right one. I'm just saying that Github's actions - to suspend the user who somehow got SSH rights to the rails org - is the right thing to do. They want to minimize his damage that he will do, and until they can do a full audit and understand how his commit got there, it's the right thing o do.
> Making a silly commit did not make anyone's data more or less vulnerable, so I don't believe that "taking this shit seriously" implies flipping out over it.
I fail to see how suspending a user is "flipping out" over it? I don't think you can color unauthorized commits to github repos with different levels of responses from Github. That's a dangerous line to walk IMO.
... if you want to take a US-centric view of things, then that last statement is correct I suppose... but not everyone is subject to US laws - including, unless I am grossly mistaken, the person you refer to.
GitHub's and its computers are governed by US laws, and many countries have extradition treaties. Many countries also have equivalent computer abuse laws.
My comment was to discourage such spectacular glory-seeking behavior by other people that claim to be trying to help. It's a serious crime, and the FBI does not care that your intentions were good.
He did it under his own name, in a way dad not damage anything, using a problem that has been well known for ages, after attempting to draw attention to the issue several times and being ignored.
It is pretty obvious that he did not have any malicious intent, nor intended to do damage (if he wanted to, he could've done massive damage from an anonymous account in ways that wouldn't draw attention to it).
The only thing suspending his account accomplished is to punish someone who has helped draw attention to a very serious problem.
It just comes across as an incredibly childish and petty response given the circumstances.
> He did it under his own name, in a way dad not damage anything, using a problem that has been well known for ages, after attempting to draw attention to the issue several times and being ignored.
He was ignored by the Rails devs, not by GitHub. Yet GitHub became the target of his attack, because he was "bored" (his word, not mine).
At no stage, as far as we know, did he raise the issue directly with GitHub. As I've said in another thread, he could have done so and requested they disclose the fix publicly (say, within 24 hours) in an effort to force the Rails devs' hands. Instead, he's punished GitHub (who, mind you, probably should have audited for this vulnerability a long while ago) because he was shunned by the Rails devs.
Sure, that was my knee jerk reaction too. Of course they should suspend the account! But I think that was the wrong choice.
Shutting down the one account will do nothing. He can create a second account in a matter of seconds. They fixed the code that permitted the exploit, right? So the fix alone should be plenty to prevent any more malicious attacks.
And now, because of the suspension, they have a wave of bad PR (justified or not). This guy is clearly infatuated with Github. He got a tattoo of the Octocat for crying out loud. Why would you punish this guy for loving your service and caring enough about it to warn you of a security vulnerability?
Seems like there are only minuses to suspending the account, not a lot of pluses.
I would expect any company to do far more than suspend the account, especially knowing exactly who did it.
I would think having a suspended GitHub account would be the least of worries with a company who has presumably access to a good lawyer or two and boatloads of cash...
All of my public comments on this topic agree that this is a problem that we should fix. Are you conflating me with other people? Am I not part of "the Rails team"?
The issue is that it needed to go this far before someone actually paid attention to it. It indicates a cultural problem that is not going to get addressed by fixing a single technical issue.
> Beyond any shadow of a doubt, a shit storm of epic proportions has just gone down...
> ...this episode has been handled is a face-palm fail of epic proportions.
I'm all for, like, the evolution of language, but can we please all, like, agree that 'fail' isn't, like, a noun, and 'epic' is, like, totally overused.
The one "good" thing that comes out of this public exploit is that github was the target.
Github is obscure enough to non-developers but quite well known in the development circle.
This means if you are using github you probably have a damn good idea what the vulnerability is. I'm not a rails developer (mostly python/django) but I get it immediately. This is mostly an issue with the framework helping me shoot myself in the foot.
Sure it's in the documentation. But realistically a good framework gives me sensible defaults so I don't have to refer to the documentation. I trust the framework does the "right thing".
"If you are one of those strange coders that don't use GitHub and think you are in the clear because you use SVN/Mercurial"
I can't believe he actually put SVN and Mercurial on the same side, or that he implies that not using GitHub must mean that you don't use Git at all. This sentence is wrong on so many levels. Sigh..
I'm strange, for my actual work I use Mercurial hosted on BitBucket.
That comment jumped out at me as well. I use Perforce at work and run my own at home. Both with local and offsite redundancy. I guess that makes me strange.
As Zed Shaw pointed out, someone appearing to be Homakov has posted a comment dating back eight years. Is Posterous vulnerable as well? If so, that may not be Homakov, of course, but his twitter comments are consistent with him posting it.
Yes, one of Egor's comments in the github bugtracker had a small (but presumably not exhaustive) list of other vulnerable sites. It included Posterous.
I'm not sure all the things you list as being possible are true.
- Every GitHub Repository could be access by anyone as if they had full administrator privileges.
- This means that anyone could commit to master.
- This means that anyone could reopen and close issues in issue tracker.
- Even the *entire* history of a project could be wiped out. Gone forever.
As I understand it from his explanation[1] he added his public key to the Rails user, which has permissions to push/pull to the repository. This doesn't mean he had web administrative access, just Git access, since you cannot log in to the web service using your private key. I hope that's the case, at least.
This is correct. People who don't understand what a mass-assignment bug is are running with this story. It's like when we witness a DDoS and have to tollerate people who think it means that the targeted party was infiltrated.
This bug allowed one to add their public key to another user's account, and make changes to comments and issues.
What are the odds that there's a similar bug which allows changes to user accounts? If that's the case, then altering the password or email address is trivial.
Nothing in the article would lead one to believe he didn't understand the distributed nature of git repositories, but a lot in the article would lead one to believe he was specifically referring to the data loss issues on GitHub if someone wiped out a project.
I'm not too worried about the ability of an attack to push a commit, or cause a commit/object deletion in a repo's history, on GH, because most smart folks (if not everybody, by default) will have multiple copies of a repo across multiple machines, with backups, so anything can be undone or restored. (And trust me, I have the imagination to understand how an unauthorized commit/push could lead to a situation enabling remote execution on client machines who've pulled down tainted commits. Think build scripts that have "install rootkit/malware/keylogger" commands added to them in the mal commits.)
What would be more bad is if this vulnerability allowed an attacker to get unauthorized read access or pull/clone access to a private GH repo.
Can anyone clarify for me whether this was possible?
Though I will say that mass assignment is about as rookie a rails mistake as you can make, I won't be leaving github.
They've delivered so much value over the past four years for me that I can forgive them this and I'd still have (a little) goodwill towards them left over.
I'm sorry, but this feels to me like an over-dramatized heap of bullshit.
First, the statement, "Rails. You clearly messed up." is self righteous bullshit at its finest. Rails didn't mess up; the programmer(s) at Github messed up. No conscientious developer lets the end user mass-assign variables carte blanche. But with that said, _every_ developer messes up every now and then despite their best efforts; some times they mess up in a big way.
Secondly, if a user discovered a vulnerability in something I wrote, and they handled it like homakov did, I'd ban the shit of them until I knew for sure that they weren't a threat.
Finally, Github handled this exactly the way many companies would handle it: it's called damage control. These guys are really good at what they do, they provide a great service and they offer-up a lot of their tools to the FOSS community.
For me the sad part is that there is an almost even split between people arguing the right way to bring this issue to every ones notice. I think this is the perfect way to show how serious the issue is and to get more sites to adopt the fix.
Exploits like this are worth a lot on black market. They are worth even more if you provide a precious and vulnerable target to go along (github).
I used GitHub and I'm not moving my stuff off. If an app gets hacked, then not long after, that app will likely be the most secure place. GitHub at least keeps it up most of the time. Who you really should be mad at are the Rails maintainers and the RoR community. I switched from Java to Ruby a few years back, and since day one, everyone using Rails has been slack on security. The reason is that they make things too easy to leave wide open. Don't believe me? Read the Rails official documentation for starting off. It is all about ease of use, not security. If you are new, you have no idea what you've really left open even when you just generate a scaffold as they show you to do. The main thing that Rails security has going for it is that the adoption of Rails is still relatively low, and because a newbie isn't likely to scale their app well, odds are you won't have an extremely popular, extremely performant Rails app that is just asking to be hacked that easily.
A system that is designed to rely on human diligence is inherently flawed. It's a mercy that drills and other dangerous tools aren't designed in the same way that a lot of software is.
Exactly. This is like saying that it was fine for older versions of windows to broadcast the users directory tree over an open file server because they should know better than to leave the server up. (This happened right? Correct me if it didn't.)
It's like no, you should have to turn the server on in the first place. The fact that it took some Russian kid in his basement breaking into the github rails master for this self evident truth to be realized by the core rails team is totally outrageous.
Oh, they've let you all down? Then stop using other people's web applications and just run your own git/hg server. With blackjack, and hookers.
You are vulnerable to someone else's fuck-ups as long as you insist on giving up control over your data and code in exchange for the convenience of someone else doing the "hard work" of development and administration for you.
Hell, restore the network to being peer-to-peer rather than hierarchical, and hosting your own whatever will no longer be such a damn problem.
The response to this makes me feel that HackerNews is now populated by a bunch of pretenders. This "bug" has been in Rails since Day 1, and any remotely experienced Rails developer is aware of this functionality. You can argue for a different default, but it's not a bug.
Github did have a bug and noone knowledgeable about Rails appears to have made even a cursory inspection of the security of their controllers - which is where attribute protection actually belongs, since different controllers and different users change different attributes. Protected attributes is a blunt tool for simple situations, which is why it's not enabled by default. Github had a pretty terrible bug, discovered, and fixed it. They may not have handled it perfectly, but the certainly don't deserve this sort of mon hatred - any competitor you go to is likely to have security flaws as well, perhaps more severe and subtle.
@homako didn't just expose the bug in github, he exploited it to make an unauthorized commit to Rails master. His account most certainly should have been at least temporarily suspended as GitHub had no idea what else he might do to prove his point.
So basically, most of the comments here are glaringly wrong or ignorant bandwagoning, and it makes me wonder about the accuracy of information here about topics I'm less familiar with. A sad day when you realize all this intelligent discussion you thought you'd been reading about new topics was probably just grandstanding by eloquent fools.
> Github did have a bug and noone knowledgeable about Rails appears to have made even a cursory inspection of the security of their controllers
> Github had a pretty terrible bug
> but the certainly don't deserve this sort of mon hatred
For all the free fun you can have on github, they are in the business of selling private repositories. What could possibly have been worse than someone finding a bunch of bugs in a matter of days just to prove a point about Rails?
What could possibly be worse? Anything actually malicious or greedy.
For all that GitHub has given to the developer community, an innocent mistake even of this proportion of incompetence is still should not evoke such hatred. And those upset about someone's account being suspended who was actively misusing security holes - well, maybe you should use a provider who looks positively upon reporting security holes by vandalizing customer data. And they suspended his account for only a few hours! Unreasonable? Hate inspiring? Outrageous? I think not.
And by the way, the fact the "issue" of the default had been reported 4 days earlier in a github issue tracker for Rails (which is certainly not followed, let alone on weekends, by github employees) does not in any way impact whether GitHub should have been aware of this vulnerability, and to suggest so is intellectually dishonest.
Normally I respond negatively to this type of post, but you've captured my feelings perfectly. The facts here are nothing that should be blown so far out of proportion. I can't help but detect a hint of schadenfreude at the idea that Rails core or Github are not infallible.
How many companies get hacked regularly like this but keep it under the rug? You think FaceBook's never been exploited? TurboTax? Mint? Stripe? PayPal? Shopify? Tumblr? Pick your app that "so so so so many businesses" use regularly, and I guarantee something like this has happened with all of them.
But were they open about it?
GitHub's been open the whole time.
Your post is like saying "All criminals are stupid". This is ridiculous, as the only sample you know of and can work with are the criminals who have been caught. You don't know how many other criminals are out there getting away with their crimes, because...they haven't been caught yet.
Who knows how many other companies have had hacks like this in the past two months alone, for example? I don't, and neither do you.
But GitHub, as an open, honest company that so so so many of use regularly (which means we know right away when there's a problem, especially with a hugely popular repo like Rails/rails) has been in the spotlight since the second this happened.
GitHub, in my opinion, has acted really cool about this. They addressed the issue, explained what the issue is, patched the hole, and even reinstated the hacker's account. DHH addressed the issue in twitter, other people in the community have admitted they fucked up, and now we as a community can work on fixing this.
That doesn't sound like "Letting us all down".
Someone who expects everything to work perfectly all the time and have no vulnerabilities is someone will be let down by anything, a pessimist, and stupid. And certainly not worthy of the front page of Hacker News.
Guy drops a zero day on a major service provider, guy gets his account suspended (temporarily, it turns out). In what possible world is disabling an account that has recently exploited your live product in a very visible way not ok? Remember, you don't have a chance to call a meeting with the C level guys and your community manager - you're one or two guys responding on a weekend.
The rest of the "oh my god the sky is falling" drivel about how terrible a bug it could have been and how they should never have had such a vulnerable bug in the first place is even worse. Security bugs are fuckups by nature - nobody sat and said well shit I was going to code this wrong but since it might allow a lot of access I won't. In terms of OH SHIT bugs this is actually rather small - I'm sure github's live infrastructure has been open to lower level remote execution vulnerabilities over the years - newsflash: we all have been. Getting user or superuser or db admin is going to almost certainly be a lot worse than an application authentication level vulnerability.
You say none of that matters because it's such an obvious bug and people have known not to do that kind of thing for years? Say hello to our old friends "buffer overflow" & "use after free" - still grabbing msft aapl & goog after all these years.
TL;DR - stop acting like children.