There are countless SaaS applications asking for full-repo access to Github (all the source code, with write access).
- Productboard
- Bugsnag
- Sentry
- Skylight
- Percy
- CodeTree
- Databox
There are heaps of others, these are just some on top the of my mind. A ticking supply chain attack waiting to happen, since these companies make themselves into alluring hacking targets.
Most of them need access only to issues (a few need read access to code or recent commits, almost none need write).
Solution:
- Let customers give granular access (only issues, only read to source code, etc) when the integration is setup. This is possible with Github's APIs.
- Try to use push instead of pull where possible, i.e. provide a CLI tool to use with Github actions or use Github's webhooks.
> There are countless SaaS applications asking for full-repo access to Github (all the source code, with write access).
Sentry does not request write access to source code. It requests read/write access to issues and read access to source code. You can also see this on the documentation for the GitHub enterprise integration which lists the exact permissions required: https://docs.sentry.io/product/integrations/source-code-mgmt...
It would still be better if I didn't have to give read access to source code, but could still use the issues integration. But I agree it's not as a bad as write access to source code.
I know it's more work, but they could also use Github Actions to push recent commits (past 90 days) to themselves, and use them. That way they wouldn't store or have access to all source code.
For a while now I've been worried about this -- either integrations asking for full read/write access for a service that might seem to need only read.
Or maybe worse, integrations asking for access to anything my account has access to, when I only want to grant it to one repo or organization, or only to public repos and not private ones.
Whenever I've reached out to inquire/complain about this, I've been told that github does not give them granular enough auth settings to ask for less than this.
Is this true? I don't know. When I've tried looking at the relevant github docs myself, i quickly get confused.
Does anyone understand the github auth architecture -- does it need to be fixed to allow more granular access, or are integrations just not using it properly? Like... who should I be complaining to?
I've been looking into this as I'm building similar per resource scoping for conjure.so's API.
GitHub has 'OAuth Apps' and 'GitHub Apps' [0]. The former's scopes do not permit such granularity (eg the `repo` scope gives access to all repos of the account [1]).
The latter is much more granular, allowing the user to select specific repos to grant permission [2].
The 'GitHub App' owner can see their installations and also determine if the user chose access for all repos or on a per repo basis.
Netlify does such granular installation and will prompt you if you don't see your repo listed in their dashboard to check permissions.
Do you think that still sounds much more confusing than it needs to be?
I don't totally understand why they offer these two mechanisms of integration -- or why the granularity of access would need to be different between them. Why not let "OAuth apps" be authorized per-repo -- if you are going to have "OAuth apps" existing as a thing still? And they don't seem to be deprecated in favor of newer "github apps", right?
Yeah agreed its pretty confusing and took me some time to reconcile!
I imagine for legacy support or business reasons they maintain the OAuth API, but I too wish they add granular repo support to the OAuth API and existing tokens were migrated to an all repos scope, while new tokens were encouraged to be per repo scoping.
> Additionally, we recommend disconnecting Heroku from your GitHub repositories.
I have respect for the Heroku/Salesforce Security team for willing to ask users to perform this action. Many companies would be too worried about losing customers or having users not reconnect it afterwards.
My thoughts are with the team working on responding to this incident on Easter Friday.
I hope we get some more clarity on the extent of this incident soon. We'll rotate our keys anyway but I really hope the attackers did not have access to the ENV vars that are commonly set on Heroku directly.
For many accounts the audit log isn’t going to show some crucial information—git activity. Clone events are only available through the REST API for Enterprise customers.
I can confirm that ~2h ago my integration was working but now it's not, so it seems they automatically disconnected everyone. Not sure if "they" is Github or Heroku here, but my master branch is not autodeploying anymore and trying to reconnect results into an error.
"Github System" removed the Heroku Dashboard key and a personal user account of someone that seems to work at neither GitHub nor Travis removed my Travis CI key (????)
If you saw the user 'shairyar', he works at Travis based on linkedin profile page (but not based on github profile page). I was confused as well when I saw that.
I guess maybe, but this is incredibly badly communicated. Just now they showed up as a member of travis GitHub orgs, but the only member? https://github.com/orgs/travis-ci/people
It's very possible (likely) this is all fine, but done in a way that feels strange/fishy. I haven't even gotten an email from travis. They must be in fire-extinguishing mode.
We're a small org with a github connected to heroku. All of our repos were cloned between April 8 and April 15 with the majority of them having no activity for several years. The audit logs don't show this, you can only see this information in the traffic graphs (/graphs/traffic). If you're seeing cloning of repos that you haven't touched in a while, you've likely been compromised.
For anyone not on a pro plan: I believe you can upgrade and still see the past two weeks of data. I cloned a few of my private repos last night to see how that affects my security logs and no logs appeared. I later upgraded to pro and visited /<username>/<repo>/graphs/traffic and can see the clone counts from before I upgraded. I also can see visitor counts from about a week ago. These clones still don't appear in the security logs though.
I tried upgrading my org to a Team to check traffic, but the upgrade seemingly did nothing. I do have sponsorships, so maybe it’s waiting until my next billing cycle? Can’t figure it out.
Are you on an enterprise plan or a personal one? I know personal plans have limited logs, but I'd hope the enterprise ones would show clones (in reality, they both probably should). Kind of defeats the purpose of an audit log if it doesn't.
Ughh here I was at 1:30AM after a hard week, checking the news one last time before going to bed, and see this. I hate this dang industry and regret ever becoming a tech lead. I know the Heroku engineers have it worse and all, but just venting.
News on HN are usually not relevant to my work and I actively avoid reading anything work related on my off time. I’d find some other activity to relax and wind down if reading HN has an impact on you. Burnout is no joke - you need yourself rested for yourself and your team. Take care of yourself.
I'm not a security expert, but if you're reading this and wondering what to do, a good start could be to just assume your repo was accessed, and so to run a tool like gitleaks against your repo. If it detects anything sensitive, I'd see about revoking/deleting those secrets right away.
In general, it's good practice not to check anything sensitive into source code for precisely this reason (if your code is compromised you don't want your secrets to be as well). So it'd also be good practice to add something like gitleaks into your CI/CD pipeline for the future.
It's such a cryptic call to action. When people reading an incident disclosure end up wondering what to do, you're not finished writing. I haven't logged into or even used Heroku in close to a decade. Am I affected? I just toyed around with a hobby project on Heroku those years ago and didn't release anything. Am I affected? If that project didn't connect Heroku with GitHub, am I affected? It's been so long, I don't even know how to use Heroku anymore or even access my repos there. Yet, I got this message in my E-mail so that would imply that maybe I'm affected and need to do... something? Who knows, the E-mail doesn't say what to do either.
Sounds like the best course of action for someone like me would be to just to log in, find any button that says "delete everything" and use it. So confusing.
> Sounds like the best course of action for someone like me would be to just to log in, find any button that says "delete everything" and use it. So confusing.
Yet that isn’t even close to the best course of action. If your GitHub account was compromised and you had secrets in private repos, deleting everything in your Heroku account isn’t going to do anything to help.
A few days before this attack campaign started, I wrote a guide to help security and engineering teams prioritize and remediate thousands of secrets-in-code incidents.
Hope it can help some of the organizations dealing with this right now!
I do remember hooking up Heroku to Github for auto-deployments and thinking to myself something along the lines of, "why does Heroku need ALL of this access?"
It'd be great if Github could allow read/write permission grants on a per-repo basis. Maybe they do already!.. in which case I'd much rather have and setup that granular detail than have a token that goes across all my public/private repos...
Edit: I do see in my Github's integration page that the Heroku connection was used within the past week... but it doesn't show how exactly it was used. Until Github can provide specific details, is it safe to assume that all repos, public and private, could have been cloned?
Thanks and I'm not surprised, it's a pretty intuitive feature! So really the only thing Heroku gets from these all-inclusive tokens is something to drive their type-a-head input on their integration page, right? Totally not worth it, I'd rather use Github's prompt.
Disappointed that the GitHub security log doesn’t show access for personal accounts. Would be rather nice if they temporarily made that available for a short period of time so we can see if any of our repositories have been cloned/downloaded.
I've been disappointed with GitHub's auditing capabilities.
I noticed that some members of my team were committing changes to our enterprise repositories with poorly-configured clients, so it was impossible to tell who made the commit or performed the push.
I understand that git itself doesn't have any protections against this, but GitHub knows who pushed it; why isn't that metadata available?
Do you know whether that's available via GitHub proper? I'd like the user identity for commits to be accessible through the API, but last I checked it wasn't.
(Updated for clarification: user identity is available via the API, but only the identity embedded in git itself, not the GitHub authenticated user.)
- Heroku might be using Travis?
- They were compromise independently but the hackers decided to start leveraging them at the same time to get the most out of them?
- The obvious common denominator: Github? It doesn't make a lot of sense though.
The attacker got _write_ access to all of these repositories as well? That's extremely worrying. I hope github or someone will be able to track down if any code changes were made.
A forced push could introduce vulnerability into code in a years old commit that is hard to detect. I guess it will look dodgy when your next push is rejected.
Honestly should be pretty easy to just compare remote to someone's recent local copy and just spot check any new commits. Definitely a time sink, though
If I want to revoke all Heroku's access to Github, is it the "Heroku Dashboard" I'm looking for under "Authrorized Oauth apps"? I revoked that one, but not sure if that is everything.
Credentials and other secrets, like API keys, should never be hard-coded in the source code repo. Use some sort of secrets management or configuration for that kind of stuff.
In this case, can we confirm that Heroku environment variables were not accessed? Because if they were, even not storing secrets in the source code wouldn’t have prevented a breach.
If Heroku could confirm environment variables were safe I’d have a much better sleep tonight.
>The compromised tokens could provide the threat actor access to customer GitHub repos, but not customer Heroku accounts. With the access to customer OAuth tokens, the threat actor may have read and write access to customer GitHub repositories connected to Heroku.
Based on the above information, my assumption is that the attacker gained access to code repos hosted on GitHub, but not access to live dynos or the Heroku dashboard (that happily shows ENV vars). We'll see if this information updates as the investigation progresses, though.
However, given 1) write access to a github repo, and 2) auto-deployments from github to production dynos (if enabled), an attacker could exfiltrate env vars (among many other nasty things). However, this would trigger events in your app's activity log (new commits + deploys) and should be quick to verify that it didn't happen.
The announcement claims that, as far as they know, the attacker can't access your Heroku account, only your GitHub repo (because they only got the GitHub-issued token). So this would imply that your environment variables are safe. That being said, it sounds like they don't yet know how the attacker managed to do what they did, which means they can't rule out the possibility that more data was stolen than is currently known. But it sounds like probably not.
at work thats fine, we use vault or secrets manager, but neither of those are really suitable for self hosting for toy apps. What do you tell someone using github's free tier and aws/gcp's free tier?
You've just moved the problem from storing the secret to storing the key in that case though. For AWS KMS, you're now paying [0] for the key storage, and if you use a GPG key, you still need to get it into the application somehow.
If you're using a GPG key, you don't necessarily need it to leave your PC. You just need to "re-deploy" the secrets manually when they change from a place that has access to the key
I’ve been using Render since January, after switching from Heroku for a project of mine with about 10,000 direct users at present. The project is a Rails app that imports hundreds of thousands of rows of data from XLSX files and spits out PDFs on demand from those imported spreadsheets.
I was worried that Heroku would end up costing me a small fortune as demand scaled. Plus the platform seemed to have stagnated.
I contemplated switching to AWS, but didn’t want to deal with the extra hassle of it. By chance, I saw someone mention Render on here, checked it out, and couldn’t be happier.
It’s a bit harder to get up and running with Render than Heroku, but orders of magnitude easier than with AWS. And once you’re operational, it’s a cinch.
(Render founder) Were you trying the free tier? Builds are significantly faster on paid plans. I'd certainly appreciate more context: email in profile.
Hi -- yes this is on the free plan for clarification of the original comment. Good to know. I was testing out the free plans until I upgraded. https://render.com/docs/build-limits I don't see this mentioned anywhere. https://render.com/pricing should maybe show a row/column on the build speed for the different tiers.
Additionally, spinning up the postgres instance was on a paid tier. I contacted support but it self-resolved eventually after I blew away the instance and rebuilt. However, it was queued up for like 8+ hours on the first pass.
I have always wondered if putting confidential info in even private git repo was a good idea, although it seems to be a common practice? I feel like that question has been answered, for me anyway.
Instead of connecting to a Github repo, you can use https://github.com/heroku/heroku-builds. It allows to create a build locally and then deploy it to Heroku. From what I've read so far, this approach has not been impacted and it should still be possible to do deployments like that.
What about simply using the heroku CLI which is the default? I've used that for every heroku project, I didn't even think to connect to github. I suppose the main benefit is that it's automated.
I just started a Discord server: https://discord.gg/K9ecetqn Please join if you're impacted by this incident, or interested in these topics in general.
I'd like to discuss mitigations around this and similar incidents with other HN:ers:
- Knowledge sharing: resources, how-tos, tips
- Discussing prevention, mitigation, etc
- Moral support and venting
If there's already such a forum (I assume there is), please send me an invite :)
The only thing in my security log over at GitHub that I'm not familiar with is a handful of "repo.change_merge_setting...Blocked a merge setting on the ${my_repo} repository" entries. Googling that provides little information. Can anyone explain what that means? Should I be worried?
That log entry happens when you change the settings on a repo for what kind of merges are allowed for PRs (merge commit, squash, rebase). It’s three checkboxes under “Pull Requests” in the settings for a repo. I can’t speak to any reason a threat actor would have wanted to change those settings though.
Heroku users using GitHub should start rotating any secrets stored in their repos (people still do), if the OAuth has been compromised it means your repositories could have been cloned. Non-enterprise GitHub users have no detailed audit history to see if their repos were cloned/zipped.
Last year when I looked into this there was no automated way to get this info; the normal audit log mentioned below does not contain any info about actions from oauth-ed applications afaik.
If you email GitHub support they can pull out detailed logs from oauth app interactions from their internal tools.
I would expect the GH security team to have relevant queries ready by now, maybe even do some proactive queries and start alerting anyone who had suspicious activity. (But this is just how I'd do it I have no special insight if they are doing this or something else).
Same. Most belong to 1 member of our team, but a few belong to others on the team. They started happening about 6 months ago (unless logs only go back 6 mo). Would really like to figure out what these are. I asked GH, they said it’s not involved with this breach, and haven’t yet answered my next question (who/what is it?)
Yeah I see lots of the same. They seem to correspond with Heroku deploys? Anyone know if that happens when a valid heroku deploy occurs? Do they download a zip?
For what it's worth, elsewhere in this comment section someone posted that Github Support says the zip downloads weren't related to this incident. Reading between the lines, the compromised repos were probably accessed using normal git clone actions.
To me, this reads like someone got read access to all the source code on Heroku, so if you have any credentials hardcoded in there, now's the time to change them, too.
I wonder if the hackers were kids who got bored around Easter holiday - meaning Heroku's security is shit - or if Heroku deliberately waited to announce this during Easter holiday to minimize the attention it gets - meaning they are as deceitful as all proper megacorps.
I haven't been able to trust their status page to accurately reflect what works and what doesn't for a long time. The only reliable signal is when their status page goes offline ;)
Based on their incident report, it sounds more like someone gained access to the OAuth tokens used to authenticate with linked GitHub repos, which would actually be both read _and_ write access to all source code of repos linked to Heroku apps.
RE kids or misdirection: as with all things, it's probably somewhere in the middle -- a somewhat-sophisticated attacker and a slow, evolving investigation unfortunately coinciding with a holiday weekend. They say the report was received 3 days ago (on April 13) from GitHub after they noticed suspicious activity on April 9.
There are countless SaaS applications asking for full-repo access to Github (all the source code, with write access).
- Productboard
- Bugsnag
- Sentry
- Skylight
- Percy
- CodeTree
- Databox
There are heaps of others, these are just some on top the of my mind. A ticking supply chain attack waiting to happen, since these companies make themselves into alluring hacking targets.
Most of them need access only to issues (a few need read access to code or recent commits, almost none need write).
Solution:
- Let customers give granular access (only issues, only read to source code, etc) when the integration is setup. This is possible with Github's APIs.
- Try to use push instead of pull where possible, i.e. provide a CLI tool to use with Github actions or use Github's webhooks.