When you validate server certificates from HTTPS clients, please be sure to use the right set of root certs. Mozilla maintain a decent list of these [1], but it's not in the PEM format that most HTTPS client libraries expect, e.g. Python's ssl.wrap_socket(sock, ca_certs="certs.pem").
Mozilla's list also includes distrusted certificates, so you need to be careful to leave them out when generating the PEM-encoded format. In fact, I'd strongly recommend using Adam Langley's excellent extract-nss-root-certs tool [2] which takes care of the subtle details for you.
And, if you are willing to trust me, you can download my pre-generated PEM-encoded cacerts file from a month or so ago [3].
I'd actually recommend just get the exact certificates you're going to work with. For example, if I'm using GitHub's API, it makes sense not to check all the chain, but just keep the GitHub's original cert.
Disclaimer: I don't particularly agree with the Certificate Authority mechanism that we currently use with TLS.
However, given that it's what we currently have, I'd strongly advice taking advantage of the security that it provides. Requiring API client library authors to ship certs will make for poor security. Not only do certificates expire, they also get compromised.
It would be easy to conduct MITM attacks using revoked certs and API client library users would be none-the-wiser. Instead, it should be the responsibility of HTTPS client libraries to use the latest cacerts data and support features like OCSP [1] for validating certificate revocations, etc.
Taking a validated known-good cert and, from that point on, simply verifying that the cert remains bit-for-bit identical is called "certificate pinning", and while it does create problems with revocations, your service is going to break as soon as Github revokes their certificate anyways, so it's not like you won't notice.
The problem is that the CA mechanism (in particular, blindly trusting whatever list of root certs your vendor ships) does not provide security. I cannot go and validate the practices of those CAs, and all it takes is one of them to get compromised (which they regularly do, have you taken out the compromised CA's root certs that ship with OS X, for example?).
The only certificates I actually trust are the self signed ones from my organization which I can actually go validate in person. While I have ZERO trust in any of the certificates that my vendor ships.
the problem here is rather than worrying about one particular cert getting compromised, you now have to worry about every CA in the world getting compromised, a much more likely possibility.
It seems the best course of action would be to trust only an individual cert, and check for revocation.
Also OCSP is basicly a joke, it works every single time, except when it matters (an attacker controlling your view of the world)
The problem here is that if your app is deployed in a corporate environment, it's possible (likely) that the corporate firewall is intercepting your HTTPS traffic and returning a different certificate, issued by the IT department.
So if you try to validate that the certificate is the specific one that your API server is using, it's going to fail in that scenario.
Depending on your app, you could just ignore that possibility of course.
The word "attack". In theory, being in a corporate environment, this is a desirable man-in-the-middle rather than a hostile one. It certainly is man-in-the-middle.
It's exactly what SSL/TLS is designed to defend against,
and neutering your apps & applying newspeak doesn't make it preserve the
security provided by SSL/TLS.
Conceptually, in a work environment you aren't accessing the website, your company is. If your company chooses to add an SSL proxy for its own purposes, there's nothing invalid, wrong, or unethical about that. Conceptually, you're all functioning as one entity.
You may note I'm using words like "theoretically" and "conceptually" in these replies, and that's basically because ctz's point is accurate. It isn't hard for someone on HN to be more competent at SSL usage than the administrator of the SSL inspector. But, well, welcome to the corporate world. Can't live with 'em, can't live without 'em. But I don't think it's wrong on any moral or technical level, it's just potentially wrong based on more mundane considerations, like competence.
Nobody is saying it is. However, this is the reality in a surprisingly large number of corporate environments. I have to support a lot of enterprise customers in my day job, and working around corporate firewalls is a large part of the issues that come up for us.
Yes of course. In a corporate environment, you would usually install the proxy server's CA certificate in your certificate store and validate that all certificates were issued by the proxy server.
My original comment was just pointing out that validating that the certificate you get when you connect to https://www.github.com in a corporate network may not be the same as the one you get on the open internet.
It's up to you to decide whether that's something you care about though.
And for the proxy's private key (which you are henceforth relying on for all transport security) to be kept secure. Given my recent experience with products like websense, this is a very poor bet to make.
Why working so hard, when M2Crypto [1] will do everything for you [2] ?
If you're doing anything serious with Python and SSL, you're going to use M2Crypto - period. Because when it comes to security, the less you "roll your own", the better.
Sadly, that M2Crypto script doesn't check for certificates which are not trusted for issuing SSL server certs. So whilst it happens to skip a few, it will include over a dozen inappropriate certs in the final output!!
In this age of high-level languages, why do I still have to worry about this? I don't mean 'security' I mean 'managing certificates.' My local framework/API should complain if I don't have a trusted root and should then make it dead simple to provide that root.
Because during most of the development cycle when these libraries are being used, the certificates aren't validating (they're dev/test/UAT systems) --- and so during actual development, certificate verification seems like just another annoying obstacle to clear as quickly as possible.
It's easy enough to generate a snakeoil cert and use its public part as your cert bundle. On Debian/Ubuntu, just install the ssl-cert package, and point to /etc/ssl/certs/ssl-cert-snakeoil.pem as your certificate bundle; the private key is at /etc/ssl/private/ssl-cert-snakeoil.key .
Or just serve your app/API/etc from different urls, plain HTTP for development and HTTPS for preprod and production.
That's exactly how it works in C# (.NET in general), and I would take a fairly large bet that's exactly the same in Java. It seems like a regression (when thinking on terms of lower level --> higher level language movement).
Security is something that should just work - arguably, the first thing with such requirement. Even though it simply can't just work in many cases, it should be as close to just-working as it can.
However, this isn't exposed in the higher-level httplib.HTTPSConnection class for some reason. I'd bet it's not too hard to write your own subclass to handle it though.
In Python, a sufficiently recent httplib2 will require valid certificates by default; same with the requests library. Both let you use your own cert bundle, as a quick-and-dirty way to do certificate pinning. That's reason enough to discourage the use of Python's standard library for http requests (another reason is that it's quite low-level).
The verification of the certificate is delegated to a TrustManager[1]. Certificate paths can be checked[2] and the TrustManager is flexible enough to support a large variety of verification scenarios[3].
If you want to check any 3rd party apps or frameworks that you use you can set up a proxy like Fiddler, configure it to do a MITM attack on you, and see if the client/API blows up with an error or just keeps on truckin'.
Getting this right was a big pain when we were implementing the various Stripe client libraries. We had to resort to hacks even in mainstream languages.
Well, having to use hacks for that is just plain wrong. I think we can consider this as bugs in these mainstream languages. Probably you could file them?
Python: Just check the certificates with Python Requests (or take one of 20 lines urllib solutions):
>>> requests.get('https://exmaple.com, verify=True)
requests.exceptions.SSLError: hostname 'exmaple.com' doesn't match either of '*.exmaple.org', 'exmaple.org'
So presumably, decent high-level languages (like Python) do use an encrypted connection - they just don't ensure the right person is on the other end of that connection?
To make it worse in many languages there not built in OCSP or CRL facilities to go with their standard TLS wrappers. e.g. The best you get in Python is checking against a CA list. So even if you do go to the trouble to turn on CA verification yourself you still accept known bad certificates.
The problem with doing this is that many/most enterprise IT shops use security devices to proxy SSL traffic, altering the cert that the client sees.
So unless you really understand you app's deployment scenario well, you should proceed with caution. Also note that 2% of enterprise IT people understand SSL at all, (and the devices that do the MITM part are usually controlled by the security dept) so troubleshooting will be close to impossible.
Java HTTPS server code allows the developer to configure the X.509 trust manager. My API uses Java for both client and server and transmits JSON messages over secure web sockets which is built over HTTPS. Works great. I authenticate on both client and server, and am just now developing an iOS Objective C client that runs wss.
i know in the the world of .NET (atleast on Windows) it'll automatically check using the cert manager. Anyone know how this might work on platforms such as Java? I'd assume the local VM would need to have some OS specific plumbing in place.
The official Oracle/Sun JVM ships its own file-based certificate store containing the root certificates of authorities it trusts. It does not, at any time, interoperate with OS-wide stores, nor does it assume they might exist. This makes it fully portable, but of course adds a further burden in terms of maintenance.
True, but if you're installing OpenJDK from Debian they have gone to the trouble of integrating it with the system-level certificate store, so it's much easier to manage.
The JDK comes with a program called key tool. You use key tool to build a "trust store". This is basically a collection of certs you trust. There is also a "key store" which contains your cert and its private key. Then when you run your app you must specify what key store and trust store to use.
iOS and OSX both use the system store represented by the "system" keychain. (On the desktop, individual users can also have keychain a with trusted roots.). Apple keeps their root store up to date via software updates and automatic OCSP checks.
NSURLConnection, the higher level resource API, will by default require a valid certificate chain, but provides for explicitly allowing an insecure connection as part of its authentication callbacks. (This is an improvement over previous versions (leopard and before) where you had to explicitly specify hostnames that should be considered safe.)
So, certificate validation is treated, at the API level, like any other sort of authentication challenge.
You can also provide a client certificate using the same mechanism, if requested by the server.
Using the lower level CFHTTP stream API, you can only fail the connection and re attempt it after disabling cert checking.
Validating certificates is a good thing and everybody should do it.
That said ... it really only tells you that a certificate is 'sound'. It by no means tells you with 100% confidence that you are talking to the right party.
The twilio-ruby library does this by packaging a set of root ca certs. There is a configuration option to override this with your own set when you construct the client.
Nginx ssl reverse proxy does not verify certificates. I tested by reverse proxying to an ssl server with a self signed certificate. Nginx did not log any warnings or complain.
You can use nginx as a proxy on your side for any APIs. Just proxy_pass it to the API, and listen only on the local side. It can provide caching and other stuff that way.
Yeah, but that's proxy, not reverse proxy. It's possible to do something fiddler-like using apache or nginx to mitm oneself and dump everything that goes through.
However, I don't know anything about using nginx that way. If you give me a link or something like that, I'll happily add it to my article. Or I'll just investigate later.
Yeah my bad I read your article assuming you were an API provider not a consumer. In the case of consuming API’s I suggest python-requests, an excellent library that is extremely well maintained.
Mozilla's list also includes distrusted certificates, so you need to be careful to leave them out when generating the PEM-encoded format. In fact, I'd strongly recommend using Adam Langley's excellent extract-nss-root-certs tool [2] which takes care of the subtle details for you.
And, if you are willing to trust me, you can download my pre-generated PEM-encoded cacerts file from a month or so ago [3].
[1] https://mxr.mozilla.org/mozilla/source/security/nss/lib/ckfw...
[2] https://github.com/agl/extract-nss-root-certs
[3] https://github.com/downloads/tav/ampify/distfile.cacerts-201...