OpenSSL 3 seems to be largely compatible (it's a much easier upgrade than previous big ones) but the part of this that breaks clients is odd.
If your peer closes the connection without sending close_notify, you get an unexpected EOF; 1.1.1 ignored this, 3 passes the error to the client, which needs to set SSL_OP_IGNORE_UNEXPECTED_EOF to recover the old behaviour - but this is only safe as long as the client checks for truncation attacks. This peer behaviour seems to be pretty common for google servers, so if you use google logins or APIs it'll trip you up (from doing these upgrades at work, almost all of the errors I saw were talking to google)
What's odd to me is most of the patches I saw for this were just setting that flag, and not doing anything about truncation attacks? I'm sure this will bite us all somewhere down the line.
Anyway, some notes for admins of old systems who got to this too late:
It was noticeable that this patch got backported into ppa's that maintain old/deprecated versions of some software. eg the ondrej php ppa has this backported to EOL'd versions of php. But, this is not the case for the python deadsnakes ppa - I didn't look at python 3.8, but 3.9 was fixed at source, and it hasn't been backported to the (EOL'd) 3.7 anywhere. You should be off 3.7 anyway, but if you're not, it's now urgent.
Another notable backport is openresty. If you're using nginx with lua and oauth, it's a popular choice; but lags the latest nginx by a long way. The latest version is 1.21.4.2 which means it's based on nginx 1.21; but nginx is on 1.25 and nginx only released with openssl 3 support in 1.22. If you proxy to https servers, this is a concern. However, digging I found the openssl3 support was backported to openresty a few months ago, so the _current_ version does work without falling over.
> What's odd to me is most of the patches I saw for this were just setting that flag, and not doing anything about truncation attacks? I'm sure this will bite us all somewhere down the line.
It sounds like those are workarounds to get the apps to work again by reintroducing the old behavior. Once they are unblocked, they can refocus on addressing the issue.
It's either that, or remain broken and unusable until further notice.
correct, and specifically with backports there is no other reasonable way to do this. Since the mitigation needs to happen outside of the openssl integration, you need to expose the flag to clients and allow them to make the choice to ignore the error or not, until they handle this. But in a backport, you can't expose new apis like this to clients, because that code isn't getting updated.
So was the truncation attack already there in 1.1.1?
I think it's OK for a patch that changes the dependency to 3 to just retain existing behavior. Addressing the problematic behavior should be a distinct patch.
Yes, the fix for the truncation attack was originally rolled into 1.1.1e (March 2020) and then withdrawn in 1.1.1f because the new errors were too disruptive; IIRC the fix was already in the 3.x prereleases by that stage. So effectively if you stuck on 1.1.1, you've been living with this risk for 3 years.
The truncation attack is one where the client thinks their ssl session has ended but a malicious actor prevents the session from ending correctly; the threat is something like, after doing this your gmail can be messed with even tho you think you've logged out. From the descriptions it seems like you'd need to layer it with another attack or two to do anything useful with it. It's not something that keeps me awake at night.
When I dug into the python code we had hitting this a few months back (python 3.7 on recent ubuntu) it appeared that the error would not only trip up a client expecting to use 1.1.1 but could mean the last buffer read from the socket was not flushed (so, you can't just catch the error in python and assume it's ok because the other end had closed the connection anyway - you're missing some of the response). I wasn't certain but that possibility was a big bucket of NOPE for me, far easier to tell people to get the upgrade done and stop using software that had been built against 1.1.1.
If your peer closes the connection without sending close_notify, you get an unexpected EOF; 1.1.1 ignored this, 3 passes the error to the client, which needs to set SSL_OP_IGNORE_UNEXPECTED_EOF to recover the old behaviour - but this is only safe as long as the client checks for truncation attacks. This peer behaviour seems to be pretty common for google servers, so if you use google logins or APIs it'll trip you up (from doing these upgrades at work, almost all of the errors I saw were talking to google)
What's odd to me is most of the patches I saw for this were just setting that flag, and not doing anything about truncation attacks? I'm sure this will bite us all somewhere down the line.
Anyway, some notes for admins of old systems who got to this too late:
It was noticeable that this patch got backported into ppa's that maintain old/deprecated versions of some software. eg the ondrej php ppa has this backported to EOL'd versions of php. But, this is not the case for the python deadsnakes ppa - I didn't look at python 3.8, but 3.9 was fixed at source, and it hasn't been backported to the (EOL'd) 3.7 anywhere. You should be off 3.7 anyway, but if you're not, it's now urgent.
Another notable backport is openresty. If you're using nginx with lua and oauth, it's a popular choice; but lags the latest nginx by a long way. The latest version is 1.21.4.2 which means it's based on nginx 1.21; but nginx is on 1.25 and nginx only released with openssl 3 support in 1.22. If you proxy to https servers, this is a concern. However, digging I found the openssl3 support was backported to openresty a few months ago, so the _current_ version does work without falling over.