That's great. Though maybe the title should acknowledge that it's only for Google Cloud customers.
It's also quite odd that the article doesn't mention Let's Encrypt or the ISRG at all. I would have expected some sort of acknowledgement to their fantastic work over the years.
That's actually a really good point. Though sometimes words are of acknowledgement are equally important. Likely not here though. I'm sure they'd rather take the money ;-)
Side note: I'm actually so proud that my company, Datto, has also been sponsoring Let's Encrypt for many years, and that I was the one to suggest it. It's not platinum, but it's still $50k per year.
yeah, words of encouragement can be great, but i don't think anybody at LetsEncrypt is in any doubt that Google is happy with the work they are doing.
also, i get the sense that getting wider adoption for FIDO outside of LetsEncrypt is a bit of a goal, so having a third-party announce support for FIDO without referring to it as "the LetsEncrypt protocol" is kind of a win for the FIDO people.
But what if I were to lend my old android tablet to my visiting 4-year-old nephew, and I found they'd used my linked credit card to spend $500 on lootboxes in whalecraft?
Google is pretty famous for their nonexistent customer support, so I can't expect satisfaction there. And a chargeback would risk getting my entire account blocked.
> But what if I were to lend my old android tablet to my visiting 4-year-old nephew, and I found they'd used my linked credit card to spend $500 on lootboxes in whalecraft?
Google cloud billing doesn't use google pay, so...nothing?
They've changed free things to not free things many times in the past. Or started restricting features once it gets popular. Don't depend on it for anything you want to last a while.
I’m certainly overlooking something but the risk seems small since it uses an open standard. You should be able to fallback to another acme provider right?
More like: first they become the dominant acme provider, then the other ones go away so they are the only one left, and then they have you where they want you.
The traditional way a big tech company becomes the dominant provider is to embrace an open interoperable protocol to minimize the friction of switching from another provider. Later, when they have captured enough of the market, they extend the protocol to gradually reduce the de facto interoperability with other providers to increase the friction of switching to any other provider.
As a Let’s Encrypt user and fan, I can answer this.
LE is great, but their SLO is significantly below our customers’ expectations. For us, Google wouldn’t replace LE, it would supplement LE for higher reliability.
Seeing more providers conforming to ACME at a price point of “free” is great for the ecosystem.
The recommended renewal cycle gives you a 30 day lead on failure becoming a problem, plenty of time for multiple retries or recovery processes to use an alternate.
The only issues I've ran into, have stemmed from DNS for wildcard certs, where a client's DNS provider is... pretty crap about updating records despite low ttls being set.
It’s a web hosting business. New customers want effortless free TLS asap. We get customers who routinely create new sites who come to expect fast provisioning.
"This API you use has changed, you have 6 months to update your code" is not unheard of.
On AWS, the equivalent is "We no longer recommend this, and it's not visible in the AWS console unless you are already using it or have asked support to enable it, but it will keep working indefinitely".
Its probably already worked into the hosting bill somehow anyway... There are numerous places to hide the cost in any CSP bill.
Even traditional host providers are raising their costs for the same mysterious reasons, although hardware costs are historically trending lower over time.
GCP, AWS, other IaaS providers, and more generally, any "post-paid" service company, needs your credit card, because the services these providers provide allow users to generate costs that can only be measured after-the-fact; and where those costs greatly exceed the free-plan limits, the provider wants to be able to pass those costs to the user.
As well, post-paid providers rely on credit-card billing information to deduplicate/KYC users. Without this sort of information, an, er, "ingenious" user could just sign up for a million free-tier accounts and lash them together into a quota-evading resource-sucking behemoth.
Sure. All that is true. And the credit card number ties your free GCE account to your profile with credit agencies that ad companies use to improve their targeting. It's not one thing or the other. It's both.
Do you think it's really that valuable for Google to spend engineering hours on an ad-targeting integration that will only improve targeting for maybe ~100k users at most?
And probably far less, actually. As many GCP users as there might be in the world, most of them are IT staff working for some-or-another enterprise; where that enterprise only has a single GCP billing-account administrator. Nobody else in the enterprise has their card on file. (And that billing-account administrator's card-on-file is just a corporate credit card, that tells you what the corporation buys, but tells you nothing about the individual. And their email is just a group/alias — billing@ or somesuch. Impossible to log into; impossible to browse the web as; no way to target ads at.)
You'd think the long tail of individual accounts could have more value, but those are the same users who GCP is least interested in recruiting to their platform, and the ones whose entered data is least trustworthy, because of all the spammers and crypto-miners attempting to use stolen credit cards to pay for service. You want to bind some poor random Joe's card-hubbed ad-profile to a spoke created by the person who stole their identity? That's negative average ad-targeting ROI!
They don't use GCP data whatsoever for ads. Thinking they'd peep into cloud customer data for the advertising boost (besides the small data point 'visited hostname console.cloud.google.com') is just as irrational as thinking Amazon would leverage AWS to look into Walmarts' suppliers' data to gain a competitive advantage.
That’s my point, it’s irrational to think that. Amazon has so little to gain by doing that compared to the absolute mass of customers that would be migrated off of aws within a few years of such a scandal.
Counterpoint: at the same time, it's something you can do and not do at the same time - everything in AWS is virtualized (save for $$$$$ physical hosts almost nobody uses) and it's not like it's impossible to undetectably analyse/monitor running VMs in realtime.
So you can both do this (silently) and "not" do it (because it's impossible to detect), and it would thus be stupid to not actually do it because it's free data.
Zooming out I think this and the opposite view are equally possible. Just articulating what this perspective looks like.
That was because Amazon is a competitor to Walmart. It's not a risk that would apply to most companies. And it's probably not a very realistic risk, either. Walmart may have just been trying to hurt a competitor.
Google Cloud is very different from Google's consumer product and advertising side. It has revenues of $13 billion which all comes directly from customers paying them money, not indirectly via advertising etc. Larger companies sign agreements with Google that cover what it does and doesn't promise, and enterprise companies are their highest priority target market. Google Cloud has its own CEO, Thomas Kurian, previously president of product development at Oracle, who was at Oracle for over 20 years.
The pattern you're referring to really doesn't apply to this business. If anything it's the opposite: Kurian's goal is to make Gcloud more like Oracle, and part of that involves making sure enterprise customers don't need to be skittish about the kinds of concerns you seem to be thinking of.
Yes, but that's because they're investing heavily to grow the business, as that article explains, which is hugely strategic for Google.
The fact that the current run rate is $22bn compared to the $13bn revenue in 2020 suggests that they're succeeding - that's some pretty significant growth.
It's a completely different business model from the consumer/ad side, and confusing the two is a mistake.
They're at different points in their lifecycle and have very different revenues and budgets. Amazon itself was famously unprofitable for many years, for exactly the same reason Gcloud is now: to grow the business as fast as possible.
Independent companies like that are funded by investors for years, all the time. The cloud market is expected to hit close to a trillion dollars by 2026. Gcloud doesn't have to take away a single existing AWS customer to win big.
> Xoogler
So what's your take? You think Google might just decide to shut down a fast-growing business with $13-20bn revenues and huge upside potential as if it were a free product like Reader? Or you think they somehow aren't able to make it profitable so will just give up?
Neither of those are really how these things work.
All I'm saying is that this business isn't a monopoly like search or a near duopoly like ads. The competition is stiff. There are two players who are well ahead of them and the ones behind aren't sitting still. I don't expect any miracles with run of the mill management consulting leadership at the helm.
I'm sure they're just "using insights from their operational infrastructure to improve the services they provide to customers". Which is still something that all but the most skeptical of engineers and managers would fail to see a potential problem with.
Quite odd indeed, maybe there are some legal reasons not to mention it. If they could, I feel the whole article could be condensed to "we just shipped a LetsEncrypt alternative gated to Google Cloud customers".
It probably doesn't mention Let's Encrypt because that's an extra opportunity for confusion since you do not (usually?) get Let's Encrypt certificates from this service.
This service has nothing to do with Let's Encrypt, you get certificates from GTS, so the connection would be that ISRG / Let's Encrypt was part of the work to develop ACME and so on. But that's increasingly ancient history. The goal of the work was that almost all CAs in the Web PKI would offer ACME (the people doing like a dozen issuances per month maybe not, but there's an open question about whether they should even exist) -- not just this one free service operated by a charity and here Google are following through.
I'd say that it's the same as if Zig announces each new compiler version or whatever and doesn't acknowledge that, yeah, Grace Hopper is in some sense responsible for the idea of compilers. [[ Hopper wrote the first "compiler" although today we would not consider her software to be a "compiler" but maybe a loader/linker. Anyway, the whole practice of having the machine do the boring job of writing machine code while a human just expresses what they wanted is down to Grace. Thinkers like Turing would have known this was possible in theory but Grace wasn't writing theory, she was doing engineering. ]]. Grace Hopper is important and worth celebrating, but it would be weird to insist on making a specific programming language that has nothing to do with Grace mention her every release.
I guess if you're an Oscar's speech writer going for that "I want to thank my parents, without whom I wouldn't be here today" vibe, then yeah. But otherwise it seems unnecessary.
On this note, I wonder if there is any way to get a google cloud account, and use it JUST for this, without any $$. I don't think so, right? because best option you still need some sort of reverse proxy to your own server?!
As far as I can tell (I now have the API access, so I might test later), you only need to provide additional authentication credentials to Certbot. The private key stays on your server. The CA server _might_ reject requests that do not come from GCP, or attempt to obtain one for a domain name not setup with GCP.
No, GCP has had arguably a superior TLS story for years.
For example they do managed TLS for their workloads like AWS but they operate their own CA rather than outsourcing to Digicert for certificate issuance which gives them a better SLA.
They have a global load balancer offering that enables TLS to terminate everywhere GCP is without having to manage a bunch of discrete load balancers, this also supports managed TLS.
They now support a very large number of certificates in the global load balancer product which allows SaaS products like hosting services to leverage the global load balancer rather than deploying a load balancer per 25 certificates (the limit per AWS LB).
And now let you enroll for certificates from the same CA they use even if you terminate TLS rather than having them do it for you. They do this via a standard API (ACME) which lets you have uniform and agile device compatibility regardless of how you deploy TLS. AWS doesn't let you do this at all.
(I should note I was the PM for most of these releases and am still the PM for Google Trust Services the CA used for this ACME release)
Also because up till, I guess now, Google Cloud TLS is just LetsEncrypt.
Which does raise an issue: I'm not sure why you'd use this, given Google's history of killing projects. What's the compelling reason to switch - especially since they feel content to release this under the `alpha` command set for the CLI tool.
GCP load balancers can automatically switch between pki.goog and letsencrypt.org if one goes down. Or you can restrict the load balancer to just get certs one of them by using a DNS CAA record.
Up till now, you could only use pki.goog with GCP load balancers. This new release allows pki.goog to be used with anything, because you actually control the private key.
> GCP load balancers can automatically switch between pki.goog and letsencrypt.org if one goes down.
Awesome, glad to see other ACME clients using issuer fallback, like Caddy does!
Do you know if there are any plans to making the Google ACME CA usable without registration? Having to register for an account and using credentials is a huge barrier to entry for many less-technical users. Caddy is able to use LE and ZeroSSL because they both don't _require_ accounts (ZeroSSL recommends it, partially as an upsell, but it's not necessary).
The more no-registration CAs exist, the more resilient ACME clients can be.
You could drop out after one day. If you drop out fast enough you might qualify for a refund.
I heard of someone who enrolls in a community college every winter, then goes skiing on a student discount, then drops out quickly and gets a refund. Not very ethical...
Sorry, this was completely unrelated to your point.
Don't forget the "going to the store for some milk" phase of google products where for a few years it doesn't get any feature additions or bugfixes.
I don't understand why anyone would go for this given LE is mature, stable, trusted, and well-supported.
Say one day it just stops issuing you new certs; now what? Call someone? Nope. Post in the forums? Not unless you want to get asked if you've cleared your Chrome cache.
Yeah, that transition didn't go as smoothly as they might have hoped. Most of us first-world programmers can just shrug and say "don't use unsupported versions," but I've had multiple non-technical clients call me up urgently and ask why a (relatively small, but not insignificant depending on the market, and definitely not in their control) subset of their users were seeing certificate errors.
So I don't recommend LE to my clients anymore. But it's a hassle to buy certificates the old way after having tasted ACME, so I'm always looking for an ACME-compatible alternative. ZeroSSL is backed by a more conservative Sectigo CA, but its ACME endpoints aren't very reliable. If this Google cert becomes widely available, I might just as well switch to it. :)
Nowadays you can get virtually unlimited 90-day certs from ZeroSSL if you use ACME through the EAB feature rather than using their API.
But their ACME support seems half-hearted at best. The endpoints often return errors for no reason, compatibility with clients is hit-and-miss, and they keep spamming you with renewal notices even if you renew the cert. For important domains these days I just get a cheap 1-year DV cert like the good ol' days.
Funny you should mention about the forums. There's been a fairly notable Chrome issue intermittently affecting users on MacOS X since late last year and Google seem oblivious to it.
You don't need direct Internet access to use Let's Encrypt, as long as you can arrange for the challenge response to appear in public DNS under the name you want to use.
Would you mind giving an example of what that might look like? Or linking to something? I've always struggled with needing to open ports temporarily on stuff behind my own reverse proxy to avoid passing the certs by hand, and it sounds like something that'd be useful to understand.
It's the DNS-01 challenge[1]. This reduces the challenge to using some DNS provider with an API supported by a client[2] / [3], as well as the server needing to be able to reach the LE-API. We use this with the CNAME delegation into an irrelevant zone everywhere to get wildcard certificates for our LBs ( meaning: the _acme_challenge.example.com record is just a CNAME for _acme_challenge.dont.ever.use.this.example.com, and the servers just have credentials to modify records in the zone <dont.ever.use.this.example.com>)
The magic phrase is “DNS-01” challenge. You place a DNS TXT record to validate control of the domain. There are lots of ACME clients that support a wide variety of DNS service providers. For example, I have a Home Assistant server which automatically issues certs using Gandi DNS and the HA Lets Encrypt support, all without being on the internet (except for the DNS entries)
I think you're confusing internet access with reachable from the internet.
If I remember correctly, you don't need your server to be reachable from the internet, but you still need to be able to contact your DNS provider and the LE server, so you need internet access
The acme client needs to reach LE though. Or you need to do a dance where the client is outside of the private network and ships the certificate into the private network.
And that they're using chrome browser to make certs a mandatory and increasingly expensive monthly subscription service for site owners even though for many sites that don't handle secure transactions it really shouldn't be a requirement.
In 2026 only rich companies will be able to maintain sites with the way we're going folks.
SSL certificates are cheaper than most DNS domain renewals are. ssls.com has them for as low as $3.88 per year; and you can use Let’s Encrypt for free.
Certs are free. What do you mean monthly subscription? Also why would secure transactions be necessary? Preventing third parties from injecting ads / mining scripts / phishing logins benefits all sites.
The big benefit of ACME is that it verifies domain ownership at the correct level.
DigiCert and the like will typically require domain verification at the TLD+1, which is meaningless gibberish that isn't even remotely an RFC standard. There's no such "concept" in DNS, which is intended to be delegated.
So for example if I'm tasked with deploying a web app to "dev1.app.project.org.parentcompany.megacorp.co.uk" where the "project team" is based out of -- say -- Australia, then DigiCert will insist that I verify that I own "megacorp.co.uk", which... I don't. The parent company might not either. MegaCorp's UK head office does. They've never heard of me, and it'll take me a month to get through to someone who cares about my tiny, outsourced project down under.
This kind of thing has happened to me repeatedly across both corporate and government projects. A 2-week project can have a 1 month delay added to it because of this.
My understanding with how TLS was originally intended to be modelled, is that Certificate Authority-ness was supposed to be delegated as well; or at least, more and smaller entities were supposed to be CAs. (About the same kinds of entities who got IPv4 /8 allocations, really.) Big companies like Microsoft were definitely supposed to be their own CAs, enabling them to assert the authenticity of their own zones' subdomains' certificates without the help of any external CA. Generic commercial CAs were supposed to be for the use of regular contractors, not enterprise megacorps.
For some reason (probably industry collusion), X.509 Name Constraints were there from the beginning to enable an extensible generalization of this, but we really missed the boat on supporting them, until eventually doing so became very very hard. We're still making progress toward enablement, though!
The problem is simply corruption. DigiCert and the like buy up the competition and entrench themselves where there is no business or technical need for their existence.
It's pure rent seeking with approximately zero value provided in exchange.
Let's Encrypt demonstrated that the marginal cost of a certificate is close enough to $0 to round down to precisely zero dollars.
To see how deeply rooted this corruption is -- and it is corruption -- ask any major cloud provider like Azure or AWS whether they are willing to provide native ACME protocol integration to enable their customers to request Let's Encrypt certificates for arbitrary DNS-hosted services.
You'll hear nothing back. "No comment" or "We're considering it".
In other words: "We considered it, but then our boss's boss made it very clear to our boss that he was getting a kick-back from DigiCert and to never mention such topics ever again."
AFAIK, while EV certificates don't verify much, they do create a "chain of custody" leading back to a real "notary"-type person who registered the cert, and from there, to a real person who applied for the cert, who can be contacted using information held by the notary (esp. by law enforcement.) It's pretty hard to register an EV cert, use it to spoof someone else's domain, and then not get in trouble.
...at least, it's pretty hard if your CA is located in a Western country. Which is the big loophole here. The fact that we trust CAs headquartered in arbitrary potentially-unfriendly countries, to make claims about the entire space of domains, is pretty silly. Trusting e.g. the Russian CAs in the trust-store only when they make claims about .ru domains (a.k.a. the X.509 Name Constraint extension), would obviate a lot of the concerns people have about X.509's centrally-curated trust store model.
EV certificates have no tie to a person. They have a tie to a corporate entity, which especially in Western countries, is basically intractable to an actual person. It takes a few minutes and $100 to create an anonymous LLC in the US.
> To check the organization's physical existence and business presence, the Certificate Authority must verify that the physical address provided by the Applicant is an address where the organization conducts business operations (not a mail drop, P.O. box or an address for an agent of the Organization). This address will be included into the body of the SSL certificate after verification.
That effectively translates to "there should be an address that criminal investigators can be sent to find and detain employees of the company."
> [Criterion 4] Operational existence
> To make sure that an organization is financially active and engaged in business activities, the Certificate Authority must verify that at least one of the following requirements is met:
> The Organization has been in existence for at least three years
> The Organization is registered in the Dun & Bradstreet database or Qualified Government Tax database
> The Organization has an active demand deposit account which can be proved by a bank statement.
That effectively translates to "you can't just make up an LLC and immediately get an EV cert for it, even if you do tell them your house is the headquarters. You'd have to put a good amount of time and effort into simulating a real business. So much that, if this were a spoofing attempt, you'd be noticed as in breach of trademark by the company you're trying to spoof, long before you got away with it."
> Professional Opinion Letter
> If you need to obtain an EV certificate urgently or prefer keeping the company details confidential, it is possible to send a professional opinion letter signed by a Lawyer, Public Notary or Certified Public Accountant. The person who signed the legal opinion or accountant letter should have a valid license within the country where the organization is registered or the country where the organization maintains an office or a physical facility. To expedite the validation process, we highly recommend requesting a Professional Opinion from a person who speaks English so that he or she can confirm the signature during phone verification with a Comodo (now Sectigo CA) validation agent.
And this, finally, translates to "you can mask your identity, but only behind someone who's notoriously signed a Code of Ethics that requires them to surrender your identity to criminal investigators when asked, without needing a subpoena."
Respectfully, you have a very optimistic view of how this plays out in practice, or even just from what is written in that article. Having obtained several EV certificates for "Stripe Inc", a company I created for $100 on the internet, this is a very basic process with very minimal safeguards. For example, Dun & Bradstreet really does not verify anything at all in its database, and most of the system is based around it for US-based certificates.
> For some reason (probably industry collusion), X.509 Name Constraints […]
For anyone curious, see RFC 5280 § 4.2.1.10:
The name constraints extension, which MUST be used only in a CA
certificate, indicates a name space within which all subject names in
subsequent certificates in a certification path MUST be located.
Restrictions apply to the subject distinguished name and apply to
subject alternative names. Restrictions apply only when the
specified name form is present. If no name of the type is in the
certificate, the certificate is acceptable.
Client support exists in OpenSSL 1.0.0, Windows 7, Mac OS 10.13.3, iOS 11.2.6. (Android?)
I think one technical 'loophole' is that while NCs apply explicitly to SANs, per the spec they do not apply to the Common Name. Though quickly skimming the RFC, I do not see anything that would prohibit them being applied to the CN. So you can probably do it under the guise of "undefined behaviour".
I think the parent poster meant that if X.509 Name Constraints were widely deployed, Visas CA could be limited to Visa's TLD+1s.
In the same way, government affiliated CAs could be limited to their own country TLDs, or a development CA could be limited to localhost.
TLS itself doesn't care about certificates. It provides messages to send and receive them, but their interpretation isn't a matter for TLS. If you want to send photos of, say an actual paper certificate you have or a particularly adorable cat, those messages would work fine but other people's software may not interoperate with yours.
The IETF's PKIX defines how X.509 certificates work for the Internet, because understandably X.509 is for the X.500 system and the Internet is not the X.500 system. PKIX defines the Subject Alternative Name (SAN) which allows the Internet's names (DNS names and IP addresses) to be subjects of X.509 certificates rather than needing the non-existent X.500 directory system for names.
I think you're mostly talking about the Web PKI. But the Web PKI was never "modelled" to work how you've described. At its outset, Netscape (who invented SSL and thus set this ball rolling) wanted pre-existing neutral services rather than they'd run everything and then obviously rival web browsers (including Microsoft's Internet Explorer) would have their own and it's pointless. Several important Certificate Authorities already existed at that time, issuing X.509 certificates in the X.500 system largely to banks, and were happy to take $$$ to issue certificates for this SSL experiment. Initially Netscape basically accepted any company that said they were in this business, and there were no rules (other than those the companies themselves decided on).
But in the modern era the Web PKI is in practice publicly overseen by m.d.s.policy, a policy discussion group of Mozilla. The lack of Name Constraints isn't because of some weird conspiracy, it's simply that Apple didn't support them for many years so if you used Name Constraints then now none of your certificates work on Safari or other Apple products (if Constraints are to work at all they must be marked Mandatory, and if you don't implement a Mandatory feature, you can't be sure if this certificate is valid, so you can't trust it).
For oversight to be effective the sort of delegation you envision is impossible, and accordingly where anything like it did exist the subCAs have moved back to being under physical control of the root CAs. In fact the big problem we had with Symtantec comes down to the lack of effective physical control, with CrossCert able to cause issuance from Symantec's systems yet having no effective oversight.
Also your timeline is badly off in thinking about the IP block allocations. CIDR happened in like 1993, Netscape's SSL doesn't happen until at least 1996. The companies that were issued class A IP blocks before CIDR are mostly smaller and few still exist in the same form, Apple, Comcast and AT&T maybe make sense, but Ford and Prudential Financial not so much. Microsoft are not on that list.
This isn’t really specific to ACME; most CAs are happy to sell domain-validated certificates only for subdomains. It sounds like this is a quirk of DigiCert mostly only selling org/extended-validation certificates, or them trying to make future sales easier.
ACME-supporting CAs and other CAs are all held to the same compliance standards in the end, so nothing prevents the other from doing what you want.
I was trying to verify ownership OF A SUBDOMAIN (e.g. sub.example.com) and I was given a TXT record to add to my DNS. I added it to the subdomain sub.example.com but the verification kept failing. It wasn't until I added it to my domain (example.com) that the verification succeeded. Who thought that was a good idea?
It was especially frustrating since I had verified ownership of the domain minutes before. Why do I have to do it again for every subdomain?
All depends on the type of the certificate you want to get and who will be your CA.
Domain Validation certificates are insecure for quite a few types of security certifications due to how stupid easy it sometimes is to get your subdomain validated through social engineering.
Manager X that manages a small project in subcontractor level can get screwed over way easier than security expert that is managing top level domain of specific corporation.
How do these certificates compare against letsencrypt on the technical dimensions? e.g. certificate chain size, rate limits, whether every certificate is published to the certificate transparency logs, what OSs the root CAs are compatible with?
Isn't it odd that Google would sign their own domains? After all, https is about ensuring your sites's authenticity to your visitor via a trusted third-party certificate. This sounds like it's mostly there to make it difficult to modify content by intermittent parties, such as ads and tracking added by your ISP. As such, "self-signing" by your own root CA would appear as a conflict of interest, and only add to SSL enforcement increasingly seen as gatekeeping.
> Isn't it odd that Google would sign their own domains? After all, https is about ensuring your sites's authenticity to your visitor via a trusted third-party certificate.
Why does it have to be a third party? My understanding is that as a user you just need to trust the certificate and it's owner, and as long as the trust is there it shouldn't matter what that certificate is signing.
> After all, https is about ensuring your sites's authenticity to your visitor via a trusted third-party certificate.
No longer especially that pre-LE, those same third-party sites already broke the rules, even with their EV offerings. Everything has been reduced to "is this domain at least controlled by them?" which is easily auditable (organisational verification has been significantly devalued). Now, Let's Encrypt only verifies domains and not trustworthiness (in fact, they won't revoke certificates known as phishing sites). Also EV certificates are nearly worthless unless you want to bypass many antivirus' HTTPS interception.
No, I don't see it as odd. GlobalSign is the root for Google's certificates, just like DigiCert is the root for Microsoft, and Starfield Technologies (GoDaddy) is the root for Amazon's certificates.
The strange thing is that some companies mix and match - aws.amazon.com is signed with their certificate, while www.amazon.com is signed with a DigiCert certificate. You'll find Microsoft seems to mix certs as well, also using DigiCert for some sites. No clue why this is.
It doesn't really matter which CA signs a certificate, as long as that CA is in the trust root of all major browsers and other clients. Which is the case for google's CA.
At least cloudflare and AWS have been doing this for a very long time. Only they weren't using ACME but their own proprietary APIs where you couldn't export the private keys to you own server. Considering Google requires you to have a GCP account, I don't think it's actually very different and they only wanted to save costs by not developing more customer tooling.
Who would trust Google with their infrastructure these days anyway? Personally I do need to work with Google services occasionally but always experience default anxiety about it.
We run our company on Google Cloud, spending about $400k a year on it. I previously worked heavily with AWS. From a technical perspective Gcloud is pretty great.
People's saltiness about the consumer/advertising side of Google having killed their favorite free product is pretty irrelevant in this context.
I wouldn’t say irrelevant as this is free service as well and same people are making decisions in upper management. There is always a catch and they don’t make anything if it is not bringing more revenue.
This is a free feature on GCP. Many features on GCP are free because they're bundled into the pricing.
> same people are making decisions in upper management.
That's not true. Google Cloud has its own CEO and a very different business model.
> they don’t make anything if it is not bringing more revenue.
Sure - they're a for-profit business. Google Cloud's revenue was $13bn in 2020, and as of last quarter it had a run rate of $22 bn. That money comes direct from paying customers. The business model of the consumer/ad side really is irrelevant here.
This line of reasoning is stupid. Are you saying that because Google killer unused consumer product X, they are going to shoot their cloud business in the head? Why? Do you not understand the difference between b2b and b2c?
Is GCP profitable? Somebody else in the comments pointed out that GCP is actually losing money and needs to be subsidized by Google's ad revenue, thus making Alphabet immensely influential.
I think my comment was conflated with others' sentiments about Google killing their RSS reader and other products. That is not my rationale at all. My hesitance for using Google Cloud or AWS stems from my mistrust of these behemoths and their track record of baseless censorship. From what I have seen, both of these companies have stepped out of their place as neutral platforms.
I understand not all large companies have the luxury of picking their service providers but I do believe it's in everyone's best interest to diversify web service infrastructure. One, to hedge against catastrophic failures that may arise from their platform going down and two, to better balance the distribution of power on the internet as a whole.
Speaking of which; is anyone else old enough to remember when it was discovered that all (Root) Certificate Authorities were compromised by the 5+1 eyes?
One major problem with Google’s L7 load balancers is that the config changes take 5-20 mins to take effect. So google trying to set up an ACME challenge on a LB, solving it, and setting the managed TLS cert on it can take non-negligible time (15-30 mins?). I hope this gets fixed someday.
Another ACME alternative to letsencrypt is zerossl.
It's especially great because letsencrypt is operated by US company ISRG and zerossl seems to be from Austria, so if you're not happy with your server being dependant on US, it might be a good option.
This is great news. One limitation with Lets Encrypt is their rate limits are a bit low for Review Apps - you can’t issue more than 50 certs a week under a given domain.
So if you’re spinning up tens or hundreds of review apps per day, you can’t get a fresh cert for each, and so you need to do something different than your production environment does. (A wildcard cert is the obvious choice.)
I hope this offering has a high enough quota that you can get enough certs to do review apps properly; the marginal cost to Google per customer is probably negligible, whereas LetsEncrypt doesn’t have other revenue generating offerings they can use to cover their operating costs.
Yes, they do. But if you use a wildcard cert for your review app, then you are not testing the same cert machinery as your production application. (Unless you want to use wildcard certs for your prod app too, which I'd rather not do.)
As part of my pre-merge pipeline I deploy the application to k8s, create a public IP, DNS entry from the branch slug, TLS-to-the-pod with wildcard cert, and then run the full end-to-end test suite. If you get your docker caching right, it can be as little as a couple minutes to deploy the whole stack
This makes it a lot easier for engineers to iterate on their changes with stakeholders, not to mention it lets you test infra and API changes before actually merging them.
We got to tens of engineers using this, so I didn't optimize utilization heavily; this ends up being a bit expensive, but I think worth it.
I take a nightly dump of my dev environment database, and then bake that into a thin docker image based on the MySQL image. I’m not dealing with huge datasets so this isn’t expensive. You can then start up a mysql pod and it comes up with a copy of the dev DB data.
(I also give developers a script to run this fixture DB locally too - it makes the onboarding process a lot easier if you don’t need to run the DB init and migrations, and generally means developers have a more robust set of test data to work with. )
We've had an internal ACME server at my dayjob for over a year now. It's one of the few things I'm proud of where we really got out early on a cool technology. Otherweise we're a big telco and move like a oil tanker.
Any status for RFC 8657, ACME CAA support? This is for restricting which account and which validation methods may issue certificates. The CPS says they may use it, which is too vague and I'm not going to test it right now.
By the way, I appreciate the support for short-term certificates as well; Letsencrypt doesn't have it, though invalidation can be detected with OCSP or draft-aaron-acme-ari.
>"Each of these have different scenarios where their use makes the most sense, for example TLS-ALPN-01 might make sense in cases where HTTPS is not used and the requestor does not have access to dynamically update DNS records."
I'm confused by TLS-ALPN-01. I understand the idea of using certs for domain verification but if there is no TLS in use how does the client verify this after the cert has been issued exactly?
This is usually used by TLS terminators, which don't have an HTTP server available (can understand TLS but not HTTP). ALPN here is a property of TLS ClientHello which specifies the capabilities of the client, usually used to signal support for HTTP 2 but in this specific case signals that the TLS client is an ACME verier on behalf of a CA, so the TLS server only sends the validator certificate when the validators specifically requesting it (ensuring interruption-free validation). The important part here is that a TLS server is available but HTTPS is not available as it is either only a TLS terminator or something that operates a non-HTTPS TLS server (for example, RTMPS).
>"The important part here is that a TLS server is available but HTTPS is not available as it is either only a TLS terminator or something that operates a non-HTTPS TLS server (for example, RTMPS)."
That makes good sense. Thanks for putting it so succinctly. Cheers.
The article has been on HN for an hour. It has 8 comments, 5 of which were my first thought - why on earth would you expect this service to hang around, based on Google's track record?
Wether it lasts or not, this surely has to be an issue for Google innovations going forward? If the perception is that any new thing will die, especially not-consumer-scale things, then how do they build traction?
One of the advantages of using the standard ACME protocol is that you will be able to very easily switch CAs should you need to. It could possibly even be automatically switched on the fly, if one CA has some sort of outage. That’s a nice advantage for improving reliability and flexibility.
This is presumably a core feature of Google Cloud, and if you’re already using their other cloud products, I can’t imagine being too worried about features randomly going away.
(Disclaimer: While I work for Let’s Encrypt, this is my own opinion and not necessarily that of my employers)
I agree 100%. So if my Initial reaction to a _standard_ service is "no way", then what chance do they have for a unique future service offering? The cert issuer is not the issue - the (general?) opinion of Google maybe is...
And that's before we take customer service into the equation.
The big difference here is that their ACME server is part of Google Cloud. Google Cloud is a paid service and in the 5 years of using it, I cannot remember any gcloud service being deprecated, without a similar alternative being introduced. (e.g cloud image registry being replaced with artifact registry)
So (had to look it up) Reader was infamously the shuttered project that first caused a lot of backlash...that went for 8 years. They closed it due to "declining usage" i.e. they could not monetize well.
Play Music died after 9 years, and that even featured paid souscriptions...the logic of "paid for and around" doesnt seem to be a solid argument, esp for a company making money primarily off of data - as of 2021 their cloud business is only 5% of their revenue, ~74% of their revenue is ad based the rest is devices (nest, phones), and Youtube.
I think its pretty reasonable to be concerned about the longevity of this product - maybe we will be wrong, but.
The history of axing stuff is not uncorrelated with the business unit and its business model. They know that B2B costumers value the stability and they are going to price that in. It's not like they are not reading what people are complaining about.
A mechanistic certificate issuance service which is explicitly capable of being re-homed under another TA: Just use ACME compliant certbot calls to letsencrypt and you will have a certificate about "you" which validates under another TA.
I get your beef with google service availability longterm and the google list of killed products, but this is an instance of a service which in true Internet form "routes around damage"
This isn't the killer "what if google stop" problem area. ACME based certification doesn't care: find another TA, and move on with your life.
I guess my point is, LE exists, I'm using it already, why would I switch?
Assuming there is a reason to switch, the switch back is not without cost. Ergo, either way, I'm not using this. Then again I don't use GCP for the same reason.
My bigger point is that logical or not, my perception appears to be common. And that may hurt Google's ability to get traction with unique services in the future.
No one cares about Google's CA, but perhaps this perception issue is a real problem?
If they change the spec and certbot follows them, I doubt it will deprecate the non-google approaches. If they do something novel like enable multi-level wildcarding not just the terminal leftmost *. form, People might get stuck on how that works, but it would probably require chrome to change too.
mainly I suspect this is to pacify people inside GKE who don't like calling outside of the protected space to get certificates to exist.
It's fascinating to me how thoroughly Google has managed to poison the well for their services. Many of the people aren't even swayed by the connection to a revenue-generating service. I understand! I wouldn't use this unless it was much, much easier than using letsencrypt - even in a GCP ecosystem. I don't trust them any more than anyone else.
> It's fascinating to me how thoroughly Google has managed to poison the well for their services.
What's fascinating to me is the apparent irrationality, from people who otherwise tend to pride themselves on being rational.
I can understand people saying they'll never deal with Google again because they killed their favorite free product or whatever. That's fair. But the attempt to justify that position as some sort of rational risk calculation, taking the actual facts of the matter into account, is misguided.
In some cases this is due to a lack of info and disinterest in correcting that, but many other cases just seem to be emotion masquerading as a thought process.
Of course it's emotional, all decisions are to some degree emotional.
And I would suggest that if they'd just killed reader, then OK, stuff like that happens. But Google has killed a Lot of things. Probably for good reason. But they get a reputation for being "flighty" when it comes to new things.
Sure it's emotional, but that reputation comes into my thinking when I make product choices. I'll use HERE maps over Google maps etc. And I'm not terribly inclined to make their stuff part of my critical work flow.
So sure I'll Google search and YouTube all day long. But I'll use AWS or Azure over GCP. 90% because I can get someone on the phone. 9% because I wonder when Google will decide to kill GCP. And 1% because I don't trust the Google AI not ti just terminate my account with no recourse.
The people who believe this are going to keep believing it, there's not really anything to dissuade them. A priori anything might be cancelled could be and is worthless. Most don't have that extreme of a position, Reader and Allo were the last big deprecations.
From what I’ve heard, which could be wrong since it’s just second-hand anecdotes —- internal Google promotions favor people who start and launch new projects rather than maintain old ones.
If that’s true, Google’s products aren’t subject to cancelation in the way that any given web service is.
I can’t think of a single google cloud product that’s gone GA and then been shut down. Google has different attitudes towards consumer and enterprise products.
That's just poor branding. Google Cloud Print was not part of Google Cloud Platform.
(I feel like Google just doesn't understand branding. I remember when they launched the Pixel C after the Chromebook Pixel, and it ran Android instead of ChromeOS. What was once a Chrome brand became an Android brand. I guess because Chrome was doing pretty well at the time, and Android just reminded people of slow phones that ran out of battery instantly. Sigh!)
Google Plus? Hangouts?
Maps got a 7000% price increase... No user support for blocked accounts?
My point is though that while you say "most don't have that extreme a position", I'm not sure what % it is. And does that perception, that it exists at all, hinder them from future rollout?
On the consumer side of things, they have. Nest guard has been discontinued, and Google play music has been replaced by YouTube music, which lacks some of the features of its predecessor.
I'm as salty as the next person about the ending of the free custom-domain gmail service, but it did last for over 10 years, which is plenty of time to get double ROI for the time and setup investment of the free google solution, and then the replacement solution if / when google rug-pulls the "free" component.
The equation does change if there are value-adds that start getting offered and used which make it more difficult to transition elsewhere (with docs, gdrive etc. being examples of value-adds for the free custom-domain email offering).
The Let's Encrypt community also made some fun references to the Roadrunner cartoons in the early days.
(1) The reference implementation of the ACME server was originally going to be called Anvil, but was renamed to Boulder. (A later lightweight testing implementation is called Pebble.)
But that documentation admits it simultaneously kills everything immediately (not great) yet you may get billed more anyway (even worse).
It's notable that Azure can do better here because they have to (the expensive Visual Studio product comes with "free" Azure credits, you aren't paying for them so there is nobody to "bill" if you run out, they just shut stuff off). They appear to do "better" by just eating the extra cost after shutting off. That's still a better user experience though.
This is a real problem. You set up a cloud service on Friday. You believe it should cost about $10 per day. On Monday you discover it already cost $500 so you switch it off immediately. Oops. Still, only $500 not a big problem right?
And then on Tuesday the billing software explains that ah, there's $1800 extra for that service, we calculate it eventually but we don't promise it'll happen immediately. Now you're $2300 down. And this can continue for several days at big cloud providers because "eventually" is apparently good enough.
It's also quite odd that the article doesn't mention Let's Encrypt or the ISRG at all. I would have expected some sort of acknowledgement to their fantastic work over the years.