Andrew Ayer

Try the following Git clone with libcurl 8.7.1 (which happens to be the version shipped in macOS 14.6 and newer) and it fails or hangs:

git clone https://software.sslmate.com/src/macosgitbug.git

Disable HTTP/2 and it works:

git clone -c http.version=HTTP/1.1 https://software.sslmate.com/src/macosgitbug.git

The bug is in libcurl 8.7.1's handling of the FAILONERROR option. FAILONERROR tells libcurl to treat unsuccessful HTTP status codes, such as 404, as a request failure. When HTTP/2 is used, the bug causes other in-flight requests on the same HTTP/2 connection to also fail, or even to hang. The bug was fixed over two years ago in curl 8.8.0, but Apple continues to ship a buggy version, even in last week's macOS 27 release.

When retrieving a repository over the "dumb" transfer protocol, Git makes certain HTTP requests with the FAILONERROR option set, notably requests to objects/info/alternates and objects/info/http-alternates, which list alternate locations where the repository's content can be found. Most repositories don't have alternate locations, so these files don't exist, and the URLs return 404 errors. When the buggy version of libcurl is used, this 404 error causes Git's other HTTP requests to also fail, and Git is unable to clone the repository.

The bug affects not just direct uses of Git, but also go get with GOPROXY=direct or a module listed in GOPRIVATE, which invoke Git under the hood.

Working around the bug on the client side is easy: just force Git to use HTTP/1.1:

git config --global http.version HTTP/1.1

Even better, install Git through MacPorts, since Apple has clearly dropped the ball. (Homebrew won't help - unlike MacPorts, they use the system libcurl.)

But most clients won't know to apply this workaround, and if you host Git repositories with the dumb protocol, you probably want macOS users to be able to clone your repositories! Fortunately, there's a really easy server-side workaround: create objects/info/alternates and objects/info/http-alternates as empty files, so they don't return a 404 error anymore. Git treats the empty files the same as it would treat a 404 error, and the libcurl bug isn't triggered.

touch /path/to/repo.git/objects/info/alternates /path/to/repo.git/objects/info/http-alternates

The bug isn't triggered when the repository supports the "smart" protocol, which is why macOS can clone repositories from GitHub and other popular forges despite them using HTTP/2. But I do not want to use the smart protocol for my repositories: although it has many advantages over the dumb protocol, it requires heavy server-side computation and even a modest load can knock a server over. In contrast, the dumb protocol can be served entirely from static files, which makes a huge difference for withstanding the horde of AI scrapers currently terrorizing the Web. I hope that we will see innovations to the dumb protocol that bring it some of the advantages of the smart protocol while still being served from static files.

Thanks to Romain of the Traefik project for noticing that SSLMate's repos couldn't be cloned on macOS, Sebastiaan van Stijn for asking that this problem be reported upstream instead of silently hacked around in Traefik's go.mod file, and Kangmin Kim for pointing me to the libcurl bug as the root cause. Claude Code proposed the empty file workaround so I didn't have to waste (too much) time on this. Zero thanks to Apple for shipping a two-year-old show-stopping bug in libcurl.

HTTP reverse proxying is a minefield. Just the other week, a researcher disclosed a desync vulnerability in Discord's media proxy that allowed spying on private attachments. This is not unusual; these vulnerabilities just keep coming.

The problem is the widespread use of HTTP as the protocol between reverse proxies and backends, even though it's unfit for the job. But we don't have to use HTTP here. There's a 30-year-old protocol for proxy-to-backend communication that avoids HTTP's pitfalls. It's called FastCGI, and its specification was released 30 years ago today.

FastCGI is a Wire Protocol, not a Process Model

It's true that some web servers can automatically spawn FastCGI processes to handle requests for files with the .fcgi extension, much like they would for .cgi files. But you don't have to use FastCGI this way - you can also use the FastCGI protocol just like HTTP, with requests sent over a TCP or UNIX socket to a long-running daemon that handles them as if they were HTTP requests.

For example, in Go all you have to do is import the net/http/fcgi standard library package and replace http.Serve with fcgi.Serve:

Everything else about your app stays the same - even your handler, which continues to use the standard http.ResponseWriter and http.Request types.

Popular proxies like Apache, Caddy, nginx, and HAProxy support FastCGI backends, and the configuration is simple:

Show more config examples

Apache HTTP

ProxyPass / http://localhost:8080/

Apache FastCGI

ProxyPass / fcgi://localhost:8080/

Caddy HTTP

reverse_proxy localhost:8080 { transport http { } }

Caddy FastCGI

reverse_proxy localhost:8080 { transport fastcgi { } }

HAProxy HTTP

backend app_backend server s1 localhost:8080

HAProxy FastCGI

fcgi-app fcgi_app docroot / backend app_backend use-fcgi-app fcgi_app server s1 localhost:8080 proto fcgi

Why HTTP Sucks for Reverse Proxies: Desync Attacks / Request Smuggling

HTTP/1.1 has the tragic property of looking simple on the surface (it's just text!) but actually being a nightmare to parse robustly. There are so many different ways to format the same HTTP message, and there are too many edge cases and ambiguities for implementations to handle consistently. As a result, no two HTTP/1.1 implementations are exactly the same, and the same message can be parsed differently by different parsers.

The most serious problem is that there is no explicit framing of HTTP messages - the message itself describes where it ends, and there are multiple ways for a message to do that, all with their own edge cases. Implementations can disagree about where a message ends, and consequently, where the next message begins. This is the foundation of HTTP desync attacks, also known as request smuggling, wherein a reverse proxy and a backend disagree about the boundaries between HTTP messages, causing all sorts of nightmare security issues, such as the Discord vulnerability I linked above.

A lot of people seem to think you can just patch the parser divergences, but this is a losing strategy. James Kettle just keeps finding new ones. After finding another batch last year, he declared "HTTP/1.1 must die".

HTTP/2, when consistently used between the proxy and backend, fixes desync by putting clear boundaries around messages, but FastCGI has been doing that since 1996 with a simpler protocol. For context, nginx has supported FastCGI backends since its first release, but only got support for HTTP/2 backends in late 2025. Apache's support for HTTP/2 backends is still "experimental".

Why HTTP Sucks for Reverse Proxies: Untrusted Headers

If desync attacks were the only problem, you could just use HTTP/2 and call it a day. Unfortunately, there's another problem: HTTP has no robust way for the proxy to convey trusted information about the request, such as the real client IP address, authenticated username (if the proxy handles authentication), or client certificate details (if mTLS is used).

The only option is to stick this information in HTTP headers, alongside the headers proxied from the client, without a clear structural distinction between trusted headers from the proxy and untrusted headers from a potential attacker. For example, the X-Real-IP header is often used to convey the client's real IP address. In theory, if your proxy correctly deletes all instances of the X-Real-IP header (not just the first, and including case variations like x-REaL-ip) before adding its own, you're safe.

In practice, this is a minefield and there are an awful lot of ways your backend can end up trusting attacker-controlled data. Your proxy really needs to delete not just X-Real-IP, but any header that's used for this sort of thing, just in case some part of your stack relies on it without your knowledge. For example, the Chi middleware determines the client's real IP address by looking at the True-Client-IP header first. Only if True-Client-IP doesn't exist does it use X-Real-IP. So even if your proxy does the right thing with X-Real-IP, you can still be pwned by an attacker sending a True-Client-IP header.

FastCGI completely avoids this class of problem by providing domain separation between headers from the client and information added by the proxy. Though trusted data from the proxy and HTTP request headers are transmitted to the backend in the same key/value parameter list, HTTP header names are prefixed with the string "HTTP_", making it structurally impossible for clients to send a header that would be interpreted as trusted data.

FastCGI defines some standard parameters such as REMOTE_ADDR to convey the real client IP address. Go's net/http/fcgi package automatically uses this parameter to populate the RemoteAddr field of http.Request, rendering middleware unnecessary. It Just Works. Proxies can also use non-standard parameters to report whether HTTPS was used, what TLS ciphersuite was negotiated, and what client certificate was presented, if any. Go automatically sets the Request's TLS field to a non-nil (but empty) value if the request used HTTPS, which is very handy for enforcing the use of HTTPS. The fcgi.ProcessEnv function can be used to access the full set of trusted parameters sent by the proxy.

Closing Thoughts

If FastCGI is the better protocol, why isn't it more popular? Maybe it's the name - while capitalizing on CGI's popularity made sense in 1996, CGI feels dated in 2026. There's also an enduring lack of awareness of the security problems with HTTP reverse proxying. Watchfire described desync attacks in 2005, and gave a prescient warning of their intractability, but the attacks were inexplicably ignored for over a decade. In an alternate timeline, Watchfire's research was taken seriously and people went looking for other protocols for reverse proxies.

FastCGI is very usable today, and has been in production use at SSLMate for over 10 years. That said, using a vintage technology has some downsides. It was never updated to support WebSockets. The tooling is not as good. For example, curl has no way to make requests to a FastCGI server. It supports FTP, Gopher, and even SMTP (however that works), but not FastCGI. When I benchmarked Go's FastCGI server behind a variety of reverse proxies, some workloads had worse throughput compared to HTTP/1.1 or HTTP/2. I don't think that's inherent to the protocol, but a reflection that FastCGI code paths have not been optimized as much as HTTP.

Despite these shortcomings, I still think FastCGI is worth using. I don't use WebSockets, and it's fast enough for my use case (and maybe yours too). If it ever became the bottleneck, I'd rather buy more hardware than deal with the nightmare of HTTP reverse proxying.

Happy 30th birthday, FastCGI!

Imagine you're connecting to an IP address over TLS. The server presents a valid, properly-issued IP address certificate from a publicly-trusted certificate authority. But you're not really talking to that IP address, but to a man-in-the-middle (MitM) who was assigned the IP address over a year ago. How could this be? Unfortunately, publicly-trusted IP address certificates just don't provide the same level of security as certificates for domain names. Unless you're operating a DNS-over-TLS or DNS-over-HTTPS resolver, you should not use IP address certificates. You'll be much more secure with domain name certificates.

(When I say IP address certificate, I mean a TLS certificate with an IP address subjectAltName, which allows a certificate to authenticate a connection to an IP address, rather than a domain name as is typical on the Web. Public certificate authorities have been allowed to issue IP address certificates for a long time, but I fear they will become more popular since Let's Encrypt added support for them last month.)

The basic security property provided by a certificate is that the certificate authority has validated that the certificate subscriber (the person who applies for the certificate and knows its private key) is authorized to represent the domain name or IP address in the certificate. This ensures that the other end of a TLS connection is truly the domain or IP address that you want to connect to, not a MitM impostor.

But the validation is not done every time a TLS connection is established; rather, it was done at some point in the past. Thus, the certificate subscriber may no longer be authorized to represent the domain or IP address.

How old might the validation be? As of February 2026, certificate authorities are allowed to issue certificates that are valid for up to 398 days. (Let's Encrypt limits their IP address certs to 6 days, but you have to worry about attackers using other CAs, not just the CA you use.) So the validation may be 398 days old. But it gets worse. When issuing a certificate, CAs are allowed to rely on a validation that was done up to 398 days prior to issuance. So when you establish a TLS connection, you may be relying on a validation that was performed a whopping 796 days ago. You could be talking not to the current assignee of the domain or IP address, but to anyone who was assigned the domain or IP address at any point in the last 2+ years.

This is a problem with both domains and IP addresses, but it's way worse with IP addresses. While it's still very possible to register a domain that no one has ever registered before, you don't have this luxury with IPv4 addresses. There are no unassigned IPv4 addresses left; when you get an IPv4 address, it has already been assigned to someone else. In some cases, particularly in cloud environments, the address could have had as many tenants as a room at a Motel 6.

For some use cases, the risk is unavoidable. For example, DNS-over-TLS and DNS-over-HTTPS resolvers have to be contacted by IP address, to avoid a circular dependency on DNS resolution. Domain name certificates are not an option. Fortunately, by 2029, the maximum certificate lifetime will be 47 days, and the maximum validation reuse period will be 10 days, so the risk window will be only 57 days. That's not too bad for a service that's likely to exist for many years with the same IP address.

But even with a 57 day window, other use cases should just use domain name certificates. One use case for IP address certificates is particularly misguided: securing short-lived cloud workloads. Let's Encrypt describes this use case as:

Securing ephemeral connections within cloud hosting infrastructure, like connections between one back-end cloud server and another, or ephemeral connections to administer new or short-lived back-end servers via HTTPS - as long as those servers have at least one public IP address available.

On Bluesky, Ryan Hurst goes into more detail:

Modern infrastructure no longer has stable hostnames, static IPs, or long-lived trust anchors. Workloads spin up before DNS exists, live briefly, and disappear. Trust has to keep up.

Short-lived and IP certificates make it possible to use TLS before a DNS name exists, reduce friction for DNS over HTTPS adoption, secure ephemeral devices and services by default, and shift trust from long-lived credentials to automated renewal.

As I understand it, the use case is running servers which accept TLS connections and exist for a very short period, like minutes. The most secure implementation would be to assign the server a unique, never-before-used hostname under your domain, publish a DNS record pointing the hostname to the server's IP address, get a domain name certificate, and have clients connect to the hostname. Thanks to the unique hostname, clients can be sure they're connecting to the right server and not to an impostor who was assigned the IP address at some point in the past.

But Ryan suggests that it might not be possible to wait for a DNS record to be published before the server needs to be usable. Fortunately, it's still possible to avoid IP address certificates. Instead, you can publish, in advance, a DNS record under your domain for every IP address you might need to use. For example, ip-192-0-2-1.yourcompany.example would resolve to 192.0.2.1. ISPs commonly publish DNS records like this for their entire IP address space, so it's possible to do at scale. Have your clients connect to your short-lived server using this DNS record, rather than an IP address. To prevent would-be attackers from completing domain validation under your domain using a non-DNS validation method, you should also publish a CAA record for your domain that uses ACME account binding to restrict issuance to your ACME account. Now clients can be sure they're connecting to a server controlled by your organization, rather than anyone who happened to be assigned the same cloud IP address in the last 2 years.

You can also monitor Certificate Transparency logs to look for unauthorized certificates for your domain. Monitoring Certificate Transparency is not practical when using IP address certificates with short-lived cloud workloads, since the set of IP addresses that you need to monitor will be constantly changing as you create and destroy your servers, and it will be hard to tell whether an unknown certificate is malicious or just an innocent issuance to someone who got assigned the same IP address.

I encourage you to think long and hard before using IP address certificates, especially for short-lived cloud servers. The high risk of stale validation, combined with the inability to effectively monitor Certificate Transparency, bring IP address certificates uncomfortably close to opportunistic encryption (TLS without certificate validation), even with post-2029 shortened lifetimes. Fortunately, you can embed IP addresses in pre-published DNS records if you can't publish DNS records on the fly. Doing a little extra DNS work is a small price to pay for avoiding the inherent risks of IP address certificates.

Twice a year, the Certificate Transparency ecosystem undergoes a transition as certificate authorities start to submit certificates to new semiannual log partitions. And recently, the ecosystem has started transitioning to the new static-ct-api specification. Unfortunately, despite efforts to make these transitions extremely easy for certificate authorities, in the past week I have detected 16 certificate authorities who have bungled these transitions, issuing certificates that are rejected by some or all mainstream web browsers with an error message like "This Connection Is Not Private" or ERR_CERTIFICATE_TRANSPARENCY_REQUIRED.

If you're not familiar, Certificate Transparency (CT) is a system for publishing SSL certificates in public logs. Certificate Transparency monitors like Cert Spotter download the logs to help you track certificate expiration and detect unauthorized certificates for your domains.

At a high level, Certificate Transparency works like this:

• Before issuing a certificate, the certificate authority (CA) creates a "precertificate" containing the details of the certificate it intends to issue.
• The CA submits the precertificate to multiple Certificate Transparency logs.
• Each log returns a receipt, called a Signed Certificate Timestamp (SCT), which confirms submission of the precertificate.
• The CA embeds the SCTs in the certificate which it gives to the site operator.
• When a browser loads a website, it makes sure the website's certificate has SCTs from a sufficient number of recognized logs. If it doesn't, the browser throws up an error page and refuses to load the website.

Billions of SSL certificates are issued and logged to CT every year. To prevent logs from growing indefinitely, logs only accept (pre)certificates which expire within a certain range, typically six months long. Every log will eventually contain only expired certificates, allowing it to be shut down. Meanwhile, new logs are created to contain certificates expiring further in the future.

How do CAs know what logs to submit precertificates to? It's easy: Apple and Chrome each publish a JSON file containing a list of logs. (Firefox and Edge use Chrome's list.) Apple's is at https://valid.apple.com/ct/log_list/current_log_list.json and Chrome's is at https://www.gstatic.com/ct/log_list/v3/log_list.json. Each log object contains the log's name, URL, public key, range of expiration dates accepted by the log, and crucially, the log's state.

{ "description": "Sectigo 'Elephant2027h1'", "log_id": "YEyar3p/d18B1Ab8kg3ImesLHH34yVIb+voXdzuXi8k=", "key": "MFkwEwYHKoZIzj0CAQYIKoZI...AScw2woA==", "url": "https://elephant2027h1.ct.sectigo.com/", "mmd": 86400, "state": { "usable": { "timestamp": "2025-07-22T01:33:20Z" } }, "temporal_interval": { "start_inclusive": "2027-01-01T00:00:00Z", "end_exclusive": "2027-07-01T00:00:00Z" } }

The state is very simple: if it's "usable", then CAs should use it. If it's something else, CAs should not use it.

The full process of logging is a bit more complicated, because CAs have to include SCTs from a sufficiently-diverse set of logs, but when it comes to finding the initial set of logs to consider, it's hard to imagine how it could be any easier for CAs. They just need to download the Apple and Chrome lists and find the logs whose state is Usable in both lists and whose expiration range covers the expiration date of the certificate.

Despite this, a number of CAs appear to either disregard the state or only consider Chrome's log list. Historically, this has not caused problems because new logs have become Usable in both Chrome and Apple before they were needed for new certificates. Since the maximum certificate lifetime is 398 days, logs for certificates expiring in the first half of 2027 (2027h1) needed to be Usable by November 29, 2025. Unfortunately, not all 2027h1 logs were Usable by this date.

First, Google's 2027h1 logs (Argon 2027h1 and Xenon 2027h1) were added to Chrome 40 days later than they should have been. Normally, new logs are added to Chrome after 30 days of successful monitoring, but this process is still very manual and human error led to Chrome setting a 70 day timer instead of a 30 day timer. Consequentially, these logs are still in the Qualified state in Chrome. Although Qualified logs are recognized by up-to-date installations of Chrome (and Firefox and Edge), there may be out-of-date installations which do not recognize them, making it a very bad idea for CAs to use Qualified logs if they care about compatibility. Chrome, Firefox, and Edge automatically disable Certificate Transparency enforcement once they become 70 days out-of-date, so Argon and Xenon 2027h1 will become Usable on December 27, 2025, which is 70 days after they became Qualified. (Argon and Xenon 2027h1 are already Usable in Apple's list.)

Second, DigiCert's 2027h1 logs (Sphinx 2027h1 and Wyvern 2027h1) don't appear at all in Apple's log list. Since Apple doesn't use a public bug tracker for their CT log program like Chrome, I have no idea what went wrong. Did DigiCert forget to tell Apple about their new logs, or is Apple slow-rolling them for some reason? Certificates which rely on either DigiCert log won't work at all on Apple platforms. (They are already Usable in Chrome's list.)

While the late addition of logs is not ideal, it should not have been a problem, because there are plenty of other 2027h1 logs which became Usable for both Apple and Chrome in time.

I first became aware of issues last Tuesday when Arabella Barks posted a message to Mozilla's dev-security-policy mailing list referencing a certificate issued by Certum with SCTs from DigiCert Wyvern 2027h1. Sensing that this could be a widespread problem, I decided to investigate. My company, SSLMate, maintains a 51TB PostgreSQL database with the contents of every Certificate Transparency log. The database's primary purpose is to power our Certificate Transparency monitoring service, Cert Spotter, and our Certificate Transparency Search API, but it's also very handy for investigating ecosystem issues.

I ran a query to find all precertificates logged to Google's and DigiCert's 2027h1 logs. This alone was not sufficient to identify broken certificates, since CAs could be submitting precertificates to these logs but not including the SCTs in the final certificate, or including more than the minimum number of required SCTs. Therefore, for every precertificate, I looked to see if the corresponding final certificate had been logged anywhere. If it had, I ran it through SSLMate's CT Policy Analyzer to see if it had enough SCTs from broadly Usable logs. If the final certificate wasn't available for analysis, I counted how many other logs the precertificate was logged to. If fewer than three of these logs were Usable, then there was no way the corresponding certificate could have enough SCTs.

I posted my findings to the ct-policy mailing list later that day, alerting CAs to the problem. Since then, I've found even more certificates relying on logs that are not broadly Usable. As of publication time, the following CAs have issued such certificates:

• Certum
• Cybertrust Japan (fixed)
• Disig
• GDCA
• GlobalSign (fixed)
• HARICA
• IdenTrust (fixed)
• Izenpe (fixed)
• Microsec
• NAVER
• SECOM
• SSL.com
• SHECA
• TWCA (fixed)
• certSIGN
• emSign

Of those, only the five indicated above have fixed their systems. The others have all issued broken certificates within the last two days, even though it has been a week since my first public posting.

Unfortunately, logging to non-Usable logs wasn't the only problem. Last Wednesday, Cert Spotter began alerting me about certificates issued by Cybertrust Japan containing SCTs with invalid signatures. I noticed that the SCTs with invalid signatures were all from static-ct-api logs.

To address shortcomings with the original Certificate Transparency specification (RFC6962), the ecosystem has been transitioning to logs based on the static-ct-api specification. Almost half of the 2027h1 logs use static-ct-api. However, while static-ct-api requires major changes for log monitors, it uses the exact same protocol for CAs to submit (pre)certificates. This was an intentional decision to make static-ct-api easier to adopt, so that it wouldn't suffer the same fate as RFC9162, which was intended to replace RFC6962 but was dead-on-arrival in part because it completely broke compatibility with the existing ecosystem.

However, there is one teeny tiny difference with static-ct-api: whereas RFC6962 logs always return SCTs with an empty extensions field, static-ct-api logs return SCTs with non-empty extensions. This should not be problem - the extensions field is just an opaque byte array and CAs do not need to understand what static-ct-api logs place it in it. They just need to copy it through to the final certificate, which they should have been doing anyways with RFC6962 logs. But Cybertrust Japan was always leaving the extension field empty regardless of what the log returned, breaking the SCT's signature. Since SCTs with invalid signatures are disregarded by browsers, this left their certificates with an insufficient number of SCTs, dooming them to rejection.

After publication of this post, Cert Spotter alerted me to invalid SCT signatures in certificates issued by NAVER. In this case, the SCT extensions were non-empty but encoded in base64, indicating that NAVER wasn't decoding the base64 from the JSON response when copying it to the SCT. On one hand, I don't love RFC6962's wording about the extensions field: while the other JSON fields, like id and signature, are clearly indicated as "base64 encoded", it's only implied that extensions is base64-encoded (it says "Clients should decode the base64-encoded data and include it in the SCT"). On the other hand, if NAVER were verifying the signature of SCTs before embedding them in certificates, they almost certainly would have caught this mistake, since successful verification relies on correctly decoding the JSON response. And we know from past incidents that it's very important for CAs to verify SCT signatures.

Unfortunately, we'll probably never learn the root cause of these failures or what CAs are doing to prevent them from happening again. Normally, when a CA violates a policy, they are required to publish a public incident report, answer questions from the community, and note the failure in their next audit. If their incident response is bad or they keep having the same incident, they run the risk of being distrusted. However, Certificate Transparency is not a policy requirement in the traditional sense - CAs are free to issue certificates which violate CT requirements; those certificates just won't work in CT-enforcing browsers. This allows to CAs to issue unlogged certificates to customers who don't want their certificates to be public knowledge (and don't need them to work in browsers). Of course, that's not what the CAs here were doing - they were clearly trying to issue certificates that work in browsers; they just did a bad job of it.

Previously:

• GoDaddy Issues Thousands of Certificates That Don't Work in Safari (again)
• The Story Behind Last Week's Let's Encrypt Downtime
• Why Your Website Is Failing With ERR_CERTIFICATE_TRANSPARENCY_REQUIRED (March 2023 Edition)
• Tens of Thousands of GoDaddy Certificates Will Break in macOS 11.4 and iOS 14.6

On each of the last two Fridays, Google has suspended SSLMate's Google Cloud access without notification, having previously suspended it in 2024 without notification. But this isn't just another cautionary tale about using Google Cloud Platform; it's also a story about usable security and how Google's capriciousness is forcing me to choose between weakening security or reducing usability.

Apart from testing and experimentation, the only reason SSLMate still has a Google Cloud presence is to enable integrations with our customers' Google Cloud accounts so that we can publish certificate validation DNS records and discover domain names to monitor on their behalf. We create a service account for each customer under our Google Cloud project, and ask the customer to authorize this service account to access Cloud DNS and Cloud Domains. When SSLMate needs to access a customer's Google Cloud account, it impersonates the corresponding service account. I developed this system based on a suggestion in Google's own documentation (under "How can I access data from my users' Google Cloud project using Cloud APIs?") and it works really well. It is both very easy for the customer to configure, and secure: there are no long-lived credentials or confused deputy vulnerabilities.

Easy and secure: I love it when that's possible!

The only problem is that Google keeps suspending our Google Cloud access.

The First Suspension

Google suspended us for the first time in 2024. Our customer integrations began failing, and logging into the Google Cloud console returned this error:

Although Google's customer support people were surprisingly responsive considering Google's rock-bottom reputation in this area, the process to recover our account was super frustrating:

1. Google required me to email them from the address associated with the account, but when I did so, the message was bounced with the error "The account [redacted] is disabled" (the redacted portion being the email address I sent from). When I emailed from a different address, the message went through, but the support people initially refused to communicate with it because it was the wrong address.

2. At one point Google asked me to provide the IDs of our Google Cloud projects - information which I could not retrieve because I couldn't log in to the console. Have you saved your project IDs in a safe place in case your account gets suspended?

3. After several emails back and forth with Google support, and verifying a phone number, I was able to log back into the Google Cloud console, but two of our projects were still suspended, including the one needed for the customer integrations. (At the time, we still had some domains registered through Google Cloud Domains, and thankfully the project for this was accessible, allowing me to begin transferring all of our domains out to a more dependable registrar.)

4. The day after I regained access to the console, I received an automated email from no-reply@accounts.google.com stating that my access to Google Cloud Platform had been restricted. Once again, I could no longer access the console, but the error message was different this time:

   

5. Twelve hours later, I received multiple automated emails from google-cloud-compliance@google.com stating that my Google Cloud projects had been "reinstated" but I still could not access the console.

6. Seven hours after that, I got another automated email from no-reply@accounts.google.com stating that my access to Google Cloud Platform had been restored. Everything began working after this.

I was never told why our account was suspended or what could be done to prevent it from happening again. Although Google claims to send emails when an account or project is suspended, they never did so for the initial suspension. Since errors with customer integrations were only being displayed in our customers' SSLMate consoles (usually an error indicates the customer made a mistake), I didn't learn about the suspension right away. I fixed this by adding a health check that fails if a large percentage of Google Cloud integrations have errors.

The Second Suspension

Two Fridays ago, that health check failed. I immediately investigated and saw that all but one Google Cloud integrations were failing with the same error as during last year's suspension ("Invalid grant: account not found"). Groaning, I tried logging into the Google Cloud console, bracing myself for another Kafkaesque reinstatement process. At least I know the project IDs this time, I reassured myself. Surprisingly, I was able to log in successfully. Then I got emails, one per Google Cloud project, informing me that my projects had been reinstated "based on information that [I] have provided." Naturally, I had received no emails that they had been suspended in the first place. The integrations started working again.

The Third Suspension

Last Friday, the health check failed again. I logged in to the Google Cloud console, unsure what to expect. This time, I was presented with a third type of error message:

Most, but not all, of SSLMate's Google Cloud projects were suspended, including the one needed for customer integrations.

I submitted an appeal on Friday. On Sunday, I received an email from Google. Was it a response to the appeal? Nope! It was an automated email stating that SSLMate's access to Google Cloud was now completely suspended.

Edited to add: On Monday, shortly after this post hit the front page of Hacker News, most projects were reinstated, including the project for the integrations. A few hours later, access was fully restored. As before, there was no explanation why access was suspended or how to prevent it from happening again.

The Lucky Customer

Incredibly, we have one lucky customer whose integration has continued to work during every suspension, even though it uses a service account in the same suspended project as all the other customer integrations.

What Now?

Clearly, I cannot rely on having a Google account for production use cases. Google has built a complex, unreliable system in which some or all of the following can be suspended: an entire Google account, a Google Cloud Platform account, or individual Google Cloud projects.

Unfortunately, the alternatives for integrations are not great.

The first alternative is to ask customers to create a service account for SSLMate and have SSLMate authenticate to it using a long-lived key. This is pretty easy, but less secure since the long-lived key could leak and can never be rotated in practice.

The second alternative is to use OpenID Connect, aka OIDC. In recent years, OIDC has become the de facto standard for integrations between service providers. For example, you can use OIDC to let GitHub Actions access your Google Cloud account without the need for long-lived credentials. SSLMate's Azure integration uses OIDC and it works well.

Unfortunately, Google has made setting up OIDC unnecessarily difficult. What is currently a simple one step process for our customers to add an integration (assign some roles to a service account) would become a complicated seven step process:

1. Enable the IAM Service Account Credentials API.
2. Create a service account.
3. Create a workload identity pool.
4. Create a workload identity provider in the pool created in step 3.
5. Allow SSLMate to impersonate the service account created in step 2 (this requires knowing the ID of the pool created in step 3).
6. Assign roles to the service account created in step 2.
7. Provide SSLMate with the ID of the service account created in step 2, and the ID of the workload identity provider created in step 4.

Since many of the steps require knowing the identifiers of resources created in previous steps, it's hard for SSLMate to provide easy-to-follow instructions.

This is more complicated than it needs to be:

• Creating a service account (steps 1, 2, and 5) should not be necessary. While it is possible to forgo a service account and assign roles directly to an identity from the pool, not all Google Cloud services support this. If you want your integration to work with all current and future services, you have to impersonate a service account. Google should stop treating OIDC like a second-class citizen and guarantee that all current and future services will directly support it.

• Creating an identity pool shouldn't be necessary either. While I'm sure some use cases are nicely served by pools, it seems like most setups are going to have just one provider per pool, making the extra step of creating a pool nothing but unnecessary busy work.

• Even creating a provider shouldn't be necessary; it should be possible to assign roles directly to an OIDC issuer URL and subject. You should only have to create a provider if you need to do more advanced configuration, such as mapping attributes.

I find this state of affairs unacceptable, because it's really, really important to move away from long-lived credentials and Google ought to be doing everything possible to encourage more secure alternatives. Sadly, SSLMate's current solution of provider-created service accounts is susceptible to arbitrary account suspensions, and OIDC is hampered by an unnecessarily complicated setup process.

In summary, when setting up cross-provider access with Google Cloud, you can have only two of the following:

1. No dangerous long-lived credentials.
2. Easy for the customer to set up.
3. Safe from arbitrary account suspensions.

Provider-created service accounts	Service account + key	OpenID Connect
No long-lived keys		No long-lived keys
Easy setup	Easy setup
	Safe from suspension	Safe from suspension

Which two would you pick?