Andrew Ayer

You can protect the users of your Go modules from supply chain attacks, such as a compromise of your GitHub account, by monitoring Go's checksum database (sumdb). Since the go command won't install a module unless its checksum is published in the sumdb, monitoring the sumdb lets you discover unauthorized versions of your modules. Since the sumdb is a transparency log, you can even detect if Google themselves go rogue and publish a malicious version of your module. Although you can only detect, not prevent, attacks, Go's Minimal Version Selection makes it possible to respond before most of your users have installed the malicious version. Dependency cooldowns, which are coming to Go, will make it even easier to respond in time.

How To Monitor

Source Spotter, which is operated by my company SSLMate as a free service to the Go community, provides Atom feeds listing all versions of your modules found in the sumdb. For example, this Atom feed returns all versions of modules under the src.agwa.name/ prefix:

https://feeds.api.sourcespotter.com/modules/versions.atom?module=src.agwa.name%2F

You can also get Prometheus-compatible metrics:

https://metrics.api.sourcespotter.com/modules?module=src.agwa.name%2F

You can scrape the metrics endpoint using Prometheus and alert in your usual way, subscribe to the Atom feed using your favorite feed reader, or use one of the free services that converts Atom feeds to emails.

Avoiding Noise

Discovery is only half the story. The more important half is deciding whether to alert on a discovered module. Approximately 100% of the records published in the sumdb are legitimate. If you're alerted every time a new version of your module is legitimately published, it will be very hard to notice the one time it's an attack.

I wasn't sure at first how Source Spotter could facilitate no-noise monitoring. Initially, I thought Source Spotter would need access to your module's Git repository so it could cross-check sumdb records against the repo's contents. But that seemed complicated, and it would fail to detect a compromise of your repository host. I also really wanted a solution that wouldn't require users to create accounts.

I finally found a lightweight solution that I really like. I'll show you how you use it, and then explain how it works.

First, you install a small command line tool called sourcespotter-authorize and generate a public/private key pair:

go install software.sslmate.com/src/sourcespotter/cmd/sourcespotter-authorize@latest

sourcespotter-authorize -keygen

Run sourcespotter-authorize again with the -feed-for or -metrics-for flags to output Atom and Prometheus URLs for the module prefix you want to monitor (src.agwa.name/ in this example):

sourcespotter-authorize -feed-for src.agwa.name/

https://feeds.api.sourcespotter.com/modules/versions.atom?module=src.agwa.name%2F&mldsa=efbcb2bcb2d4decdf1ad9cab3224b2dd0087b6c6877abe00448a00d31716dff1

sourcespotter-authorize -metrics-for src.agwa.name/

https://metrics.api.sourcespotter.com/modules?module=src.agwa.name%2F&mldsa=efbcb2bcb2d4decdf1ad9cab3224b2dd0087b6c6877abe00448a00d31716dff1

These are the same URLs shown earlier, but with a new mldsa= parameter in the query string which is the hash of the public key you generated in the previous step.

Initially, these URLs return the same contents as the URLs without the mldsa= parameter. That's because you haven't marked any module versions as authorized yet.

To mark a module version as authorized, change into the Git repository for the module and run sourcespotter-authorize with a Git tag:

cd ~/src/snid

sourcespotter-authorize v0.4.0

Now, v0.4.0 of this module is omitted from the feed and metrics URLs.

The command accepts multiple tags as arguments, so you can authorize all the tags in a repo like this:

sourcespotter-authorize $(git tag)

Moving forward, you should run sourcespotter-authorize any time you tag a new version. I've written a tiny shell script called gotag that runs git tag followed by sourcespotter-authorize:

#!/bin/sh -e git tag "$1" sourcespotter-authorize "$1"

As long as you authorize every tag you create, the feed and metrics URLs will report zero unauthorized module versions, eliminating false positive alerts. A supply chain attacker can't hide their malicious versions from your feeds unless they also compromise sourcespotter-authorize's private key.

How It Works

sourcespotter-authorize -keygen generates an ML-DSA-44 private key (stored under $XDG_CONFIG_HOME/sourcespotter-authorize). The SHA-256 hash of the corresponding public key goes in the mldsa query string parameter.

When you authorize a tag, sourcespotter-authorize uses the golang.org/x/mod/zip and golang.org/x/mod/sumdb/dirhash packages to compute the checksums of the go.mod and module zip files for the tag. It formats the module path, version, and hashes for each tag as a go.sum file (no need to invent a new format here), and signs the file with your private ML-DSA key. Finally, it uploads a JSON object to the Source Spotter server containing your public key, the go.sum file, and the signature.

The Source Spotter server verifies the signature using the public key. If valid, it records the module path, version, and hash as being authorized by that public key, and excludes that module version from the feeds and metrics endpoints for the key.

The protocol is simple and documented if you want to implement your own client.

Why Not Just Sign Your Releases?

If you're generating a private key and signing stuff anyway, why not just sign your releases and distribute the signatures? After all, this is the traditional approach to stopping unauthorized software releases.

The problem with the traditional approach is that consumers of your software have to verify the signatures, which means they need to know what your public key is. Key distribution is a hard, hard problem, and most software ecosystems do a bad job at it. I would guess that in practice, few people actually verify signatures of software releases.

In contrast, the go command automatically verifies that every module it installs is listed in the checksum database, without the user needing to do anything. Although sourcespotter-authorize needs a private key, you only need to distribute the public key to Source Spotter (or whatever other monitor you might choose to use), rather than every single consumer of your software. That's a vastly simpler problem, especially when (not if) you need to rotate your keys.

Everyone who signs their software releases will need to rotate their keys soon, as quantum computers are expected to break RSA and elliptic curves within the next few years. An earlier version of sourcespotter-authorize used elliptic curve keys. After it gained ML-DSA support, upgrading my key was super easy - generate a new key, transfer my list of authorized versions to it, and finally update the URL in my feed reader:

sourcespotter-authorize -export > tmpfile

rm ~/.config/sourcespotter-authorize/private_key

sourcespotter-authorize -keygen

sourcespotter-authorize -import < tmpfile

sourcespotter-authorize -feed-for src.agwa.name/

This was so much easier than communicating a new key to every consumer of my software would have been, and didn't require a lengthy transition period during which I was signing with two keys.

You can learn more about Source Spotter's module monitoring, get the source code on GitHub, or read my blog post about how Source Spotter also verifies Go's reproducible builds.