Andrew Ayer

I discovered today that Xpdf deterministically segfaults when you try to print any PDF under i386 Wheezy. It doesn't segfault on amd64 Wheezy which is why I had not previously noticed this. Someone reported this bug over two years ago, which you'd think would have given ample time for this to be fixed for Wheezy. This bug led me to a related bug which paints quite a sad picture. To summarize:

• Xpdf uses its own PDF rendering engine. Some other developers thought it would be nice to be able to use this engine in other applications, so they ripped out Xpdf's guts and put them in a library called Poppler. Unfortunately, Xpdf continued to use its own rendering engine instead of linking with Poppler, so now there are two diverging codebases that do basically the same thing.
• Apparently this code is bad and has a history of security vulnerabilities. The Debian maintainers quite reasonably don't want to support both codebases, so they have attempted to patch Xpdf to use Poppler instead of its own internal engine.
• Unfortunately, this is hard and they haven't done a perfect job at it, which has led to a situation where Xpdf and Poppler each define their own, incompatible, versions of the same struct... with which they then attempt to interoperate.
• As a consequence, Xpdf accesses uninitialized memory, or accesses initialized memory incorrectly, so it's a miracle any time you manage to use Xpdf without it crashing.

The Debian maintainers have three options:

1. Stop trying to patch Xpdf to use Poppler.
2. Fix their patch of Xpdf so it actually works.
3. Do nothing.

There is a patch for option 2, but it has been rejected for being too long and complicated. Indeed it is long and complicated, and will make it more difficult to package new upstream versions of Xpdf. But that's the price that will have to be paid as Xpdf and Poppler continue to diverge, if the maintainers insist that Xpdf use Poppler. Unfortunately they seem unwilling to pay this price, nor are they willing to go with option 1. Instead they have taken the easy way out, option 3, even though this results in a totally broken, and possibly insecure, package that is only going to get more broken.

Clearly I can't be using this quagmire of a PDF viewer, so I have uninstalled it after being a user for more than 10 years. Sadly, the alternatives are not great. Most of them take the position that UIs are for chumps, preferring instead to use keyboard shortcuts that I will never remember because I don't view PDFs often enough. Debian helpfully provides a version of Evince that isn't linked with GNOME and it works decently. Unfortunately, after quitting it, I noticed a new process in my process list called "evinced." As in "evince daemon." As in, a PDF viewer launched its own daemon. As in, a PDF viewer launched its own daemon. I don't even...

epdfview is like Evince but less cool because it doesn't have its own daemon has buggy text selection. qpdfview is rather nice but copies selected text in such a way that it can't be pasted by middle-clicking (it copies into XA_CLIPBOARD instead of XA_PRIMARY).

I decided to go with Evince and pretend I never saw evinced.

Update (2014-01-20): The Xpdf bug has allegedly been fixed, by defining a new version of the incompatible struct. This is option 2, which means we can expect Xpdf to break again as Poppler and Xpdf continue to diverge. It appears to be a different patch from the one that was rejected earlier, though I can't find any of the discussion which led to this fix being adopted. I will update this blog post if I learn more information.

The problem was so bizarre that for a moment I suspected I was witnessing a man-in-the-middle attack using valid certificates. Many popular HTTPS websites, including Google and DuckDuckGo, but not all HTTPS websites, were taking up to 20 seconds to load. The delay occurred in all browsers, and according to Chromium's developer tools, it was occurring in the SSL (aka TLS) handshake. I was perplexed to see Google taking several seconds to complete the TLS handshake. Google employs TLS experts to squeeze every last drop of performance out of TLS, and uses the highly efficient elliptic curve Diffie-Hellman key exchange. It was comical to compare that to my own HTTPS server, which was handshaking in a fraction of a second, despite using stock OpenSSL and the more expensive discrete log Diffie-Hellman key exchange.

Not yet willing to conclude that it was a targeted man-in-the-middle attack that was affecting performance, I looked for alternative explanations. Instinctively, I thought this had the whiff of a DNS problem. After a slow handshake, there was always a brief period during which all handshakes were fast, even if I restarted the browser. This suggested to me that once a DNS record was cached, everything was fast until the cache entry expired. Since I run my own recursive DNS server locally, this hypothesis was easy to test by flushing my DNS cache. I found that flushing the DNS cache would consistently cause the next TLS handshake to be slow.

This didn't make much sense: using tools like host and dig, I could find no DNS problems with the affected domains, and besides, Chromium said the delay was in the TLS handshake. It finally dawned on me that the delay could be in the OCSP check. OCSP, or Online Certificate Status Protocol, is a mechanism for TLS clients to check if a certificate has been revoked. During the handshake, the client makes a request to the OCSP URI specified in the certificate to check its status. Since the URI would typically contain a hostname, a DNS problem could manifest here.

I checked the certificates of the affected sites, and all of them specified OCSP URIs that ultimately resolved to ocsp.verisign.net. Upon investigation, I found that of the seven name servers listed for ocsp.verisign.net (ns100.nstld.net through ns106.nstld.net), only two of them (ns100.nstld.net and ns102.nstld.net) were returning a response to AAAA queries. The other five servers returned no response at all, not even a response to say that an AAAA record does not exist. This was very bad, since it meant any attempt to resolve an AAAA record for this host required the client to try again and wait until it timed out, leading to unsavory delays.

If you're curious what an AAAA record is and why this matters, an AAAA record is the type of DNS record that maps a hostname to its IPv6 address. It's the IPv6 equivalent to the A record, which maps a hostname to its IPv4 address. While the Internet is transitioning from IPv4 to IPv6, hosts are expected to be dual-homed, meaning they have both an IPv4 and an IPv6 address. When one system talks to another, it prefers IPv6, and falls back to IPv4 only if the peer doesn't support IPv6. To figure this out, the system first attempts an AAAA lookup, and if no AAAA record exists, it tries an A record lookup. So, when a name server does not respond to AAAA queries, not even with a response to say no AAAA record exists, the client has to wait until it times out before trying the A record lookup, causing the delays I was experiencing here. Cisco has a great article that goes into more depth about broken name servers and AAAA records.

(Note: the exact mechanics vary between operating systems. The Linux resolver tries AAAA lookups even if the system doesn't have IPv6 connectivity, meaning that even IPv4-only users experience these delays. Other operating systems might only attempt AAAA lookups if the system has IPv6 connectivity, which would mitigate the scope of this issue.)

A History of Brokenness

This is apparently not the first time Verisign's servers have had problems: A year ago, the name servers for ocsp.verisign.net exhibited the same broken behavior:

The unofficial response from Verisign was that the queries are being handled by a GSLB, which apparently means that we should not expect it to behave correctly.

"GSLB" means "Global Server Load Balancing" and I interpret that statement to mean Verisign is using an expensive DNS appliance to answer queries instead of software running on a conventional server. The snarky comment about such appliances rings true for me. Last year, I noticed that my alma matter's website was taking 30 seconds to load. I tracked the problem down to the exact same issue: the DNS servers for brown.edu were not returning any response to AAAA queries. In the process of reporting this to Brown's IT department, I learned that they were using buggy and overpriced-looking DNS appliances from F5 Networks, which, by default, do not properly respond to AAAA queries under circumstances that appear to be common enough to cause real problems. To fix the problem, the IT people had to manually configure every single DNS record individually to properly reply to AAAA queries.

I find it totally unconscionable for a DNS appliance vendor to be shipping a product with such broken behavior which causes serious delays for users and gives IPv6 a bad reputation. It is similarly outrageous for Verisign to be operating broken DNS servers that are in the critical path for an untold number of TLS handshakes. That gives HTTPS a bad reputation, and lends fuel to the people who say that HTTPS is too slow. It's truly unfortunate that even if you're Google and do everything right with IPv6, DNS, and TLS, your handshake speeds are still at the mercy of incompetent certificate authorities like Verisign.

Disabling OCSP

I worked around this issue by disabling OCSP (in Firefox, set security.OCSP.enabled to 0 in about:config). While OCSP may theoretically be good for security, since it enables browsers to reject certificates that have been compromised and revoked, in practice it's a total mess. Since OCSP servers are often unreliable or are blocked by restrictive firewalls, browsers don't treat OCSP errors as fatal by default. Thus, an active attacker who is using a revoked certificate to man-in-the-middle HTTPS connections can simply block access to the OCSP server and the browser will accept the revoked certificate. Frankly, OCSP is better at protecting certificate authorities' business model than protecting users' security, since it allows certificate authorities to revoke certificates for things like credit card chargebacks. As if this wasn't bad enough already, OCSP introduces a minor privacy leak because it reports every HTTPS site you visit to the certificate authority. Google Chrome doesn't even use OCSP anymore because it is so dysfunctional.

Finally Resolved

While I was writing this blog post, Verisign fixed their DNS servers and now every single one is returning a proper response to AAAA queries. I know for sure their servers were broken for at least two days. I suspect it was longer considering the slowness was happening for quite some time before I finally investigated.

I recently lost an afternoon dealing with a most vexing routing problem on a server which turned out to be the result of an ICMP redirect attack.

ICMP redirects are a "feature" of IP which allows a router to inform a host that there's a more efficient route to a destination and that the host should adjust its routing table accordingly. This might be OK on a trusted LAN, but on the wild Internet, where malice abounds, it may not be such a good idea to alter your routing table at someone else's whim. Nevertheless, ICMP redirects are enabled by default on Linux.

The problem manifested itself with a customer unable to contact one of our servers. The customer provided a traceroute which starred out after the last hop before our server, leading him to conclude that our firewall was blocking him. We would probably have concluded the same thing, except we knew that no such firewall existed. From our end, traceroutes to him starred out immediately before even reaching our default gateway. Even more bizarrely, if we picked a certain one of our server's four IP addresses as the source address for the traceroute, the traceroute worked! Furthermore, we were able to traceroute to adjacent IP addresses on the customer's subnet from any source address without issue.

We looked again and again at our routing table and our iptables rules. There was nothing to explain this behavior. This server is virtualized, so we were starting to suspect the underlying host, when I decided to look at the kernel's route cache with the ip route show cache command. What I saw worried me greatly:

root@tommy:~# ip route show cache | grep 198.168.103.11

198.168.103.11 via 10.254.87.146 dev eth0 src 66.228.52.206

198.168.103.11 from 66.228.52.251 via 10.254.87.146 dev eth0

198.168.103.11 from 66.228.52.209 via 10.254.87.146 dev eth0

198.168.103.11 from 66.228.52.206 via 66.228.52.1 dev eth0

198.168.103.11 from 66.228.52.250 via 10.254.87.146 dev eth0

These entries say to route packets to 198.168.103.11 (the customer's IP address, changed to protect their identity) via 10.254.87.146. However, 10.254.87.146 is not our default gateway. In fact, we don't use any private IP addresses that look remotely like that. The fourth entry, which has a legitimate gateway and applies only to packets with a source address of 66.228.52.206, explains why we could successfully use that one IP address as a source address.

I had read about ICMP redirect attacks before and suspected that they may be at play here. To test this hypothesis I spun up a test server and used an extremely useful tool called scapy to send my own fake ICMP redirect packets. The results were strange. Sending the fake ICMP redirect did not immediately put a bogus entry in the route cache. However, if I attempted to contact the host targeted by the redirect within 10 minutes of sending the redirect, then an entry appeared in the route cache that prevented me from contacting the host. Furthermore, once the entry got in the cache, there was no getting rid of it. Even if I flushed the cache, the rogue entry came back the next time I tried to contact the targeted host. The kernel was clearly keeping some separate state of redirected route entries, and as far as I could tell, there was no way to inspect it. This meant that the only way to recover the affected server was to reboot it!

There are clear denial-of-service possibilities. If you want to prevent a host (the "victim") from contacting another host (the "target"), just send a fake redirect packet for the target to the victim! Since you don't need to forge a packet's source address to send a rogue ICMP redirect, it will make its way past RP filters. The only constraint is that your victim needs to contact the target within 10 minutes of receiving the redirect for it to stick. This is easy to overcome: you can send the redirect when the victim is likely to be contacting the target, or simply send a new redirect every 10 minutes (that's hardly high volume). Or, more diabolically, if you have the ability to spoof source addresses, you can follow the redirect packet with a TCP SYN packet with its source address spoofed as the target. The victim will reply to the target with a SYN-ACK, and in doing so make permanent the effect of the ICMP redirect.

Obviously you need to disable ICMP redirect packets on any public-facing host. Unfortunately, the most intuitive and widely-documented way of disabling ICMP redirects on Linux (by writing 0 to /proc/sys/net/ipv4/conf/all/accept_redirects) doesn't always work! From Documentation/networking/ip-sysctl.txt of the Linux source:

accept_redirects - BOOLEAN Accept ICMP redirect messages. accept_redirects for the interface will be enabled if: - both conf/{all,interface}/accept_redirects are TRUE in the case forwarding for the interface is enabled or - at least one of conf/{all,interface}/accept_redirects is TRUE in the case forwarding for the interface is disabled accept_redirects for the interface will be disabled otherwise default TRUE (host) FALSE (router)

This arcane logic means that for a non-router (i.e. most servers), not only must /proc/sys/net/ipv4/conf/all/accept_redirects be 0, but so must /proc/sys/net/ipv4/conf/interface/accept_redirects. So, to recap, the following will reliably disable ICMP redirects (assuming your interface is eth0):

echo 0 > /proc/sys/net/ipv4/conf/all/accept_redirects

echo 0 > /proc/sys/net/ipv4/conf/eth0/accept_redirects

If you put that in a system startup script, you should be safe.

A final note: the precise behavior for handling ICMP redirects and the route cache may differ between kernel versions. My tests were conducted on 2.6.39.

Accurate time is essential on a server, and running ntpd is the best way to ensure it. Unfortunately, ntpd, especially on Debian, can be finicky and in the past I've had trouble with clock drift and ntpd failing to start on boot. Here are my best practices to avoid the problems.

On Debian, make sure lockfile-progs is installed

On Debian (and likely Ubuntu too), there's a nasty race condition on boot between ntpdate and ntpd. When the network comes up, ifupdown runs ntpdate to synchronize the clock. But at about the same time, ntpd starts. If ntpdate is still running when ntpd starts, ntpd can't bind to the local NTP port and terminates. Sometimes ntpd starts on boot and sometimes it doesn't!

The Debian scripts avoid this using locks, but only if the lockfile-progs package is installed. This is a Recommends: for ntpdate, but if you don't install Recommends: by default, you may miss this.

If you use DHCP, don't request ntp-servers

If your system gets its IP address from DHCP using dhclient, then by default dhclient will update your ntpd configuration with NTP server information it receives from the DHCP server. It is extremely frustrating to configure reliable upstream NTP servers only to have them replaced with unreliable servers (such as those that advertise phony leap seconds, as has happened to me multiple times). And the last thing you want is your configuration management system fighting with dhclient over what NTP servers to use.

To prevent this, edit /etc/dhcp/dhclient.conf and remove ntp-servers from the request line.

Don't use the undisciplined local clock (i.e. server 127.127.1.0)

Make sure these lines aren't in your ntp.conf:

server 127.127.1.0 fudge 127.127.1.0 stratum 10

These lines enable the Undisciplined Local Clock, and cause ntpd to start using your local clock as a time source if the real NTP servers aren't reachable. This can be useful if you want to keep a group of servers on a local network in sync even if your Internet connection goes down, but in general you don't need or want this. I've seen strange situations where the local clock becomes preferred over the real NTP servers, resulting in clock drift that goes uncorrected. Best to disable the local clock by removing all references to 127.127.1.0.

The implementation of std::basic_istream::ignore() in GCC's C++ standard library suffers from a serious flaw. After ignoring the n characters as requested, it checks to see if end-of-file has been reached. If it has, then the stream's eofbit is set. The problem is that to check for end-of-file, ignore() has to essentially peek ahead in the stream one character beyond what you've ignored. That means that if you ask to ignore all the characters currently available in the stream buffer, ignore() causes an underflow of the buffer. If it's a file stream, the buffer can be refilled by reading from the filesystem in a finite amount of time, so this is merely inefficient. But if it's a socket, this underflow can be fatal: your program may block forever waiting for bytes that never come. This is horribly unintuitive and is inconsistent with the behavior of std::basic_istream::read(), which does not check for end-of-file after reading the requested number of characters.

The origin of this problem is that the C++ standard is perhaps not as clear as it should be regarding ignore(). From section 27.7.2.3:

basic_istream<charT,traits>& ignore(streamsize n = 1, int_type delim = traits::eof());

Effects: Behaves as an unformatted input function (as described in 27.7.2.3, paragraph 1). After constructing a sentry object, extracts characters and discards them. Characters are extracted until any of the following occurs:

• if n != numeric_limits<streamsize>::max() (18.3.2), n characters are extracted
• end-of-file occurs on the input sequence (in which case the function calls setstate(eofbit), which may throw ios_base::failure (27.5.5.4));
• traits::eq_int_type(traits::to_int_type(c), delim) for the next available input character c (in which case c is extracted).

Note that the Standard does not specify the order in which the checks should be performed, suggesting that a conformant implementation may check for end-of-file before checking if n characters have been extracted, as GCC does. You may think that the order is implicit in the ordering of the bullet points, but if it were, then why would the Standard explicitly state the order in the case of getline()? From section 27.7.2.3:

basic_istream<charT,traits>& getline(char_type* s, streamsize n, char_type delim);

Effects: Behaves as an unformatted input function (as described in 27.7.2.3, paragraph 1). After constructing a sentry object, extracts characters and stores them into successive locations of an array whose first element is designated by s. Characters are extracted and stored until one of the following occurs:

1. end-of-file occurs on the input sequence (in which case the function calls setstate(eofbit));
2. traits::eq(c, delim) for the next available input character c (in which case the input character is extracted but not stored);
3. n is less than one or n - 1 characters are stored (in which case the function calls setstate(failbit)).

These conditions are tested in the order shown.

At least this is one GCC developer's justification for GCC's behavior. However, I have a different take: I believe that the only way to satisfy the Standard's requirements for ignore() is to perform the checks in the order presented. The Standard says that "characters are extracted until any of the following occurs." That means that when n characters have been extracted, ignore() needs to terminate, since this condition is among "any of the following." But, if ignore() first checks for end-of-file and blocks forever, then it doesn't terminate. This constrains the order in which a conformant implementation can check the conditions, and is perhaps why the Standard does not need to specify an explicit order here, but does for getline() where it really does want the end-of-file check to occur first.

I have left a comment on the GCC bug stating my interpretation. One problem with fixing this bug is that it will break code that has come to depend on eofbit being set if you ignore all the data remaining on a stream, though I'm frankly skeptical that much code would make that assumption. Also, both LLVM's libcxx and Microsoft Visual Studio (version 2005, at least) implement ignore() according to my interpretation of the Standard.

In the meantime, be very, very careful with your use of ignore(). Only use it on file streams or when you know you'll be ignoring fewer characters than are available to be read. And don't rely on eofbit being set one way or the other.

If you need a more reliable version of ignore(), I've written a non-member function implementation which takes a std::basic_istream as its first argument. It is very nearly a drop-in replacement for the member function (it even properly throws exceptions depending on the stream's exceptions mask), except that it returns the number of bytes ignored (not a reference to the stream) in lieu of making the number of bytes available by a call to gcount(). (It's not possible for a non-member function to set the value returned by gcount().)