http

While there are many different ways for servers to stream data to clients, the Server-sent Events / EventSource Interface is one of the simplest. Your code simply creates an EventSource and then subscribes to its onmessage callback:

Implementing the server side is almost as simple: your handler just prefaces each piece of data it wants to send to the client with the string data: and ends it with a double line-ending (\n\n). Easy peasy. You can see the API in action in this simple demo.

I’ve long been sad that we didn’t manage to get this API into Internet Explorer or the Legacy Edge browser. While many polyfills for the API exist, I was happy that we finally have EventSource in the new Edge.

Yay! \o/

Alas, I wouldn’t be writing this post if I hadn’t learned something new yesterday.

Last week, a customer reached out to complain that the new Edge and Chrome didn’t work well with their webmail application. After they used the webmail site for a some indeterminate amount time, they noticed that its performance slowed to a crawl– switching between messages would take tens of seconds or longer, and the problem reproduced regardless of the speed of the network. The only way to reliably resolve the problem was to either close the tabs they’d opened from the main app (e.g. the individual email messages could be opened in their own tabs) or to restart the browser entirely.

As the networking PM, I was called in to figure out what was going wrong over video conference. I instructed the user to open the F12 Developer Tools and we looked at the network console together. Each time the user clicked on a message, new requests were created and sat in the (pending) state for a long time, meaning that the requests were getting queued and weren’t even going to the network promptly.

But why? Diagnosing this remotely wasn’t going to be trivial, so I had the user generate a Network Export log that I could examine later.

In examining the log using the online viewer, the problem became immediately clear. On the Sockets tab, the webmail’s server showed 19 requests in the Pending state, and 6 Active connections to the server, none of which were idle. The fact that there were six connections strongly suggested that the server was using HTTP/1.1 rather than HTTP/2, and a quick look at the HTTP/2 tab confirmed it. Looking at the Events tab, we see five outstanding URLRequests to a URL that strongly suggests that it’s being used as an EventSource:

Each of these sockets is in the READING_RESPONSE state, and each has returned just ten bytes of body data to each EventSource. The web application is using one EventSource instance of the app, and the user has five tabs open to the app.

And now everything falls into place. Browsers limit themselves to 6 concurrent connections per server. When the server supports HTTP/2, browsers typically need just one connection because HTTP/2 supports multiplexing many (typically 100) streams onto a single connection. HTTP/1.1 doesn’t afford that luxury, so every long-lived connection used by a page decrements the available connections by one. So, for this user, all of their network traffic was going down a single HTTP/1.1 connection, and because HTTP/1.1 doesn’t allow multiplexing, it means that every action in the UI was blocked on a very narrow head-of-line-blocking pipe.

Looking in the Chrome bug tracker, we find this core problem (“SSE connections can starve other requests”) resolved “By Design” six years ago.

Now, I’m always skeptical when reading old bugs, because many issues are fixed over time, and it’s often the case that an old resolution is no longer accurate in the modern world. So I built a simple repro script for Meddler. The script returns one of four responses:

  • An HTML page that consumes an EventSource
  • An HTML page containing 15 frames pointed at the previous HTML page
  • An event source endpoint (text/event-stream)
  • A JPEG file (to test whether connection limits apply across both EventSources and other downloads)

And sure enough, when we load the page we see that only six frames are getting events from the EventSource, and the images that are supposed to load at the bottom of the frames never load at all:

Similarly, if we attempt to load the page in another tab, we find that it doesn’t even load, with a status message of “Waiting for available socket…”

The web app owners should definitely enable HTTP/2 on their server, which will make this problem disappear for almost all of their users.

However, even HTTP/2 is not a panacea, because the user might be behind a “break-and-inspect” proxy that downgrades connections to HTTP/1.1, or the browser might conceivably limit parallel requests on HTTP/2 connections for slow networks. As noted in the By Design issue, a server depending on EventSource in multiple tabs might use a BroadcastChannel or a SharedWorker to share a single EventSource connection with all of the tabs of the web application.

Alternatively, swapping an EventSource architecture with one based on WebSocket (even one that exposes itself as a EventSource polyfill) will also likely resolve the problem. That’s because, even if the client or server doesn’t support routing WebSockets over HTTP/2, the WebSockets-Per-Host limit is 255 in Chromium and 200 in Firefox.

Stay responsive out there!

-Eric

Note: The non-secure email link vulnerability described in this post was fixed before I posted it publicly. The Clinton campaign’s InfoSec team was polite, communicative, and fun to work with.

All emailed links to HillaryClinton.com should now use HTTPS.

Since building the MoarTLS Browser Extension to track the use of non-secure hyperlinks, I’ve found that a huge number of otherwise-secure sites and services email links to users that are non-secure. Yes, I’ve ranted about this before; today’s case is a bit more interesting.

Here’s a recent example of an email with non-secure links:

Non-secure links in email

As you can see, all of the donation links are HTTP URLs to the hostname links.hillaryclinton.com, including the link whose text claims that it points to httpS://www.hillaryclinton.com. If you click on that link, you usually end up on a secure page, eventually:

Secure server

So, what’s going on here?

Why would a site with a HTTPS certificate send users through a redirect without one?

The answer, alas, is mundane click tracking, and it impacts almost anyone using an “Email Service Provider” (ESP) like MailChimp.

For instance, here’s the original “Download our Browser” email I got from Brave:

Brave Download link uses HTTP

Brave inserted a HTTPS link to their download in their original email template, but MailChimp rewrote it to non-securely point at their click-tracking server “list-manage.com”. The click tracker allows the emailer to collect metrics about the effectiveness of the email campaign (“How many users clicked, and on which link?”). There’s no inherent reason why the tracker must be non-secure, but this appears to be common practice for most ESPs, including the one used by the Clinton campaign.

Indeed, if you change Clinton’s injected tracking URL to HTTPS, you will see a certificate error in your browser:

Bad Certificate name mismatch

… revealing the source of the problem— the links subdomain of the main site is pointed at a third-party ESP, and that ESP hasn’t bothered to acquire a proper certificate for it.

DNS reveals the "links" domain is pointed at a third party

The entire links subdomain is pointed at a 3rd-party server. A friend mused: “Clinton could’ve avoided this whole debacle if she were running her own email servers.”

So What, Who Cares?

When I complain about things like this on Twitter, I usually get at least a few responses like this:

Who cares tweet

The primary problem is that the responder assumes that the HTTP link will reliably redirect to the HTTPS page… and this is true in most cases, except when there’s an attacker present.

The whole point of HTTPS is to defend against network-based adversaries. If there’s an active man-in-the-middle (MITM) attacker on the network, he can easily perform a sslstripping attack, taking over the non-secure connection from user, fooling the user into submitting their private information. The attacker could simply keep the connection on HTTP so he can monitor and manipulate it, or he could redirect the victim to a fake domain he controls, even one with a valid HTTPS certificate (e.g. https://donations.hillary-clinton.com).

Okay, so that’s bad.

Unfortunately, it gets worse in the Clinton case. Normally a bad guy taking advantage of SSL Stripping  still needs to fool the user into giving up something of value– not a high bar, but nevertheless. In the case of Clinton’s donation’s link, there’s a bigger problem, alluded to in the text of the email itself:

Donations go through "immediately"

That’s right—if you click the link, the server collects your money, no questions asked. Security researchers immediately recognize the threat of a cross-site request forgery… any web page or MITM could direct a victim’s browser to the target link and cause money to be withdrawn from their bank account. To protect against that, a properly developed site includes a secret canary in the URL so that the attacker cannot generate a valid link. And if you look at the markup of the email you see that the campaign has done just that (behind the black boxes to protect my account):

CSRF Canary

Unfortunately, there’s a fatal flaw here: the link is HTTP, which means that the canary is sent in raw form over the network, and the canary doesn’t expire after it’s been used. Anyone who views the unprotected HTTP traffic can collect my secret token and then feed the link back to my browser, forcing me to donate over and over and over again without any kind of prompt.

Aside: Beyond the security problem, there’s a significant functionality problem here. In the HTTP protocol, GET requests link those sent in navigations are meant to be idempotent, a fancy way of saying that sending the same request more than one time shouldn’t have any side effects. The Clinton campaign’s donation page, however, will bill the user every single time the donation link is loaded no matter why the page was loaded. Even a user who is not under attack can suffer multiple-billing if they don’t immediately close the tab after donating. If the user navigates that tab to another site, then clicks the browser’s back button, they’re charged again. Clicking back and forward a few times to figure out what’s happening? Billed over and over and over.

Things are even worse on a memory-constrained device… browsers like Chrome will “page out” tabs to save memory as you switch applications. When you switch back to the browser, it swaps the page back in, reloading it. And you’re billed again:

Push notification from AMEX reveals I've been charged again

… billing continues each time the browser is activated until you have the good sense to close the tab. (You can force Desktop Chrome to manually discard a tab early by visiting chrome://discards/; you’ll note you’re billed again the next time you click the tab).

 

Whether you’re a Presidential Campaign or a streaming music site, please use HTTPS everywhere—there’s no good excuse not to protect your users. And if you’re taking users’ money, you need to be very very sure that your requests contain a nonce to require confirmation before double-billing.

Thanks for your help in securing the web!

-Eric Lawrence

Chrome has landed their change that allows you to mark unsecure (HTTP) content as insecure or dubious. Visit chrome://flags/#mark-non-secure-as to set the toggle. You can choose to mark as Dubious:

image

…or as Non-Secure:

image

The expectation is that eventually one of these modes will be the default for sites that are transferred over insecure protocols like HTTP.

Personally, I’m not really a fan of either piece of iconography; to me, showing the lock at all implies that the site has some amount of security and maybe it’s just not perfect.

I’m hoping that after some transition period, we’ll end up with a more prominent notification that explains what the issue is and why humans might care.

In December of last year, I made the following proposal with tongue only slightly in cheek:

Meet “Nosy”, your HTTP-content indicator:

Of course, Nosy’s got a lot of things to say:

nosy2

Sites and services need to use secure protocols like HTTPS because users expect it. No, not all users will expect to see the letters HTTPS and probably don’t understand hashes, ciphers, and public key encryption. But they expect that when they visit your site, it was delivered to them unmolested, privately, and as you original designed it. And the only way to realistically ensure that these expectations are met is to use HTTPS.

-Eric Lawrence

There are many interesting thing to say about HTTP caching. I’ve blogged about them a lot in the past.

Today’s public service announcement to clear up two extremely common misconceptions:

1. The no-cache directive does not mean “do not cache” (even though IE versions prior to IE10 implemented it that way).

What it really means is do not reuse this item from the cache without first validating with the server that it is fresh.

If the no-cache directive does not specify a field-name, then 
a cache MUST NOT use the response to satisfy a subsequent request
without successful revalidation with the origin server.

2. The must-revalidate directive does not mean “you must revalidate this resource with the server before using it.”

What it really means is do not reuse this item from the cache after it expires without first validating with the server that it is fresh. It’s basically saying: “Don’t ignore the Expires and max-age directives.” Which a client absolutely shouldn’t be doing anyway.

If the response includes the "must-revalidate" cache-control
directive, the cache MAY use that response in replying to a
subsequent request. But if the response is stale, all caches
MUST first revalidate it with the origin server

-Eric