2016 Brotli Update

Windows 10 Build 14986 adds support for Brotli compression to the Edge browser (but, somewhat surprisingly, not IE11). So at the end of 2016, we now have support for this improved compression algorithm in Chrome, Firefox, Edge, Opera, Brave, Vivaldi, and the long tail of browsers based on Chromium. Of modern browsers, only Apple is a holdout, with a “Radar” feature request logged against Safari but no public announcements.

Unfortunately, behavior across browsers varies at the edges:

  • Edge advertises support for and decodes Brotli compression on both HTTP and HTTPS requests.
  • Chrome advertises Brotli for HTTPS connections but will decode Brotli for both HTTPS and HTTP responses.
  • Firefox advertises Brotli for HTTPS connections and will not decode Brotli responses on HTTP responses.

There’s nothing horribly broken here: sites can safely serve Brotli content to clients that ask for it and those clients will probably decode it. The exception is when the request goes over HTTP… the reason Firefox and Chrome limit their request for Brotli to HTTPS is that, historically, middleboxes (like proxies and gateway filters) have been known to corrupt compression schemes other than gzip and deflate. This proved to be such a big problem in the rollout of SDCH (a now defunct compression algorithm Chrome supported), that the Brotli implementers decided to try to avoid the issue by requiring a secure transport.

-Eric

PS: Major sites, including Facebook and Google, have started deploying Brotli in production– if your site pulls fonts from Google Fonts, you’re already using Brotli today! In unrelated news, the 2016 Performance Calendar includes a post on serving Brotli from CDNs that don’t explicitly support it yet. Another recent post shows how to pair maximal compression for static files with fast compression for dynamically generated responses.

2016 Brotli Update

Do Not Lie to Users

Multiple people working on Outlook.com thought this was a reasonable design.

After a user deletes an email, then manually goes into the Deleted Items folder, then clicks Delete again, then acknowledges that they wish to Permanently Delete the deleted item:

Delete

… the item is still not deleted. You can “Recover deleted items” from your Deleted items folder:

Recover

… and voila, they’re all hiding out there:

Purge

Further, if you click the Purge button, you’ll find that it doesn’t actually do anything.

The poor user is expected to:

  1. Be aware of this insane behavior
  2. Individually check a box next to each unwanted message, then click Purge.

Microsoft’s design is offensively anti-privacy.

-Eric

PS: This sums it up pretty well.

image

Do Not Lie to Users

Useful Resources when Developing Chrome Extensions

I’ve built a handful of Chrome extensions this year, and I wrote up some of what I learned in a post back in March. Since then, I’ve found two more tricks that have proved useful.

First, the Chrome Canary channel includes a handy extension error console to quickly expose extension errors.

Simply open the chrome://extensions page and an Errors link will be available for side-loaded extensions (“Load Unpacked Extension”):

Errors link on Chrome://extensions

Error console

This error console can be enabled in other channels (dev/beta/stable) by launching with the --error-console command line argument.

Next, you’ll often end up with both a public version of your extension and a side-loaded developer build installed at the same time. If their Action icons are the same, it’s easy to get confused about which version is which.

To help clarify which is the development version, you can easily add a Dev badge to the icon of the developer build:
Dev badge

 

 

The code to add to your background.js file is straightforward and works even if the background page is not persistent:

 // Our background page isn't persistent, so subscribe to events.
 chrome.runtime.onStartup.addListener(()=> { init(); });
 // onInstalled fires when user uses chrome://extensions page to reload
 chrome.runtime.onInstalled.addListener(() => { init(); });

 function init()
 {
   // Add Badge notification if this is a dev-install
   chrome.management.getSelf( (o)=>{
     if (o.installType === "development") {
       chrome.browserAction.setBadgeText( {text: "dev"} );
     }
   });
 }

Finally, as mentioned in my last post, the Chrome Extension Source Viewer remains invaluable for quickly peeking at the code of existing extensions in the store.

I’ve had more fun writing Chrome extensions than anything else I’ve built this year. If you haven’t tried building an extension yet, I encourage you to try it out. Best of all, your new Chrome extension development skills will readily transfer to building extensions to Opera, Firefox, and Microsoft Edge.

-Eric

Useful Resources when Developing Chrome Extensions

Email Tracking Links are the Worst

Note: The non-secure email link vulnerability described in this post was fixed before I posted it publicly. The Clinton campaign’s InfoSec team was polite, communicative, and fun to work with.

All emailed links to HillaryClinton.com should now use HTTPS.

Since building the MoarTLS Browser Extension to track the use of non-secure hyperlinks, I’ve found that a huge number of otherwise-secure sites and services email links to users that are non-secure. Yes, I’ve ranted about this before; today’s case is a bit more interesting.

Here’s a recent example of an email with non-secure links:

Non-secure links in email

As you can see, all of the donation links are HTTP URLs to the hostname links.hillaryclinton.com, including the link whose text claims that it points to httpS://www.hillaryclinton.com. If you click on that link, you usually end up on a secure page, eventually:

Secure server

So, what’s going on here?

Why would a site with a HTTPS certificate send users through a redirect without one?

The answer, alas, is mundane click tracking, and it impacts almost anyone using an “Email Service Provider” (ESP) like MailChimp.

For instance, here’s the original “Download our Browser” email I got from Brave:

Brave Download link uses HTTP

Brave inserted a HTTPS link to their download in their original email template, but MailChimp rewrote it to non-securely point at their click-tracking server “list-manage.com”. The click tracker allows the emailer to collect metrics about the effectiveness of the email campaign (“How many users clicked, and on which link?”). There’s no inherent reason why the tracker must be non-secure, but this appears to be common practice for most ESPs, including the one used by the Clinton campaign.

Indeed, if you change Clinton’s injected tracking URL to HTTPS, you will see a certificate error in your browser:

Bad Certificate name mismatch

… revealing the source of the problem— the links subdomain of the main site is pointed at a third-party ESP, and that ESP hasn’t bothered to acquire a proper certificate for it.

DNS reveals the "links" domain is pointed at a third party

The entire links subdomain is pointed at a 3rd-party server. A friend mused: “Clinton could’ve avoided this whole debacle if she were running her own email servers.”

So What, Who Cares?

When I complain about things like this on Twitter, I usually get at least a few responses like this:

Who cares tweet

The primary problem is that the responder assumes that the HTTP link will reliably redirect to the HTTPS page… and this is true in most cases, except when there’s an attacker present.

The whole point of HTTPS is to defend against network-based adversaries. If there’s an active man-in-the-middle (MITM) attacker on the network, he can easily perform a sslstripping attack, taking over the non-secure connection from user, fooling the user into submitting their private information. The attacker could simply keep the connection on HTTP so he can monitor and manipulate it, or he could redirect the victim to a fake domain he controls, even one with a valid HTTPS certificate (e.g. https://donations.hillary-clinton.com).

Okay, so that’s bad.

Unfortunately, it gets worse in the Clinton case. Normally a bad guy taking advantage of SSL Stripping  still needs to fool the user into giving up something of value– not a high bar, but nevertheless. In the case of Clinton’s donation’s link, there’s a bigger problem, alluded to in the text of the email itself:

Donations go through "immediately"

That’s right—if you click the link, the server collects your money, no questions asked. Security researchers immediately recognize the threat of a cross-site request forgery… any web page or MITM could direct a victim’s browser to the target link and cause money to be withdrawn from their bank account. To protect against that, a properly developed site includes a secret canary in the URL so that the attacker cannot generate a valid link. And if you look at the markup of the email you see that the campaign has done just that (behind the black boxes to protect my account):

CSRF Canary

Unfortunately, there’s a fatal flaw here: the link is HTTP, which means that the canary is sent in raw form over the network, and the canary doesn’t expire after it’s been used. Anyone who views the unprotected HTTP traffic can collect my secret token and then feed the link back to my browser, forcing me to donate over and over and over again without any kind of prompt.

Aside: Beyond the security problem, there’s a significant functionality problem here. In the HTTP protocol, GET requests link those sent in navigations are meant to be idempotent, a fancy way of saying that sending the same request more than one time shouldn’t have any side effects. The Clinton campaign’s donation page, however, will bill the user every single time the donation link is loaded no matter why the page was loaded. Even a user who is not under attack can suffer multiple-billing if they don’t immediately close the tab after donating. If the user navigates that tab to another site, then clicks the browser’s back button, they’re charged again. Clicking back and forward a few times to figure out what’s happening? Billed over and over and over.

Things are even worse on a memory-constrained device… browsers like Chrome will “page out” tabs to save memory as you switch applications. When you switch back to the browser, it swaps the page back in, reloading it. And you’re billed again:

Push notification from AMEX reveals I've been charged again

… billing continues each time the browser is activated until you have the good sense to close the tab. (You can force Desktop Chrome to manually discard a tab early by visiting chrome://discards/; you’ll note you’re billed again the next time you click the tab).

 

Whether you’re a Presidential Campaign or a streaming music site, please use HTTPS everywhere—there’s no good excuse not to protect your users. And if you’re taking users’ money, you need to be very very sure that your requests contain a nonce to require confirmation before double-billing.

Thanks for your help in securing the web!

-Eric Lawrence

Email Tracking Links are the Worst

Troubleshooting Windows 10 Bluescreens

I recently bought a Dell XPS 8900 desktop system with Windows 10. It ran okay for a while, but after enabling Hyper-V, every few minutes the system would freeze for a few seconds and then reboot with no explanation. Looking at the Event Viewer’s Windows Logs > System revealed that the system had bugchecked (blue screened):

Event Viewer - BugcheckBugcheck 0x1a indicates a problem with “Memory Management” .

Run WinDbg as Administrator. File > Open Crash Dump:

WinDBG open crash dump 

Open C:\Windows\memory.dmp. Wait for symbols to download:

Debuggee not connected; symbols downloading

If symbols aren’t downloaded automatically, try typing .symfix and then .reload in the command prompt at the bottom.

Use !analyze -v says WinDBG

Then, follow the tool’s advice and run !analyze -v to have the debugger analyze the crash. WinDBG presents a surprisingly readable explanation:

WinDBG notes driver memory corruption

So a driver’s at fault, but which one?

Stack trace points at WiFi

It looks like bcmwl63a, for which symbols aren’t loaded, one clue that this isn’t Microsoft’s code. Let’s find out more about it using lm vm bcmw163a:

 

Debugger points at Wifi driver

Pop over to the listed path to examine the file’s properties, and see that it’s the WiFi driver:

Driver details

The Dell 1560 802.11ac card is the same type as found in my Dell XPS 13” notebook PC, where it was responsible for a flurry of bluescreens last year. The driver appears to have improved (the XPS 13 doesn’t crash anymore), but it looks like some corner cases got missed, likely related to the Hyper-V virtual networking code. Rather than waiting for an updated driver, the experts on Twitter suggested I simply upgrade to the Intel 7265 and install the latest Intel PROSet wireless driver. At $20 on Amazon, this seemed like a fine approach.

The upgrade was straightforward and would’ve taken less than 5 minutes to install except one of the nearly microscopic sockets broke off as I removed the Dell card’s antenna cables:

BrokenSocket

I used a needle to remove the broken pieces from the antenna’s connector before it would fit on the new card’s socket. After connecting the antenna, the new card easily slid into the slot and Windows recognized it on next boot. I used Device Manager to ensure the drivers loaded for the new card’s Bluetooth support, and installed the latest PROSet driver. Everything’s been working great since.

While WinDBG is one of the more inscrutable tools I use, it worked great in this situation and would point even a novice in the right direction.

 

-Eric

Troubleshooting Windows 10 Bluescreens

File the Bug

Two experiences this week reminded me of a very important principle for improving the quality of software… if you see something, say something. And the best way to do that is to file a bug.

Something Weird? File a bug!

The first case was last Thursday, when a user filed a bug in Chrome’s tracker noting that Chrome’s window border icons often got “stuck” in a hover state after being moused over. It was a clear, simple bug report and it was easily reproduced. I’ve probably hit this a hundred times over the years and didn’t think much of it… “probably some weird thing in my graphics card or some utility I’m running.” It never occurred to me that everybody else might be seeing this, or that it was exhibited only by Chrome.

Fortunately, the bug report showed that this issue was something others were hitting too, so I took a look. The problem proved to be almost unique to Chrome (not occurring in other Windows applications), and has existed for at least seven years, reproducing on every version from Windows Vista to Windows 10.

A scan of the bug tracker suggests that Thursday’s report was the first time in those seven years that this bug was filed; less than a week later, the simple fix is checked in and on the way to Chrome 54. Obviously, this is only a minor cosmetic issue, but we want our browser looking good at all times!

Animation of the fixed bug

Another cool aspect of this fix is that it will fix other applications too… the Opera and Vivaldi browsers are based on Chromium open-source roots and inherited this problem; they’ll probably pick up this fix shortly too.

th;df – Say Something Anyway

Even if you don’t file a bug, you should still say something. Recently, Ana Tudor noted on Twitter that her system was in a state after restart where neither Chrome nor Brave could render web content; both browsers showed the “crashed tab” experience, even after restarting and reinstalling the browsers. Running with the no-sandbox flag worked, and rebooting the system fully solved the problem. Her report sounded suspiciously similar to a problem I’d encountered back in April; fortunately, I’d filed a bug.

At the time, that bug was deemed unreproducible and I’d dismissed it as some wonkiness on my specific system, but Ana’s complaint brought this back to my attention. She’d also added another piece of data I didn’t have in my original report—the problem also occurred in Brave, but not Firefox or IE.

Even more fortunately, I hit this problem again after a system reboot yesterday, and because of Ana’s report, I was no longer convinced that this bug was some weird quirk on just my system. Playing with the repro, I found that neither Opera nor Vivaldi reproduced the problem; both of those browsers are architecturally similar to Brave, but importantly, both are 32-bit. So this was a great clue that the problem was specific to 64-bit. And I confirmed this, finding that the bug repro’d only in 64-bit Chrome Canary but not in 32-bit Canary. Now we’re cooking with gas!

I built Chromium and ran it through WinDBG, seeing that when the sandboxed content renderer process was starting up, it was hitting three debug breakpoints before dying. The breakpoints were in sandbox::InterceptionAgent::OnDllLoad, a function Chrome uses to thunk certain Windows APIs to inject security filters. At this point, and with a reliable repro in hand, my smarter colleague took over and quickly found that the code to allocate memory for the thunk was failing, due to some logic bugs. Thunks must be located at a particular place in memory – within 2gb of the thunked function – and the code to place our thunks was failing when ASLR randomly loaded the kernel32, gdi32, and user32 DLLs at the very top of the address space, leaving no room for our thunks. When the allocation failed, Chrome refused to allow the DLL to be loaded into the sandbox, and the renderer necessarily died. After the user rebooted the system again, ASLR again moved the DLLs to some other location and (usually) this location gives us room to place our thunks. With 20/20 hindsight, the root cause of this bug (and the upcoming fix) are obvious.

But we only knew to look for the problem because Ana took the time to say something.

Final Thoughts

  • Browser telemetry is great—we catch crashes and all sorts of problems with it. But debugging via telemetry can be really challenging— more akin to solving a mystery than following a checklist. For instance, in the case of the sandbox bug, the fact that the problem reproduced in Brave was a huge clue, and not something we’d ever know from telemetry.
  • Well-run projects love bug reports. Back when I was building Fiddler, a lot of users I talked to said things like: “Well, it’s free and pretty good so I didn’t want to bother you with a complaint about some bug.” This is exactly backwards. For most of Fiddler’s lifetime, bug reports from the community were the only compensation I received from making the tool available to everyone for free. Getting bug reports meant I could improve the product without having to pay for test machines and devices, hire a test organization, etc, etc. When I eventually sold Fiddler to Telerik, a large part of the value they were buying was the knowledge that the tool had been battle-tested by millions of users over 9 years and that I’d fixed thousands of bugs from that community.
  • Filing bugs is generally easy, and it’s especially easy for Chrome.
    • First, simply search for an existing bug on crbug.com
    • If you find it’s a known issue, star it so you get updates
    • If it’s not a known issue, click the New Issue button at the top-left
    • Tell us as much as you can about the problem. Try to put yourself in the reader’s shoes—we don’t know much about your system or scenario, so the more details you can provide, the better.
  • Screenshots and URLs that reproduce problems are invaluable.
  • Find a bug in another browser? Report it!

 

Thanks for your help in improving our digital world!

 

-Eric

File the Bug