File the Bug

Two experiences this week reminded me of a very important principle for improving the quality of software… if you see something, say something. And the best way to do that is to file a bug.

Something Weird? File a bug!

The first case was last Thursday, when a user filed a bug in Chrome’s tracker noting that Chrome’s window border icons often got “stuck” in a hover state after being moused over. It was a clear, simple bug report and it was easily reproduced. I’ve probably hit this a hundred times over the years and didn’t think much of it… “probably some weird thing in my graphics card or some utility I’m running.” It never occurred to me that everybody else might be seeing this, or that it was exhibited only by Chrome.

Fortunately, the bug report showed that this issue was something others were hitting too, so I took a look. The problem proved to be almost unique to Chrome (not occurring in other Windows applications), and has existed for at least seven years, reproducing on every version from Windows Vista to Windows 10.

A scan of the bug tracker suggests that Thursday’s report was the first time in those seven years that this bug was filed; less than a week later, the simple fix is checked in and on the way to Chrome 54. Obviously, this is only a minor cosmetic issue, but we want our browser looking good at all times!

Animation of the fixed bug

Another cool aspect of this fix is that it will fix other applications too… the Opera and Vivaldi browsers are based on Chromium open-source roots and inherited this problem; they’ll probably pick up this fix shortly too.

th;df – Say Something Anyway

Even if you don’t file a bug, you should still say something. Recently, Ana Tudor noted on Twitter that her system was in a state after restart where neither Chrome nor Brave could render web content; both browsers showed the “crashed tab” experience, even after restarting and reinstalling the browsers. Running with the no-sandbox flag worked, and rebooting the system fully solved the problem. Her report sounded suspiciously similar to a problem I’d encountered back in April; fortunately, I’d filed a bug.

At the time, that bug was deemed unreproducible and I’d dismissed it as some wonkiness on my specific system, but Ana’s complaint brought this back to my attention. She’d also added another piece of data I didn’t have in my original report—the problem also occurred in Brave, but not Firefox or IE.

Even more fortunately, I hit this problem again after a system reboot yesterday, and because of Ana’s report, I was no longer convinced that this bug was some weird quirk on just my system. Playing with the repro, I found that neither Opera nor Vivaldi reproduced the problem; both of those browsers are architecturally similar to Brave, but importantly, both are 32-bit. So this was a great clue that the problem was specific to 64-bit. And I confirmed this, finding that the bug repro’d only in 64-bit Chrome Canary but not in 32-bit Canary. Now we’re cooking with gas!

I built Chromium and ran it through WinDBG, seeing that when the sandboxed content renderer process was starting up, it was hitting three debug breakpoints before dying. The breakpoints were in sandbox::InterceptionAgent::OnDllLoad, a function Chrome uses to thunk certain Windows APIs to inject security filters. At this point, and with a reliable repro in hand, my smarter colleague took over and quickly found that the code to allocate memory for the thunk was failing, due to some logic bugs. Thunks must be located at a particular place in memory – within 2gb of the thunked function – and the code to place our thunks was failing when ASLR randomly loaded the kernel32, gdi32, and user32 DLLs at the very top of the address space, leaving no room for our thunks. When the allocation failed, Chrome refused to allow the DLL to be loaded into the sandbox, and the renderer necessarily died. After the user rebooted the system again, ASLR again moved the DLLs to some other location and (usually) this location gives us room to place our thunks. With 20/20 hindsight, the root cause of this bug (and the upcoming fix) are obvious.

But we only knew to look for the problem because Ana took the time to say something.

Final Thoughts

  • Browser telemetry is great—we catch crashes and all sorts of problems with it. But debugging via telemetry can be really challenging— more akin to solving a mystery than following a checklist. For instance, in the case of the sandbox bug, the fact that the problem reproduced in Brave was a huge clue, and not something we’d ever know from telemetry.
  • Well-run projects love bug reports. Back when I was building Fiddler, a lot of users I talked to said things like: “Well, it’s free and pretty good so I didn’t want to bother you with a complaint about some bug.” This is exactly backwards. For most of Fiddler’s lifetime, bug reports from the community were the only compensation I received from making the tool available to everyone for free. Getting bug reports meant I could improve the product without having to pay for test machines and devices, hire a test organization, etc, etc. When I eventually sold Fiddler to Telerik, a large part of the value they were buying was the knowledge that the tool had been battle-tested by millions of users over 9 years and that I’d fixed thousands of bugs from that community.
  • Filing bugs is generally easy, and it’s especially easy for Chrome.
    • First, simply search for an existing bug on crbug.com
    • If you find it’s a known issue, star it so you get updates
    • If it’s not a known issue, click the New Issue button at the top-left
    • Tell us as much as you can about the problem. Try to put yourself in the reader’s shoes—we don’t know much about your system or scenario, so the more details you can provide, the better.
  • Screenshots and URLs that reproduce problems are invaluable.
  • Find a bug in another browser? Report it!

 

Thanks for your help in improving our digital world!

 

-Eric

File the Bug

Using Fiddler With iOS 10 and Android 7

If you’ve tried to use Fiddler with iOS10 beta or Android 7 Nougat, you have probably found that HTTPS decryption isn’t working, even if you use the latest Fiddler and the Fiddler Certificate Maker add-on. Unfortunately, at the moment both platforms are broken, but for different reasons. In both cases, the client will fail to receive responses for HTTPS requests, and Fiddler will only show a CONNECT tunnel.

iOS 10 Bug

The current beta of iOS 10 has a bug whereby, if the response to a HTTP CONNECT tunnel request contains a Connection: close response header, the client will close the connection instead of doing as it should and waiting until the TCP/IP connection closes. A few minor platforms have had the same bug over the years, but iOS is definitely the first important platform with this issue. At least two bugs have been filed with the Apple “Radar” bug reporter.

Working around this limitation is simple. In Fiddler, click Rules > Customize Rules. Scroll to the OnBeforeResponse function. Just inside that function, add the following lines:

  if (oSession.HTTPMethodIs("CONNECT")) {
    oSession["ui-backcolor"] = "red";
    oSession.ResponseHeaders.Remove("Connection");
  }

Save the file and try connecting again.

Android 7 Feature

In contrast to the iOS regression, the change in Android 7 was intentional. The Android team has decided that, by default, HTTPS certificate validation for apps targeting API Level 24 and later will ignore all user-installed root certificates, meaning that your efforts to manually trust Fiddler’s root certificate will be fruitless. Individually application developers can temporarily override this change while debugging by updating the application’s configuration:

  <network-security-config>
    <debug-overrides>
      <trust-anchors>
        <!-- Trust user added CAs while debuggable only –>    
        <certificates src="user" />
      </trust-anchors>
    </debug-overrides>
  </network-security-config>

…or at all times…

  <network-security-config>
    <base-config>
      <trust-anchors>
        <!-- Trust preinstalled CAs --> 
        <certificates src="system" /> 
        <!-- Additionally trust user added CAs --> 
        <certificates src="user" /> 
      </trust-anchors>
    </base-config>
  </network-security-config>

…Unfortunately, these changes can only be undertaken by application developers and not end-users. End-users will probably need to jailbreak their devices, akin to what is required to circumvent certificate pinning.

-Eric

Using Fiddler With iOS 10 and Android 7

Cheating Authenticode, Redux

Back in 2014, I explained two techniques that have been used by developers to store information in Authenticode-signed executables without breaking the signature.

Recently, Kevin Jones pointed out that Chrome’s signed installer differs on each download, as you can see in this file comparison of two copies of the Chrome installer, one downloaded from IE and one from Firefox:

Binary diff shows different bytes

Surprisingly, the Chrome installer is not using either unvalidated data-injection technique I wrote about previously.

So, what’s going on?

The Extra Certificate Technique

Fortunately, Kevin wrote an awesome tool for examining Authenticode signatures at a deep level: Authenticode Lint, which he describes in this blog post. Running his tool with the default options, the solution to our mystery is immediately revealed. The signing block contains an extra certificate named (literally) “Dummy Certificate”:

Authenticode Lint reveals "Dummy Certificate"

By running the tool with the -extract argument, we can view the extra (unsigned) certificate included in the signature block and see that it contains a proprietary data field with the per-instance data:image

This technique for injecting unsigned data into a signature block could be a source of vulnerabilities unless you’re very careful; anyone using this Extra Certificate technique should follow the same best practices previously described for the Unvalidated Attributes technique.

Authenticode Lint is a great tool, and I strongly recommend that you use it to help ensure you’re following best-practices for Authenticode-signing. My favorite feature is that you can automate the tool to verify an entire folder tree of binaries. You can thus be confident that you’ve signed all of the expected files, and done so properly.

-Eric

Cheating Authenticode, Redux

TLS Fallbacks are Dead

Gravestone reading RIP Fallbacks
tl;dr

 

Just over 5 years ago, I wrote a blog post titled “Misbehaving HTTPS Servers Impair TLS 1.1 and TLS 1.2.”

In that post, I noted that enabling versions 1.1 and 1.2 of the TLS protocol in IE would cause some sites to load more slowly, or fail to load at all. Sites that failed to load were sending TCP/IP RST packets when receiving a ClientHello message that indicated support for TLS 1.1 or 1.2; sites that loaded more slowly relied on the fact that the browser would retry with an earlier protocol version if the server had sent a TCP/IP FIN instead.

TLS version fallbacks were an ugly but practical hack– they allowed browsers to enable stronger protocol versions before some popular servers were compatible. But version fallback incurs real costs:

  • security – a MITM attacker can trigger fallback to the weakest supported protocol
  • performance – retrying handshakes takes time
  • complexity – falling back only in the right circumstances, creating new connections as needed
  • compatibility – not all clients are willing or able to fallback (e.g. Fiddler would never fallback)

Fortunately, server compatibility with TLS 1.1 and 1.2 has improved a great deal over the last five years, and browsers have begun to remove their fallbacks; first fallback to SSL 3 was disabled and now Firefox 37+ and Chrome 50+ have removed fallback entirely.

In the rare event that you encounter a site that needs fallback, you’ll see a message like this, in Chrome:

Google Chrome 52 error

or in Firefox:

Firefox 45 error

Currently, both Internet Explorer and Edge fallback; first a TLS 1.2 handshake is attempted:

TLS 1.2 ClientHello bytes

and after it fails (the server sends a TCP/IP FIN), a TLS 1.0 attempt is made:

TLS 1.0 ClientHello bytes

This attempt succeeds and the site loads in IE. If you analyze the affected site using SSLLabs, you can see that it has a lot of problems, the key one for this case is in the middle:

Grade F on SSLLabs

This is repeated later in the analysis:

TLS Version intolerant for versions 1.2 and 1.3

The analyzer finds that the server refuses not only TLS 1.2 but also the upcoming TLS 1.3.

Unfortunately, as an end-user, there’s not much you can safely do here, short of contacting the site owners and asking them to update their server software to support modern standards. Fortunately, this problem is rare– the Chrome team found that only 0.0017% of TLS connections triggered fallbacks, and this tiny number is probably artificially high (since a spurious connection problem will trigger fallback).

 

-Eric

TLS Fallbacks are Dead

Non-Secure Clicktrackers–The Fastest Path from A+ to F

HTTPS only works if you use it.

Coinbase is an online bitcoin exchange backed by $106M in venture capital investment. They’ve got a strong HTTPS security posture, including the latest ciphers, a 4096bit RSA key, and advanced features like browser-preloaded HSTS and HPKP.

SSLLabs grades Coinbase’s HTTPS deployment an A+:

A+ Grade from SSLLabs

This is a well-secured site with a professional security team.

Here’s the email they just sent me:

"Add a debit card"

Let’s run the MoarTLS Analyzer on that:

All Red

That’s right… every hyperlink in this email is non-secure and any click can be intercepted and sent anywhere by a network-based attacker.

Sadly, Coinbase is far from alone in snatching security defeat from the jaws of victory; my #HTTPSFAIL folder includes a lot of other big names:

HTTPSFailures

 

It doesn’t matter how well you secure your castle if you won’t help your visitors get to it securely. Use HTTPS everywhere.

 

-Eric

Update: I filed a bug with Coinbase on HackerOne. Their security team says that they “agree” that these links should be HTTPS, but the problem is Mailchimp (their email vendor) and they can’t fix it. Mailchimp offers a security vulnerability reporting form, delivered exclusively over HTTP:

MailChimp

Coinbase isn’t the first service whose security is bypassed because their emails are sent with non-secure links; the Brave browser download announcements suffered the same problem.

Non-Secure Clicktrackers–The Fastest Path from A+ to F

File Paths in Windows

Handling file-system paths in Windows can have many subtleties, and it’s easy to forget how some of this very intricate system works under the covers.

Happily, a .NET developer has started blogging a bit about file paths, presumably as they work to improve .NET’s handling of paths longer than the legacy MAX_PATH limit of 260 characters:

Also not to be missed are blog posts about Windows’ handling of File URIs:

 

-Eric

File Paths in Windows