ericlaw talks about security, the web, and software in general
This post demonstrates a bug in WordPress. Ignore it.
This isn't a real domain. Don't hyperlink it because I explicitly did not use an A tag here https://eric-lawrence-wrote-this.com okay
This isn’t a real domain. Don’t hyperlink it because I explicitly did not use an A tag here https://eric–lawrence-wrote-this.com okay
Two experiences this week reminded me of a very important principle for improving the quality of software… if you see something, say something. And the best way to do that is to file a bug.
The first case was last Thursday, when a user filed a bug in Chrome’s tracker noting that Chrome’s window border icons often got “stuck” in a hover state after being moused over. It was a clear, simple bug report and it was easily reproduced. I’ve probably hit this a hundred times over the years and didn’t think much of it… “probably some weird thing in my graphics card or some utility I’m running.” It never occurred to me that everybody else might be seeing this, or that it was exhibited only by Chrome.
Fortunately, the bug report showed that this issue was something others were hitting too, so I took a look. The problem proved to be almost unique to Chrome (not occurring in other Windows applications), and has existed for at least seven years, reproducing on every version from Windows Vista to Windows 10.
A scan of the bug tracker suggests that Thursday’s report was the first time in those seven years that this bug was filed; less than a week later, the simple fix is checked in and on the way to Chrome 54. Obviously, this is only a minor cosmetic issue, but we want our browser looking good at all times!
Another cool aspect of this fix is that it will fix other applications too… the Opera and Vivaldi browsers are based on Chromium open-source roots and inherited this problem; they’ll probably pick up this fix shortly too.
Even if you don’t file a bug, you should still say something. Recently, Ana Tudor noted on Twitter that her system was in a state after restart where neither Chrome nor Brave could render web content; both browsers showed the “crashed tab” experience, even after restarting and reinstalling the browsers. Running with the no-sandbox flag worked, and rebooting the system fully solved the problem. Her report sounded suspiciously similar to a problem I’d encountered back in April; fortunately, I’d filed a bug.
At the time, that bug was deemed unreproducible and I’d dismissed it as some wonkiness on my specific system, but Ana’s complaint brought this back to my attention. She’d also added another piece of data I didn’t have in my original report—the problem also occurred in Brave, but not Firefox or IE.
Even more fortunately, I hit this problem again after a system reboot yesterday, and because of Ana’s report, I was no longer convinced that this bug was some weird quirk on just my system. Playing with the repro, I found that neither Opera nor Vivaldi reproduced the problem; both of those browsers are architecturally similar to Brave, but importantly, both are 32-bit. So this was a great clue that the problem was specific to 64-bit. And I confirmed this, finding that the bug repro’d only in 64-bit Chrome Canary but not in 32-bit Canary. Now we’re cooking with gas!
I built Chromium and ran it through WinDBG, seeing that when the sandboxed content renderer process was starting up, it was hitting three debug breakpoints before dying. The breakpoints were in sandbox::InterceptionAgent::OnDllLoad, a function Chrome uses to thunk certain Windows APIs to inject security filters. At this point, and with a reliable repro in hand, my smarter colleague took over and quickly found that the code to allocate memory for the thunk was failing, due to some logic bugs. Thunks must be located at a particular place in memory – within 2gb of the thunked function – and the code to place our thunks was failing when ASLR randomly loaded the kernel32, gdi32, and user32 DLLs at the very top of the address space, leaving no room for our thunks. When the allocation failed, Chrome refused to allow the DLL to be loaded into the sandbox, and the renderer necessarily died. After the user rebooted the system again, ASLR again moved the DLLs to some other location and (usually) this location gives us room to place our thunks. With 20/20 hindsight, the root cause of this bug (and the upcoming fix) are obvious.
But we only knew to look for the problem because Ana took the time to say something.
Thanks for your help in improving our digital world!
-Eric
If you’ve tried to use Fiddler with iOS10 beta or Android 7 Nougat, you have probably found that HTTPS decryption isn’t working, even if you use the latest Fiddler and the Fiddler Certificate Maker add-on. Unfortunately, at the moment both platforms are broken, but for different reasons. In both cases, the client will fail to receive responses for HTTPS requests, and Fiddler will only show a CONNECT tunnel.
After installing the FiddlerRoot certificate, one also needs to go to Settings -> General -> About -> Certificate Trust Settings and manually enable full trust for the FiddlerRoot root certificate, including accepting a dialog that says that this will allow a third-party to eavesdrop on all your communications.
The beta of iOS 10 had a bug whereby, if the response to a HTTP CONNECT tunnel request contains a Connection: close response header, the client will close the connection instead of doing as it should and waiting until the TCP/IP connection closes. A few minor platforms have had the same bug over the years, but iOS is definitely the first important platform with this issue. At least two bugs have been filed with the Apple “Radar” bug reporter.
Working around this limitation is simple. In Fiddler, click Rules > Customize Rules. Scroll to the OnBeforeResponse function. Just inside that function, add the following lines:
if (oSession.HTTPMethodIs("CONNECT")) {
oSession["ui-backcolor"] = "red";
oSession.ResponseHeaders.Remove("Connection");
}
Save the file and try connecting again.
In contrast to the iOS regression, the change in Android 7 was intentional. The Android team has decided that, by default, HTTPS certificate validation for apps targeting API Level 24 and later will ignore all user-installed root certificates, meaning that your efforts to manually trust Fiddler’s root certificate will be fruitless. Individually application developers can temporarily override this change while debugging by updating the application’s configuration:
<network-security-config>
<debug-overrides>
<trust-anchors>
<!-- Trust user added CAs while debuggable only -->
<certificates src="user" />
</trust-anchors>
</debug-overrides>
</network-security-config>
…or at all times…
<network-security-config>
<base-config>
<trust-anchors>
<!-- Trust preinstalled CAs -->
<certificates src="system" />
<!-- Additionally trust user added CAs -->
<certificates src="user" />
</trust-anchors>
</base-config>
</network-security-config>
…Unfortunately, these changes can only be undertaken by application developers and not end-users. End-users will probably need to jailbreak their devices, akin to what is required to circumvent certificate pinning.
By default, Fiddler-generated certificates are valid for five years (and backdated one year). However, this can cause an ERR_CERT_VALIDITY_TOO_LONG error in Chrome on Android. To fix this for Fiddler’s default (CertEnroll) certificate generator, run about:config in QuickExec to edit preferences. Set fiddler.certmaker.ValidDays to 820 and, if needed, reset your certificates in Fiddler using the Tools > Fiddler Options > HTTPS > Actions button.
-Eric
Here’s an image from a server with a LetsEncrypt certificate.
Back in 2014, I explained two techniques that have been used by developers to store information in Authenticode-signed executables without breaking the signature, including information about the EnableCertPaddingCheck registry flag that can be set to break the technique1.
Recently, Kevin Jones pointed out that Chrome’s signed installer differs on each download, as you can see in this file comparison of two copies of the Chrome installer, one downloaded from IE and one from Firefox:
Surprisingly, the Chrome installer is not using either unvalidated data-injection technique I wrote about previously.
So, what’s going on?
Fortunately, Kevin wrote an awesome tool for examining Authenticode signatures at a deep level: Authenticode Lint, which he describes in this blog post. Running his tool with the default options, the solution to our mystery is immediately revealed. The signing block contains an extra certificate named (literally) “Dummy Certificate”:
By running the tool with the -extract argument, we can view the extra (untrusted) certificate included in the signature block and see that it contains a proprietary data field with the per-instance data:
You can see this technique in use inside Microsoft Edge installers as well:
This technique for injecting unsigned data into a signature block could be a source of vulnerabilities (like this attack) unless you’re very careful; anyone using this Extra Certificate technique should follow the same best practices previously described for the Unvalidated Attributes technique. (Which Kevin further explored in an aborted defense against in his post about Authenticode Sealing).
Authenticode Lint is a great tool, and I strongly recommend that you use it to help ensure you’re following best-practices for Authenticode-signing. My favorite feature is that you can automate the tool to verify an entire folder tree of binaries. You can thus be confident that you’ve signed all of the expected files, and done so properly.
-Eric
1 In the years since my original “Cheating Authenticode” post, I’ve learned that even when EnableCertPaddingCheck is not enabled, the signature verification API will scan the data in the signature block of the file to attempt to detect markers of popular data archival formats (e.g. PKZIP Magic Bytes). If such markers are found, the signature will be treated as invalid. This marker scanning was a Windows Vista-era mitigation (later backported) that attempts to block the abuse of signed insecurely-self-extracting executable files: an attacker could swap out the compressed payload for a malicious payload without breaking the file’s signature.
text/plain 