Spying on HTTPS

When I launched Chrome on Thursday, I saw something unexpected:

SSLKeyLogfile

While most users probably would have no idea what to make of this, I happened to know what it means– Chrome is warning me that the system configuration has instructed it to leak the secret keys it uses to encrypt and decrypt HTTPS traffic to a stream on the local computer.

Looking at the Chrome source code, this warning was newly added last week. More surprising was that I couldn’t find the SSLKeyLogFile setting anywhere on my system. Opening a new console showed that it wasn’t set:

C:\WINDOWS\system32>set sslkeylogfile
Environment variable sslkeylogfile not defined

…and opening the System Properties > Advanced > Environment Variables UI showed that it wasn’t set for either my user account or the system at large. Weird.

Fortunately, I understood from past investigations that a process can have different environment variables than the rest of the system, and Process Explorer can show the environment variables inside a running process. Opening Chrome.exe, we see that it indeed has an SSLKEYLOGFILE set:

SSLKeyLogfileEB

The unusual syntax with the leading \\.\ means that this isn’t a typical local file path but instead a named pipe, which means that it doesn’t point to a file on disk (e.g. C:\temp\sslkeys.txt) but instead to memory that another process can see.

My machine was in this state because earlier that morning, I’d installed Avast Antivirus to attempt to reproduce a bug a Chrome user encountered. Avast is injecting the SSLKEYLOGFILE setting so that it can conduct a monster-in-the-browser attack (MITB) and see the encrypted traffic going into Chrome.

Update: In February 2024, Avast was assessed a $16.5M fine by the FTC over their handling of data gathered via this technique.

Makers of antivirus products know that browsers are one of the primary vectors by which attackers compromise PCs, and as a consequence their security products often conduct MITB attacks in order to scan web content. Antivirus developers have two common techniques to scan content running in the browser:

  1. Code injection
  2. Network interception

Code Injection

The code injection technique relies upon injecting security code into the browser process. The problem with this approach is that native code injections are inherently fragile– any update to the browser might move its functions and data structures around such that the security code will fail and crash the process. Browsers discourage native code injection, and the bug I was looking at was related to a new feature, RendererCodeIntegrity, that directs the Windows kernel to block loading of any code not signed by Microsoft or Google into the browser’s renderer processes.

An alternative code-injection approach relies upon using a browser extension that operates within the APIs exposed by the browser– this approach is more stable, but can address fewer threats.

Even well-written code injections that don’t cause stability problems can cause significant performance regressions for browsers– when I last looked at the state of the industry, performance costs for top AV products ranged from 20% to 400% in browser scenarios.

Network Interception

The Network interception technique relies upon scanning the HTTP and HTTPS traffic that goes into the browser process. Scanning HTTP traffic is straightforward (a simple proxy server can do it), but scanning HTTPS traffic is harder because the whole point of HTTPS is to make it impossible for a network intermediary to view or modify the plaintext network traffic.

Historically, the most common mechanism for security-scanning HTTPS traffic was to use a monster-in-the-middle (MitM) proxy server running on the local computer. The MITM would instruct Windows to trust a self-signed root certificate, and it would automatically generate new interception certificates for every secure site you visit. I spent over a decade working on such a MITM proxy server, the Fiddler Web Debugger. Many Enterprise security products offer a MitM (“TLS Inspection”) option, including products from Palo Alto, Broadcom, and Microsoft’s own Entra Internet Access.

There are many problems with using a MitM proxy, however. The primary problem is that it’s very very hard to ensure that it behaves exactly as the browser does and that it does not introduce security vulnerabilities. For instance, if the MITM’s certificate verification logic has bugs, then it might accept a bogus certificate from a spoof server and the user would not be warned– Avast used to use a MITM proxy and had exactly this bug; they were not alone. Similarly, the MITM might not support the most secure versions of protocols supported by the browser and server (e.g. TLS/1.3) and thus using the MITM would degrade security. Some protocol features (e.g. Client Certificates) are incompatible with MITM proxies. And lastly, some security features (specifically certificate pinning) are fundamentally incompatible with MITM certificates and are disabled when MITM certificates are used.

Given the shortcomings of using a MITM proxy, it appears that Avast has moved on to a newer technique, using the SSLKeyLogFile to leak the secret keys HTTPS negotiates on each connection to encrypt the traffic. Firefox and Chromium support this feature, and it enables decryption of TLS traffic without using the MITM certificate generation technique. While browser vendors are wary of any sort of interception of HTTPS traffic, this approach is generally preferable to MITM proxies.

There’s some worry that Chrome’s new notification bar might drive security vendors back to using more dangerous techniques, so this notification might not make its way into the stable release of Chrome.

When it comes to browser architecture, tradeoffs abound.

-Eric

PS: I’m told that Avast may be monetizing the data they’re decrypting.
Update: In February 2024, Avast was assessed a $16.5M fine by the FTC over their handling of data gathered via this technique.

Appendix: Peeking at the Keys

If we point the SSLKeyLog setting at a regular file instead of a named pipe:

chrome --ssl-key-log-file=C:\temp\sslkeys.txt

…we can examine the file’s contents as we browse to reveal the encryption keys:

ExportedKeys

This file alone isn’t very readable for a human (even if you read Mozilla’s helpful file format documentation), but you can configure tools like Wireshark to make use of it and automatically decrypt captured TLS traffic back to plaintext.

Livin’ on the Edge: Dude Where’s My Fix?!? (Redux)

In my last post, I showed you how to use OmahaProxy’s Find Releases tool to discover which versions of Chrome contain a given bugfix.

I noted that if you’re using Microsoft’s new Chromium-based Edge, you can look at the edge://version page or this extension to see the upstream Chrome version upon which Edge is based:

Oct 2022 Update: Unfortunately, the edge://version page approach no longer works. The browser has started sanitizing the User-Agent string to zero out all of the sub-build number information in the User-Agent string, such that Edge will now claim e.g. Chrome/106.0.0.0. I’ve filed a bug on the Edge team asking them to add a new field to the edge://version page. The extension works properly:

ShowChromeVersion

Until today, I never realized that this version number can be off-by-one. I noticed this discrepancy after complaints about a nasty bug in Edge that caused a bunch of webpage renderers to crash. The fix, we learned from OmahaProxy, landed in version Chrome Canary 78.0.3872.0.

When Edge next updated, I was alarmed to see that the crash was still present, but the user-agent header contains Chrome/78.0.3872.0, the version that’s supposed to have the fix.

StillCrashing.png

What’s going on?

Chromium’s src/chrome/VERSION file gets updated shortly after the prior Chrome Canary ships, as you can see in this timeline:

  1. At 03:13 on Aug 1, /src/chrome/VERSION was updated to 3872.
  2. Around 04:00 on Aug 1, Edge’s code pump ingested all of the commits from upstream Chromium into our internal integration branch, preparing for our 78.0.242 Canary build.
  3. At 11:20 on Aug 1, the fix for the crash landed.
  4. At 23:16 on Aug 1, the last commit from Chromium Master that went into Chrome Canary 78.0.3872 landed.
    Within a few hours, the Canary branch was declared an Official release.
  5. At 03:13 on Aug 2, /src/chrome/VERSION was updated to 3873.

Because Edge’s code pump pulled between when the version number was bumped to 3872 [1] and when the crasher’s fix landed [3], Edge ended up with a build that has the 3872 version number, but without the fix that went into Chrome Canary 78.0.3872.

So, for the moment at least, the safe way to be sure that a given Edge build has a given Chromium fix is to ensure that the Chrome token in the user-agent string is later than the Chrome Canary build number in which the patch first shipped.

Chrome 3889 includes some fixes and bugs not present in Edge 3889

-Eric

Livin’ on the Edge: Dude Where’s My Fix?!?

Yesterday, we covered the mechanisms that modern browsers can use to rapidly update their release channels. Today, let’s look at how to figure out when an eagerly awaited fix will become available in the Canary channels.

By way of example, consider crbug.com/977805, a nasty beast that caused some extensions to randomly be disabled and marked corrupt:

corruption

By bisecting the builds to find where the regression was introduced, we discovered that the problem was the result of a commit with hash fa8cdc81f5 that landed back on May 20th. This (probably security) change exposed an earlier bug in Chromium’s extension verification system such that an aborted request for a resource in an extension (say, because a page getting torn down just as a content script was getting injected) resulted in the verification logic thinking that the extension’s resource file was corrupted on disk.

On July 12th, the area owner landed a fix with the commit hash of cad2f6468. But how do I know whether my browser has this fix already? In what version(s) did the fix get released?

To answer these questions, we turn back to our trusted OmahaProxy. In the Find Releases box at the bottom, paste the full or partial hash value into the box and hit the Find Releases button:

UPDATE: OmahaProxy was deprecated. Find out where CLs were landed using the ChromiumDash tool instead. Just paste the commit id into the box:

… and the first release with the change will be listed. Click the title to see a full page about where the change went:

The system will churn for a bit and then return the following page:

So, now we know two things: 1) The fix will be in Chromium-based browsers with version numbers later than 77.0.3852.0, and 2) So far, the fix only landed there and hasn’t been merged elsewhere.

Does it need to be merged? Let’s figure out where the original regression was landed using the same tool with the regressing change list’s hash:

regress
Now-deprecated OmahaProxy UI, but the ChromiumDash UI is similar enough.
regress

We see that the regression originally landed in Master before the Chrome 76 branch point, so the bug is in Chrome 76.0.3801 and later. That means that after the fix is verified, we’ll need to request that it be merged from Master where it landed, over to the 76 branch where it’s also needed.

Merged

We can see what that’ll look like by looking at the fix for crbug.com/980803. This regression in the layout engine was fixed by a1dd95e43b5 in 77, but needed to be put into Chromium 76 as well. So, it was, and the result is shown as:

Note: It’s possible for a merge to be performed but not show up here. The tool looks for a particular string in the merge’s commit message, and some developers accidentally remove or alter it.

Finally, if you’re really champing at the bit for a fix, you might run Find Releases on a commit hash and see

notyetin

Assuming you didn’t mistype the hash, what this means is that the fix isn’t yet in the Canary channel. If you were to clone the Chromium master @HEAD and build it yourself, you’d see the fix, but it’s not yet in a public Canary. In almost all cases, you’ll need to wait until the next morning (Pacific time) to get an official channel build with the fix.

Now, so far we’ve mostly focused on Chrome, but what about other Chromium-based browsers?

Things are mostly the same, with the caveat that most other Chromium-based browsers are usually days to weeks to (gulp) months behind Chrome Canary. Is the extensions bug yet fixed in my Edge Canary?

The simplest (and generally reliable) way to check is to just look at the Chrome version by visiting edge://version or using my handy Show Chrome Version browser extension.

Note: Edge began sanitizing the User-Agent string to zero out all of the sub-build number information in the User-Agent string, such that Edge will now claim e.g. Chrome/106.0.0.0. The Edge team added a new field to the edge://version page to show the upstream version.

As you can see in both places, Edge 77.0.220.0 Canary is based on Chromium 77.0.3843, a bit behind the 77.0.3852 version containing the extensions verification fix:

ShowChromeVersion

So, I’ll probably have to wait a few days to get this fix into my browser.

Warning: The “Chrome” token shown in Edge might be off-by-one. See my followup post for details.

Also, note that it’s possible for Microsoft and other Chromium embedders to “cherry-pick” critical fixes into our builds before our merge pump naturally pulls them down from upstream, but this is a relatively rare occurrence for Edge Canary. 

tl;dr: OmahaProxy was awesome; its replacement, ChromiumDash is also awesome.

-Eric

PS: To view the state of a code file as compiled into a particular build/branch, you can click the build number on ChromiumDash and then navigate into the folder containing the source file.

Great Product Support

And now for something completely different…

Shortly after we moved into our house in late 2012, the control panel on our GE Oven (model #JTP30B0M1BB) started to fall apart. The faceplate of the control panel was made of a plastic that wasn’t sufficiently heat-resistant. The labeled plastic began to bubble, crack, and peel. By 2018, the plastic covering many of the buttons had fallen off entirely. It looked bad, to put it mildly, as you can see in this photo I took after I took the panel off:

IMG_20190303_161013On a whim, I searched around to see whether maybe I could buy a new overlay to stick atop the old panel.

I learned that, far from being alone, so many other people had had this problem that GE had completely redesigned the control panel. A thread on a forum led me to the magic direct phone number to the GE Parts department (866-622-6184). We called and after supplying the model number and serial number of our oven, a free replacement control panel was on its way to our house. The new panel took under an hour to install, using just a screwdriver and socket wrench. Basically, turn off the power, unscrew the old panel, unplug 7 connections from the old panel, and plug them into the new panel.

IMG_20190303_161437.jpg

The new panel looks great, and the modified ventilation and layout seem much less likely to encounter any problem like this in the future.

IMG_20190716_224532

Beyond being happy about the outcome, I’m gob-smacked about this support process. I never would have expected GE to send a free replacement panel, especially considering that the oven was over ten years old and originally purchased by another buyer. We didn’t have to supply anything other than the serial number, didn’t get stuck on hold for tens of minutes, and we didn’t even pay for shipping.

I’m approximately 25 times more likely to buy an appliance from GE in the future.

And, I’m grateful for the Internet, because there’s no chance that I would’ve ever discovered this fix for a longstanding annoyance if it wasn’t so easy to find a community of people with the same problem offering helpful steps to resolve it.

 

 

Updating Browsers Quickly: Flags, Respins, and Components

By this point, most browser enthusiasts know that Chrome has a rapid release cycle, releasing a new stable version of the browser approximately every six 4 weeks (2022 Update: now every four weeks). The Edge team adopted that rapid release cadence for our new browser, and we’re already releasing new Edge Dev Channel builds every week.

What might be less obvious is that this six four week cadence represents an upper-bound for how long it might take for an important change to make its way to the user.

Background: Staged Rollouts

Chrome uses a staged rollout plan, which means only a small percentage (1%-5%) of users get the new version immediately. If any high-priority problems are flagged by those initial users, the rollout can be paused while the team considers how to best fix the problem. That fix might involve shipping a new build, turning off a feature using the experimentation server, or dynamically updating a component.

Let’s look at each.

Respins

If a serious security or functionality problem is found in the Stable Channel, the development team generates a respin of the release, which is a new build of the browser with the specific issue patched. The major and minor version numbers of the browser stay the same. For instance, on July 15th, Chrome Stable version 75.0.3770.100 was updated to 75.0.3770.142. Users who had already installed the buggy version in the channel are updated automatically, and users who haven’t yet updated to the buggy version will just get the fixed version when the rollout reaches them.

If you’re curious, you can see exactly which versions of Chrome are being delivered from Google’s update servers for each Channel using ChromiumDash.

Field Trial Flags

In some cases, a problem is discovered in a new feature that the team is experimenting with. In these cases, it’s usually easy for the team to simply remotely disable or reconfigure the experiment as needed using the experimental flags. The browser client periodically polls the development team’s servers to get the latest experimental configuration settings. Chrome codenames their experimental system “Finch,” while Microsoft calls ours “ECS” (Experimental Control System”) or “CFR” (Controlled Feature Rollout).

You can see your browser’s current field trial configuration by navigating to

chrome://version/?show-variations-cmd

The hexadecimal Variations list is generally inscrutable, but the Command-line variations section later in the page is often more useful and allows you to better understand what trials are underway. You can even use this list to identify the exact trial causing a particular problem.

Regular readers might remember that I’ve previously written about Chrome’s Field Trials system. You can find some documentation of Edge’s ECS here: https://docs.microsoft.com/en-us/deployedge/edge-configuration-and-experiments. The tl;dr is that you must ensure https://config.edge.skype.com is reachable and you should think long and hard before setting the ExperimentationAndConfigurationServiceControl policy.

Update: Chrome does not publish their field-trial configuration in human-readable form, but here is a tool to grab the complete configuration data from Google directly. In contrast, Edge’s configuration data is sent on a per-client basis, so there’s no way for the public to see the configuration for the entire population.

Components

In other cases, a problem is found in a part of the browser implemented as a “Component.” Components are much like hidden, built-in extensions that can be silently and automatically updated by the Component Updater.

The primary benefit of components is that they can be updated without an update to Chrome itself, which allows them to have faster (or desynchronized) release cadences, lower bandwidth consumption, and avoids bloat in the (already sizable) Chrome installer. The primary drawback is that they require Chrome to tolerate their absence in a sane way.

To me, the coolest part of components is that not only can they update without downloading a new version of the browser, in some cases users don’t even need to restart their browser to begin using the updated version of a component. As soon as a new version is downloaded, it can “take over” from the prior version.

To see the list of components in the browser, visit

chrome://components

In my Chrome Canary instance, I see the following components:

Components

As you can see, many of these have rather obtuse names, but here’s a quick explanation where I know offhand:

  • MEI Preload – Policies for autoplay (see chrome://media-engagement/ )
  • Intervention Policy – Controls interventions used on misbehaving web pages
  • Third Party Module – Used to exempt accessibility and other components from the Code Integrity protections on the browser’s process that otherwise forbid injection of DLLs.
  • Subresource Filter Rules – The EasyList adblock database used by Chrome’s built-in adblocker to remove ads from a webpage when the Safe Browsing service indicates that a site violates the guidelines in the Better Ads Standard.
  • Certificate Error Assistant – Helps users understand and recover from certificate errors (e.g. when behind a known WiFi captive portal).
  • Software Reporter Tool – Collects data about system configuration / malware.
  • CRLSet – List of known-bad certificates (used to replace OCSP/CRL).
  • pnacl – Portable Native Client (overdue for removal)
  • Chrome Improved Recovery Unsure, but comments suggest this is related to helping fix broken Google Updater services, etc.
  • File Type Policies – Maps a list of file types to a set of policies concerning how they should be downloaded, what warnings should be presented, etc. See below.
  • Origin Trials – Used to allow websites to opt-in to experimenting with future web features on their sites. Explainer.
  • Adobe Flash Player – The world’s most popular plugin, gradually being phased out; slated for complete removal in late 2020.
  • Widevine Content DecryptionA DRM system that permits playback of protected video content.

If you’re using an older Chrome build, you might see:

If you’re using Edge, you might see:

  • Trust Protection Lists – The list of known trackers and organizational lists used by the Tracking Prevention feature. Presently, this list derives from Mozilla’s Content Blocking list, which in turn derives from Disconnect.me‘s list.
  • Zxcvbn Data Dictionaries – Password strength evaluator metadata(?)
  • Edge AutoLaunch Protocols – Allow certain sites to launch certain protocols without a prompt. See this post.
  • Domain Actions – Allow Edge to apply certain compatibility overrides on specific sites.
  • Well Known Domains – A list of popular and generally reputable sites used by a variety of features.

If you’re using the Chromium-derived Brave browser, you’ll see that brave://components includes a bunch of extra components, including “Ad Blocker”, “Tor Client”, “PDF Viewer”, “HTTPS Everywhere”, and “Local Data Updater.”

If you’re using Chrome on Android, you might notice that it’s only using three components instead of thirteen; the missing components simply aren’t used (for various reasons) on the mobile platform. As noted in the developer documentation, “The primary drawback [to building a feature using a Component] is that [Components] require Chrome to tolerate their absence in a sane way.

Edge checks for updated versions of components one minute after startup, then every five hours after that.

Case Study: Fast Protection via Component Update

Let’s take a closer look at my favorite component, the File Type Policies component.

When the browser downloads a file, it must make a number of decisions for security reasons. In particular, it needs to know whether the file type is potentially harmful to the user’s device. If the filetype is innocuous (e.g. plaintext), then the file may be downloaded without any prompts. If the type is potentially dangerous, the user should be warned before the download completes, and security features like SafeBrowsing/SmartScreen should scan the completed download for malicious content.

In the past, this sort of “What File Types are Dangerous?” list was hardcoded into various products. If a file type were later found to be dangerous, patching these products with updated threat information required weeks to months.

In contrast, Chrome delivers this file type policy information using the File Type Policies component. The component lets Chrome engineers specify which types are dangerous, which types may be configured to automatically open, which types are archives that contain other files that may require scanning, and so on.

How does this work in the real world? Here’s an example.

Around a year ago, it was discovered that files with the .SettingContent-ms file extension could be used to compromise the security of the user’s device. Browsers previously didn’t take any special care when such files were downloaded, and most users had no idea what the files were or what would happen if they were opened. Everyone was caught flat-footed.

Less than a day after this threat came to light, a Chrome engineer simply updated a single file to mark the settingcontent-ms file type as potentially dangerous. The change was picked up by the component builder, and Chrome users across all versions and channels were protected as their browser automatically pulled down the updated component in the background within hours.

Ever faster!

-Eric

Appendix A: Component Update Policy

This information is current as of October 2020 but will likely change over time.

The Microsoft Edge ComponentUpdatesEnabled policy can be set to default (unset), enabled, or disabled.  In the case that a component supports the policy and the policy is disabled, the update will be blocked.  Otherwise, the component will be updated when a new version is found.

Component NameCan be disabled by policyApproximate size of CRX
Adobe FlashYes13MB
Subresource Filter RulesYes77kB
Trust Protection ListsYes54kB
Edge Improved RecoveryYesNot in use
CRLSetNo22kB
Origin TrialsNoLess than 1kB
Widevine Content Decryption ModuleYes5MB

Currently, only CRLSet and Origin Trials forcibly download their updates.

The update check step is a request asking what updates exist, and is very small (>1kB typically).  The CRLSet and Origin Trials will download based upon that information either way, but they are also fairly small.

The update check also allows the browser to know if components like Flash or Widevine are too old to use safely.  Widevine ships with a version bundled into the browser, and updates every few months otherwise; leaving updates disabled for it could impact viewing of some DRM protected videos until the next browser release.

Flash updates monthly, but is only sent to users that already have it or who request it (in edge://settings/content/flash).  The Flash configuration can additionally be disabled by Flash/Plugin specific group policies; see DefaultPluginsSetting – and set that to Block.  Then Flash will not update, even if other component updates are enabled. 2024 Update: Flash was removed years ago.

The update check runs every five hours; if configured for fast update, it starts 10 seconds after browser launch, but more normally 5 minutes after browser launch.  (Extensions do update after 10 seconds though, and they likely look similar from a network perspective.)

The Edge Browser installer includes the then-current version of only some of the components, so disabling ComponentUpdates should not break your browser, but users may have a suboptimal experience (because compatibility problems or other issues may be fixed by unavailable or outdated components).