Chrome

As we’ve been working to replatform the new Microsoft Edge browser atop Chromium, one interesting outcome has been early exposure to a lot more bugs in Chromium. Rapidly root-causing these regressions (bugs in scenarios that used to work correctly) has been a high-priority activity to help ensure Edge users have a good experience with our browser.

Stabilization via Channels

Edge’s code stabilizes as it flows through release channels, from the developer’s-only ToT/HEAD (Tip-of-tree, the latest commit in the source repository) to the Canary Channel build (updated daily) to the Dev Channel updated weekly, to the Beta Channel (updated a few times over its six week lifetime) to our eventual, as-yet-unreleased Stable Channel.

Until recently, Microsoft Edge was only available in Canary and Dev channels, which meant that any bugs landed in Chromium would almost immediately impact almost all users of Edge. Even as we added a Beta channel, we still found users reporting issues that “reproduce in all Edge builds, but not in Chrome.

As it turns out, most of the “not in Chrome” comparisons turn out to mean that the problems are “not repro in Chrome Stable.” And that’s because the either the regressions simply haven’t made it to Stable yet, or because the regressions are hidden behind Feature Flags that are not enabled for Chrome’s stable channel.

A common example of this is LayoutNG, a major update to the layout engine used in Chromium. This redesigned engine is more flexible than its predecessor and allows the layout engineers to more easily add the latest layout features to the browser as new standards are finalized. Unfortunately, changing any major component of the browser is almost certain to lead to regressions, especially in corner cases. Google had enabled LayoutNG by default in the code for Chrome 76 (and Edge picked up the change), but then subsequently used the Feature Flag to disable LayoutNG for the stable channel three days before Chrome 76 shipped. As a result, the new LayoutNG engine is on-by-default for Chrome Beta, Dev, and Canary and Edge Beta, Dev, and Canary.

The key difference is that Edge doesn’t yet have a public stable channel to which any bug-impacted users can retreat. Therefore, reproducing, isolating, and fixing regressions as quickly as possible is important for Edge engineers.

Isolating Regressions

When we receive a report of a bug that reproduces in Microsoft Edge, we follow a set of steps for figuring out what’s going on.

Check Latest Canary

The first step is checking whether it reproduces in our Edge Canary builds.

If not, then it’s likely the bug was already found and fixed. We can either tell our users to sit tight and wait for the next update, or we can search on CRBug.com to figure out when exactly the fix went in and inform the user specifically when a fix will reach them.

Check Upstream

If the problem still reproduces in the latest Edge Canary build, we next try to reproduce the problem in the equivalent build of Chrome or plain-vanilla Chromium.

If Not Repro in Chromium?

If the problem doesn’t not reproduce in Chrome, this implies that the problem was caused by Microsoft’s modifications to the code in our internal branches. Alternatively, it might also be the case that the problem is hidden in upstream Chrome behind an experimental flag, so sometimes we must go spelunking into the browser’s Feature Flag configuration by visiting:

chrome://version/?show-variations-cmd 

The Command-Line Variations section of that page reveals the names of the experiments that are enabled/disabled. Launching the browser with a modified version of the command line enables launching Chrome in a different configuration2.

If the issue really is unique to Edge, we can use git blame and similar techniques on our code to see where we might have caused the problem.

If Repro in Chromium?

If the problem does reproduce in Chrome or Chromium, that’s strong evidence that we’ve inherited the problem from upstream.

Sanity Check: Does it Repro in Firefox?

If the problem isn’t a crashing bug or some other obviously incorrect behavior, I will first check the site’s behavior in the latest Firefox nightly build, on the off chance that the browsers are behaving correctly and the site’s markup or JavaScript is actually incorrect.

Try tweaking common Flags

Depending on the area where the problem occurs, a quick next step is to try toggling Feature Flags that seem relevant on the chrome://flags page. For instance, if the problem is in layout, try setting chrome://flags/#enable-layout-ng to Disabled. If the problem seems to be related to the network, try toggling chrome://flags/#network-service-in-process, and so on.

Understanding whether the problem can be impacted by flags enables us to more quickly find its root cause, and provides us an option to quickly mitigate the problem for our users (by adjusting the flag remotely from our experimental configuration server).

Bisecting Regressions

The gold standard for root-causing regressions is finding the specific commit (aka “change list” aka “pull request” aka “patch”) that introduced the problem. When we determine which commit caused a problem, we not only know exactly what builds the problem affects, we also know what engineer committed the breaking change, and we know what exactly code was changed– often we can quickly spot the bug within the changed files.

Fortunately, Chromium engineers trying to find regressing commits have a super power, known as bisect-builds.py. Using this script is simple: You specify the known-bad Chromium version and a guesstimate of the last known good version. The script then automates the binary search of the builds between the two positions, requiring the user to grade each trial as “Good” or “Bad”, until the introduction of the regression is isolated.

The simplicity of this tool belies its magic— or, at least, the power of the log2 math that underlies it. OmahaProxy informs me that the current Chrome tree contains 692,278 commits. log2(692278) is 19.4, which means that we should be able to isolate any regressing change in history in just 20 trials, taking a few minutes at most1. And it’s rare that you’d want to even try to bisect all of history– between stable and ToT we see ~27250 commits, so we should be able to find any regression within such a range in just 15 trials.

On CrBug.com, regressions awaiting a bisect are tagged with the label Needs-Bisect, and a small set of engineers try to burn down the backlog every day. But running bisects is so easy that it’s usually best to just do it yourself, and Edge engineers are doing so constantly.

One advantage available to Googlers but not the broader Chromium community is the ability to do what’s called a “Per-Revision Bisect.” Inside Google, Chrome builds a new build for every single change (so they can unambiguously flag any change that impacts performance), but not all of these builds are public. Publicly, Google provides the Chromium Continuous Build Archive, an archive of builds that are built approximately every one to fifteen commits. That means that when we do bisects, we often don’t get back a single commit, but instead a regression range. We then must look at all of the commits within that range to figure out which was the culprit. In my experience, this is rarely a problem– most commits in the range obviously have nothing to do with the problem (“Change the icon for some far off feature”), so there are rarely more than one or two obvious suspects.

The Edge team does not currently expose our Edge build archives for bisects, but fortunately almost all of our web platform work is contributed upstream, so bisecting against Chromium is almost always effective.

Recent Example

Last Thursday, we started to get reports of a “missing certificate” problem in Microsoft Edge, whereby the browser wasn’t showing the expected Lock icon for HTTPS pages that didn’t contain any mixed content:

The certificate is missing

While the lock was missing for some users, it was present for others. After also reproducing the issue in Chrome itself, I filed a bug upstream and began investigating.

Back when I worked on the Chrome Security team, we saw a bunch of bugs that manifested like this one that were caused by refactoring in Chrome’s navigation code. Users could hit these bugs in cases where they navigated back/forward rapidly or performed an undo tab close operation, or sites used the HTML5 History API. In all of these cases, the high-level issue is that the page’s certificate is missing in the security_state: either the ssl_status on the NavigationHandle is missing or it contains the wrong information.

This issue, however, didn’t seem to involve navigations, but instead was hit as the site was loaded and thus it called to mind a more recent regression from back in March, where sites that used AppCache were missing their lock icon. That issue involved a major refactoring to use the new Network Service.

One fact that immediately jumped out at me about the sites first reported to hit this new problem is that they all use ServiceWorker (e.g. Hotmail, Gmail, MS Teams, Twitter). Like AppCache, ServiceWorker allows the browser to avoid hitting the network in response to a fetch. As with AppCache, that characteristic means that the the browser must somehow have the “original certificate” for that response from the network so it can set that certificate in the security_state when it’s needed.

But where does that certificate live?

Chromium stores the certificate for a given HTTPS response after the end of the cache entry, so it should be available whenever the cached resource is used3. A quick disk search revealed that ServiceWorker stores its scripts inside the folder:

%localappdata%\Microsoft\Edge SxS\User Data\Default\Service Worker\ScriptCache

Comparing the contents of the cache file on a “good” and a “bad” PC, we see that the certificate information is missing in the cache file for a machine that reproduces the problem:

The serialized certificate chain is present in the “Good” case

So, why is that certificate missing? I didn’t know.

I performed a bisect three times, and each time I ended up with the same range of a dozen commits, only one of which had anything to do with anything to do with caching, and that commit was for AppCache, not ServiceWorker.

More damning for my bisect suspect was the fact that this suspect commit landed (in 78.0.3890) the day after the build (3889) upon which the reproducing Edge build was based. I spent a bunch of time figuring out whether this could be the off-by-one issue in Edge build numbering before convincing myself that no, it couldn’t be: build number misalignment just means that Edge 3889 might not contain everything that’s in Chrome 3889.

Unless an Edge Engineer had cherry-picked the regressing commit into our 3889 (unlikely), the suspect couldn’t be the culprit.

Edge 3889 doesn’t include all of the commits in Chromium 3889.

I posted my research into the bug at 10:39 PM on Friday and forty minutes later a Chrome engineer casually revealed something I didn’t realize: Chrome uses two different codepaths for fetching a ServiceWorker script– a “new_script_loader” and an “updated_script_loader.”

And instantly, everything fell into place. The reason the repro wasn’t perfectly reliable (for both users and my bisect attempts) was that it only happened after a ServiceWorker script updates.

  • If the ServiceWorker in the user’s cache is missing, it is downloaded by the new_script_loader and the certificate is stored.
  • If the ServiceWorker script is present and is unchanged on the server, the lock shows just fine.
  • But if the ServiceWorker in the user’s cache is present but outdated, the updated_script_loader downloads the new script… and omits the certificate chain. The lock icon disappears until the user clears their cache or performs a hard (CTRL+F5) refresh, at which point the lock remains until the next script update.

With this new information in hand, building a reliable reduced repro case was easy– I just ripped out the guts of one of my existing PWAs and configured it so that it updated itself every five seconds. That way, on nearly every load, the cached ServiceWorker would be deemed outdated and the script redownloaded.

With this repro, we can kick off our bisect thusly:

python tools/bisect-builds.py -a win -g 681094 -b 690908 --verify-range --use-local-cache -- --no-first-run --user-data-dir=/temp https://webdbg.com/apps/alwaysoutdated/

… and grade each build based on whether the lock disappears on refresh:

Grading each build based on whether the lock disappears

Within a few minutes, we’ve identified the regression range:

A culprit is found

In this case, the regression range contains just one commit— one that turns on the new ServiceWorker update check code. This confirms the Chromium engineer’s theory and that this problem is almost identical to the prior AppCache bug. In both cases, the problem is that the download request passed kURLLoadOptionNone and that prevented the certificate from being stored in the HttpResponseInfo serialized to the cache file. Changing the flag to kURLLoadOptionSendSSLInfoWithResponse results in the retrieval and storage of the ssl_info, including the certificate.

The fix was quick and straightforward; it will be available in Chrome 78.0.3902 and the following build of Edge based on that Chromium version. Notably, because the bug is caused by failure to put data in a cache file, the lock will remain missing even in later builds until either the ServiceWorker script updates again, or you hard refresh the page once.

-Eric

1 By way of comparison, when I last bisected an issue in Internet Explorer, circa 2012, it was an extraordinarily painful two-day affair.

2 You can use the command line arguments in the variations page (starting at --force-fieldtrials=) to force the bisect builds to use the same variations.

Chromium also has a bisect-variations script which you can use to help narrow down which of the dozens of active experiments is causing a problem.

If all else fails, you can also reset Chrome’s field trial configuration using chrome.exe --reset-variation-state to see if the repro disappears.

3 Aside: Back in the days of Internet Explorer, WinINET had no way to preserve the certificate, so it always bypassed the cache for the first request to a HTTPS server so that the browser process would have a certificate available for its subsequent display needs.


By this point, most browser enthusiasts know that Chrome has a rapid release cycle, releasing a new stable version of the browser approximately every six weeks. The Edge team intends to adopt that rapid release cadence for our new browser, and we’re already releasing new Edge Dev Channel builds every week.

What might be less obvious is that this six week cadence represents an upper-bound for how long it might take for an important change to make its way to the user.

Background: Staged Rollouts

Chrome uses a staged rollout plan, which means only a small percentage (1%-5%) of users get the new version immediately. If any high-priority problems are flagged by those initial users, the rollout can be paused while the team considers how to best fix the problem. That fix might involve shipping a new build, turning off a feature using the experimentation server, or dynamically updating a component.

Let’s look at each.

Respins

If a serious security or functionality problem is found in the Stable Channel, the development team generates a respin of the release, which is a new build of the browser with the specific issue patched. The major and minor version numbers of the browser stay the same. For instance, on July 15th, Chrome Stable version 75.0.3770.100 was updated to 75.0.3770.142. Users who had already installed the buggy version in the channel are updated automatically, and users who haven’t yet updated to the buggy version will just get the fixed version when the rollout reaches them.

If you’re curious, you can see exactly which versions of Chrome are being delivered from Google’s update servers for each Channel using OmahaProxy.

Field Trial Flags

In some cases, a problem is discovered in a new feature that the team is experimenting with. In these cases, it’s usually easy for the team to simply remotely disable or reconfigure the experiment as needed using the experimental flags. The browser client periodically polls the development team’s servers to get the latest experimental configuration settings. Chrome codenames their experimental system “Finch,” while Microsoft calls ours “CFR” (Controlled Feature Rollout).

You can see your browser’s current field trial configuration by navigating to

chrome://version/?show-variations-cmd

The hexadecimal Variations list is generally inscrutable, but the Command-line variations section later in the page is often more useful and allows you to better understand what trials are underway. You can even use this list to identify the exact trial causing a particular problem.

Regular readers might remember that I’ve previously written about Chrome’s Field Trials system.

Components

In other cases, a problem is found in a part of the browser implemented as a “Component.” Components are much like hidden, built-in extensions that can be silently and automatically updated by the Component Updater.

The primary benefit of components is that they can be updated without an update to Chrome itself, which allows them to have faster (or desynchronized) release cadences, lower bandwidth consumption, and avoids bloat in the (already sizable) Chrome installer. The primary drawback is that they require Chrome to tolerate their absence in a sane way.

To me, the coolest part of components is that not only can they update without downloading a new version of the browser, in some cases users don’t even need to restart their browser to begin using the updated version of a component. As soon as a new version is downloaded, it can “take over” from the prior version.

To see the list of components in the browser, visit

chrome://components

In my Chrome Canary instance, I see the following components:

Components

As you can see, many of these have rather obtuse names, but here’s a quick explanation where I know offhand:

  • MEI Preload – Policies for autoplay (see chrome://media-engagement/ )
  • Intervention Policy – Controls interventions used on misbehaving web pages
  • Third Party Module – Used to exempt accessibility and other components from the Code Integrity protections on the browser’s process that otherwise forbid injection of DLLs.
  • Subresource Filter Rules – The EasyList adblock database used by Chrome’s built-in adblocker to remove ads from a webpage when the Safe Browsing service indicates that a site violates the guidelines in the Better Ads Standard.
  • Certificate Error Assistant – Helps users understand and recover from certificate errors (e.g. when behind a known WiFi captive portal).
  • Software Reporter Tool – Collects data about system configuration / malware.
  • CRLSet – List of known-bad certificates (used to replace OCSP/CRL).
  • pnacl – Portable Native Client (overdue for removal)
  • Chrome Improved Recovery Unsure, but comments suggest this is related to helping fix broken Google Updater services, etc.
  • File Type Policies – Maps a list of file types to a set of policies concerning how they should be downloaded, what warnings should be presented, etc. See below.
  • Origin Trials – Used to allow websites to opt-in to experimenting with future web features on their sites. Explainer.
  • Adobe Flash Player – The world’s most popular plugin, gradually being phased out; slated for complete removal in late 2020.
  • Widevine Content DecryptionA DRM system that permits playback of protected video content.

If you’re using an older Chrome build, you might see:

If you’re using Edge, you might see:

If you’re using the Chromium-derived Brave browser, you’ll see that brave://components includes a bunch of extra components, including “Ad Blocker”, “Tor Client”, “PDF Viewer”, “HTTPS Everywhere”, and “Local Data Updater.”

If you’re using Chrome on Android, you might notice that it’s only using three components instead of thirteen; the missing components simply aren’t used (for various reasons) on the mobile platform. As noted in the developer documentation, “The primary drawback [to building a feature using a Component] is that [Components] require Chrome to tolerate their absence in a sane way.

Case Study: Fast Protection via Component Update

Let’s take a closer look at my favorite component, the File Type Policies component.

When the browser downloads a file, it must make a number of decisions for security reasons. In particular, it needs to know whether the file type is potentially harmful to the user’s device. If the filetype is innocuous (e.g. plaintext), then the file may be downloaded without any prompts. If the type is potentially dangerous, the user should be warned before the download completes, and security features like SafeBrowsing/SmartScreen should scan the completed download for malicious content.

In the past, this sort of “What File Types are Dangerous?” list was hardcoded into various products. If a file type were later found to be dangerous, patching these products with updated threat information required weeks to months.

In contrast, Chrome delivers this file type policy information using the File Type Policies component. The component lets Chrome engineers specify which types are dangerous, which types may be configured to automatically open, which types are archives that contain other files that may require scanning, and so on.

How does this work in the real world? Here’s an example.

Around a year ago, it was discovered that files with the .SettingsContent-ms file extension could be used to compromise the security of the user’s device. Browsers previously didn’t take any special care when such files were downloaded, and most users had no idea what the files were or what would happen if they were opened. Everyone was caught flat-footed.

In less than a day after this threat came to light, a Chrome engineer simply updated a single file to mark the settings-content.ms file type as potentially dangerous. The change was picked up by the component builder, and Chrome users across all versions and channels were protected as their browser automatically pulled down the updated component in the background.

 

Ever faster!

-Eric

Note: I expect to update this post over time. Last update: 8/29/2019.

Compatibility Deltas

As our new Edge Insider builds roll out to the public, we’re starting to triage reports of compatibility issues where Edge76+ (the new Chromium-based Edge, aka Anaheim) behaves differently than the old Edge (Edge18, aka Spartan) and/or Google Chrome.

In general, Edge76+ will behave very similarly to Chrome, with the caveat that, to date, only Beta, Dev and Canary channels have been released. When looking at Chrome behavior, be sure to compare against the corresponding Chrome Beta, Dev and Canary channels.

However, we expect there will be some behavioral deltas between Edge76+ and its Chrome-peer versions, so I’ll note those here too.

Note: I’ve previously blogged about interop issues between Edge18 and Chrome.

Navigation

  • For security reasons, Edge76 and Chrome block navigation to file:// URLs from non-file URLs. If a browser user clicks on a file: link on a webpage, nothing happens (except an error message in the Developer Tools console, noting “Not allowed to load local resource: file://host/whatever”). In contrast, Edge18 (like Internet Explorer before it) allowed HTTP/HTTPS-served pages in your Intranet Zone to navigate to URLs that use the file:// URL protocol; only pages in the Internet Zone were blocked from such navigations. No override for this block is available.
  • In Edge18 and Internet Explorer, attempting to navigate to an App Protocol with no handler installed shows a prompt to visit the Microsoft Store to find a handler. In Chrome/Edge76+, the navigation attempt is silently ignored.
  • Edge 18 and Internet Explorer offer a msLaunchUri API for launching and detecting App Protocols. This API is not available in Edge 76 or Chrome.
  • Edge 18 and Internet Explorer allow an App Protocol handler to opt-out of warning the user on open using the WarnOnOpen registry key. Edge 76 and Chrome do not support this registry key.

Downloads

  • Unlike IE/Edge18, Edge76/Chrome do not support DirectInvoke, a scheme whereby a download is converted into the launch of an application with a URL argument. DirectInvoke is most commonly used when launching Office documents and when running ClickOnce applications. For now, users can workaround the lack of ClickOnce support by installing an extensionUpdate: In Edge 78, see the edge://flags/#edge-click-once setting.
  • Edge76/Chrome do not support the proprietary msSaveBlob or msSaveOrOpenBlob APIs supported in Edge18. In most cases, you should instead use an A element with a download attribute.
  • Edge18 did not support navigation to or downloading from data URLs via the download attribute; Edge76/Chrome allow the download of data URLs up to 2mb in length. In most cases, you should prefer blob urls.

HTTPS – TLS Protocol

  • Edge76 and Chrome enable TLS/1.3 by default; Edge18 does not support TLS/1.3 prior to Windows 10 19H1, and even on that platform it is disabled by default (and known to be buggy).
  • Edge76 and Chrome support a different list of TLS ciphers than Edge18.
  • Edge76 and Chrome send GREASE tokens in HTTPS handshakes; Edge18 does not.
  • Edge76 and Chrome prohibit connections for HTTP/2 traffic from using banned (weak) ciphers, showing ERR_HTTP2_INADEQUATE_TRANSPORT_SECURITY if the server attempts to use such ciphers. Edge18 did not enforce this requirement. This has primarily impacted intranet websites served by IIS on Windows Server 2012 where the server was either misconfigured or does not have the latest updates installed. Patching the server and/or adjusting its TLS configuration will resolve the problem.

HTTPS – Certificates

  • Edge76 and Chrome require that a site’s certificate contain its domain name in the SubjectAltName (SAN) field. Edge 18 permits the certificate to omit the SAN and if the domain name is in the Subject Common Name (CN) field. (All public CAs use the SAN; certificates that chain to a local/enterprise trusted root may need to be updated).
  • Edge76 and Chrome require certificates that chain to trusted root CAs to be logged in Certificate Transparency (CT). This generally isn’t a problem because public roots are supposed to log in CT as a part of their baseline requirements. However, certain organizations (including Microsoft and CAs) have hybrid roots which are both publicly trusted and issue privately within the organization. As a result, loading pages may error out with NET::ERR_CERTIFICATE_TRANSPARENCY_REQUIRED. To mitigate this, such organizations must either start logging internal certificates in CT, or set one of three policies under HKLM\SOFTWARE\Policies\Microsoft\Edge\. Edge18 does not support CT.
  • Edge76 and Chrome use a custom Win32 client certificate picker UI, while Edge18 uses the system’s default certificate picker.

Cookies

  • Edge76 and Chrome support the Leave Secure Cookies Alone spec, which blocks HTTP pages from setting cookies with the Secure attribute and restricts the ways in which HTTP pages may interfere with cookies sent to HTTPS pages. Legacy Edge does not have these restrictions.
  • Edge76 and Chrome support Cookie prefixes (restrictions on cookies whose names begin with the prefixes __Secure- and __Host-). Legacy Edge does not enforce these restrictions.
  • Edge76, Chrome, and Firefox ignore Set-Cookie headers with values over 4096 characters in length (including cookie-controlling directives like SameSite). In contrast, IE and Edge18 permit cookies with name-value pairs up to 5118 characters in length.

Authentication and Login

  • In Edge76, Edge18, and Firefox, running the browser in InPrivate mode disables automatic Integrated Windows Authentication. Chrome and Internet Explorer do not disable automatic authentication in private mode. You can disable automatic authentication in Chrome by launching it with a command line argument: chrome.exe --auth-server-whitelist="_"
  • Edge18/Edge76 integrates a built-in single-sign-on (SSO) provider, such that configured account credentials are automatically injected into request headers for configured domains; this feature is disabled in InPrivate mode. Chrome does not have this behavior for Microsoft accounts.
  • Edge18 supports Azure Active Directory’s Conditional Access feature. For Chrome, an extension is required. Edge76 has not yet integrated support for this feature.

WebAPIs

  • Edge18 includes an API window.external.GetHostEnvironmentValue that returns some interesting information about the system, including whether it is running in the “Windows 10 S” lockdown mode. Edge76 and Chrome do not support this API. Update: Edge 78 restored this API with a limited set of tokens:
    {“os-architecture”:”AMD64″,”os-build”:”10.0.18362″,”os-sku”:”4″,”os-mode”:”2″}. The os-mode of 2 indicates a Windows 10 S configuration.
  • Google Chrome ships with the Portable Native Client plugin; Edge76 does not include this plugin. The plugin is little-used and you’re unlikely to encounter problems with its absence except on the Google Earth website. PNaCl is deprecated in favor of WebAssembly and is slated to be removed from Chrome in Q2 2019.
  • The Edge Platform Status site also includes a short list of features that are supported in Edge18 but not Chromium-derived browsers.

Group Policy and Command Line Arguments

By-default, Edge 76 shares almost all of the same Group Policies and command line arguments as Chrome 76.

If you’re using the registry to set a policy for Edge, put it under the

HKEY_CURRENT_USER\Software\Policies\Microsoft\Edge

…node instead of under the

HKEY_CURRENT_USER\Software\Policies\Google\Chrome

node.

If you’re trying to use a Chrome command line argument when launching in the new MSEdge.exe and it’s not working, check whether it has “blacklist” or “whitelist” in the name. If so, we probably renamed it.

For instance, want to tell Edge not to accept a 3DES ciphersuite for TLS? You need to use

msedge.exe --cipher-suite-denylist=0x000a

…instead of

chrome.exe --cipher-suite-blacklist=0x000a

….as you would with Chrome.

User-Agent

Browsers identify themselves to servers using a User-Agent header. A top source of compatibility problems is caused by sites that attempt to behave differently based on the User-Agent header and make incorrect assumptions about feature support, or fail to update their checks over time. Please, for the love of the web, avoid User-Agent Detection at all costs!

Chrome User-Agent string:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36

Edge77 Beta (Desktop) User-Agent string:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.19 Safari/537.36 Edg/77.0.235.9

Edge18 User-Agent string:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362

Edge73 Stable (Android) User-Agent string:
Mozilla/5.0 (Linux; Android 10; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.90 Mobile Safari/537.36 EdgA/42.0.4.3892

You’ll note that each of the Edge variants uses a different token at the end of the User-Agent string, but the string otherwise matches Chrome versions of the same build. Sites should almost never do anything with the Edge token information– treat Edge like Chrome. Failing to follow this advice almost always leads to bugs.

Sites are so bad about misusing the User-Agent header that Edge76 was forced to introduce a service-driven override list, which you can find at edge://compat/useragent. Alas, even that feature can cause problems in unusual cases. For testing, you can tell Edge to ignore the list by starting it thusly:

    msedge.exe --disable-domain-action-user-agent-override

Stay compatible out there!

-Eric

As we rebuild Microsoft Edge atop the Chromium open-source platform, we are working through various scenarios that behave differently in the new browser. In most cases, such scenarios also worked differently between 2018’s Edge (aka “Spartan”) and Chrome, but users either weren’t aware of the difference (because they used Trident-derived browsers inside their enterprise) or were aware and simply switched to a Microsoft-browser for certain tasks.

One example of a behavioral gap is related to running ClickOnce apps. Update: ClickOnce support is starting to come online in the new Edge 77+. See edge://flags/#edge-click-once

ClickOnce is a Microsoft application deployment framework that aims to allow installation of native-code applications from the web in (around) one click.

Chrome and Firefox can successfully install and launch ClickOnce’s .application files if the .application file specifies a deploymentProvider element with a codebase attribute (example):

InstallPrompt

Installation prompt when opening an .application file.

However, it’s also possible to author and deploy an .application that doesn’t specify a deploymentProvider element (example). Such files launch correctly from Internet Explorer and pre-Chromium Edge, but fail in Firefox and Chrome with an error message:

ApplicationCannotBeStarted

ClickOnce fails for a downloaded .application file.

So, what gives? Why does this scenario magically work in Edge Spartan but not Firefox or Chrome?

The secret can be found in the EditFlags for the Application.Manifest ProgId (to which the .application filename extension and application/x-ms-application MIME type are mapped):

ApplicationManifestRegistry

Registry settings for the Application.Manifest ProgId.

The EditFlags contain the FTA_AlwaysUseDirectInvoke flag, which is documented on MSDN as 

FTA_AlwaysUseDirectInvoke 0x00400000
Introduced in Windows 8. Ensures that the verbs for the file type are invoked with a URL instead of a downloaded version of the file. Use this flag only if you’ve registered the file type’s verb to support DirectInvoke through the SupportedProtocols or UseUrl registration.

If you peek in the Application.Manifest’s Shell\Open\Command value, you’ll find that it calls for running the ShOpenVerbApplication function inside dfshim.dll, passing along the .application file’s path or URL in a parameter (%1):

“C:\Windows\System32\rundll32.exe” “C:\Windows\System32\dfshim.dll”,ShOpenVerbApplication %1

And therein lies the source of the behavioral difference.

When you download and open an Application.Manifest file from Edge Spartan, it passes the source URL for the .application to the handler. When you download the file in Firefox or Chrome, it passes the local file path of the downloaded .application file. With only the local file path, the ShOpenVerbApplication function doesn’t know how to resolve the relative references in the Application Manifest’s XML and the function bails out with the Cannot Start Application error message.

Setting FTA_AlwaysUseDirectInvoke also has the side-effect of removing the “Save” button from Edge’s download manager:

NoSave

…helping prevent the user from accidentally downloading an .application file that won’t work if opened outside of the browser from the Downloads folder (since the file’s original URL isn’t readily available to Windows Explorer).

Advice to Publishers

If you’re planning to distribute your ClickOnce application from a website, specify the URL in Visual Studio’s ClickOnce Publish Wizard:

Manifest

Specify “From a Web site” in the ClickOnce Publish Wizard.

This will ensure that even if DirectInvoke isn’t used, ShOpenVerbApplication can still find the files needed to install your application.

Workarounds

A company called Meta4 offers a Chrome browser extension that aims to add fuller support for ClickOnce to Chrome. The extension comes in two pieces– a traditional JavaScript extension and a trivial “native” executable (written in C#) that simply invokes the ShOpenVerbApplication call with the URL. The JavaScript extension launches and communicates with the native executable running outside of the Chrome sandbox using Native Messaging.

Unfortunately, the extension is a bit hacky– it installs a blocking onBeforeRequest handler which watches all requests (not just downloads), and if the target URL’s path component ends in .application, it invokes the native executable. Alas, it’s not really safe to make any assumptions about extensions in URLs (the web is based on MIME types, rather than filenames).

Next Steps

For the Edge team– TBD. Update: ClickOnce support is starting to come online in the new Edge 77+. See edge://flags/#edge-click-once

Do you use ClickOnce to deploy your applications? If so, are you specifying the deployment URL in the manifest file?

-Eric

PS: Notably, Internet Explorer doesn’t rely upon the DirectInvoke mechanism; removing the EditFlags value entirely causes IE to show an additional prompt but the install still succeeds. That’s because IE activates the file using a MIME handler (see the CLSID subkey of Application.Manifest) much like it does for .ZIP files. The DirectInvoke mechanism was invented, in part, to replace the legacy MIME handler mechanism.

This issue report complains that Edge doesn’t stream AAC files and instead tries to download them. It notes that, in contrast, URLs that point to MP3s result in a simple audio player loading inside the browser.

Edge has always supported AAC so what’s going on?

The issue here isn’t about AAC, per-se; it’s instead about whether or not the browser, upon direct navigation to an audio stream, will accommodate that by generating a wrapper HTML page with an <audio> element pointed at that audio stream URL.

PlaceholderPage

A site that wants to play streaming AAC in Edge (or, frankly, any media type, for any browser) should consider creating a HTML page with an appropriate Audio or Video element pointed at the stream.

The list of audio types for which Edge will automatically generate a wrapper page does not include AAC:

audio/mp4, audio/x-m4a, audio/mp3, audio/x-mp3, audio/mpeg,
audio/mpeg3, audio/x-mpeg, audio/wav, audio/wave, audio/x-wav,
audio/vnd.wave, audio/3gpp, audio/3gpp2

In contrast, Chrome creates the MediaDocument page for a broader set of known audio types:

static const char* const kStandardAudioTypes[] = {
 "audio/aac",  "audio/aiff", "audio/amr",  "audio/basic",  "audio/flac",
 "audio/midi",  "audio/mp3",  "audio/mp4",  "audio/mpeg",  "audio/mpeg3", 
 "audio/ogg", "audio/vorbis",  "audio/wav",  "audio/webm",  "audio/x-m4a",
 "audio/x-ms-wma",  "audio/vnd.rn-realaudio",  "audio/vnd.wave"};

If the the response sends Content-Type: application/octet-stream, includes a Content-Dispostion: attachment, or puts a download attribute on the anchor <a> element that leads to the media, Edge will download the media file instead of playing it in the browser.

Note: In Windows 10 RS5, the extension model is capable enough that it’s possible to write a browser extension that intercepts navigation directly to audio/video Media types and renavigates to a wrapper page. [Sample code]

-Eric

PS: Edge has similar special handling for video types:

"application/mp4","video/mp4","video/x-m4v","video/3gpp",
"video/3gpp2","video/quicktime"

 

Update: The October 2018 Cumulative Security Update (KB4462919) brings the RS5 Cookie Control changes described below to Windows 10 RS2, RS3, and RS4.

Cookies are one of the most crucial features in the web platform, and large swaths of the web don’t work properly without them. Unfortunately, cookies are also one of the primary mechanisms that trackers and ad networks utilize to follow users around the web, potentially impacting users’ privacy. To that end, browsers have offered cookie controls for over twenty years.

Back in 2010, I wrote a summary of Internet Explorer’s Cookie Controls. IE’s cookie controls were very granular and quite powerful. The basic settings were augmented with P3P, a once-promising feature that allowed sites to advertise their privacy practices and browsers to automatically enforce users’ preferences against cookies. Unfortunately, major sites created fraudulent P3P statements, regulators failed to act, and the entire (complicated) system collapsed. P3P was removed from IE11 on Windows 10 and never implemented in Microsoft Edge.

Instead, Edge offers a very simple cookie control in the Privacy and Security section of the settings. Under the Cookies option, you have three choices: Don’t block cookies (the default), Block all cookies, and Block only third party cookies:

CookieSetting

This simple setting hides a bunch of subtlety that this post will explore.

Cookie => Cookie-Like

For the October 2018 update (aka “Redstone Five” aka “RS5”) we’ve made some important changes to Edge’s Cookie control.

The biggest of the changes is that Edge now matches other browsers, and uses the cookie controls to restrict cookie-like storage mechanisms, including localStoragesessionStorageindexedDB, Cache API, and ServiceWorkers. Each of these features can behave much like a cookie, with a similar potential impact on users’ privacy.

While we didn’t change the UI, it would be accurate to change it to:

CookieLike

This change improves privacy and can even improve site compatibility. During our testing, we were surprised to discover that some website flows fail if the browser blocks only 3rd party cookies without also blocking 3rd-party localStorage. This change brings Edge in line with other browsers with minor exceptions. For example, in Firefox 62, when 3rd-party site data is blocked, sessionStorage is still permitted in a 3rd-party context. In Edge RS5 and Chrome, 3rd party sessionStorage is blocked if the user blocks 3rd-party cookies.

Block Setting and Sending

Another subtlety exists because of the ambiguous terminology “third-party cookie.” A cookie is just a cookie– it belongs to a site (eTLD+1). Where the “party” comes into play is the context where the cookie was set and when it is sent.

In the web platform, unless a browser implements restrictions:

  • A cookie set in a first-party context will be sent to a first-party context
  • A cookie set in a first-party context will be sent to a third-party context
  • A cookie set in a third-party context will be sent to a first party context
  • A cookie set in a third-party context will be sent to a third-party context

For instance, in this sample page, if the IFRAME and IMG both set a cookie, these cookies are set in a third-party context:Contexts

  • If the user subsequently visits domain2.com, the cookie set by that 3rd-Party IFRAME will now be sent to the domain2.com server in a 1st-Party context.
  • If the user subsequently visits domain3.com, the cookie set by that 3rd-Party IMG will now be sent to the domain3.com server in a 1st-Party context.

Historically, Edge and IE’s “Block 3rd party cookies” options controlled only whether a cookie could be set from a 3rd party context, but did not impact whether a cookie initially set in a 1st party context would be sent to a 3rd party context.

As of Edge RS5, setting “Block only 3rd party cookies” will now also block cookies that were set in a 1st party context from being sent in a 3rd-party context. This change is in line with the behavior of other browsers.

Edge Controls Impacted By Zones

With the move from Internet Explorer to Edge, the Windows Security Zones architecture was largely left by the wayside.

Zones

However, cookie controls are one of a small number of exceptions to this; Edge applies the cookie restrictions only in the Internet Zone, the zone almost all sites fall into (outside of users on corporate networks).

Perhaps surprisingly, cookie-like features and the document.cookie getter are restricted, even in the Intranet and Trusted zones.

Chrome and Firefox do not take Windows Security Zones into account when applying cookie policies.

Test Cases

I’ve updated my old “Cookies” test page with new storage test cases. You can set your browser’s privacy controls:

Block3rdPartyChrome

Block3rdPartyFF

…then visit the test page to see how the browser limits features from 3rd-party contexts. You can use the Swap button on the page to swap 1st-party and 3rd-party contexts to see how restrictions have been applied. You should see that the latest versions of Chrome, Firefox, and Edge all behave pretty much the same way.

One interesting exception is that when configured to Block 3rd-party Cookies, Edge still allows 3rd-party contexts to delete their own cookies. (This is used by federated logout pages, for instance). Chrome does not allow deletion in this scenario– the attempt to delete cookies is ignored.

 

-Eric


Appendix: Chromium Audit

In the course of our site-compatibility investigations, I had a look at Chromium’s behavior with regard to their cookie controls. In Chromium, Blink asks the host application for permission to use various storages, and these chokepoints check:

cookie_settings_->IsCookieAccessAllowed(origin_url, top_origin_url);

…which is sensitive to the various “Block Cookies” settings.

Mojo messages come up through renderer_host/chrome_render_message_filter.cc, gating access to

Additionally, ChromeContentBrowserClient gates

Elsewhere, IsCookieAccessAllowed is used to limit:

  • Flash Storage (PP_FLASHLSORESTRICTIONS_BLOCK)
  • Client Hints

Of these, Edge does not support WebSQL, FileSystem, SharedWorker, or Client Hints.

Disclaimer: Hi. I’m an engineer on the Edge browser now, but worked on Chrome Security for a bit over two years. I speak for no one but myself, and I share no internal or confidential information in this post.

Update: The Chrome team announced upcoming changes based on user-feedback.

This weekend, there were a bunch of breathless articles and blogs about a very small change recently made to Chrome. Some of the claims are correct and carefully thought out, but in the swirl of clickbait and confusion, there are a bunch of inaccurate claims as well.

What Changed?

No, Chrome doesn’t “upload your browser history when you check GMail”… unless you tell it to do so.

Yes, Chrome has streamlined the opt-in to the browser’s “Sync” features, such that you no longer need to individually type your username and password when enabling Sync. Whether you consider this “Convenient!” or “Terrible!” is a matter of perception and threat model.

Screenshots

Because many people haven’t seen this change yet, here are some screenshots from Chrome 71.3558.

When you sign into a Google site or service in the browser’s HTML content area, your avatar/profile picture will now appear in the browser UI. If you click on your avatar, a flyout appears offering to enable sync:

Meeple

Similarly, if you use certain browser features that are more valuable with Sync (e.g. creating a bookmark or storing a password), the UI offers to turn on Sync:

OptIn

(Concern: This “Sync as Eric” button appears in the same X,Y location as the “Save Password” button on the preceding flyout/bubble, so you could conceivably click it by accident.)

Bookmark

If you click on one of these options to turn on Sync, a giant flyout appears to tell you what this means:

Get Google smarts

Interestingly/Concerningly, if you click “Settings”, it’s interpreted as “Yes, and show Settings“:

SettingsMeansYes

If you look through the Sync settings (also available by visiting chrome://settings/?search=sync), you’ll get a rich list of controls for what you can choose to sync:

SyncMain

Below that list, you can also find a list of all of the other ways (independent of Sync/Signin) that Chrome can talk back to Google:

OtherSettings

Change Your Mind? Disable Sync

If you change your mind after enabling Sync, or enabled it by accident, it’s easy to change. Click your Avatar and click on the “Syncing to” badge. Then click Turn off. Decide whether or not you want to keep local copies of the data that was sync’d to the server:

TurnOffSync

Note: If you accidentally signed into Chrome and enabled Sync on a device you borrowed from someone else, it’s very important that you check the box to clear local data when you’re done with it, or your information will be left behind on the device.

Fun Stuff for Nerds

If you’re a geek and are interested to better understand how Sync Works, visit the URL chrome://sync-internals/ in Chrome. You can see events in the Sync process and tons of data about what exactly is syncing.

CoolData

Opt-Out

If you don’t want Chrome to Sync, just don’t click buttons that offer to enable it.

Arguably, enabling Sync is now so streamlined that you could conceivably do it by accident (or someone borrowing your PC could do it for you).

If you’re concerned about sign-in to Chrome and want to ensure that you don’t ever activate Sync by accident, you can set a Policy inside HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Google\Chrome:

Regedit

You can set the Windows policy by simply importing this registry script.

After Sync is disabled, buttons asking to turn on Sync disappear, and (interestingly) if you try to sign into Chrome, a warning notice appears:

SyncDisabled

Motivations

Many of the articles that came out this weekend were rife with speculation that this is some new attempt from Google to erode your privacy by vacuuming up even more of your browsing data.

I don’t work for Google and I have no insider knowledge, but I think such attributions are incorrect. I think the correct explanations are much more mundane:

  1. The old UX was just really weird for humans. “Wait, you computer nerds are telling me that I just signed into Google using Google Chrome in the browser content area and now you’re telling me that I need to sign into the Google Chrome Browser using the browser chrome. WTF?!?”
  2. The old UX was dangerously misleading for people who share computers, a worryingly common practice.The new UX makes it at least a bit more clear that if you’re browsing on a borrowed computer, you really should be using a discardable Guest profile.(I think Guest Profiles are one of the coolest little-used features in Chrome).
  3. The new UX streamlines the enabling of Chrome Sync. Chrome Sync provides advantages to both the user and Google.

The Advantage to the User is that their satisfaction with the browser is higher. On average, satisfaction increases when Sync is on because the browser can do more for the user, better protecting them from phishing sites (via the password manager) and sharing their bookmarks, permission settings, etc on every device they use. Users with many devices (me) are especially excited about this benefit– I have enough to remember and configure, and I want my browser to help as much as it can.

The Obvious Advantage to Google is that users who are more satisfied with Chrome are more likely to use Chrome and not switch to some other browser. Chrome is simply stickier because switching to a browser that doesn’t have access to all of the user’s stored passwords, bookmarks, settings, etc, is less appealing.

In a world where browsers are shells around commoditized HTML parsers and script engines (E.g. compare Brave vs. Chrome), sticky features like sync are critical to marketshare. Consider, for instance, Chrome on iOS. It runs atop Apple’s WkWebView and is by-(Apple’s)-design intrinsically inferior in almost every way to the native Safari. Except for one thing… Chrome has my settings and data and Safari doesn’t. So I’ll go out of my way to get Chrome on iOS because, to me, Sync is critical time-saver that Safari can never match, because Safari doesn’t run on Windows and Chrome does. iOS represents a potentially addressable browser market of around a billion devices.

The Possible Unobvious Advantages to Google are what worry the skeptics and fearmongers– what if Google uses my data for something evil? Evil might range from a little evil (showing me ads for Beanie Babies because it sees I’m browsing a lot of Beanie Baby sites) to a lot evil (giving my data to “the Government” or my insurance company or my boss or some other bad person). Similarly, they could start using it recklessly (having my Google Home ask in front of my dinner guests “Hey, Eric, want to continue your search for the best Beanie Baby sales?“).

I personally don’t have significant concerns on this front (I got to see how seriously privacy is taken inside of Google and Chrome), but some people do.

My threat model is not your threat model.

If Evil Google is in your threat model, you can set an option to encrypt your sync data such that only your local Chrome instances can see it:

EncryptSync

If Extra Evil Google is in your threat model, you shouldn’t be using Chrome at all, because obviously Extra Evil Google could just backdoor Chrome before encryption or after decryption.

-Eric

PS: A helpful thread from a Chrome area owner.

Update: The Chrome team announced upcoming changes based on user-feedback.