security

Type https://example.com in your web browser’s address bar and hit enter.

What happens?

Before connecting to the example.com server, your browser must convert “example.com” to the network address at which that server is located.

dns

It does this lookup using a protocol called “DNS.” Today, most DNS transactions are conducted in plaintext (not encrypted) by sending UDP messages to the DNS resolver your computer is configured to use.

There are a number of problems with the 36-year-old DNS protocol, but a key one is that the unencrypted use of UDP traffic means that network intermediaries can see (and potentially modify) your lookups, such that attackers can know where you’re browsing, and potentially even direct your traffic to some other server.

The DNS-over-HTTPS (DoH) protocol attempts to address some of these problems by sending DNS traffic over a HTTPS connection to the DNS resolver. The encryption (TLS/QUIC) of the connection helps prevent network intermediaries from knowing what addresses your browser is looking up– your queries are private between your PC and the DNS resolver that is providing the answers. The expressiveness of HTTP (with request and response headers) provides interesting options for future extensibility, and the modern HTTP2 and HTTP3 protocols aim to provide high-performance and parallel transactions with a single connection.

Try It

Support for DNS-over-HTTPS is coming to many browsers and operating systems (including a future version of Windows). You can even try DoH out in the newest version of Microsoft Edge (v79+) by starting the browser with a special command line flag. The following command line will start the browser and instruct it to perform DNS lookups using the Cloudflare DoH server:

msedge.exe --enable-features="DnsOverHttps<DoHTrial" --force-fieldtrials="DoHTrial/Group1" --force-fieldtrial-params="DoHTrial.Group1:Fallback/false/Templates/https%3A%2F%cloudflare-dns.com%2Fdns-query"

You can test to see whether the feature is working as expected by visiting https://1.1.1.1/help. Unfortunately, this command line flag presently only works on unmanaged PCs, meaning it doesn’t do anything from PCs that are joined to a Windows domain.

Some Thoughts, In No Particular Order

Long-time readers of this blog know that I want to “HTTPS ALL THE THINGS” and DNS is no exception. Unfortunately, as with most protocol transitions, this turns out to be very very complicated.

SNI

The privacy benefits of DNS-over-HTTPS are predicated on the idea that a network observer, blinded from your DNS lookups by encryption, will not be able to see where you’re browsing.

Unfortunately, network observers, by definition, can observe your traffic, even if the traffic encrypted.

The network observer will still see the IP addresses you’re connecting to, and that’s often sufficient to know what sites you’re browsing.

Worse, they are usually still able to tell what specific HTTPS site you’re visiting on that IP address. That’s because one of the current limitations of HTTPS is that the browser sends, in unencrypted form a Server Name Indication (SNI), the hostname it expects to see in the server’s certificate as a part of the ClientHello HTTPS handshake message. Closing this hole requires implementation of Encrypted SNI (ESNI) and this feature is not yet implemented in Chromium.

Privacy From Observers, Not the Resolver

If your Internet Service Provider (say, for example, Comcast) is configured to offer DNS-over-HTTPS, and your browser uses their resolver, your network lookups are protected from observers on the local network, but not from the Comcast resolver.

Because the data handling practices of resolvers are often opaque, and because there are business incentives for resolvers to make use of lookup data (for advertising targeting or analytics revenue), it could be the case that the very actor you are trying to hide your traffic from (e.g. your ISP) is exactly the one holding the encryption key you’re using to encrypt the lookup traffic.

To address this, some users choose to send their traffic not to the default resolver their device is configured to use (typically provided by the ISP) but instead send the lookups to a “Public Resolver” provided by a third-party with a stronger privacy promise.

However, this introduces its own complexities.

Public Resolvers Don’t Know Private Addresses

A key problem in the deployment of DNS-over-HTTPS is that public resolvers (Google Public DNS, Cloudflare, Open DNS, etc) cannot know the addresses of servers that are within an intranet. If your browser attempts to look up a hostname on your intranet (say MySecretServer.intranet.MyCo.com) using the public resolver, the public resolver not only gets information about your internal network (e.g. now Google knows that you have a server called MySecretServer.intranet) but it also returns “Sorry, never heard of it.” At this point, your browser has to decide what to do next. It might fail entirely (“Sorry, site not found”) or it might “Fail open” and perform a plain UDP lookup using the system-configured resolver provided by e.g. your corporate network administrator.

This fallback means that a network attacker might simply block your DoH traffic such that you perform all of your queries in unprotected fashion. Not great.

Even alerting the user to such a problem is tricky: What could the browser even say that a human might understand? “Nerdy McNerdy Nerd Nerd Nerd Nerd Nerd Address Nerd Resolution Nerd Geek. Privacy. Network. Nerdery. Geekery. Continue?”

Centralization Isn’t Great

Centralizing DNS resolutions to the (relatively small) set of public DNS providers is contentious, at best. Some European jurisdictions are uncomfortable about the idea that their citizens’ DNS lookups might be sent to an American tech giant.

Some privacy-focused users are primarily worried about the internet giants (e.g. Google, Cloudflare) and are very nervous that the rise of DoH will result in browsers sending traffic to these resolvers by default. Google has said they won’t do that in Chrome, while Firefox is experimenting with using Cloudflare by default in some locales.

Content Filtering

Historically, DNS resolutions were a convenient choke point for schools, corporations, and parents to implement content filtering policies. By interfering with DNS lookups for sites that network users are forbidden to visit (e.g adult content, sites that put the user’s security at risk, or sites that might result in legal liability for the organization), these organizations were able to easily prevent non-savvy users from connecting to unwanted sites. Using DoH to a Public DNS provider bypasses these types of content filters, leaving the organization with unappealing choices: start using lower-granularity network interception (e.g. blocking by IP addresses), installing content-filters on the user’s devices directly, or attempting to block DoH resolvers entirely and forcing the user’s devices to fall back to the filtered resolver.

Geo CDNs and Other Tricks

In the past, DNS was one mechanism that a geographically distributed CDN could use to load-balance its traffic such that users get the “best” answers for their current locale. For instance, if the resolver was answering a query from a user in Australia, it might return a different server address than when resolving a query from a user in Florida.

These schemes and others get more complicated when the user isn’t using a local DNS resolver and is instead using a central public resolver, possibly provided by a competitor to the sites that the user is trying to visit.

Don’t Despair

Despite these challenges and others, DNS-over-HTTPS represents an improvement over the status quo, and as browser and OS engineering teams and standards bodies invest in addressing these problems, we can expect that deployment and use of DoH will grow more common in the coming years.

DoH will eventually be a part of a more private and secure web.

-Eric Lawrence

For security reasons, Edge 76+ and Chrome block navigation1 to file:// URLs from non-file:// urls.

If a browser user clicks on a file:// link on an https-delivered webpage, nothing visibly happens. If you open the the Developer Tools console, you’ll see a note: “Not allowed to load local resource: file://host/whatever”.

In contrast, Edge18 (like Internet Explorer before it) allowed pages in your Intranet Zone to navigate to URLs that use the file:// url protocol; only pages in the Internet Zone were blocked from such navigations2.

No option to disable this navigation blocking is available in Chrome or Edge 76+.

What’s the Risk?

The most obvious problem is that the way file:// retrieves content can result in privacy and security problems. Pulling remote resources over file:// can leak your user account information and a hash of your password to the remote site. What makes this extra horrific is that if you log into Windows using an MSA account, the bad guy gets both your global userinfo AND a hash he can try to crack.

Beyond the data leakage risks related to remote file retrieval, other vulnerabilities related to opening local files also exist. Navigating to a local file might result in that file opening in a handler application in a dangerous or unexpected way. The Same Origin Policy for file URIs is poorly defined and inconsistent across browsers, which can result in security problems.

Workaround: IE Mode

Enterprise administrators can configure sites that must navigate to file:// urls to open in IE mode. Like legacy IE itself, IE mode pages in the Intranet zone can navigate to file urls.

Workaround: Extensions

Unfortunately, the extension API chrome.webNavigation.onBeforeNavigate does not fire for file:// links that are blocked in HTTP/HTTPS pages, which makes working around this blocking this via an extension difficult.

One could write an extension that uses a Content Script to rewrite all file:// hyperlinks to an Application Protocol handler (e.g. file-after-prompt://) that will launch a confirmation dialog before opening the targeted URL via ShellExecute or explorer /select,”file://whatever”, but this would entail rewriting the extension rewriting every page which has non-zero performance implications. It also wouldn’t fix up any non-link file navigations (e.g. JavaScript that calls window.location.href=”file://whatever”).

Similarly, the Enable Local File Links extension simply adds a click event listener to every page loaded in the browser. If the listener observes the user clicking on a link to a file:// URL, it cancels the click and directs the extension’s background page to perform the navigation to the target URL, bypassing the security restriction by using the extension’s (privileged) ability to navigate to file URLs. But this extension will not help if the page attempts to use JavaScript to navigate to a file URI, and it exposes you to the security risks described above.

 

Necessary but not sufficient

Unfortunately, blocking file:// uris in the browser is a good start, but it’s not complete. There are myriad formats which have the ability to hit the network for file URIs, ranging from Office documents, to emails, to media files, to PDF, MHT, SCF files, etc, and most of them will do so without confirmation.

In an enterprise, the expectation is that the Organization will block outbound SMB at the firewall. When I was working for Chrome and reported this issue to Google’s internal security department, they assured me that this is what they did. I then proved that they were not, in fact, correctly blocking outbound SMB for all employees, and they spun up a response team to go fix their broken firewall rules. In a home environment, the user’s router may or may not block the outbound request.

Various policies are available, but I get the sense that they’re not broadly used.

 

-Eric

This post covers navigating to file:// uris. Another question which occasionally comes up is “how can I embed a subresource like an image or a script from a file:// URI into my https-served page.” This, you also cannot do, for similar security/privacy reasons. And that’s probably a good thing.

2 Interestingly, some alarmist researchers didn’t realize that this was happening on a per-zone basis, and asserted that IE/Edge would directly leak your credentials from any Internet web page. This is not correct. It is further incorrect in old Edge (Spartan) because Internet-zone web pages run in Internet AppContainers, which lack the Enterprise Authentication permission, which means that they don’t even have your credentials.

The Chrome team is embarking on a clever and bold plan to change the recipe for cookies. It’s one of the most consequential changes to the web platform in almost a decade, but with any luck, users won’t notice anything has changed.

But if you’re a web developer, you should start testing your sites and services now to help ensure a smooth transition.

What’s this all about?

As originally designed, cookies were very simple. When a browser made a request to a website, that website could return a tiny piece of text, called a cookie, to the browser. When the web browser subsequently requested any resource from that website, the cookie string would be echoed back to the server that first sent it.

Simple, right?

A bit too simple, as it turns out.

Browser designers have spent the last two decades trying to clear up the mess that this one simple feature causes, and alternatives might never gain adoption.

There are two major classes of problem with the design of cookies: Privacy, and Security.

Privacy

The top privacy problem is that cookies are sent every time a request is made for a resource, even if that request is made from a completely different context. So, if you visit A.example.com, that page might request a tracking pixel from ad.doubleclick.net. This tracking pixel might set a cookie. The tracking pixel’s cookie is called a third party cookie because it was set by a domain unrelated to the page itself.

If you later visit B.textslashplain.com, which also contains a tracking pixel from ad.doubleclick.net, the tracking pixel’s cookie set on your visit to A.example.com is sent to ad.doubleclick.net, and now that tracker knows that you’ve visited both sites. As you browse more and more sites that contain a tracking pixel from the same provider, that provider can build up a very complete profile of the sites you like to visit, and use that information to target ads to you, sell the data to a data aggregation company, etc.

Today, Brave blocks 3rd party cookies by default, while Safari’s ITP feature does something more intricate. Firefox and the new Edge have “Tracking Prevention” features that block 3rd-party cookies from known trackers.

Most browsers offer a setting to turn off ALL third party cookies, and older versions of Internet Explorer used to P3P block cookies that did not promise to abide by reasonable privacy protections. However, almost all users leave 3rd party cookies enabled, and enough sites sent fraudulent P3P declarations that the P3P support was ripped out of the only browser that supported it.

Security

The security problems with cookies are a bit more subtle.

In most cases, after you log in, a site will store your identity in a cookie, such that you don’t have to reenter your password on every page, or retap your security key every time you do anything. An authentication token is stored in a cookie, and each request you make to a site carries that cookie and token.

The problem is that this creates the possibility of a cross site request forgery attack, in which an attacker carefully crafts a request to a website to which you are logged in. When you visit the attacker’s site (say, to read a news article or view an image link posted to your social media feed), the attacker’s page instructs your browser to send its malicious request (“Transfer $1000 from me to @badguy”) to the victim site where you are logged in (e.g. https://bank.example).

Normally, such a request would be ignored or responded to by a demand for credentials, but because your browser is already logged in to bank.example and because your browser made the request, the server receives the cookie containing your authentication token and deems the request legitimate. You’ve been robbed! This class of attack is called the confused deputy attack.

Now, there are myriad ways to protect against this problem, but they all require careful work on the part of web application developers, and a long history of exploits shows that failing to protect against CSRF is a common mistake.

Other Privacy and Security Problems

I’ve described the biggest and most prominent of the security and privacy problems with the design of cookies, but those are just the tip of the iceberg. There are many other problems, including:

  • Because cookies are sent on plaintext HTTP requests, a passive network observer can watch your cookies as they’re sent over the network and correlate requests back to a single client. The NSA famously abused this vulnerability.
  • Cookies are sent to servers which respond with data that might allow Cross Site Leaks or violations of Same Origin Policy. For instance, an attacker might include in their page several guesses as to your identity. If they guess right (e.g. your cookie matches the URL identity), the images loads and now the attacker site knows exactly who you are:
    WordPressCorruptsMarkup
  • Cookies can be trivially stolen in a XSS Attack if the HTTPOnly attribute was not set.
  • Cookies can be trivially leaked if the server forget to set the Secure attribute and the site isn’t on the HSTS preload list.
  • Same Origin Policy blocks reading of cross-origin resources, but this depends on the integrity of the browser sandbox. Attacks like Spectre weaken this guarantee. Ambient authentication (like cross-origin cookies) weakens Cross-Origin-Read-Blocking‘s ability to prevent a compromised renderer from stealing data.

Cookie Design Improvements

Over the years, browsers have introduced more and more features and toggles to help lock down cookies, from the Secure and HTTPOnly attributes to Cookie Prefixes (nee Magic Named Cookies).

Perhaps the most promising improvement is a feature called SameSite cookies, by which cookies can opt-in to being sent only in a first-party context:

Set-Cookie: __HostAuth=F123ABCA; SameSite=Strict; secure; httponly;

This helps protect your site against CSRF attacks and helps mitigate leakage of the user’s identity in a cross-site context.

There’s a nice SameSite cookie explainer (with pictures!).

While broadly supported by browsers, the SameSite directive isn’t getting used everywhere it should be.

Big Changes are Coming!

So the Chrome folks plan to change that.

In Chrome 80 and later, cookies will default to SameSite=Lax. This means that cookies will automatically be sent only in a first party context unless they opt-out by explicitly setting a directive of None:

Set-Cookie: ACookieAvailableCrossSite; SameSite=None; secure; httponly


This change is small in size, and huge in scope. It has huge implications for any site that expects its cookies to be used in a cross-origin context.

 

What’s the Immediate Good?

In one fell swoop, many websites will get more secure. Sites that were previously vulnerable to CSRF and cross-site leak attacks will be protected from attack in the most popular browser.

Privacy improves, because setting and sending of 3rd party cookies are blocked-by-default:

SubframeCookiesBlocked

Who’s on board?

We plan to match this change in default for the new Edge browser (with experiments starting in v80). However, there’s no plan to match this change for Internet Explorer (please stop using it!) or the old Edge (v18 and earlier).

Per Chromium’s Intent-to-Implement announcement, Firefox is looking into matching the change, although they’ve joking-not-joking suggested that they’re going to let Chrome lead the charge (and bear the brunt of the compatibility impact) before turning the feature on by default.

Per the I2I, Safari has not yet weighed in on this change. Safari’s ITP feature already imposes many interesting restrictions on cross-site cookies.

What Can Go Wrong?

If users visit a site that expects its cookies to be available but the cookies are missing, users might get a confusing error message suggesting that they toggle a setting that won’t help:

teamserror

Or, the site might just redirect between its identity provider and itself forever.

These sorts of problems happen on sites that use Federated Identity providers that depend on accessing cookies from 3rd-party subframes:

federatedid-1

To fix this, the identity provider site will either need to set SameSite=None on its cookies, or will need to use a browser storage feature (e.g. localStorage) that is not impacted by this change. Please note that other browser features do impact the availability of DOM Storage, so it’s not a silver bullet.

Rollout Plan

The Chrome team has set an ambitious timeline which calls for turning this feature on-by-default for Chrome 80, slated for stable release on February 4th, 2020. Chrome’s rollout plan includes enabling the new default on an experimental basis in Chrome 79 pre-release [Beta/Dev/Stable] channels for machines which are not externally managed — attempting to defer walking into the compatibility minefield of enterprise intranets. Presently, this exclusion does not apply to machines that are domain-joined via AAD due to a limitation in Chromium.

The Chrome team also announced two enterprise policies for Chrome 80 that will allow admins to opt-out of the new default entirely, or to opt-out only specific sites. Edge expects to offer these same policies.

Developer Tooling

Developers who wish to enable the SameSite-by-Default feature locally for testing purposes can do so by visiting chrome://flags and searching for SameSite:

flags

Set the SameSite by default cookies feature to Enabled and restart the browser.

You can view the cookies used by the current page using the Application tab of the Developer Tools; the column at the far right shows the declared SameSite attribute:

F12Tab

The Chrome team have enabled logging in the Developer Tools to notify web developers that cookie behavior is changing. Visit chrome://flags/cookie-deprecation-messages to ensure that the warnings are enabled:

CookieMessages

If you then explore my test page, you can see the notices from the tools:

DebugInfo

The Cookies subtab of the Network tab picked up a new checkbox “show filtered out request cookies” which allows you to see (in yellow) which cookies were not sent for the selected request due to SameSite rules:

CookiesTab

Cookies restricted by SameSite rules are also logged in NetLog captures (issue 1005217).

Problems and Accommodations

While the vast majority of cookie scenarios will continue to work as expected, compatibility breaks are inevitable. Unfortunately, some of these breaks might not be trivially fixed by adding the SameSite=None attribute.

For instance, older versions of Safari treated SameSite=None as SameSite=Strict, which means that servers must avoid sending the None token to Safari 12.

The .NET Framework’s cookie writer used to simply omit the SameSite attribute when the SameSiteMode was “None.” Changing this will require the affected sites to update their framework to a version with the patch.

Early in our investigations, we found another problem related to how SameSite impacts cookies sent while navigating.

Specifically, over a million sites first set an anti-CSRF cookie on themselves, then redirect to a federated Login provider, then the Login provider POSTs the login information back to the site. That initial anti-CSRF cookie is only meant to be used in a first party context. Crucially, however, SameSite cookies are not sent on navigations if the navigations use the HTTP POST verb. Making the anti-CSRF cookies SameSite=Lax by default breaks this scenario and thus breaks tons of websites.

AntiCSRF

The Two Minute Mitigation

Demanding these security cookies be set to SameSite=None would be both onerous (many more sites would need to change) and misleading (because these cookies are really only meant to go to a 1st party context).

To address this breakage, the new default was adjusted to allow a SameSite-Lax-by-Default cookie to be sent on a subsequent POST requests for two minutes, significantly reducing the breakage without giving up all of the security benefit of the change.

Note that the 2 minute mitigation might not be enough for login scenarios that take longer than 2 minutes. For instance, consider the case where the login flow is happening in a background tab, or you have to fetch your Security Key, or a child must get a parent’s permission, etc. Sites that wish to handle scenarios like this will need to store a copy of their anti-CSRF token elsewhere (e.g. sessionStorage seems appropriate).

The Chrome team plans to eventually remove the 2 minute mitigation entirely.

Compat Landmine: document.cookie

In Firefox and Safari, the document.cookie DOM property matches the Cookie header, including omission of cookies that were restricted by SameSite navigation rules.

In contrast, in Chrome and Edge, SameSite cookies that are omitted from the Cookie header are still included in the document.cookie collection following a cross-origin navigation. I’ve been convinced that this actually makes more sense, although the reasoning is subtle [issue].

To the extent possible, you should set your cookies with the httponly attribute, so that they’re not available to JavaScript and this compatibility difference is irrelevant.

What’s Next?

Assuming the rollout happens on schedule, users will get more security and more privacy. But that’s just the start.

The next step is to combat the “non-secure-cookies-are-trackable” attack mentioned previously. To prevent non-secure cross-site cookies being used by network observers to follow users around the web, SameSite=None cookies will be blocked if set without the Secure attribute. Chrome’s timeline for enabling this change by default seems squishier, but ChromeStatus claims it is also slated for Chrome 80.

After that, the next step is to combat the inevitable abuse by trackers.

Because trackers can simply opt their own cookies out of restrictions by setting SameSite=None, trackers will do so. But this isn’t is bad as you think– by forcing sites to explicitly declare each cookie for which cross-site use is intended, browsers can then focus extra love and attention around such cookies.

If we peek at Chrome’s flags page today, we see an interesting hint about the Chrome team’s plans:

SSNoneSwitch

Enabling that option enables EnableRemovingAllThirdPartyCookies which adds a new Remove Third-Party Cookies button to the All cookies and site data page. When clicked, it pops the following dialog:Delete3rdPartyButton

The HandleRemoveThirdParty() function invoked by the Clear button clears not only the cookies from those domains, but also all of the site data for the sites on which those cookies existed. This provides a strong disincentive for sites to opt-in to SameSite=None cookies unless they really need to.

(Disclaimer: Chrome might never launch the “Clear third-party cookies” button, but it’s in the code today).

You can expect that browser designers will soon dream up new and interesting remediations for cookies marked for access across multiple sites. For instance, browsers could limit the lifetime of those cookies to days or hours, or even a single browser session (tricky).

We live in interesting times!

Update: Chrome pushed back experimenting with this feature from Chrome 78 to Chrome 79.

Update: The Chromium team have published a post about the new SameSite cookie default on their blog, including a list of incompatible legacy browsers that refuse to accept cookies that specify SameSite=None, and guidance on their rollout timeline.

-Eric

PS: If you’re a Fiddler user, you can use this script to easily visualize which cookies are being set with which SameSite values.

Cookie-dough hero photo by Pam Menegakis on Unsplash.

For a small number of users of Chromium-based browsers (including Chrome and the new Microsoft Edge) on Windows 10, after updating to 78.0.3875.0, every new tab crashes immediately when the browser starts.

Impacted users can open as many new tabs as they like, but each will instantly crash:

EveryTabCrashes

EdgeHavingAProblem

What’s going wrong? This problem relates to a security/reliability improvement made to Chromium’s sandboxing. Chromium runs each of the tabs (and extensions) within locked down (“sandboxed”) processes:

JAIL

In Chrome 78, a change was made to prevent 3rd-party code from injecting itself into these sandboxed processes. 3rd-party code is a top source of browser reliability and performance problems, and it has been a longstanding goal for browser vendors to get this code out of the web platform engine.

This new feature relies on setting a Windows 10 Process Mitigation policy that instructs the OS loader to refuse to load binaries that aren’t signed by Microsoft. Edge 13 enabled this mitigation in 2015, and the Chromium change brings parity to the new Edge 78+. Notably, Chrome’s own DLLs aren’t signed by Microsoft so they are specially exempted by the Chromium sandboxing code.

Unfortunately, the impact of this change is that the renderer is killed (resulting in the “Aw snap” page) if any disallowed DLL attempts to load, for instance, if your antivirus software attempts to inject its DLLs into the renderer processes. For example, Symantec Endpoint Protection versions before 14.2 are known to trigger this problem.

If you encounter this problem, you should follow the following steps:

Update any security software you have to the latest version.

Other than malware, security software is the other likely cause of code being unexpectedly injected into the renderers.

Temporarily disable the new protection

You can temporarily launch the browser without this sandbox feature to verify that it’s the source of the crashes.

  1. Close all browser instances (verify that there are no hidden chrome.exe or msedge.exe processes using Task Manager)
  2. Use Windows+R to launch the browser with the command line override:
msedge.exe --disable-features=RendererCodeIntegrity

or

chrome.exe --disable-features=RendererCodeIntegrity

Ensure that the tab processes work properly when code integrity checks are disabled.

If so, you’ve proven that code integrity checks are causing the crashes.

Hunt down the culprit

Visit chrome://conflicts#R to show the list of modules loaded by the client. Look for any files that are not Signed By Microsoft or Google.

If you see any, they are suspects. (There will likely be a few listed as “Shell Extension”s; e.g. 7-Zip.dll, that do not cause this problem)– check for an R in the Process types column to find modules loading in the Renderers.

You should install any available updates for any of your suspects to see if doing so fixes the problem.

Check the Event Log

The Windows Event Log will contain information about modules denied loading. Open Event Viewer. Expand Applications and Services Logs > Microsoft > Windows > CodeIntegrity > Operational and look for events with ID 3033. The detail information will indicate the name and location of the DLL that caused the crash:CodeIntegrity

Optional: Use Enterprise Policy to disable the new protection

If needed, IT Adminstrators can disable the new protection using the RendererCodeIntegrity policy for Chrome and Edge. You should outreach to the software vendors responsible for the problematic applications and request that they update them.

Other possible causes

Note that it’s possible that you could have a PC that encounters symptoms like this (all subprocesses crash) but not a result of the new code integrity check.

  • For instance, Chromium once had an obscure bug in its sandboxing code that caused all sandboxes to crash depending on the random memory mapping of Address Space Layout Randomization.
  • Similarly, Chrome and Edge still have an active bug where all renderers crash on startup if the PC has AppLocker enabled and the browser is launched elevated (as Administrator).

-Eric

Note: This post is part of a series about Web-to-App Communication techniques.

Just over eight years ago, I wrote my last blog post about App Protocols, a class of URL schemes that typically1 open another program on your computer instead of returning data to the web browser. 

App Protocols2 are both simple and powerful, allowing client app developers to easily enable the invocation of their apps from a website. For instance, ms-screenclip is a simple app protocol built into Windows 10 that kicks off the process of taking a screenshot:

    ms-screenclip:?delayInSeconds=2

When the user invokes this url, the handler waits two seconds, then launches its UI to collect a screenshot. Notably, App Protocols are fire-and-forgetmeaning that the handler has no direct way to return data back to the browser that invoked the protocol.

The power and simplicity of App Protocols comes at a cost. They are the easiest route out of browser sandboxes and are thus terrifying, especially because this exploit vector is stable and available in every browser from legacy IE to the very latest versions of Chrome/Firefox/Edge/Safari.

What’s the Security Risk?

A number of issues make App Protocols especially risky from a security point-of-view.

Careless App Implementation

The primary security problem is that most App Protocols were designed to address a particular scenario (e.g. a “Meet Now” page on a videoconferencing vendor’s website should launch the videoconferencing client) and they were not designed with the expectation that the app could be exposed to potentially dangerous data from the web at large.

We’ve seen apps where the app will silently reconfigure itself (e.g. sending your outbound mail to a different server) based on parameters in the URL it receives. We’ve seen apps where the app will immediately create or delete files without first confirming the irreversible operation with the user. We’ve seen apps that assumed they’d never get more than 255 characters in their URLs and had buffer-overflows leading to Remote Code Execution when that limit was exceeded. The list goes on and on.

Poor API Contract

In most cases3, App Protocols are implemented as a simple mapping between the protocol scheme (e.g. “alert”) and a shell command, e.g. 

AlertProtocol

When the protocol is invoked by the browser, it simply bundles up the URL and passes it on the command line to the target application. The app doesn’t get any information about the caller (e.g. “What browser or app invoked this?“, “What origin invoked this?“, etc) and thus it must make any decisions solely on the basis of the received URL.

Until recently, there was an even bigger problem here, related to the encoding of the URL. Browsers, including Chrome, Edge, and IE, were willing to include bare spaces and quotation marks in the URL argument, meaning that an app could launch with a command line like:

alert.exe "alert:This is an Evil URL" --DoSomethingDangerous --Ignore=This"

The app’s code saw the –DoSomethingDangerous “argument”, failed to recognize it as a part of the URL, and invoked dangerous functionality. This attack led to remote code execution bugs in App Protocol handlers many times over the years. 

Chrome began %-escaping spaces and quotation marks8 back in Chrome 64, and Edge 18 followed suit in Windows 10 RS5.

You can see how your browser behaves using the links on this test page.

Future Opportunity: A richer API contract that allows an App Protocol handler to determine how specifically it was invoked would allow it to better protect itself from unexpected callers. Moving the App Protocol URL data from the command line to somewhere else (e.g. stdin) might help reduce the possibility of parsing errors.

Sandbox

The application that handles the protocol typically runs outside of the browser’s sandbox. This means that a security vulnerability in the app can be exploited to steal or corrupt any data the user can access, install malware, etc. If the browser is running Elevated (at Administrator), any App Protocol handlers it invokes are launched Elevated; this is part of UAC’s design.

Because most apps are written in native code, the result is that most protocol handlers end up in the DOOM! portion of this diagram:

RuleOfTwo

Prompting

In most cases, the only4 thing standing between the user and disaster is a permission prompt.

In Internet Explorer, the prompt looked like this:

IEPermission
As you can see, the dialog provides a bunch of context about what’s happening, including the raw URL, the name of the handler, and a remark that allowing the launch may harm the computer.

Such information is lacking in more modern browsers, including Firefox:

FirefoxPermission

…and Edge/Chrome:

ChromePermission

Browser UI designers (reasonably) argue that the vast majority of users are poorly equipped to make trust decisions, even with the information in the IE prompt, and so modern UI has been greatly simplified5

Unfortunately, lost to evolution is the crucial indication to the user that they’re even making a trust decision at all [Issue].

Eliminating Prompts

Making matters more dangerous, everyone hates security prompts that interrupt non-malicious scenarios. A common user request can be summarized as “Prompt me only when something bad is going to happen. In fact, in those cases, don’t even prompt me, just protect me.

Unfortunately, the browser cannot know whether a given App Protocol invocation is good or evil, so it delegates control of prompting in two ways:

In Internet Explorer and Edge (version<=18), the browser respects a per-protocol WarnOnOpen boolean in the registry, such that the App itself may tell the browser: “No worries, let anyone launch me, no prompt needed.

In Firefox, Chrome, and Edge (version >= 76), the browser instead allows the user to suppress further prompts with a “Always open these types of links” checkbox. (UX nit: This text is wrong and should instead say “Always open this type of link” or “Always open links of this type”)

If the user selects this option, the protocol will silently launch in the future without the browser first asking permission.

However, Edge/Chrome version 77.0.3864 removed the “Always open these types of links in the associated app” checkbox.

NoCheckbox
The stated reason for the removal is found in Chrome issue #982341:

No obvious way to undo “Always open these types of links” decision for External Protocols.

We realized in a conversation around issue 951540 that we don’t have settings UI
that allows users to reconsider decisions they’ve made around external protocol
support. Until we work that out, and make longer-term decisions about the
permission model around the feature generally, we should stop making the problem
worse by removing that checkbox from the UI.

A user who had ticked the “Always open” box has no way to later review that decision6, and no obvious way to reverse it. Almost no one figured out that using the “Clear Browsing Data > Cookies and other site data” dialog box option directs Chrome to delete all previous “Always open” decisions for the user’s profile. 

Particularly confusing is that the “Always open” decision wasn’t made on a per-site basis– it applies to every site visited by the user in that browser profile.

UpdateAn Enterprise policy for v79+ allows administrators to restore the checkbox.

Future Opportunity: Much of the risk inherent in open-without-prompting behavior comes from the site that any random site (http://evil.example.com) can abuse ambient permission to launch the protocol handler. If browsers changed the option to “Always allow this site to open this protocol”, the risk would be significantly reduced, and a user could reasonably safely allow, e.g. https://teams.microsoft.com to open the msteams protocol without a prompt.

Alternatively, perhaps the Registry-based provisioning of a protocol handler should explicitly list the sites allowed to launch the protocol, akin to the SiteLock protection for legacy ActiveX controls.

For some schemes7 , Chrome will not even show a prompt because the protocol is included on a built-in allow or deny list.

Some security folks have argued that browsers should not provide any mechanism for skipping the permission prompt. Unfortunately, there’s evidence to suggest that such a firm stance might result in vendors avoiding the prompt by choosing even riskier architectures for Web-to-App communication. More on this in a future post.

Zero-Day Defense

Even when a zero day vulnerability in an App Protocol handler is getting exploited in the wild (e.g. this one), browsers have few defenses available to protect users. 

Unlike, say, file downloads, where the browser has multiple mechanisms to protect users against newly-discovered threats (e.g. file type policies and SmartScreen/SafeBrowsing), browsers do not presently have rapid update mechanisms that can block access to known vulnerable App Protocol handlers.

Future Opportunity: Use SafeBrowsing/SmartScreen or a file-type-policies style Component Update to supply the client with a list of known-vulnerable protocol handlers. If a page attempts to invoke such a protocol, either block it entirely or strongly caution the user.

To improve the experience even further, the blocklist could contain version information such that blocking/additional warnings would only be shown if the version of the handler app is earlier than the version number of the app containing the fix. 

Antivirus programs typically do monitor all calls to ShellExecute and could conceivably protect against malicious invocation of app protocol handlers, but I am not aware of any having ever done so.

Privacy Concerns Prevent Protocol Detection

One of the most common challenges for developers who want to use App Protocols for Web-to-App communication is that the web platform does not expose the list of available protocol handlers to JavaScript. This is primarily a privacy consideration: exposing the list of protocol handlers to the web would expose a significant amount of fingerprintable entropy and might even reveal things about the user’s interests and beliefs (e.g. a ConservativeNews App or a LGBTQ App might expose a protocol handler for app-to-app communication).

Internet Explorer and Edge <= 18 supply a non-standard JavaScript function msLaunchUri that allows a web page to detect that a user didn’t have a protocol handler installed, but this function is not available in other browsers.

UX When a Protocol Isn’t Installed

Browser behavior varies if the user attempts to invoke a link with a scheme for which no protocol handler is registered.

Firefox shows an error page: FirefoxNotInstalled

On Windows 8 and later, IE and Edge<=18 show a prompt that offers to take the user to the Microsoft store to search for a protocol handler for the target scheme:

Win10NotInstalled

Unfortunately, this search is rarely fruitful because most apps are not available in the Microsoft Store.

Interestingly, Chrome and Edge76+ show nothing at all when attempting to invoke a link for which no protocol handler is installed. Surprisingly, there’s no notice even in the Developer Tools console; a particularly thorough debugger will only see a “(canceled)” request in the DevTool’s Network tab.

Upcoming change – Require HTTPS to Invoke

Chrome is looking at requiring that a page be served over HTTPS in order for it to invoke an application protocol.

 

In future posts, I’ll explore some other alternatives for Web-to-App communication.

-Eric


Notes

1 In some browsers, it’s possible to register web-based handlers for “AppProtocols” (e.g. maps: and mailto: might go to Google Maps and GMail respectively). This mechanism is relatively little-used.

2 Within Chromium, App Protocols are called “External Protocols.”

3 There are other ways to handle protocols, including COM and the Windows 10 App Model’s URI Activation mechanism but these are uncommon.

4 As an anti-abuse mechanism, the browser may require a user-gesture (e.g. a mouse click) before attempting to launch an App Protocol, and may throttle invocations to avoid spamming the user with an infinite stream of prompts.

5 Chrome’s prompt used to look much like IE’s.

6 Short of opening the Preferences for the profile in Notepad or another text editor. E.g. after choosing “Always open” for Microsoft Teams and Skype for Business, the JSON file %localappdata%\Microsoft\Edge SxS\user Data\default\preferences contains

“protocol_handler”:{“excluded_schemes”:{“msteams”:false, “ms-sfb”: false}}

To see the list in IE/Edge<=18, you can run a registry query to find protocols with WarnOnOpen set to 0:

reg query "HKCU\SOFTWARE\Microsoft\Internet Explorer\ProtocolExecute" /s
reg query "HKLM\SOFTWARE\Microsoft\Internet Explorer\ProtocolExecute" /s

7 Hardcoded schemes:

kDeniedSchemes[] = {“afp”,”data”,”disk”,”disks”,”file”,”hcp”,”ie.http”,”javascript”,”ms-help”,”nntp”,”res”,”shell”,”vbscript”,”view-source”,”vnd.ms.radio”}
kAllowedSchemes[] = {“mailto”, “news”, “snews”};

The EscapeExternalHandlerValue function:

// Escapes characters in text suitable for use as an external protocol handler command. // We %XX everything except alphanumerics and -_.!~*'() and the restricted // characters (;/?:@&=+$,#[]) and a valid percent escape sequence (%XX). EscapeExternalHandlerValue()

 

When I launched Chrome on Thursday, I saw something unexpected:

SSLKeyLogfile

While most users probably would have no idea what to make of this, I happened to know what it means– Chrome is warning me that the system configuration has instructed it to leak the secret keys it uses to encrypt and decrypt HTTPS traffic to a stream on the local computer.

Looking at the Chrome source code, this warning was newly added last week. More surprising was that I couldn’t find the SSLKeyLogFile setting anywhere on my system. Opening a new console showed that it wasn’t set:

C:\WINDOWS\system32>set sslkeylogfile
Environment variable sslkeylogfile not defined

…and opening the System Properties > Advanced > Environment Variables UI showed that it wasn’t set for either my user account or the system at large. Weird.

Fortunately, I understood from past investigations that a process can have different environment variables than the rest of the system, and Process Explorer can show the environment variables inside a running process. Opening Chrome.exe, we see that it indeed has an SSLKEYLOGFILE set:

SSLKeyLogfileEB

The unusual syntax with the leading \\.\ means that this isn’t a typical local file path but instead a named pipe, which means that it doesn’t point to a file on disk (e.g. C:\temp\sslkeys.txt) but instead to memory that another process can see.

My machine was in this state because earlier that morning, I’d installed Avast Antivirus to attempt to reproduce a bug a Chrome user encountered. Avast is injecting the SSLKEYLOGFILE setting so that it can conduct a monster-in-the-browser attack (MITB) and see the encrypted traffic going into Chrome.

Makers of antivirus products know that browsers are one of the primary vectors by which attackers compromise PCs, and as a consequence their security products often conduct MITB attacks in order to scan web content. Antivirus developers have two common techniques to scan content running in the browser:

  1. Code injection
  2. Network interception

Code Injection

The code injection technique relies upon injecting security code into the browser process. The problem with this approach is that native code injections are inherently fragile– any update to the browser might move its functions and data structures around such that the security code will fail and crash the process. Browsers discourage native code injection, and the bug I was looking at was related to a new feature, RendererCodeIntegrity, that directs the Windows kernel to block loading of any code not signed by Microsoft or Google into the browser’s renderer processes.

An alternative code-injection approach relies upon using a browser extension that operates within the APIs exposed by the browser– this approach is more stable, but can address fewer threats.

Even well-written code injections that don’t cause stability problems can cause significant performance regressions for browsers– when I last looked at the state of the industry, performance costs for top AV products ranged from 20% to 400% in browser scenarios.

Network Interception

The Network interception technique relies upon scanning the HTTP and HTTPS traffic that goes into the browser process. Scanning HTTP traffic is straightforward (a simple proxy server can do it), but scanning HTTPS traffic is harder because the whole point of HTTPS is to make it impossible for a network intermediary to view or modify the plaintext network traffic.

Historically, the most common mechanism for security-scanning HTTPS traffic was to use a monster-in-the-middle (MITM) proxy server running on the local computer. The MITM would instruct Windows to trust a self-signed root certificate, and it would automatically generate new interception certificates for every secure site you visit. I spent over a decade working on such a MITM proxy server, the Fiddler Web Debugger.

There are many problems with using a MITM proxy, however. The primary problem is that it’s very very hard to ensure that it behaves exactly as the browser does and that it does not introduce security vulnerabilities. For instance, if the MITM’s certificate verification logic has bugs, then it might accept a bogus certificate from a spoof server and the user would not be warned– Avast used to use a MITM proxy and had exactly this bug; they were not alone. Similarly, the MITM might not support the most secure versions of protocols supported by the browser and server (e.g. TLS/1.3) and thus using the MITM would degrade security. Some protocol features (e.g. Client Certificates) are incompatible with MITM proxies. And lastly, some security features (specifically certificate pinning) are fundamentally incompatible with MITM certificates and are disabled when MITM certificates are used.

Given the shortcomings of using a MITM proxy, it appears that Avast has moved on to a newer technique, using the SSLKeyLogFile to leak the secret keys HTTPS negotiates on each connection to encrypt the traffic. Firefox and Chromium support this feature, and it enables decryption of TLS traffic without using the MITM certificate generation technique. While browser vendors are wary of any sort of interception of HTTPS traffic, this approach is generally preferable to MITM proxies.

There’s some worry that Chrome’s new notification bar might drive security vendors back to using more dangerous techniques, so this notification might not make its way into the stable release of Chrome.

When it comes to browser architecture, tradeoffs abound.

-Eric

PS: I’m told that Avast may be monetizing the data they’re decrypting.

Appendix: Peeking at the Keys

If we point the SSLKeyLog setting at a regular file instead of a named pipe:

chrome --ssl-key-log-file=C:\temp\sslkeys.txt

…we can examine the file’s contents as we browse to reveal the encryption keys:

ExportedKeys

This file alone isn’t very readable for a human (even if you read Mozilla’s helpful file format documentation), but you can configure tools like Wireshark to make use of it and automatically decrypt captured TLS traffic back to plaintext.

By this point, most browser enthusiasts know that Chrome has a rapid release cycle, releasing a new stable version of the browser approximately every six weeks. The Edge team intends to adopt that rapid release cadence for our new browser, and we’re already releasing new Edge Dev Channel builds every week.

What might be less obvious is that this six week cadence represents an upper-bound for how long it might take for an important change to make its way to the user.

Background: Staged Rollouts

Chrome uses a staged rollout plan, which means only a small percentage (1%-5%) of users get the new version immediately. If any high-priority problems are flagged by those initial users, the rollout can be paused while the team considers how to best fix the problem. That fix might involve shipping a new build, turning off a feature using the experimentation server, or dynamically updating a component.

Let’s look at each.

Respins

If a serious security or functionality problem is found in the Stable Channel, the development team generates a respin of the release, which is a new build of the browser with the specific issue patched. The major and minor version numbers of the browser stay the same. For instance, on July 15th, Chrome Stable version 75.0.3770.100 was updated to 75.0.3770.142. Users who had already installed the buggy version in the channel are updated automatically, and users who haven’t yet updated to the buggy version will just get the fixed version when the rollout reaches them.

If you’re curious, you can see exactly which versions of Chrome are being delivered from Google’s update servers for each Channel using OmahaProxy.

Field Trial Flags

In some cases, a problem is discovered in a new feature that the team is experimenting with. In these cases, it’s usually easy for the team to simply remotely disable or reconfigure the experiment as needed using the experimental flags. The browser client periodically polls the development team’s servers to get the latest experimental configuration settings. Chrome codenames their experimental system “Finch,” while Microsoft calls ours “CFR” (Controlled Feature Rollout).

You can see your browser’s current field trial configuration by navigating to

chrome://version/?show-variations-cmd

The hexadecimal Variations list is generally inscrutable, but the Command-line variations section later in the page is often more useful and allows you to better understand what trials are underway. You can even use this list to identify the exact trial causing a particular problem.

Regular readers might remember that I’ve previously written about Chrome’s Field Trials system.

Components

In other cases, a problem is found in a part of the browser implemented as a “Component.” Components are much like hidden, built-in extensions that can be silently and automatically updated by the Component Updater.

The primary benefit of components is that they can be updated without an update to Chrome itself, which allows them to have faster (or desynchronized) release cadences, lower bandwidth consumption, and avoids bloat in the (already sizable) Chrome installer. The primary drawback is that they require Chrome to tolerate their absence in a sane way.

To me, the coolest part of components is that not only can they update without downloading a new version of the browser, in some cases users don’t even need to restart their browser to begin using the updated version of a component. As soon as a new version is downloaded, it can “take over” from the prior version.

To see the list of components in the browser, visit

chrome://components

In my Chrome Canary instance, I see the following components:

Components

As you can see, many of these have rather obtuse names, but here’s a quick explanation where I know offhand:

  • MEI Preload – Policies for autoplay (see chrome://media-engagement/ )
  • Intervention Policy – Controls interventions used on misbehaving web pages
  • Third Party Module – Used to exempt accessibility and other components from the Code Integrity protections on the browser’s process that otherwise forbid injection of DLLs.
  • Subresource Filter Rules – The EasyList adblock database used by Chrome’s built-in adblocker to remove ads from a webpage when the Safe Browsing service indicates that a site violates the guidelines in the Better Ads Standard.
  • Certificate Error Assistant – Helps users understand and recover from certificate errors (e.g. when behind a known WiFi captive portal).
  • Software Reporter Tool – Collects data about system configuration / malware.
  • CRLSet – List of known-bad certificates (used to replace OCSP/CRL).
  • pnacl – Portable Native Client (overdue for removal)
  • Chrome Improved Recovery Unsure, but comments suggest this is related to helping fix broken Google Updater services, etc.
  • File Type Policies – Maps a list of file types to a set of policies concerning how they should be downloaded, what warnings should be presented, etc. See below.
  • Origin Trials – Used to allow websites to opt-in to experimenting with future web features on their sites. Explainer.
  • Adobe Flash Player – The world’s most popular plugin, gradually being phased out; slated for complete removal in late 2020.
  • Widevine Content DecryptionA DRM system that permits playback of protected video content.

If you’re using an older Chrome build, you might see:

If you’re using Edge, you might see:

If you’re using the Chromium-derived Brave browser, you’ll see that brave://components includes a bunch of extra components, including “Ad Blocker”, “Tor Client”, “PDF Viewer”, “HTTPS Everywhere”, and “Local Data Updater.”

If you’re using Chrome on Android, you might notice that it’s only using three components instead of thirteen; the missing components simply aren’t used (for various reasons) on the mobile platform. As noted in the developer documentation, “The primary drawback [to building a feature using a Component] is that [Components] require Chrome to tolerate their absence in a sane way.

Case Study: Fast Protection via Component Update

Let’s take a closer look at my favorite component, the File Type Policies component.

When the browser downloads a file, it must make a number of decisions for security reasons. In particular, it needs to know whether the file type is potentially harmful to the user’s device. If the filetype is innocuous (e.g. plaintext), then the file may be downloaded without any prompts. If the type is potentially dangerous, the user should be warned before the download completes, and security features like SafeBrowsing/SmartScreen should scan the completed download for malicious content.

In the past, this sort of “What File Types are Dangerous?” list was hardcoded into various products. If a file type were later found to be dangerous, patching these products with updated threat information required weeks to months.

In contrast, Chrome delivers this file type policy information using the File Type Policies component. The component lets Chrome engineers specify which types are dangerous, which types may be configured to automatically open, which types are archives that contain other files that may require scanning, and so on.

How does this work in the real world? Here’s an example.

Around a year ago, it was discovered that files with the .SettingsContent-ms file extension could be used to compromise the security of the user’s device. Browsers previously didn’t take any special care when such files were downloaded, and most users had no idea what the files were or what would happen if they were opened. Everyone was caught flat-footed.

In less than a day after this threat came to light, a Chrome engineer simply updated a single file to mark the settings-content.ms file type as potentially dangerous. The change was picked up by the component builder, and Chrome users across all versions and channels were protected as their browser automatically pulled down the updated component in the background.

 

Ever faster!

-Eric