Seamless Single Sign-On

There are many different authentication primitives built into browsers. The most common include Web Forms authentication, HTTP authentication, client certificate authentication, and the new WebAuthN standard. Numerous different authentication frameworks build atop these, and many enterprise websites support more than one scheme.

Each of the underlying authentication primitives has different characteristics: client certificate authentication is the most secure but is hard to broadly deploy, HTTP authentication works great for Intranets but poorly for most other scenarios, and Web Forms authentication gives the website the most UI flexibility but suffers from phishing risk and other problems. WebAuthN is the newest standard and is not yet supported by most sites.

Real World Authentication Flows

Many Enterprises will combine all of these schemes, using a flow something like:

  1. User navigates to https://app.corp.example
  2. The web application determines that the user is not logged in
  3. The user is redirected to https://login.corp.example
  4. The login provider checks to see whether the user has any cached authentication tokens, e.g. using the cookies accessible to the login provider.
  5. If not, the login provider tries to fetch https://clientcert.corp.example.com with a certificate filter specifying the internal CA root. If the user has a client certificate from the CA root, it is sent either silently or after a one-click prompt.
  6. If not, the login provider checks to see whether the user’s browser has any domain credentials using the HTTP Negotiate authentication scheme.
  7. If not, the login provider shows a traditional HTML form for login, ideally with a WebAuthN option that allows the user to use the new secure API rather than typing a password.
  8. After all of these steps, the user’s identity has been verified and is returned to the app.corp.example site.

In today’s post, I want to take a closer look at Step #6.

Silent HTTP Authentication

Unfortunately for our scenario, the HTTP Authentication scheme doesn’t support any sort of NoUI attribute, meaning that a server has no way to demand “Authenticate using the user’s domain credentials if and only if you can do so without prompting.”

WWW-Authenticate: Negotiate

And browsers’ HTTP Authentication prompts tend to be pretty ugly:

Depending upon client configuration and privacy mode, HTTP Authentication using the Negotiate (wrapping Kerberos/NTLM) or NTLM schemes may happen silently, or it may trigger the manual HTTP Authentication prompt.

So, at step #6, we’re stuck. If automatic HTTP authentication would’ve worked, it would be great– the user would be signed into the application with zero clicks and everything would be convenient and secure.

Load-Bearing Quirks

Fortunately for our scenario (unfortunately for understandability), there’s a magic trick that authentication flows can use to try HTTP authentication silently. As far as I can tell, it was never designed for this purpose, but it’s now used extensively.

To help prevent phishing attacks, modern browsers will prevent1 an HTTP authentication prompt from appearing if the HTTP/401 authentication response was for a cross-site image resource. The reasoning here is that many public platforms will embed images from arbitrary URLs, and an attacker might successfully phish users by posting on a message board an image reference that demands authentication. An unwary user might inadvertently supply their credentials for the message board to the third party site.

As noted in Chromium:

  if (resource_type == blink::mojom::ResourceType::kImage &&
      IsBannedCrossSiteAuth(request.get(), passed_extra_data.get())) {
    // Prevent third-party image content from prompting for login, as this
    // is often a scam to extract credentials for another domain from the
    // user. Only block image loads, as the attack applies largely to the
    // "src" property of the <img> tag. It is common for web properties to
    // allow untrusted values for <img src>; this is considered a fair thing
    // for an HTML sanitizer to do. Conversely, any HTML sanitizer that didn't
    // filter sources for <script>, <link>, <embed>, <object>, <iframe> tags
    // would be considered vulnerable in and of itself.
    request->do_not_prompt_for_login = true;
    request->load_flags |= net::LOAD_DO_NOT_USE_EMBEDDED_IDENTITY;
  }

So, now we have the basis of our magic trick.

We use a cross-site image resource (e.g. https://tryhttpauth.corp-intranet.com) into our login flow. If the image downloads successfully, we know the user’s browser has domain credentials and is willing to silently release them. If the image doesn’t download (because a HTTP/401 was returned and silently unanswered by the browser) then we know that we cannot use HTTP authentication and we must continue on to use the WebForms/WebAuthN authentication mechanism.

Update (Feb 2021): As of Chrome/Edge88, this magic trick will now fail if the user has configured their browser to “Block 3rd Party Cookies”, because the browser now treats a cross-origin authentication demand as if it were a cookie. Credentials for the cross-origin image will be omitted, and the browser will conclude that HTTP authentication is not available.

-Eric

1 Note that this magic trick is defeated if you enable the AllowCrossOriginAuthPrompt policy, because that policy permits the authentication prompt to be shown.

Post-Script: Prompting for Credentials vs. Approving for Release

As an aside, the HTTP Authentication prompt shown in this flow is more annoying than it strictly needs to be. What it’s usually really asking is “May I release your credentials to this site?“:

…but for implementation simplicity and historical reasons the prompt instead forces the user to retype their username and password.

Beating Private Mode Blockers with an Ephemeral Profile

Back in 2018, I explained how some websites use various tricks to detect that visitors are using Private Mode browsers and force such users to log-in. The most common reason that such sites do this is that they’ve implemented a “Your first five articles are free, then you have to pay” model, and cookies or similar storage are used to keep track of the user’s read count.

The New Yorker magazine is one such site:

Unfortunately, such “Private Mode blockers” make it hard for those of us who use Private Mode for other reasons (I don’t want to leave any traces of my Beanie Baby shopping research!). Private Mode detectors typically trigger for Chromium-based browsers’ Guest Profile that you might be use when borrowing a trusted friend’s computer.

So, what’s a privacy-conscious user to do?

If you’re using Firefox, you can use that browser’s “Containers” feature to isolate such sites into a partitioned container such that trackers from the site cannot follow you around the web.

If you use Microsoft Edge, you might consider creating your own “Ephemeral” browser profile for browsing sites that block InPrivate:

After you create the new profile, visit its Settings page at edge://settings/clearBrowsingDataOnClose and configure all storage areas to be cleared every time you close the browser1:

Note: Chrome does not offer a Clear on Close list, but does offer a limited Clear cookies and site data when you quit Chrome option.

You can then adjust any other settings you like, for instance, adjusting Tracking Protection to Strict in edge://settings/privacy or the like.

Then when you want to visit a site that blocks InPrivate, you can either open your Ephemeral profile from your profile icon, or use the Open link as command on a hyperlink’s context-menu:

Over time, browsers will continue to work to make Private Mode detectors less reliable, but it’s unlikely that they’ll ever be perfect. Creating an ephemeral profile that clears everything on exit is a useful trick to combat sites which prioritize their business model needs over your privacy.

-Eric

1 In Edge 85 and earlier, you must unfortunately close all browser windows (even from your main profile) to trigger the cleanup of your ephemeral profile; closing just the windows from the ephemeral profile alone is not enough. This bug was recently fixed in Edge 86.

Advanced Q&A

Q: How is this Ephemeral/ClearOnExit Profile different than a regular InPrivate Mode session?

A: There are a few key differences.

  1. InPrivate tries not to write anything to disk (although the OS memory manager might at any time decide swap process memory to the disk), while true profiles do not impose such a limitation. The “no disk write” behavior of Private Mode is the primary source of web-platform-observable differences in behavior that allow sites to build Private Mode detectors.
  2. By default, your default browser extensions do not load in InPrivate, but they can be configured to do so. In a different profile, you’ll have to install any desired extensions individually.
  3. By default, your credentials (usernames and passwords) do not autofill while InPrivate. In a different profile, your main profile’s credentials will not be available (and will be cleared on exit if configured to do so).
  4. InPrivate tabs do not perform Windows Integrated Authentication to Intranet sites automatically. Regular browser profiles do not have such a limitation.

Revealing Passwords

The Microsoft Edge browser, Edge Legacy, and Internet Explorer all offer a convenient mechanism for users to unmask their typing as they edit a password field:

Clicking the little eye icon disables the masking dots so that users can see the characters they’re typing:

This feature can be very useful for those of us who often mistype characters, and is especially important for users with various accessibility needs that can make error-free typing especially challenging. Keyboard users can hit ALT+F8 to toggle the reveal feature without using the mouse.

Nevertheless, Web Developers may disable this feature (for instance, if they offer their own version) by targeting the -ms-reveal pseudo element on an input type=password field:

.classNoReveal::-ms-reveal {
display: none;
}

If a site offers its own “reveal” feature, it should use CSS to hide the built-in feature to avoid confusing UI like this one:

Alternatively, sites may customize the Password Reveal Icon to better match their visual style.

Edge Legacy and Internet Explorer also respect a Windows policy (DisablePasswordReveal) that removes the password reveal button in various places throughout the system, including Edge Legacy and Internet Explorer. Some security configuration guides suggest setting this policy, arguing “Visible passwords may be seen by nearby persons, compromising them.” This is literally true; it is also true that such nearby persons might simply watch as the user types in their password manually.

Notably, this Windows policy is not respected by Edge 79 and above, so we’ve had a few questions about that. I’d like to point out a few non-obvious characteristics of this feature that might assuage security concerns.

The most obvious attack that administrators are worried about is that a passerby might use this mechanism to steal auto-filled passwords from an unlocked, unattended computer. This concern is misplaced1: when the browser’s Password Manager autofills a password, the reveal icon is removed:

The PasswordInputType code is smart too– an attacker cannot get the icon to appear by simply adding or deleting a few characters, it only reappears after the user completely removes all of the characters in the input field. The icon is hidden if the field is modified by JavaScript, and it’s hidden if focus leaves the input field.

All of these protections mean that the Password Reveal icon is unlikely to be abusable in any meaningful way. Of course, typing passwords at all is an anti-pattern– use the Password Manager to mitigate phishing attacks, and eliminate the use of passwords wherever possible.

-Eric

1 Notably, while concern about the reveal button is misplaced, it’s entirely possible to steal your own password using the Developer Tools or by running JavaScript from the omnibox.

Web Proxy Auto Discovery (WPAD)

Back in the mid-aughts, Adam G., a colleague on the IE team, used the email signature “IE Networking Team – Without us, you’d be browsing your hard drive.” And while I’m sure it was meant to be a bit tongue-in-cheek, it’s really true– without a working network stack, web browsers aren’t nearly as useful.

Background on Proxy Determination

One of the very first things a browser must do on startup is figure out how to send requests over the network. Typically, the host operating system already provides the transport (TCP/IP, UDP) and lower-level primitives, so the browser’s first task is to figure out whether or not web requests should be sent through a proxy. Until this question is resolved, the browser cannot send any network requests to load pages, sync profile information, update phishing blocklists, etc.

In some cases, proxy determination is simple— the browser is directly configured to ignore proxies, or to send all requests to a directly specified proxy.

However, for convenience and to simplify cases where a user might move a laptop between different networks with different proxy requirements, all major browsers support an algorithm called “Web Proxy Auto Discovery”, or WPAD. The WPAD process is meant to find and download a Proxy AutoConfiguration Script (PAC) for the current network.

The steps of the WPAD protocol are straightforward, if lengthy:

  1. Determine whether WPAD should be used, either by looking at browser settings or asking the host operating system if the browser is configured to match the OS setting.
  2. Ensure the network is ready.
  3. If WPAD is to be used, issue a DHCPINFORM query to ask for the URL of the PAC script to use.
  4. If the DHCPINFORM query fails to return a URL, perform a DNS lookup for the unqualified hostname wpad.
  5. If the DNS lookup succeeds, then the PAC URL shall be http://wpad/wpad.dat.
  6. Establish a HTTP(S) connection to discovered URL’s server and download the PAC script.
  7. If the PAC script downloads successfully, parse and optionally compile it.
  8. For each network request, call FindProxyForURL() in the PAC script and use the proxy settings returned from the function.

While conceptually simple, any of these steps might fail, and any failure might prevent the browser from using the network.

Performance

… or “Why on earth do I see Downloading proxy script… for a few seconds every time I start my browser!??!”

A Microsoft Edge feature team reached out to the networking team this week asking for help with an observed 3 second delay in the initialization of their feature. They observed that this delay magically disappeared if Fiddler happened to be running.

With symptoms like that, proxy determination is the obvious suspect, because Fiddler specifies the exact proxy configuration for browsers to use, meaning that they do not need to perform the WPAD process.

We asked the team to take an Edge network trace using the “Capture on Startup” steps. Sure enough, when we analyzed the resulting NetLog, we found almost exactly three seconds of blocking time during startup:

t= 52   PROXY_CONFIG_CHANGED
             --> new_config = Auto-detect
t= 52  +PAC_FILE_DECIDER 
t= 52  PAC_FILE_DECIDER_WAIT 
t=2007 +PAC_FILE_DECIDER_FETCH_PAC_SCRIPT 
              --> source = "WPAD DHCP"
t=2032 -PAC_FILE_DECIDER_FETCH_PAC_SCRIPT 
            --> net_error = -348 (ERR_PAC_NOT_IN_DHCP) 
t=2032 PAC_FILE_DECIDER_FALLING_BACK_TO_NEXT_PAC_SOURCE 
t=2032 +HOST_RESOLVER_IMPL_REQUEST 
              --> host = "wpad:80" 
t=3033 CANCELLED

Note: Timestamps [e.g. t=52] are shown in milliseconds.

Because the browser took a full three seconds to decide whether or not to use a proxy, every feature that relies on the network will take at least three seconds to get the data it needs.

So, where’s the delay coming from? In this case, the delay comes from two places: a two second delay for PAC_FILE_DECIDER_WAIT and a one second delay for the DNS lookup of wpad.

The two second PAC_FILE_DECIDER_WAIT [Step #2] is a deliberate delay that is meant to delay PAC lookups after a network change event is observed, to accommodate situations where the browser is notified of a network change by the Operating System before the network is truly “ready” to perform the DHCP/DNS/Download steps of WPAD. In this browser-startup case, we haven’t yet figured out why the browser thinks a network change has occurred, but the repro is not consistent and it seems likely to be a bug.

The (failing) DNS lookup [Step #3] might’ve taken even longer to return, but it timed out after one second thanks to an enabled-by-default feature called WPADQuickCheckEnabled.

This three second delay on startup is bad, but it could be even worse. We got reports from one Microsoft employee that every browser startup took around 21 seconds to navigate anywhere. In looking at his network log, we found that the wpad DNS lookup [Step #5] succeeded, returning an IP address, but the returned IP was unreachable and took 21 seconds to timeout during TCP/IP connection establishment.

What makes these delays especially galling is that they were all encountered on a network that does not actually need a proxy!

Failures

Beyond the time delays, each of these steps might fail, and if a proxy is required on the current network, the user will be unable to browse until the problem is corrected.

For example, we recently saw that [Step #7] failed for some users because the Utility Process running the PAC script always crashed due to forbidden 3rd-party code injection. When the Utility Process crashes, Chromium attempts to bypass the proxy and send requests directly to the server, which was forbidden by the Enterprise customer’s network firewall.

We’ve also found that care must be taken in the JavaScript implementation of FindProxyForURL() [Step #8] because script functions behave slightly differently across different browsers. In most cases, scripts work just fine across browsers, but sometimes corner cases are encountered that require careful handling.

Script Download

In Chromium, if a PAC script must be downloaded, it is fetched bypassing the cache.

Even if we were to comment out the LOAD_DISABLE_CACHE directive in the fetch, this wouldn’t allow reuse of a previously downloaded script file– my assumption is that the download is happening in a NetworkContext that doesn’t actually have a persistent cache, but I haven’t looked into this.

The PAC script fetches will be repeated on network change or browser restart.

Security

WPAD is something of a security threat, because it means that another computer on your network might be able to become your proxy server without you realizing it. In theory, HTTPS traffic is protected against malicious proxy servers, but non-secure HTTP traffic hasn’t yet been eradicated from the web, and users might not notice if a malicious proxy performed an SSLStripping attack on a site that wasn’t HSTS preloaded, for example.

Note: Back in 2016, it was noticed that the default Chromium proxy script implementation leaked full URLs (including HTTPS URLs’ query strings) to the proxy script; this was fixed by truncating the URL to the hostname. (In the new world of DoH, there’s some question as to whether we might be able to avoid sending the hostname to the proxy at all).

Edge Legacy and Internet Explorer have a surprising default behavior that treats sites for which a PAC script returns DIRECT (“bypass the proxy for this request“) as belonging to your browser’s Intranet Zone.

This mapping can lead to functionality glitches and security/privacy risks. Even in Chrome and the new Edge, Windows Integrated Authentication still occurs Automatically for the Windows Intranet Zone, which means this WPAD Zone Mapping behavior is still relevant in modern browsers.

Chrome performing Automatic Authentication due to Proxy Bypass

Edge Legacy and Internet Explorer

Interestingly, performance and functionality problems with WPAD might have been less common for the Edge Legacy and Internet Explorer browsers on Windows 10. That’s because both of these browsers rely upon the WinHTTP Web Proxy Auto-Discovery Service:

This is a system service that handles proxy determination tasks for clients using the WinHTTP/WinINET HTTP(S) network stacks. Because the service is long-running, performance penalties are amortized (e.g. a 3 second delay once per boot is much cheaper than a 3 second delay every time your browser starts), and the service can maintain caches across different processes.

Chromium does not, by default, directly use this service, but it can be directed to do so by starting it with the command-line argument:

--winhttp-proxy-resolver

A Group Policy that matches the command-line argument is also available.

SmartWPAD

Prior to the enhancement of the WinHTTP WPAD Service, a feature called SmartWPAD was introduced in Internet Explorer 8’s version of WinINET. SmartWPAD caches in the registry a list of networks on which WPAD has not resulted in a PAC URL, saving clients the performance cost of performing the WPAD process each time they restarted for the common case where WPAD fails to discover a PAC file:

Cache entries would be maintained for a given network fingerprint for one month. Notably, the SmartWPAD cache was only updated by WinINET, meaning you’d only benefit if you launched a WinINET-based application (e.g. IE) at least once a month.

When a client (including IE, Chrome, Microsoft Edge, Office, etc) subsequently asks for the system proxy settings, SmartWPAD checks if it had previously cached that WPAD was not available on the current network. If so, the API “lies” and says that the user has WPAD disabled.

The SmartWPAD feature still works with browsers running on Windows 7 today.

Notably, it does not seem to function in Windows 10; the registry cache is empty. My Windows 10 Chromium browsers spend ~230ms on the WPAD process each time they are fully restarted.

Update: The WinINET team confirms that SmartWPAD support was removed after Windows 7; for clients using WinINET/WinHTTP it wasn’t needed because they were using the proxy service. Clients like Chromium and Firefox that query WinINET for proxy settings but use their own proxy resolution logic will need to implement a SmartWPAD-like feature optimize performance.

Disabling WPAD

If your computer is on a network that doesn’t need a proxy, you can ensure maximum performance by simply disabling WPAD in the OS settings.

By default (if not overridden by policy or the command line), Chromium adopts the Windows proxy settings by calling WinHttpGetIEProxyConfigForCurrentUser.

On Windows, you can thus turn off WPAD by default by using the Internet Control Panel (inetcpl.cpl) Connections > LAN Settings dialog, or the newer Windows 10 Settings applet’s Automatic Proxy Setup section:

Simply untick the box and browsers that inherit their default settings from Windows (Chrome, Microsoft Edge, Edge Legacy, Internet Explorer, and Firefox) will stop trying to use WPAD.

Looking forward

WPAD is convenient, but somewhat expensive for performance and a bit risky for security/privacy. Every few years, there’s a discussion about disabling it by default (either for everyone, or for non-managed machines), but thus far none of those conversations has gone very far.

Ultimately, we end up with an ugly tradeoff– no one wants to land a change that results in users being limited to browsing their hard drives.

If you’re an end user, consider unticking the “Automatically Detect Settings” checkbox in your Internet settings. If you’re an enterprise administrator, consider deploying a policy to disable WPAD for your desktop fleet.

-Eric

Avoiding Unexpected Navigation

For over twenty years, browsers broadly supported two features that were often convenient but sometimes accidentally invoked, leading to data loss.

The first feature was that hitting backspace would send the user back one page in their navigation history. This was convenient for those of us who keep our hands on the keyboard, but could lead to data loss– for instance, when filling out a web form, if focus accidentally left a text box, hitting backspace could result in navigating away from the form. Smart websites would warn you with an OnBeforeUnload handler, and some browsers tried to restore the contents of the form if the user understood what happened and hit “forward”, but these protections are imperfect and might not work for all forms.

One of the IE browser UI leads complained about this behavior annually for a decade, and users periodically howled as they lost work. Finally, circa 2016, this feature was removed from Chrome and Microsoft Edge followed in 2018. If you happened to love the old behavior and accept the risk of data loss, you can restore it via extension or in Edge 86, via the edge://flags/#edge-backspace-key-navigate-page-back flag.

The second feature was drop to navigate. Dragging and dropping a file into the browser’s content area (the body of the page) would, (unless the page’s JavaScript was designed to handle the drop, e.g. to upload it or process it locally), navigate to that local file in the current tab. Some folks like that behavior– e.g. web developers loading HTML files from their local filesystem, but it risks the same data loss problem. If a web page doesn’t accept file uploads via drag/drop, the contents of that page will be blown away by navigation. Back in 2015, a bug was filed against Chromium suggesting that the default behavior was too dangerous, and many examples were provided where the default behavior could be problematic. Yesterday, I landed a tiny change for Chromium 85 [later merged to v84] such that dropping a file or URL into the content area of a tab will instead open the file in a new tab:

Dropping in the content area now opens it in a new tab:

If you do want to replace the content of the tab with the dropped file, you can simply drag/drop the file to the tab strip.

A small white arrow shows you which tab’s content will be replaced:

Dropping the file between tabs on the tab strip will insert a new tab at the selected location:

Chrome (85.0.4163/v84) and Microsoft Edge (85.0.541) include this change; it was also later merged to v84. Microsoft Edge Legacy didn’t support drop to navigate. Firefox still has the old behavior; the closest bug I could find is this one. Safari 13.1.1 still has the old behavior and replaces the content of the current tab with the local file. Safari Tech Preview 13.2 Release 108 instead navigates the tab to an error page (NSPOSIXErrorDomain:1 Operation not permitted”).

-Eric

Browser Basics: User Gestures

The Web Platform offers a great deal of power, and unfortunately evil websites go to great lengths to abuse it. One of the weakest (but simplest to implement) protections against such abuse is to block actions that were not preceded by a “User Gesture.” Such gestures (sometimes more precisely called User Activations) include a variety of simple actions, from clicking the mouse to typing a key; each interpreted as “The user tried to do something in this web content.”

A single user gesture can unlock any of a surprisingly wide array of privileged (“gated”) actions:

  • Allow a popup window to open
  • Allow an Application Protocol to be invoked
  • Allow an OnBeforeUnload dialog box to show
  • Allow the Vibration API to vibrate the device
  • Allow script to take the window fullscreen
  • Allow the password manager to fill the username/password into the page in a way that JavaScript can see them
  • Allow the page to prompt the user for a file to upload
  • Impact the behavior of file downloads (e.g. prompting)
  • and many more

So, when you see a site show a UI like this:

…chances are good that what they’re really trying to do is trick you into performing a gesture (mouse click) so they can perform a privileged action– in this case, open a popup ad in a new tab.

In terms of which actions can cause a gesture, the list is surprisingly limited, and includes mousedown (but not mouseup/click):

// Returns |true| if |type| is the kind of user input that should trigger user interaction observers.
bool IsUserInteractionInputType(blink::WebInputEvent::Typetype) {
// Ideally, this list would be based more off of
// https://whatwg.org/C/interaction.html#triggered-by-user-activation. return type ==
blink::WebInputEvent::Type::kMouseDown ||
type == blink::WebInputEvent::Type::kGestureScrollBegin ||
type == blink::WebInputEvent::Type::kTouchStart ||
type == blink::WebInputEvent::Type::kRawKeyDown; }

Some gestures are considered “consumable”, meaning that a single user action allows only one privileged action; subsequent privileged actions require another gesture. Web Developers do not have unlimited time to consume the action: In Chrome, when you click in a web page, the browser considers this “User Activation” valid for five seconds (as of Februrary 2019) before it expires. Here’s a simple test.

Unfortunately, even this weak protection is subject to both false positives (an unwanted granting of privilege) and false negatives (an action is unexpectedly blocked).

You can learn more about this topic (and the complexity of dealing with nested frames, etc) in the original Chromium User Activation v2 spec, and the User-Activation section of HTML5.

-Eric

A bit of GREASE keeps the web moving

For the first few years of the web, developers pretty much coded whatever they thought was cool and shipped it. Specifications, if written at all, were an afterthought.

Then, for the next two decades, spec authors drafted increasingly elaborate specifications with optional features and extensibility points meant to be used to enable future work.

Unfortunately, browser and server developers often only implemented enough of the specs to ensure interoperability, and rarely tested that their code worked properly in the face of features and data allowed by the specs but not implemented in the popular clients.

Over the years, the web builders started to notice that specs’ extensibility points were rusting shut– if a new or upgraded client tried to make use of a new feature, or otherwise change what it sent as allowed by the specs, existing servers would fail when they saw the encountered the new values. (Formally, this is called ossification).

In light of this, spec authors came up with a clever idea: clients should send random dummy values allowed by the spec, causing spec-non-compliant servers that fail to properly ignore those values to fail immediately. This concept is called GREASE (with the backronym “Generate Random Extensions And Sustain Extensibility“), and was first implemented for the values sent by the TLS handshake. When connecting to servers, clients would claim to support new ciphersuites and handshake extensions, and intolerant servers would fail. Users would holler, and engineers could follow up with the broken site’s authors and developers about how to fix their code. To avoid “breaking the web” too broadly, GREASE is typically enabled experimentally at first, in Canary and Dev channels. Only after the scope of the breakages is better understood does the change get enabled for most users.

GREASE has proven such a success for TLS handshakes that the idea has started to appear in new places. Last week, the Chromium project turned on GREASE for HTTP2 in Canary/Dev for 50% of users, causing connection failures to many popular sites, including some run by Microsoft. These sites will need to be fixed in order to properly load in the new builds of Chromium.

// Enable "greasing" HTTP/2, that is, sending SETTINGS parameters with reserved identifiers and sending frames of reserved types, respectively. If greasing Frame types, an HTTP/2 frame with a reserved frame type will be sent after every HEADERS and SETTINGS frame. The same frame will be sent out on all connections to prevent the retry logic from hiding broken servers.
NETWORK_SWITCH(kHttp2GreaseSettings, "http2-grease-settings") NETWORK_SWITCH(kHttp2GreaseFrameType, "http2-grease-frame-type")

One interesting consequence of sending GREASE H2 Frames is that it requires moving the END_STREAM flag (recorded as fin=true in the netlog) from the HTTP2_SESSION_SEND_HEADERS frame into an empty (size=0) HTTP2_SESSION_SEND_DATA frame; unfortunately, the intervening GREASE Frame is not presently recorded in the netlog.

You can try H2 GREASE in Chrome Stable using command line flags that enable GREASE settings values and GREASE frames respectively:

chrome.exe --http2-grease-settings bing.com
chrome.exe --http2-grease-frame-type bing.com

Alternatively, you can disable the experiment in Dev/Canary:

chrome.exe --disable-features=Http2Grease

GREASE is baked into the new HTTP3 protocol (Cloudflare does it by default) and the new UA Client Hints specification (where it’s blowing up a fair number of sites). I expect we’ll see GREASE-like mechanisms appearing in most new web specs where there are optional or extensible features.

-Eric

META CHARSET

Someone complained that a Japanese page is garbled in Edge/Chrome, but renders with the correct characters in Firefox and IE:

The problem is that Chromium is using an unexpected character set to interpret the response in the HTML Parser. That happens because the server doesn’t send a proper character set directive. To avoid problems like this and improve performance, document authors should specify the character set in the HTTP response headers:

Content-Type: text/html; charset=utf-8

If a charset isn’t specified in the headers, Chrome looks for a META CHARSET declaration within the response body text (“note the absurdity of encoding the character encoding in the document that you’re trying to decode“). The HTML5 spec demands that documents be encoded in UTF-8, and that the charset declaration, if any, appears within the first 1024 bytes of the response.

Chrome checks the full head for a character-set directive, and if it doesn’t find one, ensures that it’s looked through at least 1024 bytes before giving up.

Unfortunately, this site accidentally includes a div tag up in the head (ending the head section prematurely), and buries the charset down 1479 bytes into the response:

To avoid problems like this:

  1. Specify the CHARSET in the Content-Type response header, and
  2. Ensure META CHARSET appears as the first element of your HEAD.
  3. To avoid problems in legacy browsers, write it as utf-8 rather than as utf8.

Get your <HEAD> in order!

-Eric

Client Certificate Authentication

While most HTTPS sites only authenticate the server (using a certificate sent by the website), HTTPS also supports a mutual authentication mode, whereby the client supplies a certificate that authenticates the visiting user’s identity. Such a certificate might be stored on a SmartCard, or used as a part of an OS identity feature like Windows Hello.

To request mutual authentication, servers send a CertificateRequest message to the client during the HTTPS handshake, specifying a criteria filter that the browser will use to find a client certificate to satisfy the server’s request.

If a client certificate is supplied in the browser’s Certificate response to the server’s challenge, the browser proves the user’s possession of that certificate using the private key that matches that client certificate’s public key.

A client may choose not to send a certificate (either because no matching certificate is available, or because the user declined to supply a certificate that it had)—in such cases, the server may terminate the handshake (showing a Client Certificate Required error message) or it may continue the handshake and attempt to authenticate the user via other means.

Certificate Selection

The CertificateRequest message allows the server to specify criteria for the certificates it is willing to accept from the client, including details such as the certificate’s issuer, and key/signature/hash types.

The browser consults the Operating System’s trust store (Keychain on Mac OS X, certmgr.msc on Windows) to find any candidate certificates (unexpired certificates with the Client Authentication purpose set and a private key available) that match the server-supplied filtering criteria:

The private key for a given certificate might be stored on a SmartCard — when a SmartCard is inserted, the certificate(s) on it are “virtually” propagated to the OS trust store for use by browsers and other applications.

Certificates that meet the server’s filtering criteria are shown in a prompt:

If the user hits “Cancel”, the handshake is completed without sending a certificate. However, if the user selects a certificate, the browser caches that decision for the lifetime of the browser instance. The selected certificate will be resent on all new connections to the target origin and the prompt will not be shown again.

Today, there’s no good way to clear the selection decision, short of restarting the browser entirely. In contrast, legacy IE offered two very awkward mechanisms, the Clear SSLState button in the Internet Control Panel, and the ClearAuthenticationCache web API.

Automatic Selection of Client Certificate

Internet Explorer and Edge Legacy offered a behavior (Don’t prompt for client certificate selection when only one certificate exists, URLACTION_CLIENT_CERT_PROMPT), on-by-default for the Local Intranet Zone:

…whereby the browser would not prompt the user to select a certificate if the user only has one certificate that matches the server’s request. In such cases, the client would automatically send the matching certificate without showing a prompt.

For other zones, IE and Edge Legacy do prompt the user to select a certificate before any certificate is sent. This is a privacy measure, because if the browser silently sends the user’s identity to any website that asks for it, this is a “super-cookie” that would allow tracking that user across sites. Also, the client’s certificate might directly contain personally identifiable information about the user (e.g. their email address, office phone number, home address, etc).

Chromium (and thus Chrome, Edge, Brave, Opera, Vivaldi) largely does not use the concept of Zones, so instead the AutoSelectCertificateForUrls policy exists. This policy allows an IT administrator to configure clients to automatically send certificates to specified websites that request them, which can be used to satisfy the need to have, say, the user’s Windows Hello certificate sent to *.login.microsoft.com sites.

Here are two examples: the first selects the first certificate issued by “Windows Hello PIN – MSIT1” and the second rule selects the certificate with a SubjectCN=”RSACSP”.

If you’re trying to set a rule whereby multiple client certificates are valid candidates and the client should just return the first found match, just add another rule with the same pattern and a different filter.

For instance, this set will use SubjectCN=”RSACP” if a matching certificate found, or a certificate with IssuerCN=”Windows Hello PIN – MSIT1” if not:

A screenshot of a cell phone

Description automatically generated

However, as you may have noticed, the AutoSelectCertificateForUrls policy has one significant limitation, which is that it always sends the user’s first matching certificate to the selected site. Some users might have more than one certificate that matches the policy (for instance, some enterprises have both “test” and “production” certificates.

To address this shortcoming, the Edge team introduced a new policy in Edge 81. The new ForceCertificatePromptsOnMultipleMatches policy which does as it says: If the client has multiple certificates that could be used to satisfy the {OriginFilter->CertificateFilter} policy specified by a AutoSelectCertificateForUrls policy, instead of simply sending the first matching certificate, the browser will instead show a certificate selection prompt filtered to the certificates that match the policy.

If you find that Microsoft Edge shows a client certificate selection prompt in one scenario where other browsers do not, one possibility is that the site in question is not actually requesting a client certificate from those other browsers for some reason. For instance, some web authentication flows, including Microsoft’s AAD login, take the browser’s User-Agent into account when deciding what authentication mechanisms to use with the client.

In order to understand exactly what’s going on with Client Authentication, collect Network Traffic logs. SSL_HANDSHAKE_MESSAGE_RECEIVED messages of type 13 represent the client certificate request.

-Eric

Bonus trivia

  1. Some smart people persuasively argue that mutually-authenticated TLS is a misfeature.
  2. Registry-specified certificate selection policies apply across browser profiles, meaning that they are in force even when the user is in an Incognito browser session (!!)
  3. A user’s certificate selection apply for the entire browser session, making logout for ClientCert-authenticated sites an unsolved problem.
  4. Client Certificate prompting behavior on Android is weird.
  5. Client Certificate authentication is generally not available in 3rd-party browsers on iOS (Safari has access to the keychain).
  6. There are some test servers available.

The general notion of “how Client Certificates were supposed to work” was that each user would have one certificate for each organization to which they belong, issued by that organization’s root certificate. When visiting that organization’s servers, the server would send in the CertificateRequest message the identifier(s) of the root certificate(s) to which acceptable client certificates chain (using the certificate_authorities structure). The visiting client would then filter the certificates available for selection to only those that chain to that root (hopefully one certificate).

So, say I have two certificates, e.g. USA-NationalID and Microsoft-EmployeeID. When I visit https://portal.microsoft.com, Microsoft sends a CertificateRequest with a MicrosoftRootCA in the certificate_authorities field. My browser automatically filters my client certificates list to just the Microsoft-EmployeeID certificate and then sends that. In contrast, when I visit https://irs.gov, the government sends a CertificateRequest with a USGovernmentRootCA in the certificate_authorities field. My browser automatically filters my client certificates list to just the USA-NationalID certificate and sends that.

In practice, unfortunately, things haven’t worked out that way. Most organizations have not had the infrastructure or discipline to configure things to work like that, and as a consequence you end up with varying client behavior.

Firefox doesn’t seem to filter the certificate list, but it does offer a “Remember this decision” checkbox which presumably reduces user annoyance:

Firefox does not respect the Windows Trust Store, so each client certificate must be manually loaded into Firefox’s configuration (unless you set the security.osclientcerts.autoload preference to true). This is a hassle, but it tends to result in a somewhat “cleaner” experience where the user isn’t distracted by random certificates that might be cluttering Windows’ cert store.

Selection UI (as of June 2021)

Edge’s UI shows the certificate’s Friendly Name and includes an icon to try to help convey the type of client certificate (SmartCard, Windows Hello, Traditional).

In some cases, organizations are generating invalid client certificates but expecting them to work, leading us to create compat accommodations like the FEATURE_CLIENTAUTHCERTFILTER Feature Control Key.

In the browser, SmartCards can be used for two ways: HTTPS Client Certificate Authentication, and Windows Integrated Authentication.

  • Straight TLS mutual authentication, as described above.
  • Windows Integrated Authentication that occurs when visiting a website that sends a WWW-Authenticate: Negotiate header. The client may automatically send the user’s login credentials (Intranet Zone). Or, if those creds do not work or the Zone is not configured for automatic credential release (Non-intranet), the user will be prompted for credentials to use. In Edge 79, the user would get a prompt with two blank fields (“Username” and “Password”). In Edge 80 or later, upon noticing that the user has configured Windows Hello, the user will be shown the Windows Hello auth dialog that allows the user to use their face, type a PIN, use a SmartCard, etc. So, now Edge 80 matches Edge Legacy (v18 and lower).

Low Level Details 1
Low Level Details 2

Nice discussion (with pictures) of setting up client cert auth on IIS.

In Windows 10 Apps, the AppContainer must have the sharedUserCertificates capability to use certificates from the trust store.

This is the first message the client can send after receiving a ServerHelloDone message. This message is only sent if the server requests a certificate. If no suitable certificate is available, the client MUST send a certificate message containing no certificates. That is, the certificate_list structure has a length of zero. If the client does not send any certificates, the server MAY at its discretion either continue the handshake without client authentication, or respond with a fatal handshake_failure alert. Also, if some aspect of the certificate chain was unacceptable (e.g., it was not signed by a known, trusted CA), the server MAY at its discretion either continue the handshake (considering the client unauthenticated) or send a fatal alert.

CertificateVerify signs using the client certificate’s private key.

CertOpenStore “my” store

https://docs.microsoft.com/en-us/windows/desktop/api/wincrypt/nf-wincrypt-certopenstore

ClientAuthIssuer trust store.

Hard Problems: Fetch in Serviceworker scenario — how can the user select a certificate when no UI is allowed?