browsers, dev

Streaming Audio in Edge

This issue report complains that Edge doesn’t stream AAC files and instead tries to download them. It notes that, in contrast, URLs that point to MP3s result in a simple audio player inside the browser.

Edge has always supported AAC so what’s going on?

The issue here isn’t about AAC, per-se; it’s instead about whether or not the browser, upon direct navigation to an audio stream, will accommodate that by generating a wrapper HTML page with an <audio> element pointed at that audio stream URL.

PlaceholderPage

A site that wants to play streaming AAC in Edge (or, frankly, any media type, for any browser) should consider creating a HTML page with an appropriate Audio or Video element pointed at the stream.

In Edge, the list of audio types we will create the wrapper page does not include AAC:

audio/mp4, audio/x-m4a, audio/mp3, audio/x-mp3, audio/mpeg,
audio/mpeg3, audio/x-mpeg, audio/wav, audio/wave, audio/x-wav,
audio/vnd.wave, audio/3gpp, audio/3gpp2

In contrast, Chrome creates the MediaDocument page for a broader set of known audio types:

static const char* const kStandardAudioTypes[] = {
 "audio/aac",  "audio/aiff", "audio/amr",  "audio/basic",  "audio/flac",
 "audio/midi",  "audio/mp3",  "audio/mp4",  "audio/mpeg",  "audio/mpeg3", 
 "audio/ogg", "audio/vorbis",  "audio/wav",  "audio/webm",  "audio/x-m4a",
 "audio/x-ms-wma",  "audio/vnd.rn-realaudio",  "audio/vnd.wave"};

If the the response sends Content-Type: application/octet-stream, includes a Content-Dispostion: attachment, or puts a download attribute on the anchor <a> element that leads to the media, Edge will download the media file instead of playing it in the browser.

 

-Eric

PS: Edge has similar special handling for video types:

"application/mp4","video/mp4","video/x-m4v","video/3gpp",
"video/3gpp2","video/quicktime"

 

Standard
dev

ShellExecute Doesn’t

My oldest supported Windows application is a launcher app named SlickRun, and it’s ~24 years old this year. I haven’t done much to maintain it over the last few years, although it’s now available in 64-bit and runs great on Windows 10. (Thanks go to Embarcadero, who now offer a free “Community” edition of Delphi, the language/platform I ported SlickRun to circa 1994).

I still fix bugs in SlickRun from time to time, and as I was playing with Rust a few days ago I was reminded of one of the oldest limitations in my code– if you update your system’s %PATH% variable, those changes aren’t seen by applications/consoles spawned by SlickRun until you restart it. It’s particularly annoying because it’s so unexpected– users expect that command consoles launched by Win+R,cmd.exe,Enter will behave the same way as Win+Q,cmd,Enter, but the former consoles have the updated %PATH% while the latter do not.

While ShellExecute() sounds like it’s an API that causes the shell (aka Explorer) to execute something, in fact it does nothing of the sort.

Updating the Environment Block

The root cause of the “outdated path” problem is that processes launched via ShellExecute inherit the environment variables of their spawning process, and those environment variables (typically) are assigned as the process launches and never touched again. Because SlickRun starts with Windows, the %PATH% when it starts is the %PATH% that every process it launches inherits. (You can easily view a process’ environment block using the Properties > Environment tab in Process Explorer).

So, how does Explorer detect the change? That part I figured out ages ago– after updating an environment variable, the System Properties > Environment Variables Control Panel UI (or the SetX.exe console tool) broadcast a WM_SETTINGCHANGE message to all top-level windows with an lparam containing the string “Environment”. I could easily add code to SlickRun to detect that the variables had changed, but for decades I didn’t really know what to do next… I didn’t know how to read the updated variables (without doing something hacky like restarting the process) nor ensure that they were passed to the applications spawned by ShellExecute.

Yesterday, I got fed up and started Googling. A few posts on StackOverflow mentioned a promising-sounding function, RegenerateUserEnvironment. And while that function appears to be undocumented, there’s an amazing issue filed in an open-source tracker that explains exactly how Windows Explorer uses this function– basically, just wait for the WM_SETTINGCHANGE event, then call the API. The RegenerateUserEnvironment will replace the calling process’ current environment block with the latest values.

Launching at Medium Integrity

While we’re on the topic of executing applications “like the shell”, another scenario came up twelve years ago when Windows Vista was first introduced. The SlickRun installer, written in NSIS, launches SlickRun when installation completes. Unfortunately, the installer runs with Admin rights (High integrity), which means that, by default, all of the programs it launches inherit that integrity. For SlickRun, this is especially bad because it means that any programs that it, in turn, launches during that first session (e.g. your browser!) will run at High integrity too. Not good.

While you can easily use the “Runas” verb to ShellExecute to launch a High integrity application from a Medium integrity application, there (depressingly) isn’t a way to do the opposite. For years, the official recommendation was to do some fancy coding to clone Explorer’s tokens and use those. Unfortunately, this is quite complicated to implement, especially within a NSIS script.

As it turns out, however, there’s a trivial workaround which works quite well– while ShellExecute doesn’t run things as the shell, applications can easily get Explorer to launch anything they like at Explorer’s integrity. The trick is to simply invoke explorer.exe and pass the filename to be executed as the first command line argument:

While this approach isn’t technically supported, I expect it is likely to continue to work for the foreseeable future.

 

It’s depressing that together these tricks have taken me almost twenty years to discover, but I’m happy that I have. I hope they help you out.

-Eric

Standard
browsers

Cookie Limits

I’ve been writing about Cookies a lot recently, and also did so almost a decade ago.

Edge/IE cookie limits

The June 1018 Cumulative Updates increased the per-domain cookie limit from 50 to 180 for IE and Edge across Windows 7, Windows 8.1, and Windows 10 (TH1 to RS2). This higher limit matches Chrome’s cookie jar.

In IE/Edge, if the cookie length exceeds 10240 characters, document.cookie returns an empty string. (Cookies over 1023 characters can also lead to an empty document.cookie string in the event of a race condition). Cookie strings longer than 10KB will still be sent to the server in the Cookie request header, although many servers will reject headers over 16kb in size.

In IE/Edge, the browser will ignore attempts to set (and suppress attempts to send) individual cookies (`​name=value`) over 5118 characters in length.

Test Page

At the time of this writing, there’s a nice test page that attempts to exercise cookie limits using the DOM.

Standard
browsers, privacy, tech, web

Cookie Controls, Revisited

Update: The October 2018 Cumulative Security Update (KB4462919) brings the RS5 Cookie Control changes described below to Windows 10 RS2, RS3, and RS4.

Cookies are one of the most crucial features in the web platform, and large swaths of the web don’t work properly without them. Unfortunately, cookies are also one of the primary mechanisms that trackers and ad networks utilize to follow users around the web, potentially impacting users’ privacy. To that end, browsers have offered cookie controls for over twenty years.

Back in 2010, I wrote a summary of Internet Explorer’s Cookie Controls. IE’s cookie controls were very granular and quite powerful. The basic settings were augmented with P3P, a once-promising feature that allowed sites to advertise their privacy practices and browsers to automatically enforce users’ preferences against cookies. Unfortunately, major sites created fraudulent P3P statements, regulators failed to act, and the entire (complicated) system collapsed. P3P was removed from IE11 on Windows 10 and never implemented in Microsoft Edge.

Instead, Edge offers a very simple cookie control in the Privacy and Security section of the settings. Under the Cookies option, you have three choices: Don’t block cookies (the default), Block all cookies, and Block only third party cookies:

CookieSetting

This simple setting hides a bunch of subtlety that this post will explore.

Cookie => Cookie-Like

For the October 2018 update (aka “Redstone Five” aka “RS5”) we’ve made some important changes to Edge’s Cookie control.

The biggest of the changes is that Edge now matches other browsers, and uses the cookie controls to restrict cookie-like storage mechanisms, including localStoragesessionStorageindexedDB, Cache API, and ServiceWorkers. Each of these features can behave much like a cookie, with a similar potential impact on users’ privacy.

While we didn’t change the UI, it would be accurate to change it to:

CookieLike

This change improves privacy and can even improve site compatibility. During our testing, we were surprised to discover that some website flows fail if the browser blocks only 3rd party cookies without also blocking 3rd-party localStorage. This change brings Edge in line with other browsers with minor exceptions. For example, in Firefox 62, when 3rd-party site data is blocked, sessionStorage is still permitted in a 3rd-party context. In Edge RS5 and Chrome, 3rd party sessionStorage is blocked if the user blocks 3rd-party cookies.

Block Setting and Sending

Another subtlety exists because of the ambiguous terminology “third-party cookie.” A cookie is just a cookie– it belongs to a site (eTLD+1). Where the “party” comes into play is the context where the cookie was set and when it is sent.

In the web platform, unless a browser implements restrictions:

  • A cookie set in a first-party context will be sent to a first-party context
  • A cookie set in a first-party context will be sent to a third-party context
  • A cookie set in a third-party context will be sent to a first party context
  • A cookie set in a third-party context will be sent to a third-party context

For instance, in this sample page, if the IFRAME and IMG both set a cookie, these cookies are set in a third-party context:Contexts

  • If the user subsequently visits domain2.com, the cookie set by that 3rd-Party IFRAME will now be sent to the domain2.com server in a 1st-Party context.
  • If the user subsequently visits domain3.com, the cookie set by that 3rd-Party IMG will now be sent to the domain3.com server in a 1st-Party context.

Historically, Edge and IE’s “Block 3rd party cookies” options controlled only whether a cookie could be set from a 3rd party context, but did not impact whether a cookie initially set in a 1st party context would be sent to a 3rd party context.

As of Edge RS5, setting “Block only 3rd party cookies” will now also block cookies that were set in a 1st party context from being sent in a 3rd-party context. This change is in line with the behavior of other browsers.

Edge Controls Impacted By Zones

With the move from Internet Explorer to Edge, the Windows Security Zones architecture was largely left by the wayside.

Zones

However, cookie controls are one of a small number of exceptions to this; Edge applies the cookie restrictions only in the Internet Zone, the zone almost all sites fall into (outside of users on corporate networks).

Perhaps surprisingly, cookie-like features and the document.cookie getter are restricted, even in the Intranet and Trusted zones.

Chrome and Firefox do not take Windows Security Zones into account when applying cookie policies.

Test Cases

I’ve updated my old “Cookies” test page with new storage test cases. You can set your browser’s privacy controls:

Block3rdPartyChrome

Block3rdPartyFF

…then visit the test page to see how the browser limits features from 3rd-party contexts. You can use the Swap button on the page to swap 1st-party and 3rd-party contexts to see how restrictions have been applied. You should see that the latest versions of Chrome, Firefox, and Edge all behave pretty much the same way.

-Eric


Appendix: Chromium Audit

In the course of our site-compatibility investigations, I had a look at Chromium’s behavior with regard to their cookie controls. In Chromium, Blink asks the host application for permission to use various storages, and these chokepoints check:

cookie_settings_->IsCookieAccessAllowed(origin_url, top_origin_url);

…which is sensitive to the various “Block Cookies” settings.

Mojo messages come up through renderer_host/chrome_render_message_filter.cc, gating access to

Additionally, ChromeContentBrowserClient gates

Elsewhere, IsCookieAccessAllowed is used to limit:

  • Flash Storage (PP_FLASHLSORESTRICTIONS_BLOCK)
  • Client Hints

Of these, Edge does not support WebSQL, FileSystem, SharedWorker, or Client Hints.

Standard
browsers, fiddler

Firefox and Fiddler – Easier than Ever

In a world where software and systems seem to march inexorably toward complexity, I love it when things get simpler.

Years ago, Firefox required non-obvious configuration changes to even send traffic to Fiddler. Eventually, Mozilla changed their default behavior on Windows to adopt the system’s proxy, meaning that Firefox would automatically use Fiddler when it was attached, just like Chrome, IE, Edge and other major browsers.

Unlike other browsers, Firefox also has its own Trusted Root Certificates store, which means that if you attempt to load a HTTPS page through Firefox with Fiddler’s HTTPS Decryption Mode enabled, you’ll get an error page:

FirefoxMITMDetected
MOZILLA_PKIX_ERROR_MITM_DETECTED error page

To configure Firefox to trust Fiddler’s root certificate, you used to have to manually install it by opening the FiddlerRoot.cer file, ticking the “Trust this CA to identify websites” box, and clicking OK:

FirefoxCA
The old way: Manually trusting Fiddler’s certificate

Making matters more annoying, any time you reset Fiddler’s root certificates (e.g. using the Actions button inside Tools > Fiddler Options > HTTPS), you had to do the whole dance over again. If you wanted to remove the obsolete root certificates, you had to visit a buried configuration UI:

ManualTrustFF
The old way: Administering the Firefox Certificate Store

Yesterday, I was delighted to learn that Firefox added a better option way back in Firefox 49. Simply visit about:config in Firefox and toggle the security.enterprise_roots.enabled setting to true.

FirefoxEnterprise
Enable the new hotness in about:config

After you make this change, Firefox will automatically trust certificates chained to roots in the Windows Trusted Root Certificate store, just like Chrome, Edge, IE and other browsers. Unfortunately, Mozilla has declined to turn this on by default, but it’s still a great option.

 

-Eric

Standard
browsers, design, privacy

Chrome Sync

Disclaimer: Hi. I’m an engineer on the Edge browser now, but worked on Chrome Security for a bit over two years. I speak for no one but myself, and I share no internal or confidential information in this post.

Update: The Chrome team announced upcoming changes based on user-feedback.

This weekend, there were a bunch of breathless articles and blogs about a very small change recently made to Chrome. Some of the claims are correct and carefully thought out, but in the swirl of clickbait and confusion, there are a bunch of inaccurate claims as well.

What Changed?

No, Chrome doesn’t “upload your browser history when you check GMail”… unless you tell it to do so.

Yes, Chrome has streamlined the opt-in to the browser’s “Sync” features, such that you no longer need to individually type your username and password when enabling Sync. Whether you consider this “Convenient!” or “Terrible!” is a matter of perception and threat model.

Screenshots

Because many people haven’t seen this change yet, here are some screenshots from Chrome 71.3558.

When you sign into a Google site or service in the browser’s HTML content area, your avatar/profile picture will now appear in the browser UI. If you click on your avatar, a flyout appears offering to enable sync:

Meeple

Similarly, if you use certain browser features that are more valuable with Sync (e.g. creating a bookmark or storing a password), the UI offers to turn on Sync:

OptIn

(Concern: This “Sync as Eric” button appears in the same X,Y location as the “Save Password” button on the preceding flyout/bubble, so you could conceivably click it by accident.)

Bookmark

If you click on one of these options to turn on Sync, a giant flyout appears to tell you what this means:

Get Google smarts

Interestingly/Concerningly, if you click “Settings”, it’s interpreted as “Yes, and show Settings“:

SettingsMeansYes

If you look through the Sync settings (also available by visiting chrome://settings/?search=sync), you’ll get a rich list of controls for what you can choose to sync:

SyncMain

Below that list, you can also find a list of all of the other ways (independent of Sync/Signin) that Chrome can talk back to Google:

OtherSettings

Change Your Mind? Disable Sync

If you change your mind after enabling Sync, or enabled it by accident, it’s easy to change. Click your Avatar and click on the “Syncing to” badge. Then click Turn off. Decide whether or not you want to keep local copies of the data that was sync’d to the server:

TurnOffSync

Note: If you accidentally signed into Chrome and enabled Sync on a device you borrowed from someone else, it’s very important that you check the box to clear local data when you’re done with it, or your information will be left behind on the device.

Fun Stuff for Nerds

If you’re a geek and are interested to better understand how Sync Works, visit the URL chrome://sync-internals/ in Chrome. You can see events in the Sync process and tons of data about what exactly is syncing.

CoolData

Opt-Out

If you don’t want Chrome to Sync, just don’t click buttons that offer to enable it.

Arguably, enabling Sync is now so streamlined that you could conceivably do it by accident (or someone borrowing your PC could do it for you).

If you’re concerned about sign-in to Chrome and want to ensure that you don’t ever activate Sync by accident, you can set a Policy inside HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Google\Chrome:

Regedit

You can set the Windows policy by simply importing this registry script.

After Sync is disabled, buttons asking to turn on Sync disappear, and (interestingly) if you try to sign into Chrome, a warning notice appears:

SyncDisabled

Motivations

Many of the articles that came out this weekend were rife with speculation that this is some new attempt from Google to erode your privacy by vacuuming up even more of your browsing data.

I don’t work for Google and I have no insider knowledge, but I think such attributions are incorrect. I think the correct explanations are much more mundane:

  1. The old UX was just really weird for humans. “Wait, you computer nerds are telling me that I just signed into Google using Google Chrome in the browser content area and now you’re telling me that I need to sign into the Google Chrome Browser using the browser chrome. WTF?!?”
  2. The old UX was dangerously misleading for people who share computers, a worryingly common practice.The new UX makes it at least a bit more clear that if you’re browsing on a borrowed computer, you really should be using a discardable Guest profile.(I think Guest Profiles are one of the coolest little-used features in Chrome).
  3. The new UX streamlines the enabling of Chrome Sync. Chrome Sync provides advantages to both the user and Google.

The Advantage to the User is that their satisfaction with the browser is higher. On average, satisfaction increases when Sync is on because the browser can do more for the user, better protecting them from phishing sites (via the password manager) and sharing their bookmarks, permission settings, etc on every device they use. Users with many devices (me) are especially excited about this benefit– I have enough to remember and configure, and I want my browser to help as much as it can.

The Obvious Advantage to Google is that users who are more satisfied with Chrome are more likely to use Chrome and not switch to some other browser. Chrome is simply stickier because switching to a browser that doesn’t have access to all of the user’s stored passwords, bookmarks, settings, etc, is less appealing.

In a world where browsers are shells around commoditized HTML parsers and script engines (E.g. compare Brave vs. Chrome), sticky features like sync are critical to marketshare. Consider, for instance, Chrome on iOS. It runs atop Apple’s WkWebView and is by-(Apple’s)-design intrinsically inferior in almost every way to the native Safari. Except for one thing… Chrome has my settings and data and Safari doesn’t. So I’ll go out of my way to get Chrome on iOS because, to me, Sync is critical time-saver that Safari can never match, because Safari doesn’t run on Windows and Chrome does. iOS represents a potentially addressable browser market of around a billion devices.

The Possible Unobvious Advantages to Google are what worry the skeptics and fearmongers– what if Google uses my data for something evil? Evil might range from a little evil (showing me ads for Beanie Babies because it sees I’m browsing a lot of Beanie Baby sites) to a lot evil (giving my data to “the Government” or my insurance company or my boss or some other bad person). Similarly, they could start using it recklessly (having my Google Home ask in front of my dinner guests “Hey, Eric, want to continue your search for the best Beanie Baby sales?“).

I personally don’t have significant concerns on this front (I got to see how seriously privacy is taken inside of Google and Chrome), but some people do.

My threat model is not your threat model.

If Evil Google is in your threat model, you can set an option to encrypt your sync data such that only your local Chrome instances can see it:

EncryptSync

If Extra Evil Google is in your threat model, you shouldn’t be using Chrome at all, because obviously Extra Evil Google could just backdoor Chrome before encryption or after decryption.

-Eric

PS: A helpful thread from a Chrome area owner.

Update: The Chrome team announced upcoming changes based on user-feedback.

Standard
browsers, dev

Cookies and Concurrency, Redux

In yesterday’s episode, I shared the root cause of a bug that can cause document.cookie to incorrectly return an empty string if the cookie is over 1kb and the cookie grows in the middle of a DOM document.cookie getter operation.

Unfortunately, that simple bug wasn’t the root cause of the compatibility problem that I was investigating when my code-review uncovered it. The observed compatibility bug was slightly different– in the repro case, only one of the document’s cookies goes missing, and it goes missing even when only one page is setting the cookie.

After the brain-melting exercise of annotating the site’s minified framework libraries (console.log(‘…’) ftw!) via Fiddler’s AutoResponder, I found that the site uses the document.cookie API to save the same cookie (named “ld“) three times in a row, adding some information to the cookie each time. However, the ld cookie mysteriously disappears between 0.4 and 6 milliseconds after it gets set the third time. I painstakingly verified that the cookie wasn’t getting manipulated from any other context when it disappeared.

Hmm…

As I wrote up the investigation notes, I idly noted that due to a trivial typo in the website’s source code, the ld cookie was set first as a Persistent cookie, then (accidentally) as a Session cookie, then as a Persistent cookie.

In re-reading the notes an hour later, again my memory got tickled. Hadn’t I seen something like this before?

Indeed, I had. Just about five years ago, a user reported a similar bug where a HTTP response contained two Set-Cookie calls for the same cookie name and Internet Explorer didn’t store either cookie. I built a reduced test case and reported it to the engineering team.

Pushing Cookies

The root cause of the cookie disappearance relates to the Internet Explorer and Edge “loosely-coupled architecture.”

In IE and Edge, each browser tab process runs its own networking stack, in-process1. For persistent cookies, this poses no problem, because every browser process hits the same WinINET cookie storage area and gets back the latest value of the persistent cookie. In contrast, for session cookies, there’s a challenge. Session cookies are stored in local (per-process) variables in the networking code, but a browser session may include multiple tab processes. A Session cookie set in a tab process needs to be available in all other tab processes in that browser session.

As a consequence, when a tab writes a Session cookie, Edge must send an interprocess communication (IPC) message to every other process in the browser session, telling each to update its internal variables with the new value of the Session cookie. This Cookie Pushing IPC is asynchronous, and if the named cookie were later modified in a process before the IPC announcing the earlier update to the cookie is received, that later update is obliterated.

The Duplicate Set-Cookie header version of this bug got fixed in the Fall 2017 Update (RS3) to Windows 10 and thus my old Set-Cookie test case case no longer reproduces the problem.

Unfortunately, it turns out that the RS3 fix only corrected the behavior of the network stack when it encounters this pattern– if the cookie-setting calls are made via document.cookie, the problem reappears, as in this document.cookie test case.

BadBehavior

Playing with the repro page, you’ll notice that manually pushing “Set HOT as a Session cookie” or “Set as a Persistent cookie” works fine, because your puny human reflexes aren’t faster than the cookie-pushing IPC. But when you push the “Set twice” button that sets the cookie twice in fast succession, the HOT cookie disappears in Edge (and in IE11, if you have more than one tab open).

Until this bug is fixed, avoid using document.cookie to change a persistent cookie to a session cookie.

-Eric

In contrast, in Chrome, all networking occurs in the browser process (or a networking-only process), and if a tab process wants to get the current document.cookie, it must perform an IPC to ask the browser process for the cookie value. We call this “cookie pulling.”

Standard