Capture Network Logs (NetLog) from Edge and Chrome (and Electron and WebView2)

Problems in accessing websites can often be found and fixed if the network traffic between the browser and the website is captured as the problem occurs and the resulting log file is shared with engineers.

This short post explains how to capture such log files.

Capturing Network Traffic Logs

If someone asked you to read this post, chances are good that you were asked to capture a web traffic log to track down a bug in a website or your web browser.

Fortunately, in Google Chrome or the new Microsoft Edge (version 76+), capturing traffic is simple:

  1. Optional but helpful: Close all browser tabs but one.
  2. Navigate the tab to about://net-export
  3. In the UI that appears, press the Start Logging to Disk button.
  4. Choose a filename to save the traffic to. Tip: Pick a location you can easily find later, like your Desktop.
  5. Reproduce the networking problem in a new tab. If you close or navigate the //net-export tab, the logging will stop automatically.
  6. After reproducing the problem, press the Stop Logging button.
  7. Share the Net-Export-Log.json file with whomever will be looking at it. Optional: If the resulting file is very large, you can compress it to a ZIP file.
Network Capture UI

Privacy-Impacting Options

In some cases, especially when you dealing with a problem in logging into a website, you may need to set either the Include cookies and credentials or Include raw bytes options before you click the Start Logging button.

Note that there are important security & privacy implications to selecting these options– if you do so, your capture file will almost certainly contain private data that would allow a bad actor to steal your accounts or perform other malicious actions. Share the capture only with a person you trust and do not post it on the Internet in a public forum.

Tutorial Video

If you’re more of a visual learner, here’s a short video demonstrating the traffic capture process.

Alternatives

If you use Edge’s “Recreate my problem” button on the Feedback Wizard, the feedback tool will capture and include a network trace (in “include cookies and credentials” mode) as a part of your feedback report.

This method is the easiest way to get a network trace to Microsoft: the JSON is transmitted and stored securely without you having to find a way to encrypt and transfer the data. However, this method is inflexible: it does not allow you to send your traffic log to a friend or attach it to a bug in the Chromium bug database, and it does not expose the option to “include raw bytes.”

Analyzing Traffic Logs

In a followup post, I explore how developers can analyze captured traffic.

Thanks for your help in capturing network logs to diagnose and fix problems!

-Eric

Appendix A: Capture on Startup

In rare cases, you may need to capture network data early (e.g. to capture proxy script downloads and the like. To do that, close Edge, then run:

msedge.exe --log-net-log=%USERPROFILE%\Desktop\ReproNetlog.json --net-log-capture-mode=Everything

If you want to capture unsanitized cookies and authentication headers, but not the response bodies, use --net-log-capture-mode=IncludeSensitive instead. Omit the final parameter entirely if you do not want to include the raw data and want just the “Strip Private Information” mode of capture.

Appendix A.1: Capturing Electron and WebView2

Note: The command line argument approach also works for Electron JS applications like Microsoft Teams:

%LOCALAPPDATA%\Microsoft\Teams\current\Teams.exe --log-net-log=C:\temp\TeamsNetLog.json

Note: This will only capture the network traffic from the Chromium layer of Electron apps (e.g. web requests from the nodeJS side will not be captured) but it still may be very useful.

WebView2-based applications can either pass the --log-net-log command line into the WebView2 to initiate the capture, or they can add a second WebView control to their application (in the same context) and navigate it to about:net-export to allow the debugging user to manually trigger logging. It also seems that definining the WEBVIEW2_ADDITIONAL_BROWSER_ARGUMENTS environment variable ought to work as well (and that won’t require changing the WebView2 application).

Appendix B: Mobile Browsers

Appendix B.1: Android

The Net-Export feature works great on Android, but take care to ensure that you switch tabs to perform your repro after starting your capture– tab switching is less obvious on Android.

On mobile, when the capture completes, the resulting file is offered to be sent via email (because the mobile file system is not very accessible).

On modern versions of Android, the “Email Log” button will also allow you to upload the file to your Google Drive. Interestingly, if you subsequently download the file from your Drive, it will get a .eml file extension, but if you look at the file, it’s still the plain .json content; you can either fix the file extension, or you can just drag/drop the file into the NetLog viewer, which doesn’t care about the file extension.

Appendix B.2: iOS

Unfortunately, on iOS, the Network Export feature is somewhat unlikely to contain the data you need because the capture contains only the data sent by Chromium’s network stack, not the web content traffic (HTML, JS, CSS, images, etc) used inside the WkWebView control (embedded Safari). To capture data from the entire browser on iOS, you’ll need to use another approach, e.g. Telerik Fiddler.

Appendix C: Limitations

No HTTP POST Data

One important shortcoming in the current NetLog file format is that it does not contain any request body data, even if you select the “Include Raw Bytes” option. If you need the request body data, you may need to collect a HTTP Archive (HAR) file instead.

  • Hit F12 to open the Developer Tools.
  • Activate the Network tab.
  • Ensure the recording button at the top of the tab is red
  • Tick the Preserve log checkbox.
  • Reproduce the problem
  • Right-click entries in the the grid and choose Save all as HAR with content
  • Share the HAR file only with a person you trust and do not post it on the Internet in a public forum.

Alternatively, you might just capture the traffic using Fiddler.

Can’t Capture Requests that Don’t Reach the Network Stack

Perhaps surprisingly, the browser’s ServiceWorker feature lives above the network stack, so if the browser issues requests that are satisfied locally by the ServiceWorker, that “traffic” is not seen in the NetLog. (fetch requests that are sent from the ServiceWorker to the Network will appear in the log, however.) To see requests that are satisfied by the ServiceWorker, use the F12 Developer Tools.

Similar to the ServiceWorker case, the Blink engine has a memory cache for content that can be reused within a page. Certain requests (e.g. if there are ten image tags all pointed at the same URL) may be satisfied by this cache without sending the request down to the network stack.

Can’t Capture IE Mode

Pages running in Edge’s IE Mode tabs are loaded using URLMon and WinINET, the Windows Network Stacks used by Internet Explorer. Because this traffic does not go through the Chromium Network Stack, it is not recorded in NetLogs.

To work around this problem, you’re probably best off just capturing the traffic using Fiddler.

Appendix D: References

Disabling TLS/1.0 and TLS/1.1 in the new Edge Browser

UPDATE: Timelines in this post were updated in March 2020, October 2020, April 2021, and October 2021 to reflect the best available information.

HTTPS traffic is encrypted and protected from snooping and modification by an underlying protocol called Transport Layer Security (TLS). Disabling outdated versions of the TLS security protocol will help move the web forward toward a more secure future. All major browsers (including Firefox, Chrome, Safari, Internet Explorer and Edge Legacy) have publicly committed to require TLS version 1.2 or later by default starting in 2020.

Starting in Edge 84, reaching stable in July 2020, the legacy TLS/1.0 and TLS/1.1 protocols will be disabled by default. These older protocol versions are less secure than the TLS/1.2 and TLS/1.3 protocols that are now widely supported by websites:

To help users and IT administrators discover sites that still only support legacy TLS versions, the edge://flags/#show-legacy-tls-warnings flag was introduced in Edge Canary version 81.0.392. Simply set the flag to Enabled and restart the browser for the change to take effect:

Subsequently, if you visit a site that requires TLS/1.0 or TLS/1.1, the lock icon will be replaced with a “Not Secure” warning in the address box, alongside the warning in the F12 Developer Tools Console:

As shown earlier in this post, almost all sites are already able to negotiate TLS/1.2. For those that aren’t, it’s typically either a simple configuration option in either the server’s registry or web server configuration file. (Note that you can leave TLS/1.0 and TLS/1.1 enabled on the server if you like, as browsers will negotiate the latest common protocol version).

In some cases, server software may have no support for TLS/1.2 and will need to be updated to a version with such support. However, we expect that these cases will be rare—the TLS/1.2 protocol is now over 11 years old.

Obsolete TLS Blocks Subdownloads

Often a website pulls in some page content (like script or images) from another server, which might be running a different TLS version. In cases where that content server does not support TLS/1.2 or later, the content will simply be missing from the parent page.

You can identify cases like this by watching for the message net::ERR_SSL_OBSOLETE_VERSION in the Developer Tools console:

Unfortunately, a shortcoming in this console notification means that it does not appear for blocked subframes; you’ll need to look in the Network Tab or a NetLog trace for such failures.

Group Policy Details

Organizations with internal sites that are not yet prepared for this change can configure group policies to re-enable the legacy TLS protocols.

For the new Edge, use the SSLVersionMin Group Policy. This policy will remain available until the removal of the TLS/1.0 and TLS/1.1 protocols from Chromium in May 2021. Stated another way, the new Edge will always show an error page for TLS/1.0+1.1 (regardless of policy) in Edge 91 in May 2021.

For IE11 and Edge Legacy, the policy in question is the (dubiously-named) “Turn off encryption support” found inside Windows Components/Internet Explorer/Internet Control Panel/Advanced Page. Edge Legacy and IE will likely continue to support enabling these protocols via GP until they are broken from a security POV; this isn’t expected to happen for a few years.

Removal of User Bypass Option

Chromium, and thus Google Chrome, Microsoft Edge, etc, plan to remove the user’s ability to “Click through” the Legacy TLS blocking interstitial page in version 98, slated to reach stable in February 2022.

At that time, TLS/1.0 and TLS/1.1 will no longer be usable at all and the code implementing those protocols can be deleted from the codebase.

IE Mode Details

These older protocols will not be disabled in Internet Explorer and Edge Legacy until Spring 2022.

The New Edge has the ability to load administrator-configured sites in Internet Explorer Mode. IEMode tabs depend on the IE TLS settings, so if you need an IEMode site to load a TLS/1.0 website after Spring of 2022, you’ll need to enable TLS/1.0 using the “Turn off encryption support” group policy found inside Windows Components/Internet Explorer/Internet Control Panel/Advanced Page.

If you need to support a TLS/1.0 site in both Edge and IE Modes (e.g. the site is configured as “Neutral”), then you will need to set both policies (SSLVersionMin and “Turn off Encryption Support”).

Thanks for your help in securing the web!

-Eric

WebOCs and HTTP/2

If you’ve built an application using the old Web Browser Control (mshtml, aka Internet Explorer), you might notice that by default it does not support HTTP/2. For instance, a trivial WebOC host loading Akamai’s HTTP2 test page:

When your program is running on any build of Windows 10, you can set a Feature Control Key with your process’ name to opt-in to using HTTP/2.

For applications running at the OS-native bitness, write the key here:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Internet Explorer\Main\FeatureControl\FEATURE_WEBOC_ENABLE_HTTP2

For 32-bit applications running on 64-bit Windows, write it here:
HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\Internet Explorer\Main\FeatureControl\FEATURE_WEBOC_ENABLE_HTTP2

Like so:

After you make this change and restart the application, it will use HTTP/2 if the server and network path supports it.

Using HTTP/2 should help improve performance and can even resolve functional bugs.

-Eric

Update: Windows’ Internet Control Panel has a “HTTP2” checkbox, but it only controls web platform apps (IE and Legacy Edge), and unfortunately, the setting does not work properly for AppContainer/LowIL processes, which enable HTTP2 by default. This means that the checkbox, as of Windows 10 version 1909, is pretty much useless for its intended purpose (as only Intranet Zone sites outside of Protected Mode run at MediumIL).

A bug has been filed.

Update: Users of the new Chromium-based Edge browser can launch an instance with HTTP2 disabled using the disable-http2 command line argument, e.g. ms-edge.exe --disable-http2.

I’m not aware of a straightforward way to disable HTTP2 for the new Chromium-Edge-based WebView2 control, which has HTTP2 enabled by default.

The Pitfalls of EventSource over HTTP/1.1

While there are many different ways for servers to stream data to clients, the Server-sent Events / EventSource Interface is one of the simplest. Your code simply creates an EventSource and then subscribes to its onmessage callback:

Implementing the server side is almost as simple: your handler just prefaces each piece of data it wants to send to the client with the string data: and ends it with a double line-ending (\n\n). Easy peasy. You can see the API in action in this simple demo.

I’ve long been sad that we didn’t manage to get this API into Internet Explorer or the Legacy Edge browser. While many polyfills for the API exist, I was happy that we finally have EventSource in the new Edge.

Yay! \o/

Alas, I wouldn’t be writing this post if I hadn’t learned something new yesterday.

Last week, a customer reached out to complain that the new Edge and Chrome didn’t work well with their webmail application. After they used the webmail site for a some indeterminate amount time, they noticed that its performance slowed to a crawl– switching between messages would take tens of seconds or longer, and the problem reproduced regardless of the speed of the network. The only way to reliably resolve the problem was to either close the tabs they’d opened from the main app (e.g. the individual email messages could be opened in their own tabs) or to restart the browser entirely.

As the networking PM, I was called in to figure out what was going wrong over video conference. I instructed the user to open the F12 Developer Tools and we looked at the network console together. Each time the user clicked on a message, new requests were created and sat in the (pending) state for a long time, meaning that the requests were getting queued and weren’t even going to the network promptly.

But why? Diagnosing this remotely wasn’t going to be trivial, so I had the user generate a Network Export log that I could examine later.

In examining the log using the online viewer, the problem became immediately clear. On the Sockets tab, the webmail’s server showed 19 requests in the Pending state, and 6 Active connections to the server, none of which were idle. The fact that there were six connections strongly suggested that the server was using HTTP/1.1 rather than HTTP/2, and a quick look at the HTTP/2 tab confirmed it. Looking at the Events tab, we see five outstanding URLRequests to a URL that strongly suggests that it’s being used as an EventSource:

Each of these sockets is in the READING_RESPONSE state, and each has returned just ten bytes of body data to each EventSource. The web application is using one EventSource instance of the app, and the user has five tabs open to the app.

And now everything falls into place. Nearly all browsers (including Chromium-derivatives) limit themselves to 6 concurrent connections per server (this will get a bit more complicated in the future). When the server supports HTTP/2, browsers typically need just one connection because HTTP/2 supports multiplexing many (Chromium default 100, server configurable to up to 256) concurrent streams onto a single connection. HTTP/1.1 doesn’t afford that luxury, so every long-lived connection used by a page decrements the available connections by one. So, for this user, all of their network traffic was going down a single HTTP/1.1 connection, and because HTTP/1.1 doesn’t allow multiplexing, it means that every action in the UI was blocked on a very narrow head-of-line-blocking pipe.

Looking in the Chrome bug tracker, we find this core problem (“SSE connections can starve other requests”) resolved “By Design” six years ago.

Now, I’m always skeptical when reading old bugs, because many issues are fixed over time, and it’s often the case that an old resolution is no longer accurate in the modern world. So I built a simple repro script for Meddler. The script returns one of four responses:

  • An HTML page that consumes an EventSource
  • An HTML page containing 15 frames pointed at the previous HTML page
  • An event source endpoint (text/event-stream)
  • A JPEG file (to test whether connection limits apply across both EventSources and other downloads)

And sure enough, when we load the page we see that only six frames are getting events from the EventSource, and the images that are supposed to load at the bottom of the frames never load at all:

Similarly, if we attempt to load the page in another tab, we find that it doesn’t even load, with a status message of “Waiting for available socket…”

The web app owners should definitely enable HTTP/2 on their server, which will make this problem disappear for almost all of their users.

However, even HTTP/2 is not a panacea, because the user might be behind a “break-and-inspect” proxy that downgrades connections to HTTP/1.1, or the browser might conceivably limit parallel requests on HTTP/2 connections for slow networks. As noted in the By Design issue, a server depending on EventSource in multiple tabs might use a BroadcastChannel or a SharedWorker to share a single EventSource connection with all of the tabs of the web application.

Alternatively, swapping an EventSource architecture with one based on WebSocket (even one that exposes itself as a EventSource polyfill) will also likely resolve the problem. That’s because, even if the client or server doesn’t support routing WebSockets over HTTP/2, the WebSockets-Per-Host limit is 255 in Chromium and 200 in Firefox.

Stay responsive out there!

-Eric

AppOrWeb-to-WebApp Communication: Custom Scheme Handlers

I’ve previously written about Web-to-App communication via Application Protocols. App Protocols allow web content to invoke a native application outside of the browser.

WebApp advocates (like me!) want to continue to close the native/browser gaps that prevent web applications from becoming full-fledged replacements for native apps. To that end, I’ve recently spent some time looking at how the web platform allows JavaScript registration of a protocol handler, where the handling “app” is a same-origin web page.

Currently supported by Firefox and Chromium-based browsers (on platforms other than Android), the function navigator.registerProtocolHandler(scheme, url_template, description) enables a website to become a handler for a URL scheme.

Real-World Usage

The canonical use-case for this is web based email clients like Gmail. Gmail would like to be able to be the user’s handler for mailto links. When the user clicks a mailto link, the content of the link should be sent to a handler page on mail.google.com which responds accordingly (e.g. by creating a new email to the specified addressee).

https://mail.google.com/mail/u/0/?extsrc=mailto&url=mailto:u@example.com

The registerProtocolHandler API isn’t limited to the mailto scheme, however. It presently supports a short list of allowed schemes1, and any scheme named web+{one-or-more-lowercaseASCII}.

User Experience

I’ve built a page containing a number of test cases here. When you push the button to register a protocol handler, you receive a permission prompt from Chrome/Edge or Firefox:

ChromePrompt
FirefoxPrompt

To avoid annoying users, if the user declines Chrome’s prompt, the site is blocked from re-requesting permission to handle the protocol. A user must manually visit the settings page to unblock permission.

User-Activation Requirements

If a page attempts to call registerProtocolHandler() on load or before the user has interacted with the page (a so-called “gesture”), then Chromium-based browsers will not pop the permission prompt. Instead, an overlapping-diamonds icon is shown at the right-hand side of the address bar, with the text “This page wants to install a service handler.” Here’s what this looks like on Gmail:

RegisterServiceHandler

Settings

Within Chrome, you can view your currently registered handlers (and sites blocked from asking to become the registered handler) by visiting chrome://settings/content/handlers.

ProtocolSettings

Operating System Integration

One particularly interesting aspect of allowing web-based registration of protocol handlers is that it is desirable for the rest of the operating system outside of the browser to respect those protocol handler associations.

For example, clicking a mailto link in some other application should launch the browser to the web-based handler if registered. However, having the browser change system state in this manner is complicated, especially on modern versions of Windows whereby various protections have been put in place to try to prevent “hijacking” of file type and protocol handler associations.

Edge and Chrome will only attempt to become the systemwide handler for a protocol when running a channel that offers to become the default browser (e.g. Stable). On such a channel, if the browser wasn’t already the handler for the protocol, after the user clicks “Allow” on the Chrome permission bubble, a Windows UAC dialog is shown:

UAC

If the user accepts by clicking “Yes”, the RegisterChromeForProtocol function silently updates the registry:

RegistryWrites

PWAs as Handlers

There is significant interest in allowing Progressive Web Applications to register as handlers for URL schemes.

Other Things I’ve Learned

  • Chrome, Edge, and Firefox disallow registration of protocol handlers in Private/Incognito/InPrivate modes; the call fails silently.
  • With my patch landed, Chrome, Edge, and Firefox disallow registration of protocol handlers from non-secure contexts (e.g. HTTP). Due to the same-origin requirement for the handler URL, this effectively prevents the use of a non-secure page as a handler.
  • An upcoming patch proposes blocking registration from cross-origin subframes.
  • Chromium-based browsers enable IT admins to set default scheme-to-web mappings using Group Policy.
  • Chromium-based browsers do not (as of v93) offer a Group Policy to disallow protocol handler registration; an end-user setting inside about://settings does allow an individual user to turn off the prompts.
  • Chromium versions prior to v81 failed to enforce specified limits on web+ protocol names. Firefox and Chromium v81+ do enforce the limits.
  • Firefox does not support targeting a RPH registered protocol as the target of a form POST request; it silently drops the POST body.
  • Firefox does not implement the unregisterProtocolHandler API. Users must manually unregister protocol handlers using the browser UI.
  • On Windows at least, neither Firefox Stable nor Firefox Nightly seems to try to become the systemwide handler for a scheme.
  • If you have a custom scheme link in a subframe, you probably want to add a target=_blank attribute on it. Otherwise, the web app you’ve configured as the protocol handler might load within that subframe and get blocked due to privacy settings or X-Frame-Options directives. 

-Eric

1 The permitted schemes as of February 2023 are:

kProtocolSafelist[] = { 
  "bitcoin",
  "cabal",
  "dat",
  "did",
  "doi",
  "dweb",
  "ethereum",
  "geo",
  "hyper",
  "im",
  "ipfs",
  "ipns",
  "irc",
  "ircs",
  "magnet",
  "mailto", 
  "matrix", 
  "mms",  
  "news", 
  "nntp", 
  "openpgp4fpr",
  "sip",    
  "sms",   
  "smsto", 
  "ssb",
  "ssh", 
  "tel", 
  "urn",
  "webcal",
  "wtai", 
  "xmpp"
};