The Sad State of HAR

Spring 2017 Update: Some of these issues have been fixed.

The HTTP Archive Format (HAR) was designed to allow tools to exchange network traffic using a standard format; this format is akin to Fiddler’s Session Archive Zip format but is supported natively by browser developer tools. Unlike SAZ files, it is not compressed by default, and often includes redundant text to simplify parsers.

Unfortunately, none of the four major browsers (IE, Edge, Chrome, Firefox) generates HAR correctly.

Internet Explorer 11 and below: Generates the file in XML instead of the proper JSON, due to a misreading of the specification. The export is also limited by numerous bugs in the F12 Network Capture tool, including missing data and misrepresentation of certain response types (e.g. 304s).

Firefox 45: Attempts to store GZIP’d response bodies as text.

Fails to include the encoding=”base64″ attribute when storing binary bodies using base64 encoding.

Embedded image permalink

Microsoft Edge: Fails to include the encoding=base64 token when storing binary bodies using base64 encoding.

Chrome 47: “Save as HAR with content” doesn’t save the content.

Embedded image permalink

Unfortunately, there seems to be little effort to clean these problems up; the IE bug is at least four years old, the Edge bug is at least four months old. I filed bugs on Chrome and Firefox after failing to find any duplicates.

The HAR format specification itself has a number of shortcomings that have yet to be corrected, for instance:

  • No specified way to encode binary request bodies
  • No specified way to encode WebSocket messages

Perhaps the name of HAR spec-author Jan Odvarko’s blog is prescient: Software is hard.

-Eric

Reset Fiddler’s HTTPS certificates

I’ve made changes to the latest versions of Fiddler to improve the performance of certificate creation, and to avoid problems with new certificate validation logic coming to Chrome and Firefox. The biggest of the Fiddler changes is that CertEnroll is now the default certificate generator on Windows 7 and later.

Unfortunately, this change can cause problems for users who have previously trusted the Fiddler root certificate; the browser may show an error message like NET::ERR_CERT_AUTHORITY_INVALID or The certificate was not issued by a trusted certificate authority.

Please perform the following steps to recreate the Fiddler root certificate:

Fiddler 4.6.1.5+

  1. Click Tools > Fiddler Options.
  2. Click the HTTPS tab.
  3. Ensure that the text says Certificates generated by CertEnroll engine.
  4. Click Actions > Reset Certificates. This may take a minute.
  5. Accept all prompts

Fiddler 4.6.1.4 and earlier

  1. Click Tools > Fiddler Options.
  2. Click the HTTPS tab
  3. Uncheck the Decrypt HTTPS traffic checkbox
  4. Click the Remove Interception Certificates button. This may take a minute.
  5. Accept all of the prompts that appear (e.g. Do you want to delete these certificates, etc)
  6. (Optional) Click the Fiddler.DefaultCertificateProvider link and verify that the dropdown is set to CertEnroll
  7. Exit and restart Fiddler
  8. Click Tools > Fiddler Options.
  9. Click the HTTPS tab
  10. Re-check the Decrypt HTTPS traffic checkbox
  11. Accept all of the prompts that appear (e.g. Do you want to trust this root certificate)

image

If you are using Fiddler to capture secure traffic from a mobile device or Firefox, you will need to remove the old Fiddler root certificate from that device (or Firefox) and install the newly-generated Fiddler certificate.

I apologize for the inconvenience, but I believe that the new certificate generator will help ensure smooth debugging with current and future clients.

-Eric Lawrence

DotNet Makes Me Sad, In Pictures

.NET Framework KB 3088956:

image

Ouch, that sounds pretty severe.

I guess I’d better go manually install a hotfix?

image

Seriously? An email address and a CAPTCHA? Fine.

image

Oh, an email delivered HTTP URL pointed at an executable file? That seems totes legit.

image

Yup, definitely legit, it says “Microsoft” right there at the top!

image

Sure, let’s put those files in the root of the C:\ drive. I’m sure that’ll work.

image

Oh, guess it did. Magic!

Oh, wait. No files in C:\. No magic, I guess.

Let’s use Process Monitor to watch the file writes… that’s what all the cool kids have to do to install patches, right?

image

Ah, there you are! UAC virtualization like it’s 2006!

image

Of course I must, Microsoft, of course I must.

-Eric

The Budget

“Don’t tell me what you value, show me your budget, and I’ll tell you what you value.” – Joe Biden

Across the political spectrum, Americans have thoughts on how the government should spend the money we send its way. That’s great (and it’d be better if more of us voted), but many arguments about spending are based on very misguided notions of how we’re currently spending our money. So, today, some charts.

Source: OMB.

image

See that blue piece? That’s what has a lot of folks worried. That’s the interest on the $18.4T national debt, which grew by another $426 billion dollars this year. So how much is $426 billion? Just over a third of what the US government got to spend on discretionary spending this year:

image

That military chunk is huge (54%), but it’s worth keeping in mind that this money isn’t just disappearing; we’re spending it on American-built weapons, salaries for Americans, etc. It’s fair to question whether we should be bringing those resources to bear on other challenges.

So, where does the money go when you include everything?

image

Social Security and Medicare are a huge chunk of where our money goes, but a large percentage of Americans seem very confused about how these programs work; politicians on both sides of the aisle are guilty of deliberately confusing the public about them.

If you ask Americans how much we spend on Foreign Aid, the average guess is 28%; in actuality, it’s about 1%.

Interestingly the government also publishes how much money it forgoes in the form of explicit tax breaks:

image

To whom do we owe that money?

Today, we owe most of the money to ourselves.

 

Other Stuff

  • Not all debt is bad—if you’re investing what you’re borrowing and getting a greater return on that investment, you’re doing it right.
  • Comparisons of national income and spending to household income and spending are often trite and very misleading. But the counterarguments against such comparisons tend to obscure their own weaknesses too.
  • XKCD did a comic on money.
  • The White House Calligrapher’s Office spends $277k per year on salaries for its three employees.

What’s New in Fiddler 4.6.0.7

TLDR? – Get the newest Fiddler here. We’re performing a staged rollout of this build; it won’t be on autoupdate until next week.

Under the Hood

As mentioned in our notes about the Fiddler 4.6 release, we’ve started taking a very close look at Fiddler’s performance. Fiddler’s use of the CPU, system memory, and the network have gone under the microscope and this new release includes several major changes to how Fiddler uses threads and memory. If you frequently run Fiddler with a large amount of traffic in parallel, or run Fiddler on a slower or heavily-loaded PC, this new version should provide significantly improved performance. We’ve also improved overall performance by using better algorithms in scenarios like the Find Sessions (CTRL+F) experience.

The !threads and !memory QuickExec commands have been enhanced to provide insights into fine-grained performance details about Fiddler’s operation.

 

The Performance Tab

The new Performance tab in the Fiddler Options dialog offers choices that can significantly change Fiddler’s runtime performance and memory usage.

 image

The Show Memory panel in status bar controls whether Fiddler’s status bar shows a panel that indicates the current memory usage tracked by the garbage collector. For example, when Fiddler has 64mb of managed memory allocated, the panel looks like this:

image

Left-click the memory panel to instruct the .NET Framework to perform an immediate garbage collection. Right-click the panel to launch the Fiddler Options dialog box with the Performance tab activated.

The Parse WebSocket Messages checkbox controls whether Fiddler will parse WebSocket streams into individual messages, allowing display in the WebSocket tab and manipulation using the OnWebSocketMessage event handler. Disabling WebSocket Message parsing will reduce CPU usage and may save a significant amount of memory if high-traffic WebSockets are in use. Even if you disable WebSocketMessage parsing globally using this checkbox, it can be reenabled on a per-Session basis by setting the x-Parse-WebSocketMessages flag on the Session object.

The Stream and forget bodies over box controls the maximum size of a message body that Fiddler will retain. By default, the limit is just under 2 gigabytes for 64bit Fiddler and 16 megabytes for 32bit Fiddler; the much smaller default for 32bit helps avoid problems with “Out of Memory” errors when running Fiddler in a small address space. If, while reading a message body, Fiddler finds that it is larger than the threshold, it will configure the body to stream and will “drop” the bytes of the body to conserve memory. If you attempt to inspect a Session which has been dropped, you will see the following notification bar:

image

Clicking the bar will open the Fiddler Options dialog to allow you to reconfigure the limit for subsequent Sessions.

The If client aborts while streaming dropdown controls Fiddler’s behavior if a response body is streaming to the client but the client closes the connection. Depending on your choice here, Fiddler can continue to read the body from the server (useful if you’re collecting traffic) or abort the Session (useful to save memory and CPU cycles).

The Run Fiddler at AboveNormal Priority alters Fiddler’s default scheduling priority. If you enable this option, Windows will prioritize activation of Fiddler’s threads when they have work to do (e.g. reading a new request or response from the network). You can easily experiment with this option to see whether it improves the overall throughput of your client (browser) and Fiddler.

 

New QuickFilters

The Session list’s Filter Now context menu has been enhanced with two new filters:

image

The Hide /1stpath/ filter hides any traffic whose Url path component starts with the specified string.

The Hide Url… option uses the current Session’s Url as the default of a Url filter; you can edit the string to apply more broadly by removing text from the start or end of the string:

image

 

High DPI Improvements

Today, only a small number of Fiddler users (< 4%) run Fiddler on Windows systems with a non-default screen DPI, but we want Fiddler to work great for those users too. The latest build of Fiddler includes a number of DPI-related fixes. Fiddler is not yet marked DPI aware in its manifest; if you’d like to see Fiddler in its DPI-aware mode, run Fiddler with the -dpiAware command line argument:

    fiddler.exe -dpiAware

We will continue to make improvements as problems are discovered or reported and expect to eventually set the dpiAware flag by default.

 

New HTTPS Cipher Option

Fiddler 4 on Windows 7 and later supports modern TLS versions (TLS 1.1 and TLS 1.2) and the HTTPS tab on the Fiddler Options dialog enables you to easily enable these protocols. However, TLS 1.1 and 1.2 remain off-by-default for compatibility reasons.

The Enabled Protocols link on the HTTPS dialog now supports a new token <client>; if present, this token adds to the list of versions offered to the server the latest protocol that has been offered by the client. For instance, with these settings:

image

… a request from Internet Explorer offering TLS 1.2 will be presented to the server with TLS 1.2 and TLS 1.0 enabled. The advantage of using the <client> token is that if the request fails, many browser clients are configured to “fall back” and attempt negotiation with an earlier protocol version. In this example, if the TLS 1.2 connection fails, the browser will retry with TLS 1.0 and the connection may succeed.

You must include at least one specific TLS version in the HTTPS Protocols list to handle cases where the request was generated by Fiddler itself (e.g. the Composer tab).

 

Brotli Compression Support

Researchers at Google have developed a new compression algorithm named Brotli that offers significantly better compression than the DEFLATE algorithm used by Gzip. This new compression algorithm is already in use by many browsers today (inside the WOFF2 font format) and it is expected to appear as a HTTP Content-Encoding in early 2016.

Fiddler now supports Brotli as a Content-Encoding everywhere compression is supported; simply download the Authenticode-signed Windows Brotli.exe and place it in the Fiddler2\Tools\ subfolder in your Program Files folder. After you restart Fiddler, you will find a new Brotli option on the Transformer tab and Fiddler APIs like utilDecodeResponse() will be able to decompress Brotli-encoded content:

image

The new version of Fiddler also has better handling of unsupported compression schemes like SDCH—if a response with Content-Encoding: sdch,gzip is encountered, for instance, the various decoding APIs will decompress with GZIP and then stop without removing the SDCH token.

 

Hidden Tabs

FiddlerScript’s BindUITab attribute now supports a <hidden> token:

image

If this token is present, the tab is not shown until the user manually activates it via the View > Tabs menu:

image

This feature helps you “unclutter” Fiddler by keeping uncommonly-used script tabs hidden.

 

More Powerful ImageView Extensions

Fiddler’s ImageView Extensions feature allows you to add new commands to the Tools context menu on the ImageView Inspector:

image

Now you can use a new Options parameter to specify that Fiddler should show the <stdout> or <stderr> results of running the target tool, and a new {out:extension} token enables you to specify that the target tool writes a file that Fiddler should load as a new Session in the Web Sessions list.

For instance, here’s the logic to add a new ToWebP Lossless command to the list:

image

To use it, add the registry entries and place CWebP.exe in Fiddler’s Tools subfolder. When you invoke the command, Fiddler will run the tool, passing an input temporary file containing the JPEG image to the tool in the {in} parameter and specifying an autogenerated filename with a .webp file extension in the {out:webp} parameter. The cwebp.exe tool will be run, any text from Standard Error will be collected and displayed to the user, and the file named by the {out} token will be reloaded as a new Web Session:

image 

 

Updated – Show Image Bloat

I’ve also updated the Show Image Bloat add-on (described here) with some additional tweaks and features; the add-on has improved bloat detection for JPEG and GIF files and has other minor improvements. Install the latest build of Show Image Bloat (v2.6) and activate it from the Fiddler Rules menu.

image

 

I hope you enjoy these new improvements to Fiddler – Keep sending in your feedback to ensure we’re evolving the tool to best meet your needs.

 

-Eric Lawrence