Fiddler Certificate Generators

Fiddler and FiddlerCore offer three different choices for generating interception certificates:

  • MakeCert
  • CertEnroll
  • Bouncy Castle

If you’re so inclined, you can even write your own certificate generator (say, by wrapping OpenSSL) and expose it to Fiddler using the ICertificateProvider3 interface.

On Windows, Fiddler includes the MakeCert and CertEnroll certificate generators by default; you can download the Bouncy Castle Certificate Generator if you like. In contrast, when Fiddler is running on Linux and Mac, only the Bouncy Castle certificate generator is available, and it is included by default.

If you’re using Windows, however, you may wonder which Certificate Generator you should use in Fiddler or for your applications based on FiddlerCore.

In general, I recommend the Bouncy Castle generator, as it has better performance than the default MakeCert generator and it offers more configuration choices than the CertEnroll generator. Another advantage of the Bouncy Castle certificate generator is that the only certificate that (typically) goes in the Windows Certificate store is the root certificate. The server (end-entity) certificates generated for each website are kept in memory and discarded when Fiddler exits; because the Bouncy Castle generator reuses a single private key for all certificates by default, the performance impact of this behavior is minimal.

The only downside to the Bouncy Castle generator is its size: it is ~200KB when compressed, which is 25% larger than FiddlerCore itself.

The CertEnroll generator was added to Fiddler relatively recently; it offers better performance and standards-compliance than the legacy MakeCert generator but it is available only on Windows 7 and later. You can easily switch Fiddler to use CertEnroll inside Tools > Fiddler Options > HTTPS.

The MakeCert generator is the original certificate generator used by Fiddler and it remains the default on Windows today (mostly) for legacy compatibility reasons. It suffers from a number of shortcomings, including the fact that the certificates it generates are not compatible with iOS and (some) Android devices. It generates certificates with a 1024 bit RSA key (which may soon trigger warnings in some browsers) and each certificate has a unique key (meaning that each new secure site you visit triggers the somewhat costly key generation code).

Both the CertEnroll and MakeCert-based certificate generators must store all server certificates in the Windows Certificate store which some users may find confusing:

image

The storage of (potentially thousands of) server certificates in the user profile can also cause some problems for corporate users who have roaming user profiles, as these certificates are roamed to each workstation as the user logs in. To mitigate that, the Clear server certs on exit checkbox can be set inside the Tools > Fiddler Options > HTTPS > Certificate Provider dialog, or via:

    FiddlerApplication.Prefs.SetBoolPref("fiddler.certmaker.CleanupServerCertsOnExit", true);

… however, the downside of doing that is that Fiddler must then re-create the server certificates every time it starts. This performance penalty is smaller when using CertEnroll, which reuses a single 2048-bit RSA key, than for MakeCert, which generates unique 1024-bit RSA keys for each site.

FiddlerCore Considerations

To determine which Certificate Generator is in use, be sure to attach the following event handlers:

Fiddler.FiddlerApplication.OnNotification +=
  delegate(object sender, NotificationEventArgs oNEA) { Console.WriteLine(“** NotifyUser: ” + oNEA.NotifyString); };
Fiddler.FiddlerApplication.Log.OnLogString +=
  delegate(object sender, LogEventArgs oLEA) { Console.WriteLine(“** LogString: ” + oLEA.LogString); };

You can then view information about the Certificate Generator in the console when it loads.

Developers building applications atop FiddlerCore should keep the following in mind when deciding which Certificate Generator to use:

MakeCert

  • MakeCert.exe is a Microsoft Visual Studio 2008 redistributable file, meaning that you’re licensed to redistribute it if you have an appropriate license to that version of Visual Studio. Microsoft may offer MakeCert.exe as a redistributable in other circumstances, but licensing is provided by Microsoft, not Telerik.
  • To use MakeCert.exe, you must include it adjacent to your application’s .exe file.
  • MakeCert-generated certificates are not compatible with iOS and some Android devices.
  • MakeCert-generated certificates “pollute” the user’s Certificate Store and you should consider offering a mechanism to clear them.

CertEnroll

  • The CertEnroll API is available on Windows 7 and later.
  • Use CertEnroll by either omitting makecert.exe from the application’s folder or by explicitly setting the preference:
  •     FiddlerApplication.Prefs.SetBoolPref("fiddler.certmaker.PreferCertEnroll", true);

  • CertEnroll-generated certificates “pollute” the user’s Certificate Store and you should consider offering a mechanism to clear them.

Bouncy Castle

  • Bouncy Castle is an open-source PKI and crypto library distributed under the MIT license.

  • To use Bouncy Castle, you must include CertMaker.dll and BCMakeCert.dll adjacent to your application’s .exe file.
  • Bouncy Castle does not store certificates in the Windows Certificate Store (yay!) but this also means that your application needs to keep track of its root certificate and private key (unless you recreate and retrust it every time the application runs).

    Two preferences are used to hold the key and certificate, fiddler.certmaker.bc.key and fiddler.certmaker.bc.cert. After you first call createRootCert, you should retrieve these preferences using FiddlerApplication.Prefs.GetStringPref and store them somewhere within your application’s settings (registry, XML, etc); the private key should be considered sensitive data and protected as such.  When your application next runs, it should detect whether the key and certificate have already been created, and if so, they should be provided to the certificate generator using FiddlerApplication.Prefs.SetStringPref before any certificates are requested, lest you inadvertently create a new root certificate.

    Rick Strahl wrote a great blog post on this process, including some sample code.

 

-Eric

What’s New in Fiddler 4.6

Telerik Fiddler version 4.6 (and v2.6 targeting .NET2) is now available for download. The new version includes several new features and dozens of tweaks and bugfixes, described in this article.

View > Tabs Menu

The new View > Tabs menu offers a list of tabs that are hidden by default.

image

The Preferences command displays a tab that allows you to edit Fiddler’s Name/Value preferences.

The new APITest command displays the new Fiddler APITest tab that enables easy testing of web APIs.

The AutoSave command (formerly located in the Tools menu) permits you to automatically save Session Archive Zip files on a regular schedule.

Note: In the future, this submenu will become an extension point to allow developers to easily expose optional UI tabs.

Lint Filters

Fiddler’s Lint feature enables you to control how Fiddler reports violations of the HTTP protocol and other errors. Controlled on the Tools > Fiddler Options > General tab, you can leave the If protocol violations are observed setting at the basic “Warn on Critical errors” section or adjust it to “Warn on all errors”; the latter setting performs more tests on traffic to find mistakes that could cause clients or servers to misbehave. However, this setting can get pretty noisy. To allow you to suppress specific warnings, you can now use the Filter link on the Protocol Violation Report dialog.

image

Each Lint warning now has a unique code consisting of a letter and three digits. The letter prefix indicates the severity:

  • L – Common “problem”, low impact, unlikely to break anything.
  • M – Significant problem, likely to break functionality in one client.
  • H – Important problem likely to significantly break functionality in multiple clients

You can exclude an entire class of warnings by simply including the prefix in the list of exclusions.

Single Session Timeline

The Timeline tab in Fiddler allows you to visualize the download of multiple Sessions at once to permit you to view parallelism and connection reuse. In Fiddler 4.6, the view of a single Session has been enhanced to permit you to visualize how the content of the Session was downloaded. You can determine, for instance, whether the headers were flushed immediately or the client was forced to wait for headers until the body was ready, and you can determine whether download speed improved or regressed as the download proceeded. For instance, in this chart, the server took 2.5 seconds to return the headers and began to slowly stream the body in small chunks. As the download progressed, the speed improved until completion at just over 9.2 seconds:

image

 

Interface Tweaks

A new Copy as Image command on each Inspector’s tab context menu allows you to copy an image of the tab’s contents to your clipboard for easy pasting in an email or blog post. The Headers, XML, and JSON Inspectors allow you to easily highlight nodes of interest—simply press the spacebar on a node to temporarily render it with a yellow highlight:

image

The new Math context menu appears when you right-click on a numeric column in the Web Sessions list; the menu currently offers a single command: Sum and Average, which shows these attributes of the selected Sessions:

image

The Composer tab now offers a splitter between the headers and body boxes so you can adjust their height as desired:

image

Pressing the / key while in the Web Sessions list now enters QuickSearch mode in the QuickExec box, selecting any Sessions whose URL contains the text you type.

Hash Support

Fiddler now offers enhanced support for computing hashes of blocks of bytes or strings.

The Tools > TextWizard feature now offers a quick way to get a hash (MD5 to SHA512) of a block text, in either base64 or dashed hexadecimal format:

image

New methods are also available for FiddlerScript to compute hashes. For instance, you can copy the following block to just inside your Rules > Customize Rules > Handlers class:

 
public BindUITab("Resource Integrity Hashes", "<nowrap><nolink>")
static function ShowSRI(arrSess: Session[]):String
{
var oSB: System.Text.StringBuilder = new System.Text.StringBuilder();
for (var i:int = 0; i<arrSess.Length; i++)
{
if (arrSess[i].HTTPMethodIs("CONNECT")) continue;
        if (!arrSess[i].bHasResponse)
{
oSB.AppendFormat("\r\n// Skipping incomplete response '{0}'\r\n",
arrSess[i].fullUrl);
continue;
}
if (arrSess[i].responseCode != 200)
{
oSB.AppendFormat("\r\n// Skipping non-HTTP/200 response '{0}'\r\n",
arrSess[i].fullUrl);
continue;
}
var sType: String = arrSess[i].oResponse.MIMEType.ToLower();
var bIsScript = sType.Contains("script");
var bIsCSS = sType.Contains("css");
if (!bIsScript && !bIsCSS)
{
oSB.AppendFormat("\r\n// Skipping non-CSS/JS response '{0}'\r\n", arrSess[i].fullUrl);
continue;
}
var sIntegrity = "sha256-" + arrSess[i].GetResponseBodyHashAsBase64("sha256").Replace("-", "")
+"\n\tsha384-" + arrSess[i].GetResponseBodyHashAsBase64("sha384").Replace("-", "")
+"\n\tsha512-" + arrSess[i].GetResponseBodyHashAsBase64("sha512").Replace("-", "");
        if (bIsScript)
{
oSB.AppendFormat('\r\n<script src="{0}"\r\n\tintegrity="{1}"></script>\r\n',
arrSess[i].fullUrl, sIntegrity);
}
else
{
oSB.AppendFormat('\r\n<link rel="stylesheet"\r\n\thref="{0}"\r\n\tintegrity="{1}">\r\n',
arrSess[i].fullUrl, sIntegrity);
}
}
return oSB.ToString();
}

When you save the script, Fiddler adds a new tab to display the Subresource Integrity attributes for the selected response bodies:

image

 

FiddlerScript Improvements

BindUITab Enhancements

As seen in the Resource Integrity Hashes example above, the BindUITab attribute allows you to create new tabs inside Fiddler that are populated based on the selected Sessions. BindUITab now offers a second parameter that allows you to specify one or more of the following options:

  • <nowrap> – The RichEdit control should not wordwrap lines
  • <nolink> – The RichEdit control should not detect or underline urls
  • <html> – Instead of using a RichEdit control, the function’s response will be rendered as HTML inside a Web Browser Control

BeforeFiddlerShutdown event

Fiddler now exposes a  BeforeFiddlerShutdown event that enables extensions or FiddlerScript to block shutdown of Fiddler; this may be useful if you wish to prompt the user for permission to lose unsaved work, etc.

A Fiddler extension should attach an event handler:

    FiddlerApplication.BeforeFiddlerShutdown += (o, c) =>
{
c.Cancel = (DialogResult.Cancel ==
MessageBox.Show(“Allow Fiddler to close????”,
“Go Bye-bye?”, MessageBoxButtons.OKCancel));
};

Within FiddlerScript, you can add a method to the existing Handlers class:

    static function OnBeforeShutdown(): Boolean {
return ((0 == FiddlerApplication.UI.lvSessions.TotalItemCount()) ||
(DialogResult.Yes == MessageBox.Show(“Allow Fiddler to exit?”,
“Go Bye-bye?”, MessageBoxButtons.YesNo, MessageBoxIcon.Question,
MessageBoxDefaultButton.Button2)));
}

AutoResponder Improvements

The AutoResponder now supports the NOT: operator inside the METHOD:, HEADER:, and FLAG:, operators. For instance, if you’d like the AutoResponder only to impact requests from Google Chrome, add this rule to the top of your list:

image

Microsoft Edge Support

Fiddler has been updated to recognize Microsoft’s new Edge browser as a Web Browser, so features like the Process Filter in the Status Bar:

image

…and in the Browse command in the Fiddler toolbar:

image

work as expected.

Note: You should use the WinConfig button at the left of Fiddler’s toolbar to enable Windows 10 “Store” applications to run Fiddler. By default, you shouldn’t need to use the WinConfig button for Edge, because Edge’s about:flags enables access to loopback by default.

 

ImageView Enhancements

FavIcon Preview

When a site uses an .ICO file as its favicon, the icon may contain multiple different images that are used depending on the user’s device and the context in which the icon is rendered. Fiddler’s ImageView Inspector now renders all of the images contained within the .ICO file like so:

ImageView showing icon

 

JPEG Thumbnails

Websites should generally strip embedded thumbnails from JPEG files. Embedded thumbnails are a common source of bloat, wasteful bytes that aren’t used by the client. However, some sites fail to optimize their images by removing thumbnails. In some cases, an image thumbnail may even contain data which was never meant to be made public, if, for instance, the larger image was cropped without regeneration of the thumbnail. The ImageView now allows you to extract the thumbnail image as a new Session that is added to the Web Sessions list:

Extract Thumbnail

 

AutoSave Enhancements

Fiddler’s AutoSave feature (now found under View > Tabs > AutoSave) now supports several preferences to control its behavior.

Set fiddler.extensions.AutoSave.AlwaysOn to true to have Fiddler automatically activate AutoSave mode when it starts. Set fiddler.extensions.AutoSave.Minutes to the number of minutes to collect traffic between each save operation; the default is 5. Set fiddler.extensions.AutoSave.HeadersOnly to true if you’d like the SAZ file to contain only the request and response headers, omitting the bodies. Set fiddler.extensions.AutoSave.Path to the folder path under which auto-saved SAZ files should be stored.

Additional Upgrade Notifications

By default, Fiddler’s automatic update notifications will only show if a significant change in version number occurs. For instance, say you’re running version 4.6.0.2 and version 4.6.0.3 becomes available. By default, Fiddler will only tell you about this minor update if you manually check for new versions by clicking Help > Check for Updates. When Fiddler 4.6.1.0 becomes available, the updater detects this larger change in version number and prompts you to upgrade.

If you would prefer Fiddler to notify you of every update automatically, use the black QuickExec box below Fiddler’s Web Sessions list to enter the following command:

    prefs set fiddler.updater.BleedingEdge true

With this preference set, you’ll see more frequent notice of upgrades. On one hand, that’s great—you’ll get the latest Fiddler improvements ahead of most other people. On the other hand, as a bleeding edge user, you’re also more likely to uncover any bugs we inadvertently introduce in new versions.

Fiddler Improvement Program

You may now opt-in to sending telemetry information about your PC environment and Fiddler usage. Within your first few boots of Fiddler, you’ll see the following dialog:

image

Telemetry data is reported over HTTPS and its usage is governed by Telerik’s Privacy Policy. If you later change your mind, you can control your participation using the checkbox Participate in the Fiddler Improvement Program on the Tools > Fiddler Options > General tab. Note: If an administrator has set the BlockUpdateCheck registry key in the HKLM registry hive, users cannot opt-in to the Fiddler Improvement Program.

The Telerik Analytics integration into Fiddler has already yielded several bugfixes and has helped us prioritize our investments in performance and feature improvements. We’ll write more about what we’ve learned from Fiddler Telemetry in a future blog post.

Performance Improvements

One early finding from our Fiddler Telemetry is that a surprising number of users are running on 32bit Windows, where the address space limitations mean that they often run into “out of memory” errors.  To help mitigate this, we’ve made some under-the-hood changes to how Fiddler allocates memory, and this is only the beginning of a larger project to improve Fiddler’s overall performance for everyone.

As a part of our performance investigations, two new commands were added to the QuickExec box, !memory and !threads. Invoke these commands to add information to the Log tab for troubleshooting purposes. If you find Fiddler is running more slowly than expected, sending the output of these commands to us will help narrow down the problem.

 

We hope you enjoy the new version of Fiddler!

-Eric Lawrence

API Testing with Telerik Fiddler

Fiddler has long been the tool of choice for developers and testers who are building and verifying APIs exposed over HTTP(S). In this post, we’ll explore the existing features Fiddler offers for API Testing and announce new capabilities we’ve released in Fiddler 2.6/4.6.

Composing API Calls

The Composer tab enables the authoring of arbitrary HTTP(S) requests using any HTTP method, url, headers, and body, and the many Inspectors permit examination of responses of any type, including XML, JSON, BSON, and myriad other formats. Over the last few years, the power of the Composer has grown, and it now offers many conveniences, including a Request History pane:

image

… to allow simple reuse of existing requests. The powerful Scratchpad tab allows you to easily display a list of requests and execute those requests as needed—a great feature for live demonstrations of your API for tech talks and the like:

image

As you can see in the screenshot above, the Scratchpad even supports executing requests expressed curl syntax, so you can easily exercise APIs that are documented in terms of the parameters passed to that popular tool.

 

Capturing API Calls

Of course, Fiddler is best known as a proxy debugger, and its ability to capture traffic from nearly any source means that the fastest way to generate API tests is to simply capture the API traffic from an existing client and modify and augment those tests as desired. You can start with a baseline of the calls your clients presently send, and add new tests to probe for performance, security, error handling, and other problems.

One of the most powerful capabilities Fiddler offers is capture of traffic from almost any device (iOS, Android, Windows, Mac, etc); you can easily exercise your mobile clients’ API endpoints without adding any new software to the device itself!

 

Resending Requests from the Web Sessions List

After you’ve collected a set of requests in Fiddler, you can easily save them to a Session Archive Zip (.saz) file for convenient reuse at any time. To resend the traffic, just load the SAZ file, select the desired requests, and use the Replay submenu on the Web Sessions list’s context menu to resend the requests. Common choices include:

  • Reissue Requests – Resend the selected requests as quickly as possible
  • Reissue Sequentially – Resend the selected requests, waiting for a response to each before sending the next
  • Reissue and Verify – Resend the selected requests, verifying each response’s status code and body match the original response

Of these options, the last is perhaps the most interesting—you can very easily regression test an API set by simply collecting “good” traffic in a SAZ file and then using that traffic as a baseline against which later verifications will be conducted. The verification performed is rather crude, which is why we’re excited to announce…

Automated API Testing

The new APITest extension to Fiddler greatly enhances its power to validate the behavior of web APIs. Click View > Tabs > APITest to show the APITest tab. The bulk of the tab displays a Test List, a list of API requests and Validators.

image

Tests can be added to the list by simply dragging and dropping Sessions from Fiddler’s Web Sessions list. You can then use the context menu to specify a Validator and optionally mark it with a Note (comment) and optionally assign it to a group.

Validators

Validators offer a lightweight way to judge the success or failure of a test, and support many of the same operators used by Fiddler’s AutoResponder feature. As seen in the screenshot above:

  • EXACT:This is a Test – Ensure the status code matches and response body contains (case-sensitively) the text This is a Test 
  • Updated via API – Ensure the status code matches and response body contains (case-insensitively) the text Updated via API
  • {CODE} – Only ensure the response status code matches the original status, e.g. 201 for the first test
  • {BODY} (Not pictured above) Ensure the response status code matches and the response body is identical 

In addition to the EXACT: operator prefix, the REGEX: and NOT: prefixes are also supported for text searches:

image

The HEADER: prefix allows you to ensure that the response bears a specified header containing a given value (e.g. HEADER:Content-Encoding=gzip ensures that the response has a Content-Encoding header showing that the response used GZIP compression).

The SCRIPT: prefix allows you to specify a FiddlerScript function in your CustomRules.js file to use to evaluate the response; the named function is passed both the test Session and the baseline (original) Session and can evaluate any criteria you like to return whether or not the test has passed. Beyond examining the headers and body of the response, you could, for instance, evaluate the values in the Session’s Timers object and fail the test if the response took more than 100 milliseconds.

  // This method is called by "SCRIPT:CheckTest" validators; it
  // returns TRUE if the Test passes, and false otherwise.
  static function CheckTest(oRun: Session, oBase: Session): boolean
  {
    // Simplest possible validator
    if (oBase.responseCode != oRun.responseCode)
    {
        oBase["api-lastfailreason"] = "Mismatched status code...";
        return false;
    }

    var timeTotal = 0;
    timeTotal = (oRun.Timers.ServerDoneResponse - oRun.Timers.ClientDoneRequest).TotalMilliseconds;
    if (timeTotal > 100)
    {
        oBase["api-lastfailreason"] = "Performance Too slow...";
        return false;
    }


    return true;   
  }

Script Events

Before your Test List is run, the BeforeTestList method (if present in your FiddlerScript) is run, permitting you to adjust the requests as needed:

  static function BeforeTestList(arrSess: Session[]): boolean
  {
    // In this method, you can do any setup you need for the test,
    // e.g. adding an Authorization: token OAUTH value that you
    // obtained programmatically...
    var sOAUTHToken = obtainToken();
    if (String.IsNullOrEmpty(sOAUTHToken)) return false;

    for (var i: int=0; i<arrSess.Length; i++)
    {
      arrSess[i].oRequest["Authorization"] =  sOAUTHToken;
    }
   
    MessageBox.Show("Token Set. Running " + arrSess.Length.ToString() +
                    " tests.", "BeforeTestList");

    return true; // Test should proceed; return false to cancel
  }

After all of the individual test cases are executed, the AfterTestList method allows you to validate any post-conditions, log the results of the Test list, or perform other operations. The method is passed a TestListResults object which contains an enumerable list of TestResult objects. Each result contains the baseline (original) Session, the test Session, and an IsPassing boolean indicating whether the test passed. The WriteFailuresToSAZ convenience method will write all failing TestResults to a SAZ file for later analysis.

  static function AfterTestList(listResults: TestListResults)
  {
    for (var oResult in listResults)
    {
      FiddlerApplication.Log.LogString(oResult);
    }

    listResults.WriteFailuresToSAZ("C:\\temp\\lastfailure.saz");
  }

 

Working with Test Lists

A Test List is simply a set of Sessions each of which contains several custom Session Flags (api-testitem, api-Validator, api-LastResult, and api-LastFailReason). As a consequence, these lists can be saved and loaded as regular SAZ files; their “Test List” functionality only lights up when loaded into the APITest tab using its context menu. The menu commands are largely self-explanatory:

image

  • Run Selected Tests runs only those tests that are currently selected in the UI
  • Rerun Failed Tests runs all tests that are marked as failing
  • Set Comment… sets the Notes column for the selected tests
  • Set Validator… assigns the validation criteria for the selected tests
  • Set Group… assigns the tests to a UI group, useful for organizing your test list
  • Inspect baseline… opens the original Session in Inspectors for viewing or editing
  • Promote moves the selected test earlier in the run order
  • Demote moves the selected test later in the run order
  • Clone creates a copy of the current test, useful when you need to use multiple validators
  • Save Test List saves the test list to a SAZ file
  • Load Test List adds the tests from a SAZ file to the current test list

You can temporarily disable one or more tests from running by pressing the spacebar with the desired tests selected.

FiddlerCore

While the new APITest extension is offers powerful functionality, many larger enterprises instead choose to use the FiddlerCore .NET Class Library to build their API Testing infrastructure. FiddlerCore allows you to fully customize the behavior of your testing logic, driving the test using the .NET Language of your choice (typically C#). Because FiddlerCore does not utilize any of Fiddler’s UI, it can easily be integrated into existing test automation suites and can be used in console and service applications.

Load Testing APIs

The Fiddler-native Session Archive Zip file format is supported by Telerik Test Studio’s Load Testing product, allowing you to simply import a SAZ file that exercises your API set, configure the number of virtual users and run time, and generate load instantly or on the schedule you set. If you’d like to apply validation logic while your server is under load, run the Load Test in Test Studio and in parallel execute your Test List in Fiddler.

 

We hope you find Fiddler’s API Testing support useful, and we look forward to your feedback as we continue to enhance the tool.

 

Eric Lawrence
Telerik

Attribution Error

In life, you sometimes encounter people with “high standards”—folks who often find others’ behavior lacking in some way. Such people usually explain: “Sure, I have high standards… but I hold myself to an even higher standard!”

Except… they rarely do.

The problem is that, as humans, we’re subject to both fundamental attribution error and actor-observer bias. These phenomenon mean that we attribute our own good behaviors as intrinsic demonstrations of our good character, and we explain away bad behaviors as a temporary consequence of a given situation. When observing others, however, we tend to do the opposite—good behavior is dismissed as situational and bad behavior is deemed evidence of poor character.

These errors are made every day, at the small scale of our personal relationships to the global stage of policy-making. They’re hard to avoid, especially if you’re not consciously aware of them and don’t actively strive to consider events from different perspective.

Tuning MemoryStream

By day, I build the Fiddler Web Debugger. I’ve recently started integrating telemetry collection into the application for automated exception reporting and to collect information about the user’s environment to ensure that Fiddler testing environments match real-world usage.

The data is fascinating (and I’ll probably blog more about it later), but one data point in particular jumped out at me:

23.09% of users on 32bit

Early data suggests that nearly a quarter of Fiddler users are using the tool under 32bit Windows. This number is significantly higher than I expected and it’s worrisome. Why?

Because well over half of the exceptions reported via telemetry are System.OutOfMemoryException, and 100% of those are from users running 32bit. Now, given the exception’s name, most users assume this means that the system has “run out of RAM”, but this isn’t really the case. In virtually all instances, this exception would more correctly be named System.InsufficientContiguousAddressSpace – the user has plenty of memory available (both RAM and virtual memory backed by the swap file), but that memory is fragmented such that a large allocation (of, say, 100megabytes) cannot be fit within a single address range of the 2 billion addresses available:

Fragmented Memory

If that 100mb allocation were split into two smaller chunks of 50 megabytes each, it could easily fit into the available address space, and you’d never hit the OutOfMemory exception.

When Fiddler is running in a 64bit process, this error is almost never seen because the address space is so overwhelmingly huge (8 terabytes, 128tb on Win8.1+) that fragmentation isn’t a problem like it is in the 2GB address space.

So, what can we do?

The “simple” approach is to just throw the /3GB switch so that the system assigns up to 3GB of address space to the application, but this requires that the user manually adjust their overall operating system configuration, and it doesn’t buy us very much additional space.

The better approach is to stop using contiguous memory and start using data structures that keep portions of the data in reasonably-sized chunks of only a few megabytes each. This is arguably the “right” fix, but it can be considerably more complex, and it would be a major breaking change for Fiddler because Fiddler exposes HTTP request and response bodies as byte[] requestBodyBytes and byte[] responseBodyBytes. Even if we were to change that, popular APIs like GetResponseBodyAsString would end up making contiguous allocations anyway.

Users can use rules to stream large requests and responses without Fiddler keeping a copy, but this limits a lot of Fiddler’s power—streamed and forgotten traffic can’t be modified and isn’t available for inspection. As a consequence, the “stream and forget” approach is best reserved for bodies over 2 gigabytes (since .NET, even in 64bit, cannot use more than 2gb in a single allocation).

So are we out of luck?

No, not quite.

Before creating the byte arrays, Fiddler has to collect all of the traffic. It does so using a MemoryStream because Fiddler can’t know ahead of time how big the request or response will be.

MemoryStreams are tremendously useful objects, but they’re really just a bit of syntactic sugar around a backing array that resizes itself when you attempt to write more data to it than it can hold. The Write method checks the size of the write operation and if space is needed, it calls a private EnsureCapacity method to grow the backing array. Let’s have a look at this method, trimmed for clarity:

image

The first thing to notice is the code in the orange box; the MemoryStream has an optimization that says: “If the required capacity is less than double the current capacity, then double the current capacity.” This optimization helps ensure that if you’re writing small amounts of data to the MemoryStream over and over, it doesn’t cause the MemoryStream to reallocate the byte array over and over. Reallocations are especially expensive because the data from the old backing array must be copied to the new backing array upon each allocation.

However, this optimization quickly becomes problematic for 32bit Fiddler. Say we’re reading a large file download of 75 megabytes, reading at 15kb per second. The MemoryStream starts out at 16kb, then quickly grows to 32kb, 64kb, 128kb, 256kb, 512kb, 1mb, 2mb, 4mb, 8mb, 16mb, 32mb, 64mb, and finally ends at 128mb. When the download completes, before Fiddler can generate the 75mb responseBodyBytes array, the MemoryStream has tried to allocate 128 contiguous megabytes, an allocation that is 53 megabytes larger than we truly need. Even if we have a 75 megabyte slot available in memory, we need a 128 megabyte slot for the download to succeed. You’ll also notice that the MemoryStream makes no attempt to catch an OutOfMemoryException error and allocate only what is needed if the double-size allocation fails.

Not good.

Also, have a look at the code in the blue box, which is new for .NET 4.6. Prior to the addition of this code, once more than 1GB of data was written to the MemoryStream, any additional writes to the stream would only grow the MemoryStream to the exact size needed. That was awful for Fiddler performance because every single read from the network would result in reallocating the entire array, from 1.000000gb to 1.000064gb to 1.000128gb to 1.000192gb to 1.000256gb, etc. That performance problem has been fixed, but it introduces a massive overallocation—when a MemoryStream with 1GB of data is reallocated, it automatically jumps to just under 2 gigabytes of data capacity—potentially wasting up to nearly a gigabyte of memory. That’s not awesome, even on 64bit.

So, what to do?

While we can’t fix all of these problems, we can improve the situation over the stock MemoryStream because we know a little bit more about how the caller will be using it than the .NET Framework designers could assume.

We replace MemoryStream with a new class which descends from MemoryStream called PipeReadBuffer. The PipeReadBuffer has an overridden Write method that evaluates the bitness of the process, the size of the current response, and, if available, a “hint” about the final body size. The growth algorithm is enhanced such that:

  1. Capacity avoids growing past 85kb as long as possible, to keep the allocation off the large object heap.
  2. After that, Capacity grows quickly at small sizes, but growth caps at 16mb (on 32bit) to 64mb (on 64bit) at large sizes.
  3. The caller can send the PipeReadBuffer a “Hint” about the expected final capacity based on the Content-Length header, if any.

With these optimizations, the PipeReadBuffer can help ensure that waste is minimized in allocations and the risk of OutOfMemoryExceptions is reduced as much as possible.

Elsewhere in the code, we found great opportunities to tune MemoryStream’s growth. For instance, we have a method called GZipExpand that decompresses data that was compressed using gzip. The API we use unzips into a MemoryStream and previously was subject to the same overallocation problem seen in the network reads. Fortunately, the gzip format includes a hint about the size of the uncompressed data in the footer, so we can use that to set the capacity of our MemoryStream ahead of time:

Trimmed GzipExpand source

You can also see that here we take advantage of another optimization—if the stream is sized appropriately, we can avoid the allocation inside the ToArray() call by using the GetBuffer() method instead.

These improvements will appear in the next release of Fiddler.

Further explorations: The Bing team has released a promising new RecyclableMemoryStream (announcement) which might be an even better fit for the PipeReadBuffer, as it can avoid allocations on the Large Object Heap; those allocations are particularly a problem for older versions of .NET that don’t offer any way to compact the LOH.

-Eric Lawrence