text/plain

For simplicity (and because I didn’t know any better at the time), Fiddler uses plain public byte[] array fields to represent the request and response bodies. This makes working with the body data trivial for authors of extensions and FiddlerScript, but it also creates significant shortcomings. Using fields rather than properties improves performance in some scenarios, but it muddles the contract about mutability of the data and means that developers can easily accidentally create inconsistent state (e.g. by decompressing the body but forgetting to change the Content-Length, Content-Encoding, and Transfer-Encoding headers).

The more serious problem with the use of byte arrays is that they require contiguous memory allocations. In a 64-bit process, this isn’t a major problem, but in a 32-bit process, address space fragmentation means that finding an open address range larger than a few hundred megabytes is often impossible:

If Fiddler cannot allocate contiguous memory of the required size, the resulting .NET System.OutOfMemoryException kills the Web Session. While 64-bit processes rarely suffer from address space fragmentation, the use of byte arrays still leads to a problem with large downloads—the .NET Framework imposes a cap of 0x7FFFFFC7 elements in an array, meaning that even 64-bit Fiddler is limited to storing request and response bodies that are just under two gigabytes. In practice, this is rarely a huge problem, but it’s occasionally annoying.

From an API point-of-view, I should have exposed message bodies as a Stream, so the backing data structure could be selected (and changed) as needed for performance and reliability reasons.

Now, as it happens, Fiddler internally uses a Stream buffer when it’s reading the body from the socket—it uses a MemoryStream for this purpose. Unfortunately, the MemoryStream built into the .NET Framework itself uses a plain byte[] to store the data, which means it suffers from the same problems described before, and some additional problems. Its biggest problem is that the growth algorithm for byte array backing the MemoryStream, and I’ve written at length about the issue and how I worked around it in Fiddler by creating a PipeReadBuffer object with smarter growth rules.

I thought things were as good as they could be without swapping to use a different object underlying the PipeReadBuffer, but last night Rafael Rivera pointed out a scenario that’s really broken in Fiddler today. His client was trying to download a 13.6gb ZIP file through Fiddler, and at just below 2gb the download slowed to a crawl. Looking at Fiddler.exe in Process Monitor, nearly 50% of the time was logged in garbage collection.

What’s going on?

The problem is that 64-bit Fiddler defaults to Stream and Forget bodies only when they reach 0x7FFFFFC7 bytes. For various reasons (which are likely not very compelling), Fiddler doesn’t trust the Content-Length response header and will instead keep buffering the response body until the StreamAndForget threshold is reached, at which point the response bytes will be streamed to the client, dropped, and subsequent bytes will be blindly streamed to the client without recording to a buffer. Despite that wasted buffering for the first 2 gigabytes, however, everything ought to work reasonably quickly.

Except.

When I coded the PipeReadBuffer, I made it grow by 64mb at a time, until we got to within 64mb of the .NET max array index of 0x7FFFFFC7. When we got within 64mb of the end, instead of correctly growing to 0x7FFFFFC7 bytes, it instead grows to exactly the length needed with no slack bytes. Which means that when the next network read comes along a millisecond later, the MemoryStream’s byte array has no free space and must be reallocated. And it gets reallocated to exactly the needed size, leaving no slack. This process repeats, with each network read meaning that .NET must:

• Allocate an array of just under 2gb
• Copy the 2 billion bytes from the old array to the new array
• Copy in the ~16kb from the network read to the end of the new array
• Free the old array

This is not a fast pattern, but things get even worse. Ordinarily, those last 64mb below the threshold will download reasonably quickly, the StreamAndForget threshold will get hit, and then all of the memory is freed and the download will proceed without buffering.

But.

TCP/IP includes a behavior called flow control, which means that the server tries to send data only as fast as the client is able to read it. When Fiddler hits the bad reallocation behavior described, it dramatically slows down how quickly it reads from the network. This, in turn, causes the server to send smaller and smaller packets. Which means Fiddler performs more and more network reads of smaller and smaller data, slowing the download of the 64mb to a virtual crawl.

Before Telerik ships a fix for this bug, anyone hitting this can avoid it with a trivial workaround—just set the Stream and Forget threshold inside Tools > Fiddler Options > Performance to something smaller than 2gb (for most users, 100mb would actually work great).

-Eric