Fiddler

In Fiddler, the Caching tab will attempt to calculate the cache freshness lifetime for responses that lack an explicit Expires or Cache-Control: max-age directive. The standard suggests clients use (0.1 * (DateTime.Now – Last-Modified)) as a heuristic freshness lifetime.

An alert Fiddler user noticed that the values he was seeing were slightly off what he expected: sometimes the values were 6 minutes shorter than he thought they should be.

Consider the following scenarios:

Last-Modified: February 28, 2016 01:00:00
Date: February 29, 2016 01:00:00
These are 24 hours apart (1440 minutes); 10% of that is 144 minutes.
Last-Modified: March 13, 2016 01:00:00
Date: March 14, 2016 01:00:00
Due to the “spring forward” adjustment of Daylight Savings Time, these values are just 23 hours apart (1380 minutes); 10% of that is 138 minutes.
Last-Modified: November 6, 01:00:00
Date: November 7, 01:00:00

Due to the “fall back” adjustment of Daylight Savings Time, these values are 25 hours apart (1500 minutes); 10% of that is 150 minutes.

So when a timespan encompasses an even number of those DST transitions, the effect cancels out. When a timespan encompasses an odd number of these DST transitions, the span is either an hour longer or an hour shorter than it would be if Daylight Savings Time did not exist.

-Eric

Fiddler’s Transformer tab has long been a simple way to examine the use of HTTP compression of web assets, especially as new compression engines (like Zopfli) and compression formats (like Brotli) arose. However, the one-Session-at-a-time design of the Transformer tab means it is cumbersome to use to evaluate the compressibility of an entire page or series of pages.

Introducing Compressibility

Compressibility is a new Fiddler 4 add-on1 which allows you to easily find opportunities for compression savings across your entire site. Each resource dropped on the compressibility tab is recompressed using several compression algorithms and formats, and the resulting file sizes are recorded:

Compressibility tab

You can select multiple resources to see the aggregate savings:

Total savings text

WebP savings are only computed for PNG and JPEG images; Zopfli savings for PNG files are computed by using the PNGDistill tool rather than just using Zopfli directly. Zopfli is usable by all browsers (as it is only a high-efficiency encoder for Deflate) while WebP is supported only by Chrome and Opera. Brotli is available in Chrome and Firefox, but limited to use from HTTPS origins.

Download the Addon…

To show the Compressibility tab, simply install the add-on, restart Fiddler, and choose Compressibility from the View > Tabs menu2.

View > Tabs > Compressibility menu screenshot

The extension also adds ToWebP Lossless and ToWebP Lossy commands to the ImageView Inspector’s context menu:

ImagesMenuExt

I hope you find this new addon useful; please send me your feedback so I can enhance it in future updates!

-Eric

1 Note: Compressibility requires Fiddler 4, because there’s really no good reason to use Fiddler 2 any longer, and Fiddler 4 resolves a number of problems and offers extension developers the ability to utilize newer framework classes.

2 If you love Compressibility so much that you want it to be shown in the list of tabs by default, type prefs set extensions.Compressibility.AlwaysOn true in Fiddler’s QuickExec box and hit enter.

TLDR? – Get the newest Fiddler here.

It’s been just over two months since the last significant release, and Fiddler 4.6.2.0 (and v2.6.2.0) are now available.

As always, the latest build includes a slew of bugfixes and minor tweaks, as well as a number of features described in this post.

Default Certificate Generator Changed

Changes coming to certificate validation in browsers and other clients mean that certificates generated by makecert.exe (previously Fiddler’s default generator) will soon be rejected. To address this problem, the default certificate generator on Windows 7 and later has been changed to CertEnroll. (Windows XP and Vista users should consider installing the similar CertMaker Addon).

Unfortunately, if you’re upgrading from an earlier version of Fiddler which used a different certificate generator, you may need to explicitly reset Fiddler’s certificates. Doing so is simple:

  1. Click Tools > Fiddler Options.
  2. Click the HTTPS tab.
  3. Ensure that the text says Certificates generated by CertEnroll engine.
  4. Click Actions > Reset Certificates. This may take a minute.
  5. Accept all prompts
  6. If you are using Fiddler to capture secure traffic from a mobile device or Firefox, you will need to remove the old Fiddler root certificate from that device (or Firefox) and install the newly-generated Fiddler certificate.

If necessary, you can read more about resetting Fiddler’s Certificates or read more about Fiddler’s Certificate Generators.

SAZ Repair

From time to time, users have asked for help with Fiddler Session Archive files (.SAZ or .RAZ files) that are corrupt, either because they are incomplete (e.g. power failed) or they were mangled by an incomplete download or a failing disk drive.

Fiddler 4.6.2 includes a new feature that can recover data from corrupt Session Archive files. If the Session Archive fails to load due to corruption, you’ll be prompted to attempt a repair of the file. Data recovered from the SAZ file will be stored in a new archive and loaded for display.

Notably, this feature may also be useful to recover corrupt .zip, .docx, .xlsx, .pptx, etc files that have nothing to do with Fiddler; give it a try!

FiddlerHook Removed

This release removes the FiddlerHook extension for Firefox. Mozilla is changing their add-on model for Firefox extensions. Short-term, Firefox requires that extensions be signed (and Mozilla has declined to sign FiddlerHook) and over the next year, Mozilla will be removing the XUL Overlay extension model upon which FiddlerHook was based.

Fortunately, you don’t really need FiddlerHook to use Fiddler with Firefox. For HTTP traffic, it will often “just work” and for HTTPS traffic, only minor configuration updates are needed. You can read this post for tips on using Fiddler with Firefox.

Decryption Control

Previously, Fiddler UI only allowed you to exempt certain hosts from HTTPS decryption; if you wanted to only decrypt from a small number of hosts, you were forced to use the script engine. That limitation has been removed via a new option on the Tools > Fiddler Options > HTTPS tab. Simply click the link to toggle between exclusion and inclusion:

imageimage

Extensibility Improvements

This release adds a number of improvements to Fiddler’s extensibility, from both FiddlerScript and .NET extensions.

BindUIButton

You can now add buttons to Fiddler’s toolbar in a supported way. Simply add a new BindUIButton attribute to a static method in your FiddlerScript file; the string argument is the caption with which to label the button.

For instance:

    BindUIButton("Copy HAR")

yields:

toolbar button

Toolbar buttons are added to the left of the toolbar in the order opposite of their listing within the FiddlerScript. Adding images is not supported from FiddlerScript but you can use Unicode Emoji symbols if you’d like.

Fiddler extensions may add to the toolbar using the static method FiddlerToolbar.AddToolstripItem() and may remove entries using .RemoveToolstripItem().

Export to String

In many cases, you may wish to generate a string representing one or more Sessions in either HTTPArchive (HAR) or cURL format. While you can manually use File > Export to generate files of either format, you can now skip the middle-man and export to these types in memory. To do so, simply set the ExportToString option and do not set a Filename option. After the DoExport command completes, the output is found in the OutputAsString option.

For instance, you can add the following to your FiddlerScript:

BindUIButton("Copy HAR")
ContextAction("Copy HAR")
public static function doCopyHar(arrSess: Session[])
{
var oExportOptions = FiddlerObject.createDictionary();

// If you’d prefer to save to a file, set the Filename instead
//oExportOptions.Add(“Filename”, “C:\\users\\lawrence\\desktop\\out1.har”);

oExportOptions.Add(“ExportToString”, “true”);
oExportOptions.Add(“MaxBinaryBodyLength”, 1000000);
oExportOptions.Add(“MaxTextBodyLength”, 1000000);

FiddlerApplication.DoExport(“HTTPArchive v1.2”, arrSess, oExportOptions, null);
var sOutput: String = oExportOptions[“OutputAsString”];

Utilities.CopyToClipboard(sOutput);
FiddlerApplication.UI.SetStatusText(“Copied Sessions as HAR”);
}

…and Fiddler will add a toolbar button and context menu command that copies the Selected Sessions to the clipboard in HAR format. (Tip: You can choose Edit > Paste As Sessions in Fiddler to create new Sessions based on HAR text that you’ve copied from browser tools.)

Alternatively, you can add a similar command to copy the Selected Sessions as a set of cURL commands:

BindUIButton("Copy cURL")
ContextAction("Copy cURL")
public static function doCopyCurl(arrSess: Session[])
{
var oExportOptions = FiddlerObject.createDictionary();

// If you’d prefer to save to a file, set the Filename instead
//oExportOptions.Add(“Filename”, “C:\\users\\lawrence\\desktop\\out1.bat”);

oExportOptions.Add(“ExportToString”, “true”);

  FiddlerApplication.DoExport("cURL Script", arrSess, oExportOptions, null);
var sOutput: String = oExportOptions["OutputAsString"];

Utilities.CopyToClipboard(sOutput);
FiddlerApplication.UI.SetStatusText(“Copied Sessions as cURL”);
}

Invoking on the UI Thread

Fiddler processes Sessions on background threads, but you should only ever manipulate Fiddler’s UI using the UI thread. Only a few of Fiddler’s UI calls are thread-safe; if you’re not sure, your script should use the new FiddlerObject.uiInvoke or FiddlerObject.uiInvokeAsync methods to avoid crashing or corrupting the user-interface.

Load Extensions at Runtime

To support some exciting new work from the community, Fiddler now has the ability to load additional Extensions and Inspectors at runtime; this enables building of more complex add-on systems atop Fiddler’s existing system. To use these APIs, invoke any of these four methods from the UI thread:


FiddlerApplication.oExtensions.InstantiateInspectorsFromPath(string sPathToInspectors)
FiddlerApplication.oExtensions.InstantiateExtensionsFromPath(string sPathToExtensions)
FiddlerApplication.oExtensions.InstantiateExtensionsInFile(FileInfo oFile, bool bWriteToLog, bool bRethrowExceptions)
FiddlerApplication.oExtensions.InstantiateExtensionByType(Type typeExtension, bool bWriteToLog)

Thank You!

Lastly, I’d like to thank everyone for all of your support over the last twelve years, as Fiddler has evolved from a side project to a fully-supported debugging platform used around the world. I’m excited to see where Telerik takes Fiddler next, and while I’ll be keeping plenty busy in my new job, I expect I’ll remain involved in the Fiddler community (updating the book, and haunting the forum) for quite some time.

Wishing you all the best in 2016 and beyond!

-Eric Lawrence

Sadly, you’re unlikely to get wealthy by writing a book. You should definitely write one anyway.

My Background

People I respect suggest you shouldn’t write (or buy) books on specific technologies, going so far as to say that writing a book was on their top-10 lists of life regrets. Top-10… whoa!

As a consequence, when I was approached to write a book about Internet Explorer in 2009, I turned it down. “No one reads books anymore,” I asserted to the Vice President of the Internet Explorer team. At the time, I was sitting about 6 feet from my bookshelf full of technical books that I’d been buying over the last decade, including a few I’d purchased in the last month.

My counter-factual stance continued for the next few years, even as I served as a technical reviewer for five books on web development and performance. Then, in 2011, as I started pondering the sale of Fiddler, I met some product managers at the proposed acquirer and watched them use Fiddler for a few minutes. I was appalled—these guys had been looking at Fiddler for six months, and seemed to have settled on the most cumbersome and complicated ways to get anything accomplished. It wasn’t really their fault—Fiddler was incredibly feature rich, and I couldn’t fault them for not reading the manual—there wasn’t one. I felt a moral obligation, whether I sold Fiddler or not, to at least write down how to use it.

At the time, my wife was training to run marathons and I had quite a bit of free time in the mornings. Not knowing any better, I did what I assumed most writers do—I took my laptop to the coffee shop in the mornings and started writing. My resolve was aided by two crutches—

  1. I was happy to describe Fiddler, feature-by-feature, from top to bottom, left to right (Hemmingway, this wasn’t), and
  2. I decided that even if I abandoned the project without finishing a proper book, I could just save the results to a PDF, title it “Fiddler-Manual.pdf” and upload it to the Fiddler website.

I’ll cover the mechanics of actual writing in a future post (surprisingly straightforward, but I have some advice that may save you some time), but for now it suffices to say that after nine months of work a few times a week, I had a book.

My Investment

Writing the first edition took about 110 hours authoring, 20 hours of editing, and 30 hours of designing the cover, fixing formatting, and futzing with the printer/publisher webapp. I spent about $50 on draft copies, $40 or so on the cover photo, $20 on the fiddlerbook.com domain name registration, and about $650 for coffee and pastries consumed while writing. From September 2011 to June 2012, I periodically took a “snapshot” of my progress by printing the work:

Fiddler draft copies

Writing took about three months longer than my prediction:

Notebook with dates crossed out

… in part because as I wrote, I discovered what I’ve come to call Book-Driven Development (covered in the next section).

I was proud of the final product, but skeptical that it would earn back even what I spent on coffee.

So, Why Write?

First, it’s something parents and other folks can tangibly hold and appreciate. It’s probably the only way my grandmother will ever have any idea what the heck a “Fiddler Web Debugger” is and why someone might use one.

Second, it’s tangible. Many people have contributed to Fiddler over the years, and I can inscribe a paperback copy and send it to them as a “Thank you.” When the book was finished, I bought a dozen copies and dropped them off in the offices of colleagues who’d made contributions (in the form of bug reports or suggestions) over the years. One of the proudest moments of my life was when I got an email from Mark Russinovich thanking me for the signed copy and noting that it would nicely complement the ebook he’d already bought.

Third, writing a book makes you think very very hard about what you’re writing about, and with a different mindset. The Fiddler book took quite a bit longer to write because I made hundreds of improvements to Fiddler while I was writing the book, because I was writing the book. Almost every time I thought of something interesting to explain, I began to write about it, then realized that whatever it was shouldn’t have been so complicated in the first place. I’d then go fix Fiddler itself to avoid the problem. In other cases, explicitly writing out everything you can do with Fiddler made me recognize some important (and in hindsight, obvious) gaps, and go implement those features. I started calling this process Book-Driven Development and Fiddler was dramatically improved over the authoring of the book. Having said that, this also made writing the book take quite a bit longer—I’d write three or four pages, realize that the feature in question shouldn’t be so complicated, and go implement a checkbox or a button that would do everything the user needed without explanation. Then I’d have to go delete those three or four pages and replace it with “To do <X>, just push the <X> button.

Fourth, I got to choose what to write about. Fiddler is insanely powerful, but after watching people “in the field” use it, it was plain that most of its functionality is completely unknown to the vast majority of users. While some users watch the videos and read the blog posts, it was clear that there are some number of folks for which a complete book with an end-to-end explanation of the tool is the best way to learn it.

Fifth, it gives you an appreciation for other authors that you may never get otherwise. Marathon runners probably have more respect for other marathon runners than the general public ever will, simply because they know how grueling it is to run 26.2 in a way that someone who hasn’t never will. I think the same is probably true for book-writers.

So, in summary:

  1. It’s tangible.
  2. You can gift it to contributors.
  3. You’re forced to think like a new user.
  4. You can drive the direction of usage.
  5. You learn to appreciate authors.
  6. Self-publishing significantly changes your book’s financial prospects.

Money Matters

One of the challenges with almost any profit-making endeavor is that folks are so coy about the numbers involved. Inspired by John Resig’s post on Programming Book profits, I’m going to share my numbers here. My goal isn’t to brag—I think these are solid numbers, not runaway success numbers, but I want to show why “You’ll never make any money selling a book” is simply untrue.

Having read a bunch of posts like Jeff Atwood’s and Resig’s, I realized that going the traditional publisher route was a bad deal for both the reader and for me– the Fiddler book would have been ~$30 and I’d see maybe two or three dollars of that. Self-publishing is a better deal for the reader (lower prices) and it’s a better deal for me (I get about $6 and $8 respectively). While a traditional publisher would have probably netted me an advance of a few thousand bucks (more than I expected to make) I frankly prefer the “honesty” of being solely responsible for my book’s sales, and the often happy feeling I get when I (obsessively?) check sales stats and find that I sold a handful more copies overnight.

The first edition of Debugging with Fiddler was released in June 2012. The book was self-published on Lulu for the ebook (a PDF) and via CreateSpace (paperback) which was mostly sold on Amazon. The Lulu ebook was sold for a flat $10, while Amazon set the price for the paperback, usually at a small discount off the nominal $18.99 cover price.

Here are the sales figures for the ebook on Lulu:

$20116

The paperback sold slightly better, with 2713 English copies sold; the CreateSpace report below includes the 319 copies sold (so far) of the Second Edition:

3032 copies sold

Beyond the sales of my book, I also agreed to let the book be translated to Korean, Chinese, and Japanese by three local publishers. Each publisher agreed to an advance of about $1500, as well as three or four copies of the translated paperback. Of these, only one publisher (O’Reilly Japan) sends follow-up royalty statements; the book sold approximately 1400 copies in Japanese, and their advance was followed by a royalty check of $1450 in February of 2014.

On March 5th of 2015, I released the Second Edition update, revised to describe the changes in Fiddler over the last three years. This too proved far more successful than I’d ever dreamed. The $14.99 PDF (usually “on sale” for $9.99) took the lion’s share of sales with 840 copies sold, vs. 319 copies of the revised paperback. While the paperback stayed at CreateSpace, I moved to GumRoad for the new ebook for reasons I’ll describe in a future post.

$7453 royalties

So, how much did I earn? A bit more than $53318 or so thus far– the Euro/GBP exchange rates make the math a bit tricky. I spent about 200 hours of solid work on the First and Second Editions, so this works out to a bit over $250 per hour. Pretty amazing, for a project that yielded so many non-financial rewards!

Results Will Vary

It’s worth mentioning that my sales numbers are almost certainly much higher than they would’ve been “naturally”, but for one critical factor— as the developer of Fiddler, I was in a position to advertise the book both on the Fiddler website and in the Fiddler application itself. Exposed to millions of Fiddler users, this exposure was obviously invaluable and not, alas, something available to most writers.

It’s also the case that as the tool’s author, I benefit from credibility and name recognition (people expect that I’ll know what I’m writing about). As the primary source, I also have the luxury of writing more quickly since I didn’t need to do much research (subject to the “Book driven development” caveats earlier).

My (long overdue) next book project, Blue Badge, is a memoir of my time working at Microsoft, and it won’t benefit from the incredible exposure I had for Debugging with Fiddler. I’m intrigued to see how it sells.

 

If you’re an aspiring author, or simply interested in book publishing, I hope you found this post useful!

-Eric

Fiddler’s default file format is the SAZ Format, which is just a ZIP file with a particular structure. Unfortunately, sometimes users’ SAZ files get corrupted due to failing disks or incomplete downloads, and when this happens, Fiddler can no longer open them.

Corrupt Archive dialog

Because Fiddler uses a standard ZIP file, surely a good ZIP reader will be able to read some data, right?

Windows Explorer’s primitive ZIP implementation can’t do anything useful:

Windows Cannot Open dialog

Alas, not even 7-zip offers any help.

Cannot Open dialog

Okay, well, surely you can just use any of the many ZIP Repair tools to extract the data that isn’t corrupt from the file, right?

Alas, a few hour’s worth of research suggests that almost all of the public ZIP repair tools are terrible, unable to handle most forms of corruption. Some claim to work, but the resulting “repaired” archive remains unreadable:

Error 0x800040005 Unspecified Error when extracting

Those tools that seem promising aren’t free, and require spending $30 or so before you can even determine whether they’ll get your data back.

What to do?

Write my own, of course. Most SAZ files are internally quite simple, and it shouldn’t be too hard to recover most data from archives that aren’t encrypted.

Fiddler 4.6.2 will offer a Repair Corrupt option on the dropdown in the Load dialog box:

Repair Corrupt option

When you choose this option, Fiddler will enter its archive recovery mode:

Explanation of recovery process

Notably, the recovery mode doesn’t especially care whether the recovered ZIP file is a SAZ file. If not, Fiddler will alert you that the file couldn’t be interpreted as a SAZ:

Fiddler Alert - Not A SAZ

… but the repaired file on your desktop:

image

… should now be openable by your ZIP reader of choice:

Windows Explorer View

I hope you find this new capability useful, both for Fiddler-generated files as well as any other corrupt ZIP or ZIP-based (e.g. docx, pptx) files you may encounter.

-Eric Lawrence

I’ve made changes to the latest versions of Fiddler to improve the performance of certificate creation, and to avoid problems with new certificate validation logic coming to Chrome and Firefox. The biggest of the Fiddler changes is that CertEnroll is now the default certificate generator on Windows 7 and later.

Unfortunately, this change can cause problems for users who have previously trusted the Fiddler root certificate; the browser may show an error message like NET::ERR_CERT_AUTHORITY_INVALID or The certificate was not issued by a trusted certificate authority.

Please perform the following steps to recreate the Fiddler root certificate:

Fiddler 4.6.1.5+

  1. Click Tools > Fiddler Options.
  2. Click the HTTPS tab.
  3. Ensure that the text says Certificates generated by CertEnroll engine.
  4. Click Actions > Reset Certificates. This may take a minute.
  5. Accept all prompts

Fiddler 4.6.1.4 and earlier

  1. Click Tools > Fiddler Options.
  2. Click the HTTPS tab
  3. Uncheck the Decrypt HTTPS traffic checkbox
  4. Click the Remove Interception Certificates button. This may take a minute.
  5. Accept all of the prompts that appear (e.g. Do you want to delete these certificates, etc)
  6. (Optional) Click the Fiddler.DefaultCertificateProvider link and verify that the dropdown is set to CertEnroll
  7. Exit and restart Fiddler
  8. Click Tools > Fiddler Options.
  9. Click the HTTPS tab
  10. Re-check the Decrypt HTTPS traffic checkbox
  11. Accept all of the prompts that appear (e.g. Do you want to trust this root certificate)

image

If you are using Fiddler to capture secure traffic from a mobile device or Firefox, you will need to remove the old Fiddler root certificate from that device (or Firefox) and install the newly-generated Fiddler certificate.

I apologize for the inconvenience, but I believe that the new certificate generator will help ensure smooth debugging with current and future clients.

-Eric Lawrence