Category Archives: fiddler

fiddler, Uncategorized

FiddlerCore and Brotli compression

Recently, a developer asked me how to enable Brotli content-compression support in FiddlerCore applications, so that APIs like oSession.GetResponseBodyAsString() work properly when the entity body has been compressed using brotli.

Right now, support requires two steps:

  1. Put brotli.exe (installed by Fiddler or off Github) into a Tools subfolder of the folder containing your application’s executable.
  2. Ensure that the Environment.SpecialFolder.MyDocuments folder exists and contains a FiddlerCore subfolder (e.g. C:\users\username\documents\FiddlerCore).

Step #1 allows FiddlerCore to find brotli.exe. Alternatively, you can set the fiddler.config.path.Tools preference to override the folder.

Step #2 allows FiddlerCore to create necessary temporary files. Sadly, this folder cannot presently be overridden [Bug].

One day, Fiddler might not need brotli.exe any longer, as Brotli compression is making its way into the framework.

 

 

Standard
dev, fiddler

Fiddler And LINQ

Since moving to Google at the beginning of 2016, I’ve gained some perspective about my work on Fiddler over the prior 12+ years. Mostly, I’m happy about what I accomplished, although I’m a bit awed about how much work I put into it, and how big my “little side project” turned out to be.

It’s been interesting to see where the team at Telerik has taken the tool since then. Some things I’m not so psyched about (running the code through an obfuscator has been a source of bugs and annoyance), but the one feature I think is super-cool is support for writing FiddlerScript in C#. That’s a feature I informally supported via an extension, but foolishly (in hindsight) never invested in baking into the tool itself. That’s despite the fact that JScript.NET is a bit of an abomination which is uncomfortable for both proper JavaScript developers and .NET developers. But I digress… C# FiddlerScript is really neat, and even though it may take a bit of effort to port the many existing example FiddlerScript snippets, I think many .NET developers will find it worthwhile.

I’ve long been hesitant about adopting the more fancy features of the modern .NET framework, LINQ key among them. For a while, I justified this as needing Fiddler to work on the bare .NET 2.0 framework, but that excuse is long gone. And I’ll confess, after using LINQ in FiddlerScript, it feels awkward and cumbersome not to.

To use LINQ in FiddlerScript, you must be using the C# scripting engine and you must add System.core.dll inside Tools > Fiddler Options > Scripting. Then, add using System.Linq; to the top of your C# script file.

After you make these changes, you can do things like:

    var arrSess = FiddlerApplication.UI.GetAllSessions();
    bool b = arrSess.Any(s=>s.HostnameIs("Example.com"));
    FiddlerApplication.UI.SetStatusText((b) ? "Found it!":"Didn't find it.");

-Eric Lawrence

Standard
fiddler

Using Fiddler With iOS 10 and Android 7

If you’ve tried to use Fiddler with iOS10 beta or Android 7 Nougat, you have probably found that HTTPS decryption isn’t working, even if you use the latest Fiddler and the Fiddler Certificate Maker add-on. Unfortunately, at the moment both platforms are broken, but for different reasons. In both cases, the client will fail to receive responses for HTTPS requests, and Fiddler will only show a CONNECT tunnel.

iOS 10 Change

After installing the FiddlerRoot certificate, one also needs to go to Settings -> General -> About -> Certificate Trust Settings and manually enable full trust for the FiddlerRoot root certificate, including accepting a dialog that says that this will allow a third-party to eavesdrop on all your communications.

iOS 10 Beta Bug (Fixed for final version)

The beta of iOS 10 had a bug whereby, if the response to a HTTP CONNECT tunnel request contains a Connection: close response header, the client will close the connection instead of doing as it should and waiting until the TCP/IP connection closes. A few minor platforms have had the same bug over the years, but iOS is definitely the first important platform with this issue. At least two bugs have been filed with the Apple “Radar” bug reporter.

Working around this limitation is simple. In Fiddler, click Rules > Customize Rules. Scroll to the OnBeforeResponse function. Just inside that function, add the following lines:

  if (oSession.HTTPMethodIs("CONNECT")) {
oSession["ui-backcolor"] = "red";
oSession.ResponseHeaders.Remove("Connection");
}

Save the file and try connecting again.

Android 7 Feature

In contrast to the iOS regression, the change in Android 7 was intentional. The Android team has decided that, by default, HTTPS certificate validation for apps targeting API Level 24 and later will ignore all user-installed root certificates, meaning that your efforts to manually trust Fiddler’s root certificate will be fruitless. Individually application developers can temporarily override this change while debugging by updating the application’s configuration:

<network-security-config>  
      <debug-overrides>  
           <trust-anchors>  
                <!-- Trust user added CAs while debuggable only -->
                <certificates src="user" />  
           </trust-anchors>  
      </debug-config>  
 </network-security-config>

…or at all times…

<network-security-config>  
      <base-config>  
            <trust-anchors>  
                <!-- Trust preinstalled CAs -->  
                <certificates src="system" />  
                <!-- Additionally trust user added CAs -->  
                <certificates src="user" />  
           </trust-anchors>  
      </base-config>  
 </network-security-config>

…Unfortunately, these changes can only be undertaken by application developers and not end-users. End-users will probably need to jailbreak their devices, akin to what is required to circumvent certificate pinning.

-Eric

Standard
fiddler, security

Silliness – Fiddler Blocks Malware

Enough malware researchers now depend upon Fiddler that some bad guys won’t even try to infect your system if you have Fiddler installed.

The Malware Bytes blog post has the details, but the gist of it is that the attackers use JavaScript to probe the would-be victim’s PC for a variety of software. Beyond Kaspersky, TrendMicro, and MBAM security software, the fingerprinting script also checks for VirtualBox, Parallels, VMWare, and Fiddler. If any of these programs are thought to be installed, the exploit attempt is abandoned and the would-be victim is skipped.

This isn’t the only malware we’ve seen hiding from Fiddler—earlier attempts use tricks to see whether Fiddler is actively running and intercepting traffic and only abandon the exploit if it is.

This behavior is, of course, pretty silly. But it makes me happy anyway.

Preventing Detection of Fiddler

Malware researchers who want to help ensure Fiddler cannot be detected by bad guys should take the following steps:

  1. Do not put Fiddler directly on the “victim” machine.
    – If you must, at least install it to a non-default path.
  2. Instead, run Fiddler as a proxy server external to the victim machine (either use a different physical machine or a VM).
    – Tick the Tools > Fiddler Options > Connections > Allow remote computers to connect checkbox. Restart Fiddler and ensure the machine’s firewall allows inbound traffic to port 8888.
    – Point the victim’s proxy settings at the remote Fiddler instance.
    – Visit http://fiddlerserverIP:8888/ from the victim and install the Fiddler root certificate
  3. Click Rules > Customize Rules and update the FiddlerScript so that the OnReturningError function wipes the response headers and body and replaces them with non-descript strings. Some fingerprinting JavaScript will generate bogus AJAX requests and then scan the response to see whether there are signs of Fiddler.

-Eric

Standard
browsers, fiddler, perf

Automatically Evaluating Compressibility

Fiddler’s Transformer tab has long been a simple way to examine the use of HTTP compression of web assets, especially as new compression engines (like Zopfli) and compression formats (like Brotli) arose. However, the one-Session-at-a-time design of the Transformer tab means it is cumbersome to use to evaluate the compressibility of an entire page or series of pages.

Introducing Compressibility

Compressibility is a new Fiddler 4 add-on1 which allows you to easily find opportunities for compression savings across your entire site. Each resource dropped on the compressibility tab is recompressed using several compression algorithms and formats, and the resulting file sizes are recorded:

Compressibility tab

You can select multiple resources to see the aggregate savings:

Total savings text

WebP savings are only computed for PNG and JPEG images; Zopfli savings for PNG files are computed by using the PNGDistill tool rather than just using Zopfli directly. Zopfli is usable by all browsers (as it is only a high-efficiency encoder for Deflate) while WebP is supported only by Chrome and Opera. Brotli is available in Chrome and Firefox, but limited to use from HTTPS origins.

Download the Addon…

To show the Compressibility tab, simply install the add-on, restart Fiddler, and choose Compressibility from the View > Tabs menu2.

View > Tabs > Compressibility menu screenshot

The extension also adds ToWebP Lossless and ToWebP Lossy commands to the ImageView Inspector’s context menu:

ImagesMenuExt

I hope you find this new addon useful; please send me your feedback so I can enhance it in future updates!

-Eric

1 Note: Compressibility requires Fiddler 4, because there’s really no good reason to use Fiddler 2 any longer, and Fiddler 4 resolves a number of problems and offers extension developers the ability to utilize newer framework classes.

2 If you love Compressibility so much that you want it to be shown in the list of tabs by default, type prefs set extensions.Compressibility.AlwaysOn true in Fiddler’s QuickExec box and hit enter.

Standard
fiddler

What’s New in Fiddler 4.6.2

TLDR? – Get the newest Fiddler here.

It’s been just over two months since the last significant release, and Fiddler 4.6.2.0 (and v2.6.2.0) are now available.

As always, the latest build includes a slew of bugfixes and minor tweaks, as well as a number of features described in this post.

Default Certificate Generator Changed

Changes coming to certificate validation in browsers and other clients mean that certificates generated by makecert.exe (previously Fiddler’s default generator) will soon be rejected. To address this problem, the default certificate generator on Windows 7 and later has been changed to CertEnroll. (Windows XP and Vista users should consider installing the similar CertMaker Addon).

Unfortunately, if you’re upgrading from an earlier version of Fiddler which used a different certificate generator, you may need to explicitly reset Fiddler’s certificates. Doing so is simple:

  1. Click Tools > Fiddler Options.
  2. Click the HTTPS tab.
  3. Ensure that the text says Certificates generated by CertEnroll engine.
  4. Click Actions > Reset Certificates. This may take a minute.
  5. Accept all prompts
  6. If you are using Fiddler to capture secure traffic from a mobile device or Firefox, you will need to remove the old Fiddler root certificate from that device (or Firefox) and install the newly-generated Fiddler certificate.

If necessary, you can read more about resetting Fiddler’s Certificates or read more about Fiddler’s Certificate Generators.

SAZ Repair

From time to time, users have asked for help with Fiddler Session Archive files (.SAZ or .RAZ files) that are corrupt, either because they are incomplete (e.g. power failed) or they were mangled by an incomplete download or a failing disk drive.

Fiddler 4.6.2 includes a new feature that can recover data from corrupt Session Archive files. If the Session Archive fails to load due to corruption, you’ll be prompted to attempt a repair of the file. Data recovered from the SAZ file will be stored in a new archive and loaded for display.

Notably, this feature may also be useful to recover corrupt .zip, .docx, .xlsx, .pptx, etc files that have nothing to do with Fiddler; give it a try!

FiddlerHook Removed

This release removes the FiddlerHook extension for Firefox. Mozilla is changing their add-on model for Firefox extensions. Short-term, Firefox requires that extensions be signed (and Mozilla has declined to sign FiddlerHook) and over the next year, Mozilla will be removing the XUL Overlay extension model upon which FiddlerHook was based.

Fortunately, you don’t really need FiddlerHook to use Fiddler with Firefox. For HTTP traffic, it will often “just work” and for HTTPS traffic, only minor configuration updates are needed. You can read this post for tips on using Fiddler with Firefox.

Decryption Control

Previously, Fiddler UI only allowed you to exempt certain hosts from HTTPS decryption; if you wanted to only decrypt from a small number of hosts, you were forced to use the script engine. That limitation has been removed via a new option on the Tools > Fiddler Options > HTTPS tab. Simply click the link to toggle between exclusion and inclusion:

imageimage

Extensibility Improvements

This release adds a number of improvements to Fiddler’s extensibility, from both FiddlerScript and .NET extensions.

BindUIButton

You can now add buttons to Fiddler’s toolbar in a supported way. Simply add a new BindUIButton attribute to a static method in your FiddlerScript file; the string argument is the caption with which to label the button.

For instance:

    BindUIButton("Copy HAR")

yields:

toolbar button

Toolbar buttons are added to the left of the toolbar in the order opposite of their listing within the FiddlerScript. Adding images is not supported from FiddlerScript but you can use Unicode Emoji symbols if you’d like.

Fiddler extensions may add to the toolbar using the static method FiddlerToolbar.AddToolstripItem() and may remove entries using .RemoveToolstripItem().

Export to String

In many cases, you may wish to generate a string representing one or more Sessions in either HTTPArchive (HAR) or cURL format. While you can manually use File > Export to generate files of either format, you can now skip the middle-man and export to these types in memory. To do so, simply set the ExportToString option and do not set a Filename option. After the DoExport command completes, the output is found in the OutputAsString option.

For instance, you can add the following to your FiddlerScript:

BindUIButton("Copy HAR")
ContextAction("Copy HAR")
public static function doCopyHar(arrSess: Session[])
{
var oExportOptions = FiddlerObject.createDictionary();

// If you’d prefer to save to a file, set the Filename instead
//oExportOptions.Add(“Filename”, “C:\\users\\lawrence\\desktop\\out1.har”);

oExportOptions.Add(“ExportToString”, “true”);
oExportOptions.Add(“MaxBinaryBodyLength”, 1000000);
oExportOptions.Add(“MaxTextBodyLength”, 1000000);

FiddlerApplication.DoExport(“HTTPArchive v1.2”, arrSess, oExportOptions, null);
var sOutput: String = oExportOptions[“OutputAsString”];

Utilities.CopyToClipboard(sOutput);
FiddlerApplication.UI.SetStatusText(“Copied Sessions as HAR”);
}

…and Fiddler will add a toolbar button and context menu command that copies the Selected Sessions to the clipboard in HAR format. (Tip: You can choose Edit > Paste As Sessions in Fiddler to create new Sessions based on HAR text that you’ve copied from browser tools.)

Alternatively, you can add a similar command to copy the Selected Sessions as a set of cURL commands:

BindUIButton("Copy cURL")
ContextAction("Copy cURL")
public static function doCopyCurl(arrSess: Session[])
{
var oExportOptions = FiddlerObject.createDictionary();

// If you’d prefer to save to a file, set the Filename instead
//oExportOptions.Add(“Filename”, “C:\\users\\lawrence\\desktop\\out1.bat”);

oExportOptions.Add(“ExportToString”, “true”);

  FiddlerApplication.DoExport("cURL Script", arrSess, oExportOptions, null);
var sOutput: String = oExportOptions["OutputAsString"];

Utilities.CopyToClipboard(sOutput);
FiddlerApplication.UI.SetStatusText(“Copied Sessions as cURL”);
}

Invoking on the UI Thread

Fiddler processes Sessions on background threads, but you should only ever manipulate Fiddler’s UI using the UI thread. Only a few of Fiddler’s UI calls are thread-safe; if you’re not sure, your script should use the new FiddlerObject.uiInvoke or FiddlerObject.uiInvokeAsync methods to avoid crashing or corrupting the user-interface.

Load Extensions at Runtime

To support some exciting new work from the community, Fiddler now has the ability to load additional Extensions and Inspectors at runtime; this enables building of more complex add-on systems atop Fiddler’s existing system. To use these APIs, invoke any of these four methods from the UI thread:


FiddlerApplication.oExtensions.InstantiateInspectorsFromPath(string sPathToInspectors)
FiddlerApplication.oExtensions.InstantiateExtensionsFromPath(string sPathToExtensions)
FiddlerApplication.oExtensions.InstantiateExtensionsInFile(FileInfo oFile, bool bWriteToLog, bool bRethrowExceptions)
FiddlerApplication.oExtensions.InstantiateExtensionByType(Type typeExtension, bool bWriteToLog)

Thank You!

Lastly, I’d like to thank everyone for all of your support over the last twelve years, as Fiddler has evolved from a side project to a fully-supported debugging platform used around the world. I’m excited to see where Telerik takes Fiddler next, and while I’ll be keeping plenty busy in my new job, I expect I’ll remain involved in the Fiddler community (updating the book, and haunting the forum) for quite some time.

Wishing you all the best in 2016 and beyond!

-Eric Lawrence

Standard