Developer Advocacy

The Microsoft Edge (nee Internet Explorer) team held one of their “#AskMSEdge chats” on Twitter yesterday.

image

After watching the stream, @MarkXA neatly summarized the chat:

image

The folks over on WindowsCentral built out a larger summary of the tidbits of news that did get answered on the chat, some of which were just pointers to their Status and UserVoice sites.

After the chat ended, I complained that none of my questions had been answered:

image

In response, an IE Engineer retorted:

image

I don’t think that’s fair. Here are my questions, and a few remarks on each:

image

As far as I know, I’ve never asked the IE/Edge team about Brotli before, as I hadn’t spent any time looking at it until very recently. I’m interested in the team’s plans for Content-Encoding: Brotli because it can significantly improve browser performance, and if the team implements WOFF2, they must integrate Brotli decoding logic anyway.

image

I don’t think I’ve ever asked the IE/Edge team about their plans here before. More efficient HTTPS algorithms are important for both performance and battery life on mobile devices in particular, and thus I think they’re a great investment.

image

I have asked this before. IE has had non-standard network export for four years and I was really excited that Edge moved from HTTP Archive XML to the standard HTTP Archive JSON format. Unfortunately, this bug makes their code non-interoperable. The fix will be one or two lines of code. I feel justified in asking for status since weeks or months have passed without update.

image

I have asked this before. Edge regressed a significant piece of functionality and created a denial-of-service condition in their browser. I feel justified in asking for status since weeks or months have passed without update.

image

I have asked this one over and over again. I find it galling that Microsoft products are less secure together, and especially when Microsoft’s new President promised to close these sort of gaps nearly two years ago. It’s clear that the team agrees that the behavior is bad, because Edge uses Bing securely and doesn’t even allow users to add non-HTTPS search providers.

image

This one is probably the least “fair” of the questions, insofar as I already know the answer and I’m effectively just calling the team out on the specious nature of the promise to “watch demand” they made when the original concerns about the absence of Windows 7 support were raised.

However, I’ll note that the team answered several repeats of the question “When will it run on Mac? When will it run on iOS and Android.” Given the Windows 7 marketshare dominance, I think this question remains fair.

Other Unanswered

Other folks asked several great questions that didn’t get answered:

image

I really want this feature.

image

I think the Edge team is making a huge mistake if they’re not piloting their new extension model with critical extension developers like uBlock, NoScript, etc.

image

Because of the nature of the legacy Win32 Address Bar’s context menu, Paste-and-Go was always prohibitively expensive. The UI replacement for Metro IE and now Edge makes this a trivially added feature that was requested by several questions.

Unwanted Answers

Some questions got answers that I’m just not happy with, but I’m tired of complaining about:

image

The Edge team replied “No” and suggested they consider this a scenario for the new extension model. I think this is a mistake and a case where “different” isn’t likely to be “better.”

 

Several folks asked when the new extension model would be released. “Stay tuned” was the answer.

Subtext

I think the subtext of Adrian’s complaint is that “You’ve worked here, you know we don’t announce things on IEDevChats.” There’s some truth to that frustration – I know that announcements are carefully vetted and published on the blog and I understand why live chats aren’t a source of new information. However…  

Rabble-Rousing and Information Asymmetry

I know some folks think my questions are just rabble-rousing and that, as an ex-teammate and current MVP I should be asking these questions in private, directly to the IE team. A few points on that:

  • Microsoft has basically requested we provide feedback in this manner, with the “we’re watching feedback to influence our decisions” position on everything from features to bugfixes. Feedback that isn’t getting public traction is largely ignored.
  • A significant number of my friends and colleagues are no longer on the IE team. In the photo of the team answering questions, I recognize seven of the seventeen engineers.
  • My emails to the IE MVP discussion list generally do not receive replies.
  • Direct emails to individual engineers on the IE team often do not get replies.
  • Status on bugs I’ve filed with MSRC is similarly hard to get– issues have languished for months without so much as a “working on it” status update.

Having been on both sides of the fence now, it’s plain to me that one serious problem Microsoft has is that they don’t realize how incredibly opaque things are from outside the company. As an engineer racing from one issue to the next, it’s easy to deprioritize status updates and justify doing so when there are so many higher-priority things to fix. From outside the company, however, “working on it and coming as soon as we can”  is often indistinguishable from “ignoring—really hope this goes away.” That problem is exacerbated by Microsoft’s tendency not to deliver hard messages like “Silverlight is dead dead dead, get off it now!” in a timely manner to allow customers and partners to plan appropriately.

Brotli

2022 Update: Brotli is requested by 94% of browsers, offers great performance, and works amazingly well on Web Assembly code. If you’re still using GZIP today, you should update!

Regular readers of my blog know how much I love Zopfli, Google’s compression engine that often shrinks output by 5% or better when compared to the popular zlib engine. The beauty of Zopfli is that its output is compatible with all of the billions of existing DEFLATE encoders deployed worldwide, making its use an easy choice for any static content.

But imagine for a moment what compression ratios we could achieve if we weren’t limited by compatibility with existing decoders? If we could add a new compression engine to the web, what might it look like?

The Brotli compression engine, co-written by Jyrki Alakuijala (inventor of Zopfli), provides one answer. Brotli combines the LZ77 and Huffman algorithms of DEFLATE with a larger sliding window (up to 16mb1 vs. DEFLATE’s 32kb) and context modeling; the specification also calls for a 122kb static dictionary.

Brotli In Browsers

Today, Brotli is the compression engine behind the newish WOFF2 font format, providing savings of approximately 25% over WOFF 1.0 fonts compressed with Zopfli. Not content to rest on their laurels, Google has announced their Intent to Implement Brotli as a general purpose HTTP Content-Encoding, allowing web developers to use it to compress script, stylesheets, svg, xml, and the like. Firefox beat Google to the finish and shipped Brotli support in the Firefox 44 Dev build.

Probably HTTPS only

Past attempts to add new compression algorithms (bzip2 and SDCH) have demonstrated that a non-trivial number of intermediaries (proxies, gateway scanners) fail when Content-Encodings other than GZIP and DEFLATE are specified, so Brotli will probably only be supported over HTTPS connections, where intermediaries are less likely to interfere.

    Accept-Encoding: br, gzip, deflate, sdch

Results

Facebook investigated Brotli and found it would save about 17% of CSS bytes and 20% of JavaScript bytes (compared with Zopfli). When run on the CSS and JavaScript from the Alexa top-300k, Brotli saved 12% of CSS bytes and 9% of JavaScript bytes when compared to gzip.

Running a few simple tests with Fiddler, I saw great results with Brotli:

    Content-Encoding: br

jQueryMobileMin.js
image

Microsoft homepage:
image

A random giant XML documentation file:
image

Microsoft Word Online WordEditor.js
image

Microsoft Word Online WordEditor.Wac.TellMeModel.js
image

Cloudflare’s blog post on Brotli includes some benchmarks too.

Brotli is optimized for decompression speed. When compressing, Brotli is slower than zlib’s deflate, but considerably faster than zopfli, lzma and bzip2; given 1gb of extremely compressible content, Brotli finished compressing it to 3339 bytes after 301 seconds of CPU time. After 8040 seconds of CPU time, zopfli.exe crashed when a memory allocation failed.

Running Brotli.exe

To make things simpler for Windows users, I’ve built the latest release (v0.3) from GitHub for Win32 using Visual Studio 2015. You can download the Authenticode-signed Windows Brotli.exe from my site.

To compress a file, specify the input and output filenames:

  • --in filename
  • --out compressed_filename

… and optionally specify any of the following arguments:

  • --quality n
  • --force
  • --verbose

The quality parameter controls the compression-speed vs. compression-ratio tradeoff; the higher the quality, the slower but denser the compression. The supported range is 0 to 11, and 11 is the default.

The force parameter instructs Brotli to overwrite the output file if it already exists.

The verbose parameter instructs Brotli to display its compression speed in megabytes per second upon completion.

To decompress a file, use the --decompress parameter and specify the input and output filenames:

  • --in compressed_filename
  • --out filename

… and optionally specify the --verbose parameter to instruct Brotli to display its decompression speed in megabytes per second upon completion.

If you’d like to expose Brotli inside Fiddler 4.6.0.5+, place brotli.exe inside Fiddler’s \Tools\ subfolder and restart to see it appear on the Transformer tab:

image

Tracking Brotli

If you’d like to follow along:

Alas, the Brotli Discussion forum is currently empty.

Assorted Further Investigations

1. Someone needs to register the brotli token in the IANA registry (although Google’s SDCH and Microsoft’s Xpress aren’t listed there either).

2. Implementers should consider protections against “brotli bombing” DoS attacks. Brotli’s high compression ratio makes attacks even cheaper for the bad guys.  A trivial test of compressing a file containing all 0s shows that Brotli can achieve a compression ratio of at least 386516:1, meaning that 1389 bytes of compressed data can blow up to 512MB when uncompressed. In contrast, DEFLATE has a maximum compression ratio approaching 1032 to 1, so an attacker would need to send 375 times as much data over the network to achieve a similar result. That being said, even DEFLATE can result in a denial-of-service, as with this 5.8mb PNG file that can require allocation of up to 141GB of memory.

3. Brotli’s use in WOFF2 means that browsers have already taken on its attack surface. However, not all attack surface is created equal; WOFF2 fonts can be decoded inside a very restricted sandbox. When Chrome 1.0 released, I was surprised to learn its HTTP decompressors were in a full-trust process; it turns out that is still the case today, which makes fuzzing against decompressors very interesting to an attacker.

4. Brotli’s static dictionary was generated from a broad corpus of content, but considering the most likely use cases (static files), it may not be optimal for this use. At this point, it’s probably too late to change it.

5. When used as a Content-Encoding, will brotli be used “bare” or in some framing format (e.g. with a trailing CRC and size marker)? Will it have magic bytes that will allow sniffing? (Per @mcmanusducksong, Firefox is going with a bare stream and no magics. boo)

6. While not terribly relevant to my scenarios, it turns out Google builds a lot of compression engines I’d never heard of, e.g. Snappy and Gipfeli. When compression speed is more important than ratio, they’re worth a look.

7. Brotli makes the most sense for pre-compression of static content; to that end, someone needs to xcopy the http_gzip_static module for nginx and make a few tweaks to create a new http_brotli_static module. While the nginx team may eventually release one, Google released a brotli module that supports both dynamic and static compression.

1 While Brotli can use a 16mb window, for performance reasons it appears that constraining the window to 4mb is the plan for most scenarios.

Fiddler Certificate Generators

Fiddler and FiddlerCore offer three different choices for generating interception certificates:

  • MakeCert
  • CertEnroll
  • Bouncy Castle

If you’re so inclined, you can even write your own certificate generator (say, by wrapping OpenSSL) and expose it to Fiddler using the ICertificateProvider3 interface.

On Windows, Fiddler includes the MakeCert and CertEnroll certificate generators by default; you can download the Bouncy Castle Certificate Generator if you like. In contrast, when Fiddler is running on Linux and Mac, only the Bouncy Castle certificate generator is available, and it is included by default.

If you’re using Windows, however, you may wonder which Certificate Generator you should use in Fiddler or for your applications based on FiddlerCore.

In general, I recommend the Bouncy Castle generator, as it has better performance than the default MakeCert generator and it offers more configuration choices than the CertEnroll generator. Another advantage of the Bouncy Castle certificate generator is that the only certificate that (typically) goes in the Windows Certificate store is the root certificate. The server (end-entity) certificates generated for each website are kept in memory and discarded when Fiddler exits; because the Bouncy Castle generator reuses a single private key for all certificates by default, the performance impact of this behavior is minimal.

The only downside to the Bouncy Castle generator is its size: it is ~200KB when compressed, which is 25% larger than FiddlerCore itself.

The CertEnroll generator was added to Fiddler relatively recently; it offers better performance and standards-compliance than the legacy MakeCert generator but it is available only on Windows 7 and later. You can easily switch Fiddler to use CertEnroll inside Tools > Fiddler Options > HTTPS.

The MakeCert generator is the original certificate generator used by Fiddler and it remains the default on Windows today (mostly) for legacy compatibility reasons. It suffers from a number of shortcomings, including the fact that the certificates it generates are not compatible with iOS and (some) Android devices. It generates certificates with a 1024 bit RSA key (which may soon trigger warnings in some browsers) and each certificate has a unique key (meaning that each new secure site you visit triggers the somewhat costly key generation code).

Both the CertEnroll and MakeCert-based certificate generators must store all server certificates in the Windows Certificate store which some users may find confusing:

image

The storage of (potentially thousands of) server certificates in the user profile can also cause some problems for corporate users who have roaming user profiles, as these certificates are roamed to each workstation as the user logs in. To mitigate that, the Clear server certs on exit checkbox can be set inside the Tools > Fiddler Options > HTTPS > Certificate Provider dialog, or via:

    FiddlerApplication.Prefs.SetBoolPref("fiddler.certmaker.CleanupServerCertsOnExit", true);

… however, the downside of doing that is that Fiddler must then re-create the server certificates every time it starts. This performance penalty is smaller when using CertEnroll, which reuses a single 2048-bit RSA key, than for MakeCert, which generates unique 1024-bit RSA keys for each site.

FiddlerCore Considerations

To determine which Certificate Generator is in use, be sure to attach the following event handlers:

Fiddler.FiddlerApplication.OnNotification +=
  delegate(object sender, NotificationEventArgs oNEA) { Console.WriteLine(“** NotifyUser: ” + oNEA.NotifyString); };
Fiddler.FiddlerApplication.Log.OnLogString +=
  delegate(object sender, LogEventArgs oLEA) { Console.WriteLine(“** LogString: ” + oLEA.LogString); };

You can then view information about the Certificate Generator in the console when it loads.

Developers building applications atop FiddlerCore should keep the following in mind when deciding which Certificate Generator to use:

MakeCert

  • MakeCert.exe is a Microsoft Visual Studio 2008 redistributable file, meaning that you’re licensed to redistribute it if you have an appropriate license to that version of Visual Studio. Microsoft may offer MakeCert.exe as a redistributable in other circumstances, but licensing is provided by Microsoft, not Telerik.
  • To use MakeCert.exe, you must include it adjacent to your application’s .exe file.
  • MakeCert-generated certificates are not compatible with iOS and some Android devices.
  • MakeCert-generated certificates “pollute” the user’s Certificate Store and you should consider offering a mechanism to clear them.

CertEnroll

  • The CertEnroll API is available on Windows 7 and later.
  • Use CertEnroll by either omitting makecert.exe from the application’s folder or by explicitly setting the preference:
  •     FiddlerApplication.Prefs.SetBoolPref("fiddler.certmaker.PreferCertEnroll", true);

  • CertEnroll-generated certificates “pollute” the user’s Certificate Store and you should consider offering a mechanism to clear them.

Bouncy Castle

  • Bouncy Castle is an open-source PKI and crypto library distributed under the MIT license.

  • To use Bouncy Castle, you must include CertMaker.dll and BCMakeCert.dll adjacent to your application’s .exe file.
  • Bouncy Castle does not store certificates in the Windows Certificate Store (yay!) but this also means that your application needs to keep track of its root certificate and private key (unless you recreate and retrust it every time the application runs).

    Two preferences are used to hold the key and certificate, fiddler.certmaker.bc.key and fiddler.certmaker.bc.cert. After you first call createRootCert, you should retrieve these preferences using FiddlerApplication.Prefs.GetStringPref and store them somewhere within your application’s settings (registry, XML, etc); the private key should be considered sensitive data and protected as such.  When your application next runs, it should detect whether the key and certificate have already been created, and if so, they should be provided to the certificate generator using FiddlerApplication.Prefs.SetStringPref before any certificates are requested, lest you inadvertently create a new root certificate.

    Rick Strahl wrote a great blog post on this process, including some sample code.

 

-Eric

What’s New in Fiddler 4.6

Telerik Fiddler version 4.6 (and v2.6 targeting .NET2) is now available for download. The new version includes several new features and dozens of tweaks and bugfixes, described in this article.

View > Tabs Menu

The new View > Tabs menu offers a list of tabs that are hidden by default.

image

The Preferences command displays a tab that allows you to edit Fiddler’s Name/Value preferences.

The new APITest command displays the new Fiddler APITest tab that enables easy testing of web APIs.

The AutoSave command (formerly located in the Tools menu) permits you to automatically save Session Archive Zip files on a regular schedule.

Note: In the future, this submenu will become an extension point to allow developers to easily expose optional UI tabs.

Lint Filters

Fiddler’s Lint feature enables you to control how Fiddler reports violations of the HTTP protocol and other errors. Controlled on the Tools > Fiddler Options > General tab, you can leave the If protocol violations are observed setting at the basic “Warn on Critical errors” section or adjust it to “Warn on all errors”; the latter setting performs more tests on traffic to find mistakes that could cause clients or servers to misbehave. However, this setting can get pretty noisy. To allow you to suppress specific warnings, you can now use the Filter link on the Protocol Violation Report dialog.

image

Each Lint warning now has a unique code consisting of a letter and three digits. The letter prefix indicates the severity:

  • L – Common “problem”, low impact, unlikely to break anything.
  • M – Significant problem, likely to break functionality in one client.
  • H – Important problem likely to significantly break functionality in multiple clients

You can exclude an entire class of warnings by simply including the prefix in the list of exclusions.

Single Session Timeline

The Timeline tab in Fiddler allows you to visualize the download of multiple Sessions at once to permit you to view parallelism and connection reuse. In Fiddler 4.6, the view of a single Session has been enhanced to permit you to visualize how the content of the Session was downloaded. You can determine, for instance, whether the headers were flushed immediately or the client was forced to wait for headers until the body was ready, and you can determine whether download speed improved or regressed as the download proceeded. For instance, in this chart, the server took 2.5 seconds to return the headers and began to slowly stream the body in small chunks. As the download progressed, the speed improved until completion at just over 9.2 seconds:

image

 

Interface Tweaks

A new Copy as Image command on each Inspector’s tab context menu allows you to copy an image of the tab’s contents to your clipboard for easy pasting in an email or blog post. The Headers, XML, and JSON Inspectors allow you to easily highlight nodes of interest—simply press the spacebar on a node to temporarily render it with a yellow highlight:

image

The new Math context menu appears when you right-click on a numeric column in the Web Sessions list; the menu currently offers a single command: Sum and Average, which shows these attributes of the selected Sessions:

image

The Composer tab now offers a splitter between the headers and body boxes so you can adjust their height as desired:

image

Pressing the / key while in the Web Sessions list now enters QuickSearch mode in the QuickExec box, selecting any Sessions whose URL contains the text you type.

Hash Support

Fiddler now offers enhanced support for computing hashes of blocks of bytes or strings.

The Tools > TextWizard feature now offers a quick way to get a hash (MD5 to SHA512) of a block text, in either base64 or dashed hexadecimal format:

image

New methods are also available for FiddlerScript to compute hashes. For instance, you can copy the following block to just inside your Rules > Customize Rules > Handlers class:

 
public BindUITab("Resource Integrity Hashes", "<nowrap><nolink>")
static function ShowSRI(arrSess: Session[]):String
{
var oSB: System.Text.StringBuilder = new System.Text.StringBuilder();
for (var i:int = 0; i<arrSess.Length; i++)
{
if (arrSess[i].HTTPMethodIs("CONNECT")) continue;
        if (!arrSess[i].bHasResponse)
{
oSB.AppendFormat("\r\n// Skipping incomplete response '{0}'\r\n",
arrSess[i].fullUrl);
continue;
}
if (arrSess[i].responseCode != 200)
{
oSB.AppendFormat("\r\n// Skipping non-HTTP/200 response '{0}'\r\n",
arrSess[i].fullUrl);
continue;
}
var sType: String = arrSess[i].oResponse.MIMEType.ToLower();
var bIsScript = sType.Contains("script");
var bIsCSS = sType.Contains("css");
if (!bIsScript && !bIsCSS)
{
oSB.AppendFormat("\r\n// Skipping non-CSS/JS response '{0}'\r\n", arrSess[i].fullUrl);
continue;
}
var sIntegrity = "sha256-" + arrSess[i].GetResponseBodyHashAsBase64("sha256").Replace("-", "")
+"\n\tsha384-" + arrSess[i].GetResponseBodyHashAsBase64("sha384").Replace("-", "")
+"\n\tsha512-" + arrSess[i].GetResponseBodyHashAsBase64("sha512").Replace("-", "");
        if (bIsScript)
{
oSB.AppendFormat('\r\n<script src="{0}"\r\n\tintegrity="{1}"></script>\r\n',
arrSess[i].fullUrl, sIntegrity);
}
else
{
oSB.AppendFormat('\r\n<link rel="stylesheet"\r\n\thref="{0}"\r\n\tintegrity="{1}">\r\n',
arrSess[i].fullUrl, sIntegrity);
}
}
return oSB.ToString();
}

When you save the script, Fiddler adds a new tab to display the Subresource Integrity attributes for the selected response bodies:

image

 

FiddlerScript Improvements

BindUITab Enhancements

As seen in the Resource Integrity Hashes example above, the BindUITab attribute allows you to create new tabs inside Fiddler that are populated based on the selected Sessions. BindUITab now offers a second parameter that allows you to specify one or more of the following options:

  • <nowrap> – The RichEdit control should not wordwrap lines
  • <nolink> – The RichEdit control should not detect or underline urls
  • <html> – Instead of using a RichEdit control, the function’s response will be rendered as HTML inside a Web Browser Control

BeforeFiddlerShutdown event

Fiddler now exposes a  BeforeFiddlerShutdown event that enables extensions or FiddlerScript to block shutdown of Fiddler; this may be useful if you wish to prompt the user for permission to lose unsaved work, etc.

A Fiddler extension should attach an event handler:

    FiddlerApplication.BeforeFiddlerShutdown += (o, c) =>
{
c.Cancel = (DialogResult.Cancel ==
MessageBox.Show(“Allow Fiddler to close????”,
“Go Bye-bye?”, MessageBoxButtons.OKCancel));
};

Within FiddlerScript, you can add a method to the existing Handlers class:

    static function OnBeforeShutdown(): Boolean {
return ((0 == FiddlerApplication.UI.lvSessions.TotalItemCount()) ||
(DialogResult.Yes == MessageBox.Show(“Allow Fiddler to exit?”,
“Go Bye-bye?”, MessageBoxButtons.YesNo, MessageBoxIcon.Question,
MessageBoxDefaultButton.Button2)));
}

AutoResponder Improvements

The AutoResponder now supports the NOT: operator inside the METHOD:, HEADER:, and FLAG:, operators. For instance, if you’d like the AutoResponder only to impact requests from Google Chrome, add this rule to the top of your list:

image

Microsoft Edge Support

Fiddler has been updated to recognize Microsoft’s new Edge browser as a Web Browser, so features like the Process Filter in the Status Bar:

image

…and in the Browse command in the Fiddler toolbar:

image

work as expected.

Note: You should use the WinConfig button at the left of Fiddler’s toolbar to enable Windows 10 “Store” applications to run Fiddler. By default, you shouldn’t need to use the WinConfig button for Edge, because Edge’s about:flags enables access to loopback by default.

 

ImageView Enhancements

FavIcon Preview

When a site uses an .ICO file as its favicon, the icon may contain multiple different images that are used depending on the user’s device and the context in which the icon is rendered. Fiddler’s ImageView Inspector now renders all of the images contained within the .ICO file like so:

ImageView showing icon

 

JPEG Thumbnails

Websites should generally strip embedded thumbnails from JPEG files. Embedded thumbnails are a common source of bloat, wasteful bytes that aren’t used by the client. However, some sites fail to optimize their images by removing thumbnails. In some cases, an image thumbnail may even contain data which was never meant to be made public, if, for instance, the larger image was cropped without regeneration of the thumbnail. The ImageView now allows you to extract the thumbnail image as a new Session that is added to the Web Sessions list:

Extract Thumbnail

 

AutoSave Enhancements

Fiddler’s AutoSave feature (now found under View > Tabs > AutoSave) now supports several preferences to control its behavior.

Set fiddler.extensions.AutoSave.AlwaysOn to true to have Fiddler automatically activate AutoSave mode when it starts. Set fiddler.extensions.AutoSave.Minutes to the number of minutes to collect traffic between each save operation; the default is 5. Set fiddler.extensions.AutoSave.HeadersOnly to true if you’d like the SAZ file to contain only the request and response headers, omitting the bodies. Set fiddler.extensions.AutoSave.Path to the folder path under which auto-saved SAZ files should be stored.

Additional Upgrade Notifications

By default, Fiddler’s automatic update notifications will only show if a significant change in version number occurs. For instance, say you’re running version 4.6.0.2 and version 4.6.0.3 becomes available. By default, Fiddler will only tell you about this minor update if you manually check for new versions by clicking Help > Check for Updates. When Fiddler 4.6.1.0 becomes available, the updater detects this larger change in version number and prompts you to upgrade.

If you would prefer Fiddler to notify you of every update automatically, use the black QuickExec box below Fiddler’s Web Sessions list to enter the following command:

    prefs set fiddler.updater.BleedingEdge true

With this preference set, you’ll see more frequent notice of upgrades. On one hand, that’s great—you’ll get the latest Fiddler improvements ahead of most other people. On the other hand, as a bleeding edge user, you’re also more likely to uncover any bugs we inadvertently introduce in new versions.

Fiddler Improvement Program

You may now opt-in to sending telemetry information about your PC environment and Fiddler usage. Within your first few boots of Fiddler, you’ll see the following dialog:

image

Telemetry data is reported over HTTPS and its usage is governed by Telerik’s Privacy Policy. If you later change your mind, you can control your participation using the checkbox Participate in the Fiddler Improvement Program on the Tools > Fiddler Options > General tab. Note: If an administrator has set the BlockUpdateCheck registry key in the HKLM registry hive, users cannot opt-in to the Fiddler Improvement Program.

The Telerik Analytics integration into Fiddler has already yielded several bugfixes and has helped us prioritize our investments in performance and feature improvements. We’ll write more about what we’ve learned from Fiddler Telemetry in a future blog post.

Performance Improvements

One early finding from our Fiddler Telemetry is that a surprising number of users are running on 32bit Windows, where the address space limitations mean that they often run into “out of memory” errors.  To help mitigate this, we’ve made some under-the-hood changes to how Fiddler allocates memory, and this is only the beginning of a larger project to improve Fiddler’s overall performance for everyone.

As a part of our performance investigations, two new commands were added to the QuickExec box, !memory and !threads. Invoke these commands to add information to the Log tab for troubleshooting purposes. If you find Fiddler is running more slowly than expected, sending the output of these commands to us will help narrow down the problem.

 

We hope you enjoy the new version of Fiddler!

-Eric Lawrence

API Testing with Telerik Fiddler

Fiddler has long been the tool of choice for developers and testers who are building and verifying APIs exposed over HTTP(S). In this post, we’ll explore the existing features Fiddler offers for API Testing and announce new capabilities we’ve released in Fiddler 2.6/4.6.

Composing API Calls

The Composer tab enables the authoring of arbitrary HTTP(S) requests using any HTTP method, url, headers, and body, and the many Inspectors permit examination of responses of any type, including XML, JSON, BSON, and myriad other formats. Over the last few years, the power of the Composer has grown, and it now offers many conveniences, including a Request History pane:

image

… to allow simple reuse of existing requests. The powerful Scratchpad tab allows you to easily display a list of requests and execute those requests as needed—a great feature for live demonstrations of your API for tech talks and the like:

image

As you can see in the screenshot above, the Scratchpad even supports executing requests expressed curl syntax, so you can easily exercise APIs that are documented in terms of the parameters passed to that popular tool.

 

Capturing API Calls

Of course, Fiddler is best known as a proxy debugger, and its ability to capture traffic from nearly any source means that the fastest way to generate API tests is to simply capture the API traffic from an existing client and modify and augment those tests as desired. You can start with a baseline of the calls your clients presently send, and add new tests to probe for performance, security, error handling, and other problems.

One of the most powerful capabilities Fiddler offers is capture of traffic from almost any device (iOS, Android, Windows, Mac, etc); you can easily exercise your mobile clients’ API endpoints without adding any new software to the device itself!

 

Resending Requests from the Web Sessions List

After you’ve collected a set of requests in Fiddler, you can easily save them to a Session Archive Zip (.saz) file for convenient reuse at any time. To resend the traffic, just load the SAZ file, select the desired requests, and use the Replay submenu on the Web Sessions list’s context menu to resend the requests. Common choices include:

  • Reissue Requests – Resend the selected requests as quickly as possible
  • Reissue Sequentially – Resend the selected requests, waiting for a response to each before sending the next
  • Reissue and Verify – Resend the selected requests, verifying each response’s status code and body match the original response

Of these options, the last is perhaps the most interesting—you can very easily regression test an API set by simply collecting “good” traffic in a SAZ file and then using that traffic as a baseline against which later verifications will be conducted. The verification performed is rather crude, which is why we’re excited to announce…

Automated API Testing

The new APITest extension to Fiddler greatly enhances its power to validate the behavior of web APIs. Click View > Tabs > APITest to show the APITest tab. The bulk of the tab displays a Test List, a list of API requests and Validators.

image

Tests can be added to the list by simply dragging and dropping Sessions from Fiddler’s Web Sessions list. You can then use the context menu to specify a Validator and optionally mark it with a Note (comment) and optionally assign it to a group.

Validators

Validators offer a lightweight way to judge the success or failure of a test, and support many of the same operators used by Fiddler’s AutoResponder feature. As seen in the screenshot above:

  • EXACT:This is a Test – Ensure the status code matches and response body contains (case-sensitively) the text This is a Test 
  • Updated via API – Ensure the status code matches and response body contains (case-insensitively) the text Updated via API
  • {CODE} – Only ensure the response status code matches the original status, e.g. 201 for the first test
  • {BODY} (Not pictured above) Ensure the response status code matches and the response body is identical 

In addition to the EXACT: operator prefix, the REGEX: and NOT: prefixes are also supported for text searches:

image

The HEADER: prefix allows you to ensure that the response bears a specified header containing a given value (e.g. HEADER:Content-Encoding=gzip ensures that the response has a Content-Encoding header showing that the response used GZIP compression).

The SCRIPT: prefix allows you to specify a FiddlerScript function in your CustomRules.js file to use to evaluate the response; the named function is passed both the test Session and the baseline (original) Session and can evaluate any criteria you like to return whether or not the test has passed. Beyond examining the headers and body of the response, you could, for instance, evaluate the values in the Session’s Timers object and fail the test if the response took more than 100 milliseconds.

  // This method is called by "SCRIPT:CheckTest" validators; it
  // returns TRUE if the Test passes, and false otherwise.
  static function CheckTest(oRun: Session, oBase: Session): boolean
  {
    // Simplest possible validator
    if (oBase.responseCode != oRun.responseCode)
    {
        oBase["api-lastfailreason"] = "Mismatched status code...";
        return false;
    }

    var timeTotal = 0;
    timeTotal = (oRun.Timers.ServerDoneResponse - oRun.Timers.ClientDoneRequest).TotalMilliseconds;
    if (timeTotal > 100)
    {
        oBase["api-lastfailreason"] = "Performance Too slow...";
        return false;
    }


    return true;   
  }

Script Events

Before your Test List is run, the BeforeTestList method (if present in your FiddlerScript) is run, permitting you to adjust the requests as needed:

  static function BeforeTestList(arrSess: Session[]): boolean
  {
    // In this method, you can do any setup you need for the test,
    // e.g. adding an Authorization: token OAUTH value that you
    // obtained programmatically...
    var sOAUTHToken = obtainToken();
    if (String.IsNullOrEmpty(sOAUTHToken)) return false;

    for (var i: int=0; i<arrSess.Length; i++)
    {
      arrSess[i].oRequest["Authorization"] =  sOAUTHToken;
    }
   
    MessageBox.Show("Token Set. Running " + arrSess.Length.ToString() +
                    " tests.", "BeforeTestList");

    return true; // Test should proceed; return false to cancel
  }

After all of the individual test cases are executed, the AfterTestList method allows you to validate any post-conditions, log the results of the Test list, or perform other operations. The method is passed a TestListResults object which contains an enumerable list of TestResult objects. Each result contains the baseline (original) Session, the test Session, and an IsPassing boolean indicating whether the test passed. The WriteFailuresToSAZ convenience method will write all failing TestResults to a SAZ file for later analysis.

  static function AfterTestList(listResults: TestListResults)
  {
    for (var oResult in listResults)
    {
      FiddlerApplication.Log.LogString(oResult);
    }

    listResults.WriteFailuresToSAZ("C:\\temp\\lastfailure.saz");
  }

 

Working with Test Lists

A Test List is simply a set of Sessions each of which contains several custom Session Flags (api-testitem, api-Validator, api-LastResult, and api-LastFailReason). As a consequence, these lists can be saved and loaded as regular SAZ files; their “Test List” functionality only lights up when loaded into the APITest tab using its context menu. The menu commands are largely self-explanatory:

image

  • Run Selected Tests runs only those tests that are currently selected in the UI
  • Rerun Failed Tests runs all tests that are marked as failing
  • Set Comment… sets the Notes column for the selected tests
  • Set Validator… assigns the validation criteria for the selected tests
  • Set Group… assigns the tests to a UI group, useful for organizing your test list
  • Inspect baseline… opens the original Session in Inspectors for viewing or editing
  • Promote moves the selected test earlier in the run order
  • Demote moves the selected test later in the run order
  • Clone creates a copy of the current test, useful when you need to use multiple validators
  • Save Test List saves the test list to a SAZ file
  • Load Test List adds the tests from a SAZ file to the current test list

You can temporarily disable one or more tests from running by pressing the spacebar with the desired tests selected.

FiddlerCore

While the new APITest extension is offers powerful functionality, many larger enterprises instead choose to use the FiddlerCore .NET Class Library to build their API Testing infrastructure. FiddlerCore allows you to fully customize the behavior of your testing logic, driving the test using the .NET Language of your choice (typically C#). Because FiddlerCore does not utilize any of Fiddler’s UI, it can easily be integrated into existing test automation suites and can be used in console and service applications.

Load Testing APIs

The Fiddler-native Session Archive Zip file format is supported by Telerik Test Studio’s Load Testing product, allowing you to simply import a SAZ file that exercises your API set, configure the number of virtual users and run time, and generate load instantly or on the schedule you set. If you’d like to apply validation logic while your server is under load, run the Load Test in Test Studio and in parallel execute your Test List in Fiddler.

 

We hope you find Fiddler’s API Testing support useful, and we look forward to your feedback as we continue to enhance the tool.

 

Eric Lawrence
Telerik

Attribution Error

In life, you sometimes encounter people with “high standards”—folks who often find others’ behavior lacking in some way. Such people usually explain: “Sure, I have high standards… but I hold myself to an even higher standard!”

Except… they rarely do.

The problem is that, as humans, we’re subject to both fundamental attribution error and actor-observer bias. These phenomenon mean that we attribute our own good behaviors as intrinsic demonstrations of our good character, and we explain away bad behaviors as a temporary consequence of a given situation. When observing others, however, we tend to do the opposite—good behavior is dismissed as situational and bad behavior is deemed evidence of poor character.

These errors are made every day, at the small scale of our personal relationships to the global stage of policy-making. They’re hard to avoid, especially if you’re not consciously aware of them and don’t actively strive to consider events from different perspective.

Tuning MemoryStream

By day, I build the Fiddler Web Debugger. I’ve recently started integrating telemetry collection into the application for automated exception reporting and to collect information about the user’s environment to ensure that Fiddler testing environments match real-world usage.

The data is fascinating (and I’ll probably blog more about it later), but one data point in particular jumped out at me:

23.09% of users on 32bit

Early data suggests that nearly a quarter of Fiddler users are using the tool under 32bit Windows. This number is significantly higher than I expected and it’s worrisome. Why?

Because well over half of the exceptions reported via telemetry are System.OutOfMemoryException, and 100% of those are from users running 32bit. Now, given the exception’s name, most users assume this means that the system has “run out of RAM”, but this isn’t really the case. In virtually all instances, this exception would more correctly be named System.InsufficientContiguousAddressSpace – the user has plenty of memory available (both RAM and virtual memory backed by the swap file), but that memory is fragmented such that a large allocation (of, say, 100megabytes) cannot be fit within a single address range of the 2 billion addresses available:

Fragmented Memory

If that 100mb allocation were split into two smaller chunks of 50 megabytes each, it could easily fit into the available address space, and you’d never hit the OutOfMemory exception.

When Fiddler is running in a 64bit process, this error is almost never seen because the address space is so overwhelmingly huge (8 terabytes, 128tb on Win8.1+) that fragmentation isn’t a problem like it is in the 2GB address space.

So, what can we do?

The “simple” approach is to just throw the /3GB switch so that the system assigns up to 3GB of address space to the application, but this requires that the user manually adjust their overall operating system configuration, and it doesn’t buy us very much additional space.

The better approach is to stop using contiguous memory and start using data structures that keep portions of the data in reasonably-sized chunks of only a few megabytes each. This is arguably the “right” fix, but it can be considerably more complex, and it would be a major breaking change for Fiddler because Fiddler exposes HTTP request and response bodies as byte[] requestBodyBytes and byte[] responseBodyBytes. Even if we were to change that, popular APIs like GetResponseBodyAsString would end up making contiguous allocations anyway.

Users can use rules to stream large requests and responses without Fiddler keeping a copy, but this limits a lot of Fiddler’s power—streamed and forgotten traffic can’t be modified and isn’t available for inspection. As a consequence, the “stream and forget” approach is best reserved for bodies over 2 gigabytes (since .NET, even in 64bit, cannot use more than 2gb in a single allocation).

So are we out of luck?

No, not quite.

Before creating the byte arrays, Fiddler has to collect all of the traffic. It does so using a MemoryStream because Fiddler can’t know ahead of time how big the request or response will be.

MemoryStreams are tremendously useful objects, but they’re really just a bit of syntactic sugar around a backing array that resizes itself when you attempt to write more data to it than it can hold. The Write method checks the size of the write operation and if space is needed, it calls a private EnsureCapacity method to grow the backing array. Let’s have a look at this method, trimmed for clarity:

image

The first thing to notice is the code in the orange box; the MemoryStream has an optimization that says: “If the required capacity is less than double the current capacity, then double the current capacity.” This optimization helps ensure that if you’re writing small amounts of data to the MemoryStream over and over, it doesn’t cause the MemoryStream to reallocate the byte array over and over. Reallocations are especially expensive because the data from the old backing array must be copied to the new backing array upon each allocation.

However, this optimization quickly becomes problematic for 32bit Fiddler. Say we’re reading a large file download of 75 megabytes, reading at 15kb per second. The MemoryStream starts out at 16kb, then quickly grows to 32kb, 64kb, 128kb, 256kb, 512kb, 1mb, 2mb, 4mb, 8mb, 16mb, 32mb, 64mb, and finally ends at 128mb. When the download completes, before Fiddler can generate the 75mb responseBodyBytes array, the MemoryStream has tried to allocate 128 contiguous megabytes, an allocation that is 53 megabytes larger than we truly need. Even if we have a 75 megabyte slot available in memory, we need a 128 megabyte slot for the download to succeed. You’ll also notice that the MemoryStream makes no attempt to catch an OutOfMemoryException error and allocate only what is needed if the double-size allocation fails.

Not good.

Also, have a look at the code in the blue box, which is new for .NET 4.6. Prior to the addition of this code, once more than 1GB of data was written to the MemoryStream, any additional writes to the stream would only grow the MemoryStream to the exact size needed. That was awful for Fiddler performance because every single read from the network would result in reallocating the entire array, from 1.000000gb to 1.000064gb to 1.000128gb to 1.000192gb to 1.000256gb, etc. That performance problem has been fixed, but it introduces a massive overallocation—when a MemoryStream with 1GB of data is reallocated, it automatically jumps to just under 2 gigabytes of data capacity—potentially wasting up to nearly a gigabyte of memory. That’s not awesome, even on 64bit.

So, what to do?

While we can’t fix all of these problems, we can improve the situation over the stock MemoryStream because we know a little bit more about how the caller will be using it than the .NET Framework designers could assume.

We replace MemoryStream with a new class which descends from MemoryStream called PipeReadBuffer. The PipeReadBuffer has an overridden Write method that evaluates the bitness of the process, the size of the current response, and, if available, a “hint” about the final body size. The growth algorithm is enhanced such that:

  1. Capacity avoids growing past 85kb as long as possible, to keep the allocation off the large object heap.
  2. After that, Capacity grows quickly at small sizes, but growth caps at 16mb (on 32bit) to 64mb (on 64bit) at large sizes.
  3. The caller can send the PipeReadBuffer a “Hint” about the expected final capacity based on the Content-Length header, if any.

With these optimizations, the PipeReadBuffer can help ensure that waste is minimized in allocations and the risk of OutOfMemoryExceptions is reduced as much as possible.

Elsewhere in the code, we found great opportunities to tune MemoryStream’s growth. For instance, we have a method called GZipExpand that decompresses data that was compressed using gzip. The API we use unzips into a MemoryStream and previously was subject to the same overallocation problem seen in the network reads. Fortunately, the gzip format includes a hint about the size of the uncompressed data in the footer, so we can use that to set the capacity of our MemoryStream ahead of time:

Trimmed GzipExpand source

You can also see that here we take advantage of another optimization—if the stream is sized appropriately, we can avoid the allocation inside the ToArray() call by using the GetBuffer() method instead.

These improvements will appear in the next release of Fiddler.

Further explorations: The Bing team has released a promising new RecyclableMemoryStream (announcement) which might be an even better fit for the PipeReadBuffer, as it can avoid allocations on the Large Object Heap; those allocations are particularly a problem for older versions of .NET that don’t offer any way to compact the LOH.

-Eric Lawrence

On Defaults

Yesterday, Chris Beard, the CEO of Mozilla, wrote an open letter to Microsoft complaining about Windows 10’s behavior related to default applications. Reactions were all over the board, but in my Twitter feed, at least, they mostly skewed against Mozilla. With the perspective of having been both inside and outside Microsoft, I feel compelled to say a few things.

First, Defaults Matter

Defaults matter. Regardless of its relative merits, the default application is disproportionally likely to get used versus any alternative, even if those alternatives are all free. When I bought my car, it had floor mats— they seem fine; I’m sure they’re not the best floor mats, but I’m going to devote exactly zero thought to replacing them unless they fail me in some way.

It’s no accident that the Edge icon is a little blue “E”—that was the default browser icon for so many years that changing it would’ve been very foolish.

The most compelling argument for the fact that defaults matter is how much gets spent trying to influence them. Most obviously, Microsoft has spent billions in legal settlements and fines related to the default applications on Windows. “But,” you might argue, “that isn’t relevant because they didn’t choose to spend that money.” That’s true, but look at the budget that companies spend to get their software and services distributed and made default. Yahoo, Google, and Microsoft have paid organizations like Mozilla huge sums (on the order of $300,000,000 per year) for over a decade for the opportunity to be the default search engine in Firefox and other browsers.

By way of personal example, around 2010, I was approached with an offer of $1 per install to bundle the Bing Toolbar with Fiddler’s installer in order to increase the number of users defaulted to searching with Bing. I turned down this opportunity (worth somewhere between $300K to $3M per year) because I’m fundamentally opposed to bundling of this sort.

Want to learn more, or prefer your evidence in dead-tree form? Check out Nudge.

“It’s Easy To Change”

Even assuming the user does want to switch to a different browser, any friction in that process impedes change and makes a meaningful difference in adoption. This topic was deeply explored during the Microsoft-E.U. case that led to the browser ballot screen, and every tiny bit of friction was analyzed and negotiated at length. When we were redesigning how ClickOnce worked with IE in Windows 8, we were very constrained by being forbidden to add even a single click to the user flow by which Google Chrome was installed.

In Windows 8 and Windows 8.1, code was changed in Windows such that (well-behaved) applications could not automatically set themselves as the default handler. Instead, a prompt is shown:

image

In Windows 10, the same API call yields a very different result:

image

There are a number of significant problems with this. First, it’s not at all clear that this dialog is coming from Windows itself, not the caller, inviting the user to go on a wild goose chase around the app looking for some sort of “Settings” thing. More annoyingly, there’s no “Go there now” button—the computer should serve me, not the other way around. When Picard shouts “Computer: Shields up”, the Enterprise doesn’t reply with “To raise the shields, go to Ship > Controls > Security > Shields.” The omission of an entry point into the experience is glaring to the point of being offensive.

When you get to the “Default apps” controls, how do you change your browser to Chrome or Firefox? Perhaps amazingly, by clicking on Edge:

image

…only then do you see the alternatives:

image

Unfortunately, even if the user makes it all the way here, I think it’s likely to cause problems, and the reason is pretty subtle. In the Windows 8/8.1 UX, the “How do you want to handle this type of link” experience only assigns a “basket” of protocols (HTTP, HTTPS, and maybe FTP) to the selected application. In Windows 10, the “Choose an app” experience appears to give the target all of the protocols it requests. This is a problem, because Google Chrome lists itself as able to handle the mailto: protocol (used for opening email links). But unless you’ve followed some rather obscure steps, Chrome doesn’t actually do anything other than launch when you click a mailto link. So the user experience is bizarrely and confusingly broken.

Claims and Retorts

API Parity

In their blog post announcing the change, Microsoft makes the following (oddly phrased) argument for the change:

In Windows 8.1, Classic Windows applications (Win32) could invoke the prompt asking you to change your defaults, so you may have seen multiple prompts during install and after they launched. However, Windows Store apps could not invoke this prompt.
[…]
We know your defaults matter to you. With Windows 10, all apps – both Classic Windows apps and Universal Windows apps – will be unable to invoke a prompt to change your defaults, only Windows.

Some have leapt to the defense of the change noting that because Store apps couldn’t invoke this prompt, this means that Desktop APIs needed to change. I’d counter that there’s no reason why a similar API couldn’t be exposed for Store apps.

API Deprecation

In the same post, Microsoft notes that applications just need to update their code to avoid the new ugly and unactionable dialog. That’s true as far as it goes– Firefox and Chrome will probably update in their next builds, but this is a problem that could’ve been mitigated in the first place. We know from past experience that requiring applications to change is an adoption hindrance for a new operating system, and there are a long tail of applications which might take a long time to be updated.

What About ABUSE?

Some have argued that this change helps protect users from unwanted changes to their providers, the same argument made for the Windows 8 change. As far as I can tell, no evidence has been provided that the Windows 8 mechanism was ineffective.

“But what about ‘drive-by downloaded’ malware?” some have asked. The first law of computer security is still in effect: a malicious program running at Admin can do anything it likes, including circumventing the speed bumps introduced in Windows 10.

Is Microsoft Evil?

In my twelve years at Microsoft, I can’t recall any cases where anyone sat around twirling their mustache while planning some evil Machiavellian plot. Instead, the problem is that Microsoft too often fails to consider that they are a huge bull and the sole proprietor managing a tightly-packed consignment shop full of china. Any change, regardless of intent, can end up breaking a lot of nice things. For a time, this realization was formalized in the design tenet “Change is bad, unless it’s great.”

Is this Memo A PR Move for Mozilla?

Some commentators are noting that, since other operating systems are even more restrictive in terms of defaults, Mozilla is only complaining about Windows 10 to get attention from the press… as if this were somehow a clever line of argument.

Of course Mozilla is only releasing this complaint now to get attention!

You can’t build a great third-party browser for iOS because of Apple’s lockdowns, and both Google and Mozilla lament this. Mozilla is going so far as to tilt at the windmill of creating their own mobile OS in the hopes of having a shot at relevance in a mobile world. While I liken this to setting fire to a large pit of money and rate its chances of success in the neighborhood of zero, what choice does Mozilla have?

They’re at the mercy of Apple, Google, and Microsoft, companies whose market caps are measured in fractions-of-a-$Trillion USD. Failing to use their bully pulpit as the champion of openness to push back on lockout would represent severe negligence.

Why do I care?

I care because I want the best for Microsoft and I want the best for users. I think Edge is going to become a great browser, and it would be terrible if it were shackled with accusations of dirty tricks in the same way that IE4 and IE5 were, even if the “tricks” aren’t nearly as dirty this time around.

I care because I’m very worried about user-choice. I think Microsoft’s decision to make it much harder for humans to change the default search engine was an awful one and I hope they reverse course. As it stands, Microsoft’s browser share is standing on a precipice—because Edge is exclusive to Windows 10, its share has Windows 10 upgrade adoption shackled to its throat. If Windows 10 lags, Edge might well die with barely a whimper. Similarly, because Edge makes it so hard to switch away from Bing, users who like Edge but find Bing lacking are incentivized to switch to Chrome.

-Eric Lawrence

Stupid (useful) Windows Shell Tricks

RunAsUser

Back in the Windows Vista timeframe, the existing RunAs verb for ShellExecute was changed. Instead of prompting the user to run the target as a different user (e.g. the experience you get when you Shift-Right-Click on an app) it instead would treat the call as “Run the target as an Elevated Administrator”. Well, what if you wanted the old experience, where the user was prompted for an account to use?

It turns out there’s another (as far as I can tell, undocumented) verb, RunAsUser that has the old behavior.

This trick has been integrated in SlickRun 4.3 and you can use it by hitting Ctrl+Alt+Shift+Enter when executing a command. The existing Ctrl+Shift+Enter shortcut for “Run as Administrator” remains unchanged. You can also configure any given MagicWord to always prompt for a user-account to use via the new checkbox at the bottom:

MagicWord UI

 

Launching Unelevated

A longstanding problem with SlickRun was that the Setup program launched it when installation was complete, and because the Setup program runs as Administrator, SlickRun itself ran as Administrator. And when SlickRun runs at Administrator, everything it runs launches at Administrator as well. This has horrific consequences for things like Internet Explorer, which disables its sandbox when run as Administrator. It’s just awful.

So, you’d expect that ShellExecute would’ve been updated to offer some sort of RunUnelevated verb that enabled an elevated application to run a target application without elevation. Sadly, it doesn’t; the Windows team considered this a really hard problem (the corner cases are ugly) so they punted on solving it. The problem with this approach, of course, is that leaves solving the problem to the poor ISV developers, who have even less understanding of how to solve it, and thus they typically don’t even bother trying, leaving their users at risk.

Fortunately, a while ago I discovered an ugly hack that works everywhere I’ve tried it. A kind developer on the Shell team confirmed that the fact that this works is basically a bug and they could remove it at any time. But they haven’t yet, and compared to the (absurd) alternatives, I think it’s the best option for most of us.

The trick is simple—just execute Explorer and pass in the path to your application. In a NullSoft Installer Script, it looks like this:

    exec '"$WINDIR\explorer.exe" "$INSTDIR\sr.exe"'

There are, of course, corner cases where this could fail (e.g. if the user isn’t using Explorer as their Windows shell) but in the last few years I’ve been using this trick there have been a total of zero complaints.

The famous Raymond Chen has provided a more robust approach that asks Explorer to launch your app for you, depending on COM rather than the “bug”, but that approach is only feasible if you can call arbitrary native code, which is rather complicated for NSIS scripts.

 

-Eric Lawrence

An A+ HTTPS site for $20

After the CEO of an Internet Security company “explained” that it didn’t matter that his company website was only accessible using insecure HTTP (“it’s only marketing pages so we don’t need security”), I decided to build out a new website: https://WhyTLS.com. Here, I’ll be making my case that all websites need to move to HTTPS, and providing links and resources to help site owners do so.

Naturally, I needed to use HTTPS for my site, but my current host already has a certificate for a different hostname, and changing the site to use a multi-domain certificate would be an expensive hassle. Fortunately, I have recently started migrating my domain registrations over to Namecheap, and it turns out that they offer a great deal on their first year of hosting and HTTPS; adding these to my order was as simple as clicking two buttons.

image

So, for under $20, I’m now the proud owner of a new HTTPS-secured site.

When I bought a certificate for my old site, it was a bit of a pain—I had to create an account with a Certificate Authority and do some complicated dances to prove my ownership of the domain. I then had to give the hosting company $20 to “install” my new certificate on the domain, and pay them the same amount each time I renewed to a new certificate.

Fortunately, Namecheap’s integrated process was much simpler—the form to obtain the certificate was on Namecheap’s Control Panel, and was completely pre-filled out except for “Job Title”, “Company” and “Phone Number” fields (the CA wanted these). With the click of two buttons and a wait of about 10 minutes, I got email telling me that my certificate was ready to install. I clicked “Activate” in the Control Panel and my HTTPS site was live!

I immediately headed over to SSLLabs.com’s Server Test to see how the security of my site measures up. It got an “A”, beating the “B” my other site gets (that one is hindered by running on Windows Server 2008, which only supports older ciphers). Now, an A is good, but I want an A+. That’s easy—I just need to add a HTTP Strict Transport Security (HSTS) policy to combat SSLStripping attacks.

Using my editor, I created /www/.htaccess and added the following lines:

RewriteEngine on
# force ssl
RewriteCond     %{SERVER_PORT} ^80$
RewriteRule     ^(.*)$
https://%{SERVER_NAME}%{REQUEST_URI} [R=301,L]
# Send HSTS policy
Header set Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" env=HTTPS

I saved the file, and here we go– A+ HTTPS configuration, with minimal hassle, for under $20.

Now, is it perfect?

No. Let’s take a closer look at the SSLLabs report.

SNI Required

At the top, there’s a small banner: image. I’ve written about SNI before but SSLLabs’ “Handshake Simulation” report shows exactly what this means: My site will show a certificate error on Android 2.3, IE on Windows XP, and with Java 6, none of which support SNI.

The SNI TLS handshake extension allows a web hosting provider to cut costs by hosting multiple unrelated sites at a single IP address; without receiving the extension from the client, the server doesn’t know which certificate to return.

Fortunately for me, these platforms are fading in importance and I have the luxury of ignoring them. Or so I thought. I later tried to set up WebDAV support on this server so I could use Windows Explorer to manage its files and I found the SNI extension was not getting sent by the SvcHost process:

This bug was supposedly fixed in Windows 8.1, but my results here show otherwise; the problem was fixed in Windows 10 and Microsoft is looking at bringing the fix downlevel.

Extra Certificates

The configuration contains one other minor problem – it sends one certificate more than necessary in the certificate chain sent when the client connects:

The “Contains anchor” notice means that the server sent to the client the root Certificate Authority certificate:

This is an small waste of precious bandwidth, because either the client already has this certificate in its Trusted Root store, or the connection will fail anyway (because a client isn’t going to start trusting the root just because the server sent it). Note: There are some obscure cases (related to Extended Validation EV certificates) where sending a root may be useful to help a client recognize the certificate should get the “green bar” EV UI treatment, but those doesn’t apply here.

I’m excited to see that deployment of HTTPS is getting easier with each passing month, and I’m very much looking forward to the launch of the Let’s Encrypt project (certificates free of hassle and cost) later this year.

-Eric Lawrence