text/plain

Ad Publishers–A TODO List

Where’s Google’s* blog on how they’re doing everything they can to make ads they serve as fast and small as possible?

Where’s Google’s blog on how many ads they’ve nuked as “deceptive” and trumpeting how policy forbids ads for “adware-wrapped” installers?

Where’s Google’s blog about how many billions of ad-generated dollars they’ve supplied to content sites and open-source products that people care about like Firefox?

Where’s Google’s blog on how much smaller they’ve made ads served using WebP instead of older formats? And Zopflinated PNGs for legacy browsers?

Ad publishers can’t expect a company like EmbarcaderoTech to know how to make fast ads. Publishers need to help.

Where’s the argument that the worst privacy impact of ads comes from trying to recover revenue lost through blocks and fraud?

Without good answers to these questions, ad publishers are going to have a very hard time regaining any control of the narrative. The entire industry is demonized for performance, security, and privacy problems, even though each publisher has different practices.

-Eric

*Note: Google does plenty of things right from an advertising point-of-view; I only mention them by name because they’re probably the biggest and I expect the best technology from them. They’ve invented much of the interesting technology in this space, including Zopfli and WebP.

Ad-Block: Current Status

Microsoft used to joke about cutting off a competitor’s air supply. Apple instead handed shears to devs, pointed at a hose, and walked out.

As iOS9 launches, here’s the list of top paid apps in the iTunes AppStore:

AppStore - Top Paid Screenshot

It’s only a question of when, not if, the top slot of the Free Apps category is an open-source alternative that will save users $3-$4 and offer the same functionality.

I’m pretty sure the Web is at exactly this moment:

The next year or so is going to be very very messy.

Update (Sept-18-2015): After 36 hours, the maker of the top-grossing blocker bowed out of the market. This is unexpected windmill tilting. I love it.

Developer Advocacy

The Microsoft Edge (nee Internet Explorer) team held one of their “#AskMSEdge chats” on Twitter yesterday.

After watching the stream, @MarkXA neatly summarized the chat:

The folks over on WindowsCentral built out a larger summary of the tidbits of news that did get answered on the chat, some of which were just pointers to their Status and UserVoice sites.

After the chat ended, I complained that none of my questions had been answered:

In response, an IE Engineer retorted:

I don’t think that’s fair. Here are my questions, and a few remarks on each:

As far as I know, I’ve never asked the IE/Edge team about Brotli before, as I hadn’t spent any time looking at it until very recently. I’m interested in the team’s plans for Content-Encoding: Brotli because it can significantly improve browser performance, and if the team implements WOFF2, they must integrate Brotli decoding logic anyway.

I don’t think I’ve ever asked the IE/Edge team about their plans here before. More efficient HTTPS algorithms are important for both performance and battery life on mobile devices in particular, and thus I think they’re a great investment.

I have asked this before. IE has had non-standard network export for four years and I was really excited that Edge moved from HTTP Archive XML to the standard HTTP Archive JSON format. Unfortunately, this bug makes their code non-interoperable. The fix will be one or two lines of code. I feel justified in asking for status since weeks or months have passed without update.

I have asked this before. Edge regressed a significant piece of functionality and created a denial-of-service condition in their browser. I feel justified in asking for status since weeks or months have passed without update.

I have asked this one over and over again. I find it galling that Microsoft products are less secure together, and especially when Microsoft’s new President promised to close these sort of gaps nearly two years ago. It’s clear that the team agrees that the behavior is bad, because Edge uses Bing securely and doesn’t even allow users to add non-HTTPS search providers.

This one is probably the least “fair” of the questions, insofar as I already know the answer and I’m effectively just calling the team out on the specious nature of the promise to “watch demand” they made when the original concerns about the absence of Windows 7 support were raised.

However, I’ll note that the team answered several repeats of the question “When will it run on Mac? When will it run on iOS and Android.” Given the Windows 7 marketshare dominance, I think this question remains fair.

Other Unanswered

Other folks asked several great questions that didn’t get answered:

I really want this feature.

I think the Edge team is making a huge mistake if they’re not piloting their new extension model with critical extension developers like uBlock, NoScript, etc.

Because of the nature of the legacy Win32 Address Bar’s context menu, Paste-and-Go was always prohibitively expensive. The UI replacement for Metro IE and now Edge makes this a trivially added feature that was requested by several questions.

Unwanted Answers

Some questions got answers that I’m just not happy with, but I’m tired of complaining about:

The Edge team replied “No” and suggested they consider this a scenario for the new extension model. I think this is a mistake and a case where “different” isn’t likely to be “better.”

Several folks asked when the new extension model would be released. “Stay tuned” was the answer.

Subtext

I think the subtext of Adrian’s complaint is that “You’ve worked here, you know we don’t announce things on IEDevChats.” There’s some truth to that frustration – I know that announcements are carefully vetted and published on the blog and I understand why live chats aren’t a source of new information. However…

Rabble-Rousing and Information Asymmetry

I know some folks think my questions are just rabble-rousing and that, as an ex-teammate and current MVP I should be asking these questions in private, directly to the IE team. A few points on that:

Microsoft has basically requested we provide feedback in this manner, with the “we’re watching feedback to influence our decisions” position on everything from features to bugfixes. Feedback that isn’t getting public traction is largely ignored.
A significant number of my friends and colleagues are no longer on the IE team. In the photo of the team answering questions, I recognize seven of the seventeen engineers.
My emails to the IE MVP discussion list generally do not receive replies.
Direct emails to individual engineers on the IE team often do not get replies.
Status on bugs I’ve filed with MSRC is similarly hard to get– issues have languished for months without so much as a “working on it” status update.

Having been on both sides of the fence now, it’s plain to me that one serious problem Microsoft has is that they don’t realize how incredibly opaque things are from outside the company. As an engineer racing from one issue to the next, it’s easy to deprioritize status updates and justify doing so when there are so many higher-priority things to fix. From outside the company, however, “working on it and coming as soon as we can” is often indistinguishable from “ignoring—really hope this goes away.” That problem is exacerbated by Microsoft’s tendency not to deliver hard messages like “Silverlight is dead dead dead, get off it now!” in a timely manner to allow customers and partners to plan appropriately.

Brotli

2022 Update: Brotli is requested by 94% of browsers, offers great performance, and works amazingly well on Web Assembly code. If you’re still using GZIP today, you should update!

Regular readers of my blog know how much I love Zopfli, Google’s compression engine that often shrinks output by 5% or better when compared to the popular zlib engine. The beauty of Zopfli is that its output is compatible with all of the billions of existing DEFLATE encoders deployed worldwide, making its use an easy choice for any static content.

But imagine for a moment what compression ratios we could achieve if we weren’t limited by compatibility with existing decoders? If we could add a new compression engine to the web, what might it look like?

The Brotli compression engine, co-written by Jyrki Alakuijala (inventor of Zopfli), provides one answer. Brotli combines the LZ77 and Huffman algorithms of DEFLATE with a larger sliding window (up to 16mb¹ vs. DEFLATE’s 32kb) and context modeling; the specification also calls for a 122kb static dictionary.

Brotli In Browsers

Today, Brotli is the compression engine behind the newish WOFF2 font format, providing savings of approximately 25% over WOFF 1.0 fonts compressed with Zopfli. Not content to rest on their laurels, Google has announced their Intent to Implement Brotli as a general purpose HTTP Content-Encoding, allowing web developers to use it to compress script, stylesheets, svg, xml, and the like. Firefox beat Google to the finish and shipped Brotli support in the Firefox 44 Dev build.

Probably HTTPS only

Past attempts to add new compression algorithms (bzip2 and SDCH) have demonstrated that a non-trivial number of intermediaries (proxies, gateway scanners) fail when Content-Encodings other than GZIP and DEFLATE are specified, so Brotli will probably only be supported over HTTPS connections, where intermediaries are less likely to interfere.

Accept-Encoding: br, gzip, deflate, sdch

Results

Facebook investigated Brotli and found it would save about 17% of CSS bytes and 20% of JavaScript bytes (compared with Zopfli). When run on the CSS and JavaScript from the Alexa top-300k, Brotli saved 12% of CSS bytes and 9% of JavaScript bytes when compared to gzip.

Running a few simple tests with Fiddler, I saw great results with Brotli:

Content-Encoding: br

jQueryMobileMin.js

Microsoft homepage:

A random giant XML documentation file:

Microsoft Word Online WordEditor.js

Microsoft Word Online WordEditor.Wac.TellMeModel.js

Cloudflare’s blog post on Brotli includes some benchmarks too.

Brotli is optimized for decompression speed. When compressing, Brotli is slower than zlib’s deflate, but considerably faster than zopfli, lzma and bzip2; given 1gb of extremely compressible content, Brotli finished compressing it to 3339 bytes after 301 seconds of CPU time. After 8040 seconds of CPU time, zopfli.exe crashed when a memory allocation failed.

Running Brotli.exe

To make things simpler for Windows users, I’ve built the latest release (v0.3) from GitHub for Win32 using Visual Studio 2015. You can download the Authenticode-signed Windows Brotli.exe from my site.

To compress a file, specify the input and output filenames:

--in filename
--out compressed_filename

… and optionally specify any of the following arguments:

--quality n
--force
--verbose

The quality parameter controls the compression-speed vs. compression-ratio tradeoff; the higher the quality, the slower but denser the compression. The supported range is 0 to 11, and 11 is the default.

The force parameter instructs Brotli to overwrite the output file if it already exists.

The verbose parameter instructs Brotli to display its compression speed in megabytes per second upon completion.

To decompress a file, use the --decompress parameter and specify the input and output filenames:

--in compressed_filename
--out filename

… and optionally specify the --verbose parameter to instruct Brotli to display its decompression speed in megabytes per second upon completion.

If you’d like to expose Brotli inside Fiddler 4.6.0.5+, place brotli.exe inside Fiddler’s \Tools\ subfolder and restart to see it appear on the Transformer tab:

Tracking Brotli

If you’d like to follow along:

GitHub
Brotli on CanIUse.com
Chrome Bugs: 452335, 472009
Firefox (shipped in FF44): 366559
Status.Modern.IE #291

Alas, the Brotli Discussion forum is currently empty.

Assorted Further Investigations

1. Someone needs to register the brotli token in the IANA registry (although Google’s SDCH and Microsoft’s Xpress aren’t listed there either).

2. Implementers should consider protections against “brotli bombing” DoS attacks. Brotli’s high compression ratio makes attacks even cheaper for the bad guys. A trivial test of compressing a file containing all 0s shows that Brotli can achieve a compression ratio of at least 386516:1, meaning that 1389 bytes of compressed data can blow up to 512MB when uncompressed. In contrast, DEFLATE has a maximum compression ratio approaching 1032 to 1, so an attacker would need to send 375 times as much data over the network to achieve a similar result. That being said, even DEFLATE can result in a denial-of-service, as with this 5.8mb PNG file that can require allocation of up to 141GB of memory.

3. Brotli’s use in WOFF2 means that browsers have already taken on its attack surface. However, not all attack surface is created equal; WOFF2 fonts can be decoded inside a very restricted sandbox. When Chrome 1.0 released, I was surprised to learn its HTTP decompressors were in a full-trust process; it turns out that is still the case today, which makes fuzzing against decompressors very interesting to an attacker.

4. Brotli’s static dictionary was generated from a broad corpus of content, but considering the most likely use cases (static files), it may not be optimal for this use. At this point, it’s probably too late to change it.

5. When used as a Content-Encoding, will brotli be used “bare” or in some framing format (e.g. with a trailing CRC and size marker)? Will it have magic bytes that will allow sniffing? (Per @mcmanusducksong, Firefox is going with a bare stream and no magics. boo)

6. While not terribly relevant to my scenarios, it turns out Google builds a lot of compression engines I’d never heard of, e.g. Snappy and Gipfeli. When compression speed is more important than ratio, they’re worth a look.

7. Brotli makes the most sense for pre-compression of static content; to that end, someone needs to xcopy the http_gzip_static module for nginx and make a few tweaks to create a new http_brotli_static module. While the nginx team may eventually release one, Google released a brotli module that supports both dynamic and static compression.

¹ While Brotli can use a 16mb window, for performance reasons it appears that constraining the window to 4mb is the plan for most scenarios.

Fiddler Certificate Generators

Fiddler and FiddlerCore offer three different choices for generating interception certificates:

MakeCert
CertEnroll
Bouncy Castle

If you’re so inclined, you can even write your own certificate generator (say, by wrapping OpenSSL) and expose it to Fiddler using the ICertificateProvider3 interface.

On Windows, Fiddler includes the MakeCert and CertEnroll certificate generators by default; you can download the Bouncy Castle Certificate Generator if you like. In contrast, when Fiddler is running on Linux and Mac, only the Bouncy Castle certificate generator is available, and it is included by default.

If you’re using Windows, however, you may wonder which Certificate Generator you should use in Fiddler or for your applications based on FiddlerCore.

In general, I recommend the Bouncy Castle generator, as it has better performance than the default MakeCert generator and it offers more configuration choices than the CertEnroll generator. Another advantage of the Bouncy Castle certificate generator is that the only certificate that (typically) goes in the Windows Certificate store is the root certificate. The server (end-entity) certificates generated for each website are kept in memory and discarded when Fiddler exits; because the Bouncy Castle generator reuses a single private key for all certificates by default, the performance impact of this behavior is minimal.

The only downside to the Bouncy Castle generator is its size: it is ~200KB when compressed, which is 25% larger than FiddlerCore itself.

The CertEnroll generator was added to Fiddler relatively recently; it offers better performance and standards-compliance than the legacy MakeCert generator but it is available only on Windows 7 and later. You can easily switch Fiddler to use CertEnroll inside Tools > Fiddler Options > HTTPS.

The MakeCert generator is the original certificate generator used by Fiddler and it remains the default on Windows today (mostly) for legacy compatibility reasons. It suffers from a number of shortcomings, including the fact that the certificates it generates are not compatible with iOS and (some) Android devices. It generates certificates with a 1024 bit RSA key (which may soon trigger warnings in some browsers) and each certificate has a unique key (meaning that each new secure site you visit triggers the somewhat costly key generation code).

Both the CertEnroll and MakeCert-based certificate generators must store all server certificates in the Windows Certificate store which some users may find confusing:

The storage of (potentially thousands of) server certificates in the user profile can also cause some problems for corporate users who have roaming user profiles, as these certificates are roamed to each workstation as the user logs in. To mitigate that, the Clear server certs on exit checkbox can be set inside the Tools > Fiddler Options > HTTPS > Certificate Provider dialog, or via:

FiddlerApplication.Prefs.SetBoolPref("fiddler.certmaker.CleanupServerCertsOnExit", true);

… however, the downside of doing that is that Fiddler must then re-create the server certificates every time it starts. This performance penalty is smaller when using CertEnroll, which reuses a single 2048-bit RSA key, than for MakeCert, which generates unique 1024-bit RSA keys for each site.

FiddlerCore Considerations

To determine which Certificate Generator is in use, be sure to attach the following event handlers:

Fiddler.FiddlerApplication.OnNotification +=
delegate(object sender, NotificationEventArgs oNEA) { Console.WriteLine(“** NotifyUser: ” + oNEA.NotifyString); };
Fiddler.FiddlerApplication.Log.OnLogString +=
delegate(object sender, LogEventArgs oLEA) { Console.WriteLine(“** LogString: ” + oLEA.LogString); };

You can then view information about the Certificate Generator in the console when it loads.

Developers building applications atop FiddlerCore should keep the following in mind when deciding which Certificate Generator to use:

MakeCert

MakeCert.exe is a Microsoft Visual Studio 2008 redistributable file, meaning that you’re licensed to redistribute it if you have an appropriate license to that version of Visual Studio. Microsoft may offer MakeCert.exe as a redistributable in other circumstances, but licensing is provided by Microsoft, not Telerik.
To use MakeCert.exe, you must include it adjacent to your application’s .exe file.
MakeCert-generated certificates are not compatible with iOS and some Android devices.
MakeCert-generated certificates “pollute” the user’s Certificate Store and you should consider offering a mechanism to clear them.

CertEnroll

The CertEnroll API is available on Windows 7 and later.
Use CertEnroll by either omitting makecert.exe from the application’s folder or by explicitly setting the preference:

FiddlerApplication.Prefs.SetBoolPref("fiddler.certmaker.PreferCertEnroll", true);

CertEnroll-generated certificates “pollute” the user’s Certificate Store and you should consider offering a mechanism to clear them.

Bouncy Castle

Bouncy Castle is an open-source PKI and crypto library distributed under the MIT license.
To use Bouncy Castle, you must include CertMaker.dll and BCMakeCert.dll adjacent to your application’s .exe file.
Bouncy Castle does not store certificates in the Windows Certificate Store (yay!) but this also means that your application needs to keep track of its root certificate and private key (unless you recreate and retrust it every time the application runs).
Two preferences are used to hold the key and certificate, fiddler.certmaker.bc.key and fiddler.certmaker.bc.cert. After you first call createRootCert, you should retrieve these preferences using FiddlerApplication.Prefs.GetStringPref and store them somewhere within your application’s settings (registry, XML, etc); the private key should be considered sensitive data and protected as such. When your application next runs, it should detect whether the key and certificate have already been created, and if so, they should be provided to the certificate generator using FiddlerApplication.Prefs.SetStringPref before any certificates are requested, lest you inadvertently create a new root certificate.

Rick Strahl wrote a great blog post on this process, including some sample code.

-Eric