Zopfli All The Things

I’ve written about Zopfli quite a bit in the past, and even wrote a tool to apply it to PNG files. For fun, I had a look at one of the most optimized pages in the world: Google.com, through the lens of Zopfli.

Here are the basic resources delivered by the Google homepage:

Zopfli WhatIf

This breakdown shows that Google isn’t optimizing their own compression using the compressor they wrote. The Savings column shows the number of bytes saved by using Zopfli over whatever Google used to compress the asset. Using the default settings in an ideal world, Google could save up to 16.5k, almost 5% of the bytes transferred, by using Zopfli.

I’ve color-coded the column based on how practical I believe the savings to be—the green numbers are the static images where there’s no question the size benefit could be realized. The yellow numbers are cases where script files are compressed; given the complicated query string parameters, I’m betting these scripts are dynamically generated and the compression cost of Zopfli might not be reasonable. The red number is the homepage itself, which probably isn’t reasonable to Zopfli compress as it certainly is generated dynamically.

So, most likely the savings of a practical Zopfli deployment on the homepage page would be about 3.7kb; savings are much greater on other pages on other sites.

More interesting, however, is the Google API CDN, which hosts scripts for other sites; optimizing these would take a minute or two at most and make every site that uses them faster.

Zopfli savings

Use Zopfli; give the tubes a little bit more room.

-Eric

PS: You may already have zopfli.exe on your system; Fiddler installs a copy to its \Tools\ subfolder!

What I Use–Software Edition

I’ll update this list from time-to-time.

Criteria

The #1 criteria for any software I use is first, do no harm. There’s a lot of great software out there that’s ruined by side-effects, including security problems, performance problems, advertising, and anything else that makes my computer worse for having it installed. In some cases, I’ve simply written my own software (usually uglier and with fewer features) because I’m not willing to compromise on this principle.

What I’m using

Fiddler (free) – For someone who doesn’t really build or test web applications for a living, I still find myriad uses for Fiddler, and I’m always adding more. Current boot count: 13,689.

Chrome (free) – I recently changed my default browser to Chrome on most of my computers. After years of suffering daily crashes in Internet Explorer (known to the IE team, but unfixed), I got tired of waiting for relief. I’m less pleased with Chrome than I hoped to be (their add-ons site is a cesspool of bugs and malware, just like IE’s) but the browser itself is great, and it’s clear that most web developers are building in Chrome first and only later testing in everything else.

Internet Explorer (free) – I use Internet Explorer because it works well with most of the sites I visit, it’s familiar (muscle memory built over a decade), and it supports TPLs, making for a more pleasant browsing experience.

Visual Studio 2013 (commercial) – While I gripe about Visual Studio a fair bit, I can’t imagine using anything else. (I still play with Delphi XE4 once in a while to remind myself how bad things could have gotten.)

SlickRun (free) – This powerful application launcher is one of the first GUI programs I ever built, and it’s now old enough to drink. I’ve modified it over the years to support the latest Microsoft OS’s and hardware (a 64-bit version is now available, for instance) and it remains the first thing I install on every new PC I use. Commands executed: 142,672.

MezerTools (free) – I wrote this simple Software Designer’s toolbox to quickly collect screen-snips, get pixel-perfect measurements (via calipers), and collect color information. You can also quickly convert to/from hex and interact with clipboard text.

Windows Live Writer 2012 (free) – It’s buggy, but better than web-based editors. ~~This tool is on-track to be open-sourced, per Scott Hanselman~~. Now open-source (minus a few features) as OpenLiveWriter.

Windows Live Mail (free) – No-frills email software with solid integration to Hotmail/WindowsLiveMail/Outlook.com/WhateverItIsCalledThisWeek.

File Locator Pro (trialware, freeware) – Windows has flailed around for almost twenty years trying to create a working file search experience. File Locator Pro (and its free cousin, Agent Ransack) neatly fill the gap with powerful search.

EditPad Pro (trialware, freeware) – My favorite text editor offers high-performance (even on obscenely large files), syntax highlighting, a great hex mode, and much more. A freeware version (EditPad Lite) is available, but this software is worth buying. I originally thought that its support for FTP/FTPS was utterly ridiculous “feature bloat”. Then my ISP stopped working with Expression Web (FPSE fell out of support with Win2k3’s retirement) so EditPad has become my primary web authoring tool.

Camtasia (commercial) – The industry-leading screen recording software. It has more features than I’ll ever use, and it’s not cheap. But if I had it to do all over again, I’d buy Camtasia immediately and save myself the hours of wasted effort trying to get lesser software to work.

VLC (freeware) – This media player seems to be able to play back everything I throw at it, and gets updated as new formats arise.

Microsoft Word 2010 (commercial) – I wrote my book using Word 2010 and it worked much better than anything else I tried (more on this in a later post). I tried Office 2013 and uninstalled it quickly—beyond the confusingly “extra flat” user-interface, the later version of Office couldn’t handle my book without slowing to a crawl (“background save” locks the UI for 5-15 seconds).

Paint.NET (freeware, be careful) – When Microsoft Paint can’t do the job, I turn to Paint.NET, a powerful alternative. Warning: Be sure to click the right download link, there are many misleading advertisements on the download page. Also, note that it does a terrible job encoding PNG files, so be sure to recompress them.

Axialis IconWorkshop (trialware) – When I need to build icons, this tool takes the pain away.

Start8 (trialware) – Makes Windows 8 and Windows 8.1 bearable.

7-zip (freeware) – Archive compression and decompression software.

Day One (Mac, commercial) – Journaling software, with the right mix of beauty and power.

-Eric

Google Search Provider in Microsoft Edge

Back in the IE7 days, I built a simple Search Provider Builder that allowed IE users (and later users of other browsers) to add custom search engines to their browser without any changes to the site. Trivia: This hour-long little prototype soon led to a formal effort to put this tool on the IEAddons site; the PM for that project was a new hire who eventually married me. :-)

Microsoft Edge has decided to change course and break the AddSearchProvider API used to add custom search providers based on user-initiated actions. The API works in IE7-IE11, Firefox, and Chrome, but not in Edge. Instead, search providers can only be “discovered” by sites that advertise them. (For the avoidance of doubt, let me say explicitly that I think this is terrible; if you agree, vote here).

For now, you can workaround the Edge browser limitation by visiting this page: Install Edge Search Providers for Google, DuckDuckGo, Wikipedia, and Amazon.

Two other changes were made as a part of the Edge search changes:

Search provider URLs must be HTTPS (yay!)
Search providers may not provide Search Suggestions. All Search Suggestions now come from Bing (boo!) over HTTPS (yay)

-Eric Lawrence

Update: Feb 6, 2017 — This post is still accurate for the very latest Microsoft Edge Insider’s Build 15025.

What I Read–Book Edition

This is a list of books I’ve read recently, with a Twitter-fitting review for each. I’ll update it periodically.

Fiction

The Martian – I greatly enjoyed this book; I was planning to try to get it some attention, but just before I tweeted, I learned it’s about to be a major motion picture. Oops. :-)

Wool – Great dystopian sci-fi. The writer is the closest thing the self-publishing industry has to an evangelist, and he’s awesome at it.

Ready Player One: A Novel – a light, fun read; I loved it.

Mr. Penumbra’s 24-Hour Bookstore – Fun and odd.

Seveneves: A Novel – I love Stephenson’s earlier work, and some of his later work (e.g. Reamde). This one was a mixed bag—it managed to reduce the magic of spaceflight to a boring set of “delta-v”s. On the other hand, every time I considered putting it down, there was a twist that pulled me back in.

Non-Fiction

Bulletproof SSL and TLS: Understanding and Deploying SSL/TLS and PKI to Secure Servers and Web Applications – If you want an accurate, up-to-date book on TLS, this is the one to buy.

Command and Control: Nuclear Weapons, the Damascus Accident, and the Illusion of Safety – A terrifying and great book; if you don’t know why you should still be afraid of nuclear weapons, you need to read this book.

Countdown to Zero Day: Stuxnet and the Launch of the World’s First Digital Weapon – A great book; reads like a techno-thriller… but it’s non-fiction.

An Astronaut’s Guide to Life on Earth: What Going to Space Taught Me About Ingenuity, Determination, and Being Prepared for Anything – Canadian astronaut Chris Hadfield’s memoir. I wanted to be an astronaut as a kid and this reawoke that interest to a surprising degree. But it also clearly pointed out the trade-offs (37 out of 52 weeks a year on the road) that I couldn’t imagine making with a family.

How to Fail at Almost Everything and Still Win Big: Kind of the Story of My Life – Cartoonist Scott Adams’ memoir and suggestions for success in life; it’s similar in ways to Hadfield’s memoir in being a mix of stories and advice. There were parts of this I really disliked, but there were some great parts too. The best was the repeated advice that goals are for suckers, systems are for winners — similar to Hadfield’s advice, this points out that life is more about the journey than the destination, and if you ever make it about the destination, you’re going to be in very bad shape after you realize you’ve reached it and can’t imagine what to do next.

On Writing Stephen King writes about writing — how he does it and how to do it well. It’s awesome.

Stories I Only Tell My Friends – Rob Lowe’s memoir; I had low expectations, but this book crushed them– it was funny, surprisingly interesting and very well-written.

HTTPS Only Works If You Use It

It should be obvious, but everyone seems to be making the same mistake.

HTTPS only works if you use it. Everywhere.

If you don’t use HTTPS everywhere, a bad guy can intercept an insecure request and prevent the user from reaching your secure site. HSTS is a good start to mitigating the threat of accidentally using an insecure link, but it only helps if you have an HSTS policy set for every domain you will be using.

There’s a big collection of failures to use HTTPS here, but the following are ongoing problems that I’ve been complaining about for a long time now…

IE’s “Domain Suggestions” feature can prioritize insecure suggestions over secure suggestions:

Many major companies (including OS vendors, investment firms, etc) offer HTTPS links in their email… Except they’re not really HTTPS; they’re HTTP links to a “click counter” that is meant to redirect to the secure link. These redirects can be intercepted:

Microsoft OneDrive’s Sharing experience generates secure links by default:

…but the link is made insecure if you click the “Shorten link” button:

The IE Team still hasn’t changed the default Bing search provider to use HTTPS:

Surprisingly, both the Google and Yahoo providers offered are secure, and the Bing provider is secure in Firefox and Chrome. Only IE+Bing is insecure.

The list, sadly, goes on and on.

One of the more esoteric problems I’ve seen is on a site that generally does security quite well: Twitter.

Consider what happens if a user posts a tweet: “I invest with wealthfront.com.” Now I, as a normal human, didn’t spell out https:// in front of that link and Twitter sees it as http://wealthfront.com. This, in itself, might be okay, because WealthFront.com sends a 24 month HSTS policy with the preload attribute, meaning that many browsers will automatically upgrade any http:// reference to https://. That’s great.

Except.

Twitter has some interesting logic in their site. They use a redirector (t.co) to rewrite all hyperlinks, presumably so they can track clicks and block spam or dangerous URLs. When you paste a link into Twitter, it looks to see if the link is to a HTTP target or to a HTTPS target. If it’s to a HTTP target, they use http://t.co and if it’s to a HTTPS target, they use https://t.co.

And here we find the problem. My innocent wealthfront.com reference, which should have been protected by HSTS, has been made insecure because the Twitter folks decided not to use HTTPS everywhere.

Update: They fixed this, now all t.co links are HTTPS.

If you think you’re smart enough not to use HTTPS everywhere, you’re probably wrong.

-Eric