Browser Benchmarks

Back in December, I predicted that Microsoft wouldn’t release the Project Spartan beta until it bested all of its competitors on the major benchmarks: SunSpider, Kraken, and Octane. I was wrong—the first beta was released with only minor script performance improvements. That changed with build 10061 of Windows 10, and Spartan now does beat everyone else on their own benchmarks.

Running Windows 10 on my new 2015 Dell XPS13 i5-5200U, I get the following scores:

Browser SunSpider Kraken Octane
Spartan/IE 10061 122.7ms 1444.44ms 23652
Chrome 43
beta 2357.37
255.5ms 1557.7ms 22656
Firefox 37.0.2 204.2ms 1498.4ms 21762

Now, some of these “victories” are within the margin of error, and it’s very possible that upcoming versions of Chrome and Firefox will improve their performance on slow outliers (e.g. Chrome’s score on Octane’s MandreelLatency is just 22% as fast as Spartan’s). But anyone surprised at Microsoft’s great results is overlooking the fact that some of the world’s best compiler developers and architects work for Microsoft and their attention has increasingly been turned toward JavaScript.

Of course, script performance is important, but it’s far from the only way to measure a browser. Standards-compliance, network performance, ease-of-use, security, end-user features and many other aspects determine your experience with a browser. There are many different tests (subjective and less-subjective) for these aspects, although each has its own biases. But just to give one example, with all its feature flags turned on, Spartan ekes out a score of 402/555 on the (questionable, but easily run) HTML5Test.com while Firefox and Chrome score 449 and 526 respectively.

Hamstringing JavaScript

Of course, your numbers might be wildly different than those above, for one major reason: security software.

Every year for Microsoft’s annual AV summit, the IE Team puts together a chart of the impact of AV on browser performance, showing the variation across the top 20 AV products (the variation is huge). They don’t want to publish this data, but the impact ranges from “bad” to “absurdly unbelievably bad.” The best products impact performance by ~15%, the worst slow the browser by 400% or more. Several of the products crash the browser entirely and can’t be benchmarked fully. Conducting these benchmarks correctly is difficult—you need to account for every piece of software running on the machine and ensure that the test conditions are entirely fair (hardware, software, updates, etc); as a consequence, many of the “public” benchmarks are rather inaccurate.

Why hasn’t the IE team released their numbers? My guess is that it’s to try not to anger the AV companies, all of whom have been muttering “antitrust antitrust antitrust” under their collective breath ever since Microsoft integrated an entirely decent antivirus package into Windows 8.

Personal anecdote: I have Symantec Endpoint Protection running on a machine with a high-end i7-4771 CPU; even after unticking all of the “optional” protection features I can find in the Symantec control panel, the Octane score in Chrome 43 is 11659. On the same hardware in the same browser version without Symantec installed, the Octane score is 32555, 279% of the original score.

The devastating impact of antivirus on browsing performance is one reason why your portable devices feel magically fast—on a AV-unhindered i7, IE11 runs SunSpider in 70ms. Add AV and it runs in 350ms. The IPad Air, running with Safari’s slower script engine, runs it in 380ms. Mobile devices offer “Desktop Class” performance only because your desktop has been wrecked.

Antivirus software is too often a cure that’s as bad as the disease. The business model of AV rewards noisy products, and the desire for “checkbox parity” leads to a race to shove its tentacles in all sorts of places they don’t belong (e.g. the internal data structures of the browser). Unfortunately, even beyond antitrust concerns, Microsoft is very limited in its ability to deal with horrible AV products due to court precedents that say that AV can pretty much get away with doing anything it wants in the name of “protecting the user.”

You might ask: “Without my security software, aren’t I at risk?”

Yes, you are. But security software provides surprisingly little protection, as this delightful photo of a colleague’s laptop shows. In the foreground, the AV software promising that the user is protected. In the background, the ransom UI demanding payment to decrypt the documents that have just been mangled.

Awkward screenshot

Even worse, “security” software itself often introduces vulnerabilities into otherwise secure systems.

Advice

Want to be protected and stay fast?

  • Upgrade to Windows 8.1 or later.
  • Leave SmartScreen Application Reputation enabled.
  • Leave the built-in antivirus enabled.

Or get a Chromebook.

– Eric Lawrence

Velocity Conference – 2015 Santa Clara

I’ll be speaking at my third (or fourth?) Velocity Conference in Santa Clara California one month from now. I’m always excited to go to Velocity; it’s a conference full of great talks, and more importantly, smart people working on important problems.

image

My talk, HTTPS in 2015, delivers a whirlwind overview of HTTPS, from how it works to why you need to use it to secure your sites and services.

Anyone watching the news knows that securing your websites and services using HTTPS has never been more important.  In this talk, a former browser security program manager covers the best practices for using HTTPS today. Topics covered in this session include ciphers and hash algorithms, forward secrecy, handshakes and protocols, and new browser features like HSTS and PKP. We’ll talk about exciting developments in 2015 that will make securing your site simpler than ever before, even as attackers get ever more sophisticated.  Explore how attackers circumvent HTTPS, and what you can do to help protect your visitors and keep customers safe from snoops and bad guys.  Secure all the things!

Leading up to the conference, the fine folks from O’Reilly conducted a little mini-interview with me you can read here HTTPS – The S stands for User Experience.

I hope to meet some of you at the conference! If you haven’t registered yet, you can use code ericl20 to save 20%, but hurry—prices go up May 1st.

-Eric

PS: O’Reilly has got you covered if you need to convince your boss.

Security UI

Over a decade ago, Windows started checking the signature of downloaded executables. When invoked, Attachment Execute Services’ (AES) UI displays the publisher’s information for signed executables; unsigned executables instead show a security prompt with a red shield and a bolded warning that the publisher of the file is unknown:

image

In contrast, signed executables show a yellow shield and the name of the publisher and the publisher’s declared name of the application.

When Windows Vista released in late 2006, an “elevation dialog” was introduced to prompt the user for permission to run an executable with elevated (administrator) rights. The new prompt’s design somewhat mirrored that of the earlier AES prompt, where unsigned executables are scary:

image

… and signed executables are less so:

image

As you can see, the prompt’s icon, program name, and publisher name are all pulled from the downloaded file.

To avoid double-prompting the user, the system detects whether a given executable will be elevated, and if so the AES dialog is suppressed and only the elevation prompt is shown.

As a consequence, the security UI in modern Windows is a bit backwards… the lower-risk “run as user” dialog seems complex and scary, while the higher-risk “run as administrator” dialog seems simpler and more trustworthy:

BadDesign

From a security design point-of-view, this seems unfortunate. Application designers should never be in the position of choosing higher-permission requests to get friendlier prompt behavior.

-Eric Lawrence

OpenSearch

History

The new UI of Internet Explorer 7 included a dedicated search box adjacent to the address bar, like the then-new Firefox. As IE7 was built between 2004 and 2006, Microsoft didn’t have a very credible entry into the search engine market—Bing wouldn’t appear until 2009. The IE team made a wise decision in support of the open web—we embraced the nascent OpenSearch specification developed by Amazon for search provider specification, allowing the browser to easily discover search providers offered by the site and enabling users to easily add those providers to IE’s search box.

This was a huge win for openness– it ensured that IE users had their choice of the best search engines the web had to offer. There was no lock-in.

Aside: The Narrative

Part of the Internet Explorer team’s internal narrative1 for years was that only two browsers were properly aligned with user’s interest—the only browsers where the customer was also the user were Safari and Internet Explorer. Safari and IE were the browsers bought by the customers who purchased their vendor’s hardware and software respectively. In contrast, the story went, Firefox had one customer, Google, who paid hundreds of millions of dollars for the right to be the default search engine. Later, Chrome had many thousands of customers, the AdSense advertisers who were buying access to the real product (millions of users’ eyeballs). As a consequence, the narrative went, the IE team were champions of the user and thus we’d make every decision with only our customers’ experience in mind.

What Happened Next

Fortunately, OpenSearch was quickly successful, and both Chrome and Firefox adopted it and the window.external.AddSearchProvider API that allowed a site (upon a user-initiated action) to offer to add a new Search Provider to the browser. This enabled customers to easily access search engines both large (Google, Yahoo, Bing, etc) and niche (Amazon, MSDN, etc) within their browser of choice. Some browsers even used OpenSearch to allow users to access search providers without installing them.

Openness won…

Today

… until it didn’t. The Internet Explorer team has indicated that they don’t plan to support the de facto standard AddSearchProvider API they invented in their next browser, currently codenamed Project Spartan. They’ve offered a variety of defenses of the decision (e.g. “Safari doesn’t support it so we don’t have to!”) that they’ve previously ridiculed in other contacts (e.g. Pointer Events).

Currently, in fact, Spartan is hardcoded to use just one search engine—Bing. I have no doubt that the Spartan team will add additional search engines to their browser before they ship, but only an open API provides the freedom for sites and users to interact without any confounding politics and economic decisions. If I want to switch to a privacy-focused engine like DuckDuckGo, that should be trivial. If I want the ability to quickly run MSDN searches, this shouldn’t require petitioning the IE development team.

Security And Privacy

Making matters worse, Spartan users’ searches are sent to Bing over the network in plaintext, despite the fact that Bing supports HTTPS and the latest versions of both Chrome and Firefox use that HTTPS provider.

Some have argued that AddSearchProvider is “too powerful” and browsers shouldn’t offer APIs that enable changes to their configuration, even with user consent. This argument is compelling until you notice what actually happens—when you take away the sandboxed, restricted API, sites don’t just throw up their hands and say “Ah well, guess we’ll go home.” Instead, they send the user executable downloads that can take any action they like on the system, changing the search provider and, while they’re at it, reconfiguring the user’s other browsers, changing the search page, throwing in a toolbar or two, or whatever. Once the user’s suckered into running your code, why not maximize the value? And the Windows ecosystem continues its swirl toward the drain…

Other users have argued that a “gallery” of search providers, like IEAddons.com is the right way to go. There are many problems with this approach. First, it requires that each site go to Microsoft, hat in hand, and register a provider. It requires that users go out of their way to go to the Gallery. Worst of it, it doesn’t provide any user workaround when the Gallery gets things wrong: for example, both Bing and Google offer HTTPS-based searches, and have for years. But if you install their official providers from the IE Gallery, you get insecure search and leak of your keystrokes as you type in the address bar. Microsoft Security Response Center (MSRC) has indicated that they do not consider this a security vulnerability.

In contrast, when AddSearchProvider is supported, the search engine can itself offer the proper, secure, search URLs. Or a user can build their own provider.

Please join me in begging the Internet Explorer team to reconsider: Support freedom. #SupportOpenSearch.

Vote here to fix Spartan: Bug Tracker link

Update: Hours after this post, the April security update for IE broke the AddSearchProvider API in existing IE versions. :-(

-Eric Lawrence

1 The validity of this narrative is itself worthy of its own post, so please don’t bother flaming the comments below.

On Appreciation

Over on Twitter today, there’s a trending hashtag #HonoringWebFolk, started by the very honorable Molly E. Holzschlag. My feed is full of friends, colleagues, acquaintances, and folks I’ve yet to meet expressing their gratitude to “unsung leaders” for their contributions to the web. Of course, some of the folks being honored aren’t exactly unsung, but even Internet famous tends to rate below a C-list Hollywood celebrity:

image

The list of people I should thank is long—from the folks who took a chance on me back before I’d really written anything of note, to the colleagues who challenged me and helped me grow. One of the recurring themes developing as I slowly crank out my Microsoft-years memoir is how unbelievably lucky I am to have had the chance to work with so many fantastic people over the years.

However, when I think about the unsung heroes I’ve worked with, the first name that comes to mind is Ed Praitis. Ed was a developer on the IE Fundamentals team who made many contributions over the years; one of his most important tasks in the IE9 timeframe was the grueling work of improving add-on compatibility as we continually ratcheted up the security and reliability of the browser. As we made fundamental architecture changes along the way, add-ons written in the era of single-process, non-Protected Mode browsers blew up, often in spectacular fashion. Many of these add-ons were broadly used and essential to everyday users, especially in countries like South Korea. While we evangelized the HTML5 future as loudly as we could, we know we’d be unable to get these users to even upgrade to browsers with better standards support if their banks and other critical sites didn’t work on Day 1. As if debugging sites and add-ons written in Korean wasn’t hard enough, most of these add-ons were designed with obfuscation and anti-debugging code in an attempt to foil malware. And yet Ed soldiered on and helped ensure IE9 had a smooth launch, unblocking deployments and helping drive toward a HTML5 future.

More personally, Ed provided unexpected and fruitful encouragement at exactly the right time. As we built IE9, I lamented that the dev team couldn’t muster the resources to fix around a dozen performance bugs in the network cache code. As I explained the changes needed and how important they were, he listened thoughtfully and then quietly noted: “It seems like you understand this stuff pretty well. Why don’t you just fix it yourself?”

I chuckled until I saw he was serious. “But I’m a PM!” I protested, “we don’t check-in code. At least, nothing like this.”

“I’ll review it for you if you want,” he offered.

And this was just the push I needed. Within a few weeks, I checked in my fixes, and this was probably the work I’m proudest of in over a decade at the company… helping save hundreds of millions of users untold billions of seconds in downloading pages.

Unfortunately, I never properly thanked Ed for his encouragement.

Two other Microsoft stories come to mind.

Early one morning, the IE PM team was having some sort of internal team-building event in a nearby building. One of the exercises called for us to form two long lines facing one another, and we did so in the entry hallway, one line to each side of the hall. Before the exercise started, a few of the employees who worked in that building arrived for work. They looked confused at what must have looked like a receiving line of some sort, until one of the PMs (I cannot recall who) started clapping and the rest of us joined in, cheering on the arrivals as if they were rockstars who’d arrived at a venue. They were beaming as they walked off to their offices, doubtless still confused, but certainly happier than when they’d come in. It was random and beautiful, and later I often mused that teams ought to consider doing “Rockstar Hellos” as a morale event.

When I left Microsoft, I sent out the typical farewell mail to my immediate coworkers. This mail was circulated more broadly to various folks and I received many extremely kind farewells from all over the company. There’s nothing quite like getting a personal mail thanking you for specific contributions you’ve made from someone you admire, especially when you didn’t expect that person to even know who you are.

Express your appreciation today, and as often as you can. You never know who you might inspire.

-Eric Lawrence

Authenticode and ClickOnce

On my old IEInternals blog, I posted a fair bit about using Authenticode to sign your programs so that their origins could be identified and to avoid triggering warnings from SmartScreen. My last post on that blog before Microsoft took it away was about using a hardware token to improve security of your certificate’s private key.

Unfortunately, one topic I didn’t cover was the use of Authenticode with ClickOnce applications. Fortunately @RobinDotNet did a deep dive on this topic nearly two years ago and published her results in this two-part series:

A simple summary of these thorough posts: You should use Authenticode to sign both your app’s executable and the setup.exe bootstrapper to help ensure that your ClickOnce invocation isn’t blocked by SmartScreen. This signing is slightly trickier than it sounds because you must sign the files in the proper sequence or the ClickOnce manifest file will contain the wrong hash value. The most reliable place to invoke signcode.exe is in the AfterCompile or BeforePublish steps.

Note: Signing ClickOnce apps is especially confusing because there are so many different things to sign: the manifest, the assemblies, setup.exe, and the executable. When you specify a signing certificate in the project properties, that certificate is used to sign setup.exe (used to install dependencies if the required frameworks aren’t yet installed) and the ClickOnce manifest, but it isn’t used to sign the executable. If you tick the Sign Assembly box, you might expect this to code-sign the executable too, but this doesn’t perform Authenticode signing– it instead applies a strongname to the assemblies. Strongnames aren’t examined by SmartScreen (or much of anything else for that matter).

-Eric Lawrence

How Hard Could It Be

I have an idea for a new reality show; it’ll be called “How Hard Could It Be?” The show will feature everyday people approaching straightforward (but new to them) tasks, for instance, minor home repairs.

For instance, in the kickoff show, a software developer will be faced with filling a gap introduced in his front porch when a professional replaced an exterior door.

gap

To make it more interesting, we’ll have the work done on the highly-visible front porch, and we’ll have the man’s wife be pregnant to ensure a steady flow of family and friends visiting over the next few months.

The show will start with the man rubbing his chin and looking at the gap. “I know”, he’ll say, “I can fill this with concrete. I mean, ‘How hard could it be?'”

We’ll follow our character as he researches by watching assorted videos on the Internet, wherein a wide variety of workaday joes install concrete successfully in a wide variety of projects.

Confirming the ease of his project, he’ll drive over to the local home depot, grab an 80lb bag of concrete, a few stirs, a 5 gallon bucket, and some red concrete dye. That’s right, dye. The dye serves two purposes: One– to attempt to match the final result to the nearby brick and other décor. Two– so that the viewers at home (and any passers-by) can easily see all of the unintended places that the concrete ends up. To up the ante a bit, this project will begin late on Sunday morning (ensuring that only one day can be spared for the work) and we’ll film on a May day in Austin with afternoon temperatures in the mid-90s.

Reading the bag of concrete mix, he’ll scoff as he reads “Do not let concrete dry in your hair” and feel confident that this project is going to be trivial: “How hard could it be, if the package needs to warn the typical user of such things.”

As the afternoon proceeds, we’ll watch our hapless hero increasingly drenched in sweat, as he learns valuable lessons like: ‘Why you don’t want to pour concrete mix without a facemask’, and ‘Why do professionals mix concrete in wheelbarrows instead of buckets.’ A counter on the screen will track the skyrocketing obscenity count as the afternoon proceeds toward dusk.

The show will end with our red-tinted grimy hero and his very-pregnant wife on the porch looking forlornly at the final lumpy, off-color result.

She’ll look on the bright side. “It won’t look so bad after we cover it with the welcome mat.”