windmills

Security response isn’t just about the “sexy” analysis of vulnerabilities, reverse-engineering of malware, and so on… it’s probably mostly about getting the basics right.

Every morning, I forward all of the PayPal phishing scams I receive to PhishTank, Netcraft, and Spoof@Paypal. Today, I took a closer look at the response I got to the last of these:

HTTP link to paypal in email

PayPal was carefully instructing me to visit their site using an unsecure HTTP url. While plenty of sites make this mistake, PayPal has a large security team and they should certainly know better… especially when the email is from their security team!

I dutifully dashed off a note to security@paypal, suggesting that they stop giving out bad security advice. Only to learn that they’ve decided not to follow an obvious best practice, instead auto-replying to say that they don’t accept email at that address and suggesting I follow some convoluted process on their website:

Autoreply - go to our website

Gah. Okay, fine, I’ll just Google around to find out how to report a security vulnerability to PayPal. I find a very nice “Reporting Security Issues” page, which contains a LOT more text than I’d like to read, but hey, it’s a bug bounty program too so I guess they need lots of legalese. Okay, so I’ll just click the link to register:

Link to registration for bounty

Except that, oops… this link doesn’t go anywhere, automatically redirecting to PayPal’s homepage.

 

If you can’t manage your own security response process, you should seriously look into getting an account at HackerOne; researchers are far less likely to get fed up when the process is clear, simple, and well-managed.

-Eric Lawrence

.NET Framework KB 3088956:

image

Ouch, that sounds pretty severe.

I guess I’d better go manually install a hotfix?

image

Seriously? An email address and a CAPTCHA? Fine.

image

Oh, an email delivered HTTP URL pointed at an executable file? That seems totes legit.

image

Yup, definitely legit, it says “Microsoft” right there at the top!

image

Sure, let’s put those files in the root of the C:\ drive. I’m sure that’ll work.

image

Oh, guess it did. Magic!

Oh, wait. No files in C:\. No magic, I guess.

Let’s use Process Monitor to watch the file writes… that’s what all the cool kids have to do to install patches, right?

image

Ah, there you are! UAC virtualization like it’s 2006!

image

Of course I must, Microsoft, of course I must.

-Eric

Where’s Google’s* blog on how they’re doing everything they can to make ads they serve as fast and small as possible?

Where’s Google’s blog on how many ads they’ve nuked as “deceptive” and trumpeting how policy forbids ads for “adware-wrapped” installers?

Where’s Google’s blog about how many billions of ad-generated dollars they’ve supplied to content sites and open-source products that people care about like Firefox?

Where’s Google’s blog on how much smaller they’ve made ads served using WebP instead of older formats? And Zopflinated PNGs for legacy browsers?

image

Ad publishers can’t expect a company like EmbarcaderoTech to know how to make fast ads. Publishers need to help.

image

Where’s the argument that the worst privacy impact of ads comes from trying to recover revenue lost through blocks and fraud?

Without good answers to these questions, ad publishers are going to have a very hard time regaining any control of the narrative. The entire industry is demonized for performance, security, and privacy problems, even though each publisher has different practices.

-Eric

*Note: Google does plenty of things right from an advertising point-of-view; I only mention them by name because they’re probably the biggest and I expect the best technology from them. They’ve invented much of the interesting technology in this space, including Zopfli and WebP.

The Microsoft Edge (nee Internet Explorer) team held one of their “#AskMSEdge chats” on Twitter yesterday.

image

After watching the stream, @MarkXA neatly summarized the chat:

image

The folks over on WindowsCentral built out a larger summary of the tidbits of news that did get answered on the chat, some of which were just pointers to their Status and UserVoice sites.

After the chat ended, I complained that none of my questions had been answered:

image

In response, an IE Engineer retorted:

image

I don’t think that’s fair. Here are my questions, and a few remarks on each:

image

As far as I know, I’ve never asked the IE/Edge team about Brotli before, as I hadn’t spent any time looking at it until very recently. I’m interested in the team’s plans for Content-Encoding: Brotli because it can significantly improve browser performance, and if the team implements WOFF2, they must integrate Brotli decoding logic anyway.

image

I don’t think I’ve ever asked the IE/Edge team about their plans here before. More efficient HTTPS algorithms are important for both performance and battery life on mobile devices in particular, and thus I think they’re a great investment.

image

I have asked this before. IE has had non-standard network export for four years and I was really excited that Edge moved from HTTP Archive XML to the standard HTTP Archive JSON format. Unfortunately, this bug makes their code non-interoperable. The fix will be one or two lines of code. I feel justified in asking for status since weeks or months have passed without update.

image

I have asked this before. Edge regressed a significant piece of functionality and created a denial-of-service condition in their browser. I feel justified in asking for status since weeks or months have passed without update.

image

I have asked this one over and over again. I find it galling that Microsoft products are less secure together, and especially when Microsoft’s new President promised to close these sort of gaps nearly two years ago. It’s clear that the team agrees that the behavior is bad, because Edge uses Bing securely and doesn’t even allow users to add non-HTTPS search providers.

image

This one is probably the least “fair” of the questions, insofar as I already know the answer and I’m effectively just calling the team out on the specious nature of the promise to “watch demand” they made when the original concerns about the absence of Windows 7 support were raised.

However, I’ll note that the team answered several repeats of the question “When will it run on Mac? When will it run on iOS and Android.” Given the Windows 7 marketshare dominance, I think this question remains fair.

Other Unanswered

Other folks asked several great questions that didn’t get answered:

image

I really want this feature.

image

I think the Edge team is making a huge mistake if they’re not piloting their new extension model with critical extension developers like uBlock, NoScript, etc.

image

Because of the nature of the legacy Win32 Address Bar’s context menu, Paste-and-Go was always prohibitively expensive. The UI replacement for Metro IE and now Edge makes this a trivially added feature that was requested by several questions.

Unwanted Answers

Some questions got answers that I’m just not happy with, but I’m tired of complaining about:

image

The Edge team replied “No” and suggested they consider this a scenario for the new extension model. I think this is a mistake and a case where “different” isn’t likely to be “better.”

 

Several folks asked when the new extension model would be released. “Stay tuned” was the answer.

Subtext

I think the subtext of Adrian’s complaint is that “You’ve worked here, you know we don’t announce things on IEDevChats.” There’s some truth to that frustration – I know that announcements are carefully vetted and published on the blog and I understand why live chats aren’t a source of new information. However…  

Rabble-Rousing and Information Asymmetry

I know some folks think my questions are just rabble-rousing and that, as an ex-teammate and current MVP I should be asking these questions in private, directly to the IE team. A few points on that:

  • Microsoft has basically requested we provide feedback in this manner, with the “we’re watching feedback to influence our decisions” position on everything from features to bugfixes. Feedback that isn’t getting public traction is largely ignored.
  • A significant number of my friends and colleagues are no longer on the IE team. In the photo of the team answering questions, I recognize seven of the seventeen engineers.
  • My emails to the IE MVP discussion list generally do not receive replies.
  • Direct emails to individual engineers on the IE team often do not get replies.
  • Status on bugs I’ve filed with MSRC is similarly hard to get– issues have languished for months without so much as a “working on it” status update.

Having been on both sides of the fence now, it’s plain to me that one serious problem Microsoft has is that they don’t realize how incredibly opaque things are from outside the company. As an engineer racing from one issue to the next, it’s easy to deprioritize status updates and justify doing so when there are so many higher-priority things to fix. From outside the company, however, “working on it and coming as soon as we can”  is often indistinguishable from “ignoring—really hope this goes away.” That problem is exacerbated by Microsoft’s tendency not to deliver hard messages like “Silverlight is dead dead dead, get off it now!” in a timely manner to allow customers and partners to plan appropriately.

In life, you sometimes encounter people with “high standards”—folks who often find others’ behavior lacking in some way. Such people usually explain: “Sure, I have high standards… but I hold myself to an even higher standard!”

Except… they rarely do.

The problem is that, as humans, we’re subject to both fundamental attribution error and actor-observer bias. These phenomenon mean that we attribute our own good behaviors as intrinsic demonstrations of our good character, and we explain away bad behaviors as a temporary consequence of a given situation. When observing others, however, we tend to do the opposite—good behavior is dismissed as situational and bad behavior is deemed evidence of poor character.

These errors are made every day, at the small scale of our personal relationships to the global stage of policy-making. They’re hard to avoid, especially if you’re not consciously aware of them and don’t actively strive to consider events from different perspective.

Yesterday, Chris Beard, the CEO of Mozilla, wrote an open letter to Microsoft complaining about Windows 10’s behavior related to default applications. Reactions were all over the board, but in my Twitter feed, at least, they mostly skewed against Mozilla. With the perspective of having been both inside and outside Microsoft, I feel compelled to say a few things.

First, Defaults Matter

Defaults matter. Regardless of its relative merits, the default application is disproportionally likely to get used versus any alternative, even if those alternatives are all free. When I bought my car, it had floor mats— they seem fine; I’m sure they’re not the best floor mats, but I’m going to devote exactly zero thought to replacing them unless they fail me in some way.

It’s no accident that the Edge icon is a little blue “E”—that was the default browser icon for so many years that changing it would’ve been very foolish.

The most compelling argument for the fact that defaults matter is how much gets spent trying to influence them. Most obviously, Microsoft has spent billions in legal settlements and fines related to the default applications on Windows. “But,” you might argue, “that isn’t relevant because they didn’t choose to spend that money.” That’s true, but look at the budget that companies spend to get their software and services distributed and made default. Yahoo, Google, and Microsoft have paid organizations like Mozilla huge sums (on the order of $300,000,000 per year) for over a decade for the opportunity to be the default search engine in Firefox and other browsers.

By way of personal example, around 2010, I was approached with an offer of $1 per install to bundle the Bing Toolbar with Fiddler’s installer in order to increase the number of users defaulted to searching with Bing. I turned down this opportunity (worth somewhere between $300K to $3M per year) because I’m fundamentally opposed to bundling of this sort.

Want to learn more, or prefer your evidence in dead-tree form? Check out Nudge.

“It’s Easy To Change”

Even assuming the user does want to switch to a different browser, any friction in that process impedes change and makes a meaningful difference in adoption. This topic was deeply explored during the Microsoft-E.U. case that led to the browser ballot screen, and every tiny bit of friction was analyzed and negotiated at length. When we were redesigning how ClickOnce worked with IE in Windows 8, we were very constrained by being forbidden to add even a single click to the user flow by which Google Chrome was installed.

In Windows 8 and Windows 8.1, code was changed in Windows such that (well-behaved) applications could not automatically set themselves as the default handler. Instead, a prompt is shown:

image

In Windows 10, the same API call yields a very different result:

image

There are a number of significant problems with this. First, it’s not at all clear that this dialog is coming from Windows itself, not the caller, inviting the user to go on a wild goose chase around the app looking for some sort of “Settings” thing. More annoyingly, there’s no “Go there now” button—the computer should serve me, not the other way around. When Picard shouts “Computer: Shields up”, the Enterprise doesn’t reply with “To raise the shields, go to Ship > Controls > Security > Shields.” The omission of an entry point into the experience is glaring to the point of being offensive.

When you get to the “Default apps” controls, how do you change your browser to Chrome or Firefox? Perhaps amazingly, by clicking on Edge:

image

…only then do you see the alternatives:

image

Unfortunately, even if the user makes it all the way here, I think it’s likely to cause problems, and the reason is pretty subtle. In the Windows 8/8.1 UX, the “How do you want to handle this type of link” experience only assigns a “basket” of protocols (HTTP, HTTPS, and maybe FTP) to the selected application. In Windows 10, the “Choose an app” experience appears to give the target all of the protocols it requests. This is a problem, because Google Chrome lists itself as able to handle the mailto: protocol (used for opening email links). But unless you’ve followed some rather obscure steps, Chrome doesn’t actually do anything other than launch when you click a mailto link. So the user experience is bizarrely and confusingly broken.

Claims and Retorts

API Parity

In their blog post announcing the change, Microsoft makes the following (oddly phrased) argument for the change:

In Windows 8.1, Classic Windows applications (Win32) could invoke the prompt asking you to change your defaults, so you may have seen multiple prompts during install and after they launched. However, Windows Store apps could not invoke this prompt.
[…]
We know your defaults matter to you. With Windows 10, all apps – both Classic Windows apps and Universal Windows apps – will be unable to invoke a prompt to change your defaults, only Windows.

Some have leapt to the defense of the change noting that because Store apps couldn’t invoke this prompt, this means that Desktop APIs needed to change. I’d counter that there’s no reason why a similar API couldn’t be exposed for Store apps.

API Deprecation

In the same post, Microsoft notes that applications just need to update their code to avoid the new ugly and unactionable dialog. That’s true as far as it goes– Firefox and Chrome will probably update in their next builds, but this is a problem that could’ve been mitigated in the first place. We know from past experience that requiring applications to change is an adoption hindrance for a new operating system, and there are a long tail of applications which might take a long time to be updated.

What About ABUSE?

Some have argued that this change helps protect users from unwanted changes to their providers, the same argument made for the Windows 8 change. As far as I can tell, no evidence has been provided that the Windows 8 mechanism was ineffective.

“But what about ‘drive-by downloaded’ malware?” some have asked. The first law of computer security is still in effect: a malicious program running at Admin can do anything it likes, including circumventing the speed bumps introduced in Windows 10.

Is Microsoft Evil?

In my twelve years at Microsoft, I can’t recall any cases where anyone sat around twirling their mustache while planning some evil Machiavellian plot. Instead, the problem is that Microsoft too often fails to consider that they are a huge bull and the sole proprietor managing a tightly-packed consignment shop full of china. Any change, regardless of intent, can end up breaking a lot of nice things. For a time, this realization was formalized in the design tenet “Change is bad, unless it’s great.”

Is this Memo A PR Move for Mozilla?

Some commentators are noting that, since other operating systems are even more restrictive in terms of defaults, Mozilla is only complaining about Windows 10 to get attention from the press… as if this were somehow a clever line of argument.

Of course Mozilla is only releasing this complaint now to get attention!

You can’t build a great third-party browser for iOS because of Apple’s lockdowns, and both Google and Mozilla lament this. Mozilla is going so far as to tilt at the windmill of creating their own mobile OS in the hopes of having a shot at relevance in a mobile world. While I liken this to setting fire to a large pit of money and rate its chances of success in the neighborhood of zero, what choice does Mozilla have?

They’re at the mercy of Apple, Google, and Microsoft, companies whose market caps are measured in fractions-of-a-$Trillion USD. Failing to use their bully pulpit as the champion of openness to push back on lockout would represent severe negligence.

Why do I care?

I care because I want the best for Microsoft and I want the best for users. I think Edge is going to become a great browser, and it would be terrible if it were shackled with accusations of dirty tricks in the same way that IE4 and IE5 were, even if the “tricks” aren’t nearly as dirty this time around.

I care because I’m very worried about user-choice. I think Microsoft’s decision to make it much harder for humans to change the default search engine was an awful one and I hope they reverse course. As it stands, Microsoft’s browser share is standing on a precipice—because Edge is exclusive to Windows 10, its share has Windows 10 upgrade adoption shackled to its throat. If Windows 10 lags, Edge might well die with barely a whimper. Similarly, because Edge makes it so hard to switch away from Bing, users who like Edge but find Bing lacking are incentivized to switch to Chrome.

-Eric Lawrence

I’ve written about Zopfli quite a bit in the past, and even wrote a tool to apply it to PNG files. For fun, I had a look at one of the most optimized pages in the world: Google.com, through the lens of Zopfli.

Here are the basic resources delivered by the Google homepage:

Zopfli WhatIf

This breakdown shows that Google isn’t optimizing their own compression using the compressor they wrote. The Savings column shows the number of bytes saved by using Zopfli over whatever Google used to compress the asset. Using the default settings in an ideal world, Google could save up to 16.5k, almost 5% of the bytes transferred, by using Zopfli.

I’ve color-coded the column based on how practical I believe the savings to be—the green numbers are the static images where there’s no question the size benefit could be realized. The yellow numbers are cases where script files are compressed; given the complicated query string parameters, I’m betting these scripts are dynamically generated and the compression cost of Zopfli might not be reasonable. The red number is the homepage itself, which probably isn’t reasonable to Zopfli compress as it certainly is generated dynamically.

So, most likely the savings of a practical Zopfli deployment on the homepage page would be about 3.7kb; savings are much greater on other pages on other sites.

More interesting, however, is the Google API CDN, which hosts scripts for other sites; optimizing these would take a minute or two at most and make every site that uses them faster.

Zopfli savings

Use Zopfli; give the tubes a little bit more room.

-Eric

PS: You may already have zopfli.exe on your system; Fiddler installs a copy to its \Tools\ subfolder!