Out-of-Memory is (Usually) a Lie

  • The most common exception logged by Fiddler telemetry is OutOfMemoryException.
  • Yesterday, a Facebook friend lamented: “How does firefox have out of memory errors so often while only taking up 1.2 of my 8 gigs of ram?”
  • This morning, a Python script running on my machine as a part of the Chromium build process failed with a MemoryError, despite 22gb of idle RAM.

Most platforms return an “Out of Memory error” if an attempt to allocate a block of memory fails, but the root cause of that problem very rarely has anything to do with truly being “out of memory.” That’s because, on almost every modern operating system, the memory manager will happily use your available hard disk space as place to store pages of memory that don’t fit in RAM; your computer can usually allocate memory until the disk fills up (or a swap limit is hit; in Windows, see System Properties > Performance Options > Advanced > Virtual memory).

So, what’s happening?

In most cases, the system isn’t out of RAM—instead, the memory manager simply cannot find a contiguous block of address space large enough to satisfy the program’s allocation request.

In each of the failure cases above, the process was 32bit. It doesn’t matter how much RAM you have, running in a 32bit process nearly always means that there are fewer than 3 billion addresses1 at which the allocation can begin. If you request an allocation of n bytes, the system must have n unused addresses in a row available to satisfy that request.

Making matters much worse, every active allocation in the program’s address space can cause “fragmentation” that can prevent future allocations by splitting available memory into chunks that are individually too small to satisfy a new allocation with one contiguous block.

Out-of-address-space

Running out of address space most often occurs when dealing with large data objects like arrays; in Fiddler, a huge server response like a movie or .iso download can be problematic. In my Python script failure this morning, a 1.3gb file (chrome_child.dll.pdb) needed to be loaded so its hash could be computed. In some cases, restarting a process may resolve the problem by either freeing up address space, or by temporarily reducing fragmentation enough that a large allocation can succeed.

Running 64-bit versions of programs will usually eliminate problems with address space exhaustion, although you can still hit “out-of-memory” errors before your hard disk is full. For instance, to limit their capabilities and prevent “runaway” allocations, Chrome’s untrusted rendering processes run within a Windows job object with a 4gb memory allocation limit:

Job limit 4gb shown in SysInternals Process Explorer

Elsewhere, the .NET runtime restricts individual array dimensions to 2^31 entries, even in 64bit processes2.

-Eric Lawrence

1 If a 32bit application has the LARGEADDRESSAWARE flag set, it has access to s full 4gb of address space when run on a 64bit version of Windows.

2 So far, four readers have written to explain that the gcAllowVeryLargeObjects flag removes this .NET limitation. It does not. This flag allows objects which occupy more than 2gb of memory, but it does not permit a single-dimensional array to contain more than 2^31 entries.

Things I’ve Learned in my first weeks on Chrome

This is a stub post which will be updated periodically.

It would be impossible to summarize how much I’ve learned in the last six weeks working at Google, but it’s easy to throw together some references to the most interesting and accessible things I’ve learned. So that’s this post.

Developing Chrome

Searching the code is trivial. You don’t need to know C++ to read C++. And if you can write C++, the process of adding new code to Chrome isn’t too crazy.

Creating bugs is easy: https://crbug.com

Developing Chrome extensions is easy and approximately 5% as hard as building IE extensions.

Using Chrome

PM’ing at Microsoft was all about deleting email. Surviving at Google is largely an exercise in tab management, since nearly everything is a web page. QuickTabs allows you to find that tab you lost with a searchable most-recently-used list.

You can CTRL+Click *multiple* Chrome tabs and drag your selections out into a new window (unselected tabs temporarily dim). Use SHIFT+Click if you’d prefer to select a range of tabs.

Hit SHIFT+DEL when focused on unwanted entries in the omnibox (addressbar) dropdown to get rid of them.

Want to peek under the hood? Load chrome://chrome-urls to see all of the magic URLs that Chrome supports for examining and controlling its state, caches, etc. For instance, the ability to view network events and export them to a JSON log file (see chrome://net-internals), later importing those events to review them using the same UI is really cool. Other cool pages are chrome://tracing, chrome://crashes, and chrome://plugins/.

Chrome uses a powerful system of experimental field trials; your Chrome instance might behave differently than everyone else’s.

Web Developers and Footguns

If you offer web developers footguns, you’d better staff up your local trauma department.

In a prior life, I wrote a lot about Same-Origin-Policy, including the basic DENY-READ principle that means that script running in the context of origin A.com cannot read content from B.com. When we built the (ill-fated) XDomainRequest object in IE8, we resisted calls to enhance its power and offer dangerous capabilities that web developers might misconfigure. As evidence that even experienced web developers could be trusted to misconfigure almost anything, we pointed to a high-profile misconfiguration of Flash cross-domain policy by a major website (Flickr).

For a number of reasons (prominently including unwillingness to fix major bugs in our implementation), XDomainRequest received little adoption, and in IE10 IE joined the other browsers in supporting CORS (Cross-Origin-Resource-Sharing) in the existing XMLHttpRequest object.

The CORS specification allows sites to allow extremely powerful cross-origin access to data via the Access-Control-Allow-Origin and Access-Control-Allow-Credentials headers. By setting these headers, a site effectively opts-out of the bedrock isolation principle of the web and allows script from any other site to read its data.

Evan Johnson recently did a scan of top sites and found over 600 sites which have used the CORS footgun to disable security, allowing, in some cases, theft of account information, API keys, and the like. One of the most interesting findings is that some sites attempt to limit their sharing by checking the inbound Origin request header for their own domain, without verifying that their domain was at the end of the string. So, victim.com is vulnerable if the attacker uses an attacking page on hostname victim.com.Malicious.com or even AttackThisVictim.com. Oops.

Vulnerability Checklist

For your site to be vulnerable, a few things need to be true:

1. You send Access-Control-Allow headers

If you’re not sending these headers to opt-out of Same-Origin-Policy, you’re not vulnerable.

2. You allow arbitrary or untrusted Origins

If your Access-Control-Allow-Origin header only specifies a site that you trust and which is under your control (and which is free of XSS bugs), then the header won’t hurt you.

3. Your site serves non-public information

If your site only ever serves up public information that doesn’t vary based on the user’s cookies or authentication, then same-origin-policy isn’t providing you any meaningful protection. An attacker could scrape your site from a bot directly without abusing a user’s tokens.

Warning: If your site is on an Intranet, keep in mind that it is offering non-public information—you’re relying upon ambient authorization because your sites’ visitors are inside your firewall. You may not want Internet sites to be able to be able to scrape your Intranet.

Warning: If your site has any sort of login process, it’s almost guaranteed that it serves non-public information. For instance, consider a site where I can log in using my email address and browse some boring public information. If any of the pages on the site show me my username or email address, you’re now serving non-public information. An attacker can scrape the username or email address from any visitor to his site that also happens to be logged into your site, violating the user’s expectation of anonymity.

Visualizing in Fiddler

In Fiddler, you can easily see Access-Control policy headers in the Web Sessions list. Right-click the column headers, choose Customize Columns, choose Response Headers and type the name of the Header you’d like to display in the Web Sessions list.

Add a custom column

For extra info, you can click Rules > Customize Rules and add the following inside the Handlers class:


public static BindUIColumn("Access-Control", 200, 1)
function FillAccessControlCol(oS: Session): String {
  if (!oS.bHasResponse) return String.Empty;
  var sResult = String.Empty;
  var s = oS.ResponseHeaders.AllValues("Access-Control-Allow-Origin");
  if (s.Length > 0) sResult += ("Origin: " + s);

  if (oS.ResponseHeaders.ExistsAndContains(
    "Access-Control-Allow-Credentials", "true"))
  {
    sResult += " +Creds";
  }

  var s = oS.ResponseHeaders.AllValues("Access-Control-Allow-Methods");
  if (s.Length > 0) sResult += (" Methods: " + s);

  var s = oS.ResponseHeaders.AllValues("Access-Control-Allow-Headers");
    if (s.Length > 0) sResult += (" SendHdrs: " + s);

  var s = oS.ResponseHeaders.AllValues("Access-Control-Expose-Headers");
  if (s.Length > 0) sResult += (" ExposeHdrs: " + s);

  return sResult;
}

-Eric

Leaking Keystrokes

Windows 10’s IE11 continues to send your keystrokes over the internet in plaintext as you type in the address bar, a part of the “Search Suggestions” feature:

failing

“But I don’t search from the address bar,” you might say.

That may be, but if you fail to type or paste a URL (sans protocol) into the address bar, all of that text gets leaked too:

Danger

This problem doesn’t exist in Edge (which always gets search suggestions from Bing, regardless of your Search Provider, but it at least uses HTTPS). It also doesn’t occur in Firefox’s or Chrome’s provider for Bing Search, or if you use Google or Yahoo search providers in Internet Explorer.

Extended Validation Certificates – The Introduction

In 2005, one of my first projects on the Internet Explorer team was improving the user-experience for HTTPS sites (“SSLUX”).

Our first task was to change the certificate error experience from the confusing and misleading modal dialog box:

Certificate errors UX

… to something that more clearly conveyed the risk and which more clearly discouraged users from accepting invalid certificates. We quickly settled upon using a blocking page for bad certificates, a pattern seen in all major browsers today.

Next, we wanted to elevate the security information from the lowly lock buried at the bottom of the window (at best, since the status bar could be hidden entirely):

IE6 status bar

As a UI element, the lock resonated with users, but it wasn’t well understood (“I look for the lock and if it’s there, I’m safe”). We felt it was key to ensure that users not only saw that the connection was secure, but also with whom a secure connection had been made. This was especially important as some enterprising phishers had started obtaining HTTPS certificates for their spoofing sites, with domain names like BankOfTheVVest.com. Less urgently, we also wanted to help users understand that a secure connection didn’t necessarily mean the site is safe – the common refrain was that we’d happily set up a secure connection to a site run by the Russian Mafia, so long as the user recognized who they were talking to.

We decided to promote the HTTPS certificate information to a new UI element next to the address bar1. Called the “Trust Badge”, the button would prominently display the information about the owner and issuer of the HTTPS certificate, and clicking it would allow users to examine the certificate in full:

Displaying the Issuer of the certificate was deemed especially important– we knew some CAs were doing a much better job than others. High-volume-Low-vetting CAs’ $20 certificates were, to users, indistinguishable from the certificates from CAs who did a much more thorough job of vetting their applicants (usually at a much higher price point). The hope was that the UI would both shame lazy CAs and also provide a powerful branding incentive for those doing a good job.

We were pretty excited to show off our new work in IE7 Beta 1, but five months before our beta shipped, Opera beat us to it with Opera 8 beta 2 with a UI that was nearly identical to what we were building.

During those five months, however, we spoke to some of the Certificate Authorities in the Microsoft Root CA program and mentioned that we’d be making some changes to IE’s certificate UI. They expressed excitement to hear that their names would no longer be buried in the depths of a secondary dialog, but cautioned: “Just so long as you don’t do what Opera did.”

“Why’s that?” we asked innocently.

“Well, they show the Subject organization and location information in their UI.”

“And that’s a problem because…” we prompted.

“Well, we don’t validate any of the information in the certificate beyond the domain name.” came the reply.

“But you omit any fields you don’t validate, right?” we asked with growing desperation.

“Nah, we just copy ‘em over.”

After the SSLUX feature team picked our collective jaws off the floor, we asked around and determined that, yes, the ecosystem “race to the bottom” had been well underway over the preceding few years, and so-called “Domain validation” (DV) of certificates was extremely prevalent. While not all DV certificates contained inaccurate information, there was no consistent behavior across CAs.

Those CAs who were doing a good job of vetting certificates were eager to work with browsers to help users recognize their products, and even the “cheap” CAs felt that their vetting was better than that of their competitors2. Soon the group that evolved into the CA/Browser forum was born, pulling in stakeholders of all types (technologists, policy wonks, lawyers) from all over the industry (Microsoft, Mozilla, Konquerer, etc). Meetings were had. And more meetings. And calls. And much sniping and snarking. And more meetings. Eventually, the version 1.0 guidelines for a new sort of certificate were adopted. These Extended Validation (nee “Enhanced Validation”, nee “High Assurance”) certificates required specific validation steps that every CA would be required to undertake.

EV certificates were far from perfect, but we thought they were a great first step toward fixing the worst problems in the ecosystem.

Browsers would clearly identify when a connection was secured with EV (IE flood-filled the address bar with green) to improve user confidence and provide sites with a business reason to invest (time and money) in a certificate with more vetting. For the EV UI treatment, browsers could demand sites and CAs use stronger algorithms and support features like revocation checking. Importantly, this new class of certificates finally gave browsers a stick to wield against popular CAs who did a poor job—in the past, threats to remove a CA from the trust store rang hollow, because the CA knew that users would blame the browser vendor more than the CA (“Why do I care if bad.com got a certificate, good.com should work fine!”); with EV, browsers could strip the EV UX from a CA (leading their paying customers to demand refunds) without issuing an “Internet Death Sentence” for the entire CA itself.

Our feature was looking great. Then the knives really came out.

…to be continued… (preview)

-Eric

1 Other SSLUX investments, like improving the handling of Mixed Content, were not undertaken until later releases.

2 Multiple CAs who individually came to visit Redmond for meetings brought along fraudulent certificates they’d tricked their competitors to issue in our names, perhaps not realizing how shady this made them look.