It started innocently enough. I saw a tweet about a new product offering from one of my favorite companies, Cloudflare.
Neat! I clicked through to the site and there it is:
And huzzah!, my preferred handle, @ericlaw is still available. I’d better hurry to claim it before someone else gets it!
Since I’m already a long-time Cloudflare user, I just need to sign in. That makes sense, how else will they bind the handle to my account?
Easy peasy. I’m in. Looks like there’s just one more step, I gotta authorize the new feature?
But wait a sec!
This looks exactly like one of those Consent Phishing attacks that have been so popular over the last few years!
And wait, why is the entry point on cloudflare.pay, a site that doesn’t already have my credentials, rather than something within the cloudflare.com domain which does (e.g. cloudflare.com/pay)? There is no inherent technical relationship between a .com domain and a .pay domain. Domain names under the.pay sTLD are available to anyone with $20 (unlike, e.g. .bank which requires more vetting), so there’s nothing that would stop me from registering my own cloudflarepayments.pay domain name in just a few minutes.
And why doesn’t Cloudflare’s permission site recognize its own company’s feature? And that green checkmark looks suspicious as heck– an attacker could probably just shove that emoji inside their misleading display name, the same way that folks trying to phish Microsoft email accounts use misleading app names and icons:
Fake Outlook OAuth phishing request
The guys at Cloudflare are geniuses who know their stuff. This has got to be an attack. It’s a clever one — I was feeling such a sense of urgency because I wanted to “win” the race to get my desired handle. Very very clever!
Unfortunately, the Cloudflare permission page doesn’t follow best practices, so there’s no “Report suspicious request” link I can use to let the Cloudflare folks know that their customers are under attack.
Let me go back to my Cloudflare dashboard and try to get to the Wallet feature from its sidebar. Hrm. It’s not there. Now, Wallet purports to be “a new feature”, so maybe the Dashboard just isn’t updated yet. A search of the docs turns up nothing. Let’s ask the AI agent in chat.
The very first thing the chat agent wants is access to my account:
This feels a little weird, but the page is still cloudflare.com so I guess I can give the thing access to things it already has access to. Weirdly, the AI agent first proposes that I grant it full control rather than read only access, which feels like a failure of the principle of least privilege, but I don’t actually need to ask an account specific question anyway. After granting read permission, the agent allows me to ask my question:
After a few minutes of further frantic searching, it turns out that this is, in fact, a legitimate new Cloudflare product and a legitimate site, despite giving every indication of being a clever phishing attack.
It further turns out that that suspicious green checkmark is not part of the app’s untrustworthy display name but instead a (poorly placed) security UI element that a user is expected to hover over to get the security details:
The Cloudflare folks apparently want security issues reported via HackerOne (which wouldn’t let me log in because the Cloudflare CAPTCHA HackerOne uses seems to be broken…).
When legitimate websites sometimes act very very phishy, consider how hard it must be for URL Reputation services like Microsoft SmartScreen and Google SafeBrowsing to block malicious sites without false positives as millions of new sites are added to the web every week.
Lessons
Web Developers, please follow every best practice, I’m begging you:
Host apps and content under your trusted domain name (e.g. cloudflare.com/pay or pay.cloudflare.com. If you must add a new name, link to it directly from a page on your trusted domain name.
Show relevant security information in a trustworthy place when asking the user to make security decisions.
When a user attempts to run a file with elevated privilege, Windows will show a User Account Control elevation prompt that asks whether the user trusts the file to run.
For a regular file, the user will see a prompt like this:
For a file signed by a certificate in the Untrusted Certificates store, elevation is explicitly blocked:
For a file with a trustworthy signature, the user is expected to see a prompt like this one:
However, I’m sometimes sometimes surprised to encounter a different behavior when running signed files.
Surprise #1 – Windows Files are Special
The first surprise occurred last year when we were working testing the Defender Deployment tool. If we looked at the executable in File Explorer’s Properties dialog, we saw that it was correctly signed:
However, when double-clicking the file in Explorer, the UAC prompt behaved as if the file were not signed.
What gives?
After investigation, I learned that UAC has a special carve out for files that are signed by the Windows Build lab. If such a file is encountered outside of a limited set of directories deemed “secure” (e.g. %systemroot%\system32) the file is treated as unsigned.
Windows executables are non-hermetic so running them from untrustworthy locations can have dangerous outcomes. Showing the file as unsigned is intended to discourage a user from doing this.
Surprise #2 – Chain Building
Today, I was downloading the updater for the latest update to Telerik Fiddler and unexpectedly encountered the Unknown Publisher UAC prompt. Before I fired off a flaming tweet to the team, I double-checked the binary and…
What? I immediately thought of the “only special folders” case I hit last year, but of course this third-party file wasn’t signed by the Windows build lab:
To double-check, I looked at the same file on a different machine, where the correct UAC prompt appeared showing the expected publisher. Hmmm…
I asked Gemini, which hallucinated a plausible but incorrect answer:
I knew this guess was wrong right away because I’ve studied the MoTW behaviorextensively for decades. Gemini’s second guess was better:
This guess was non-hallucinated and plausible, but the fact that the File Properties dialog showed the file as correctly signed implied that this guess was wrong.
Still. I looked at my affected machine’s Trusted Root Certification Authorities stores. Note the s there– your trusted roots are a merge of two storage areas, one for your local user account, and one for your local machine.
And here’s where things get interesting: I’ve got the GCC R45 Intermediate certificate in my local user account but not for the local machine:
That seems… weird. Weird enough to be an explanation. I complained to Gemini that it had gotten it wrong, but didn’t wait around for its insightful answer:
Instead, I used an old signature troubleshooting trick, enabling CAPI2 Logging in the Windows Event Viewer, and getting another data point. The problem was indeed that consent.exe (the UAC prompt) was failing to build a chain to a trusted root.
At that point, I used the certificate viewer to export the GCC root from the local user store and imported it into the local machine store.
Success!
Now, unlike Gemini, I have access to the source code to Windows, so I went off to discover exactly what it had already told me in my forgotten browser window:
Gemini further suggested that I try using an Elevated command prompt to verify the file’s signature:
… but Gemini failed to recognize that using an elevated command prompt isn’t going to use the non-working SYSTEM context. My user account, even elevated, will verify the signature just fine:
Finally, only one real mystery remained: How did I have the correct intermediate in the local user store and why didn’t whatever magic put it there get it into the local machine store too?
AIA Fetching
By default, when Windows builds a certificate chain, if it cannot find the Intermediate certificate that the signing certificate chains to, it will look for a place to download that Intermediate using a process called Authority Information Access (AIA). And indeed, you can see the intermediate certificate’s URL right there:
As a part of chain building for a normal certificate validation, WinVerifyTrust will fetch the intermediate, confirm that it validly chains to a trusted root, and then use it to complete the chain.
Unless you tell WinVerifyTrust not to do that.
And, indeed, it turns out that UAC‘s consent.exe code disables URL fetching by passing the WTD_CACHE_ONLY_URL_RETRIEVAL flag into its WinVerifyTrust call. This flag prevents hitting the network to fetch information related to the certificate. Most of the time, that flag means not fetching a CRL or OCSP URL to see if the certificate was revoked, but the flag also controls whether AIA is used to fetch any needed intermediates in the rare instance that such an intermediate is missing locally. Like this one.
And there we have it. Explorer’s File Prompting and UAC’s Consent Prompt exhibit different behavior because Explorer’s signature verification uses the user’s certificate context and fetches intermediates via AIA, while UAC uses the system context and refuses to fetch intermediates. One way a software developer can help prevent this problem is to embed the intermediate certificate into the file’s Authenticode signature directly using the ac parameter to signtool.exe.
When a device first joins a network, the upstream network hardware has full control over its traffic and can allow/block any packets sent from the device from reaching the Internet. Many public networks (typically Wi-Fi, but sometimes wired, located in hotels, coffee shops, mass transit, schools, etc.) require that the user accept Terms of Use or otherwise interact with a webpage before gaining broader/unrestricted access to the network.
The restricted client is called a “captive” and the webpages to allow removal of the access limitation are called Captive Portals, the subject of a previous post.
When Windows detects that a network is blocking internet access with a Captive Portal, a link is provided to launch the captive user’s default browser:
The captive browser is navigated to a non-secure HTTP url, and the network is expected to intercept the non-secure request and redirect to the Captive Portal webpage.
The Attack
Over the last few years, there have been a series of attacks (2025, 2026) where attackers have used fake “captive portal” web pages that entice users to download and run malware…
… or follow malicious instructions that result in compromise of the device, with this initial access being abused to steal credentials and move laterally within the victim’s organization.
Defenses
Beyond educating users not to follow any unusual instructions on captive portal pages, users can set ShellSmartScreenLevel to BLOCK to help prevent users from running downloaded malware when working offline with security services unreachable.
With the policy set, users may not override the “SmartScreen Unreachable” warning dialog
For enterprise-managed devices, organizations can choose to prevent Wi-Fi connections to networks that have not been provisioned via MDM. See AllowManualWifiConfiguration.
Threat Analysis
From a security perspective, untrusted networks attacks are mostly equivalent to the fake captive portal threat described here. In both, an attacker on the network path can observe and tamper with all non-secure (e.g. HTTP) requests, and can partially-observe, block, or delay any secure (e.g. HTTPS, VPN) connections. Network attackers can behave selectively, allowing some connections while interfering with others.
Compared to traditional network attacks, the key differences for Captive Portal attacks are:
Users have been primed (by prior experience) to expect Captive Portal pages and comply with their instructions. There’s no standard for how Captive Portals behave and virtually every venue has different instructions for connections. This non-uniformity means that users are more easily socially-engineered into performing unsafe operations.
All captive portal flows inherently start with a non-secure HTTP request. Using HTTP is what allows a legitimate guest network to direct the user to the Captive Portal page because HTTPS encryption prevents tampering. Features like “automatic HTTPS upgrades” have to be disabled in Captive Portal scenarios because they break the expected flow.
This non-secure protocol usage allows the network-based attacker to take over the flow.
Microsoft Defender for Endpoint is a paid security product that extends Microsoft Defender Antivirus (included for free in Windows) with enterprise capabilities, including a full EDR product that reports security telemetry into a security operations center (SOC).
Some recently-retired Microsoft employees have been surprised to discover that their personal PCs that had previously been used in a BYOD (bring your own device) manner are still applying Microsoft Internal IT security policies. For example, an ex-employee might see this block from Network Protection when visiting Grammarly.com:
When this happens, the user is often confused, either thinking Microsoft Defender is blocking everyone from visiting Grammarly’s website (it’s not), or realizing that it’s only their device previously used for work but not understanding why/how.
Some ex-employees are especially confused because they know that they remembered to “offboard” their device from Microsoft Corp, leaving the domain and/or performing the Intune offboarding process.
The confusion arises because Microsoft Defender for Endpoint doesn’t care whether your device has been disjoined from the domain or offboarded from Intune policies: neither of those actions will offboard your device from the enterprise’s Microsoft Defender tenant.
Today, the Windows Security app makes no mention of the fact that a given device is onboarded to a Microsoft Defender tenant:
Unfortunately, Defender-onboarded machines don’t mention this in the Windows Security App
To see whether your device is onboarded, you can check the registry. Look inside the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Advanced Threat Protection key for the OnboardedInfo for information about which tenant your device is onboarded to. The senseId and senseGuid are the unique identifiers sent to the SOC when reporting security telemetry.
To offboard your device from an enterprise’s Defender tenant, you can’t simply delete a few registry keys and call it a day. Many of Defender’s registry keys and services are protected from modification by Tamper Prevention code inside Defender’s kernel module that’s designed to prevent unauthorized modifications.
Instead, to offboard you must run a signed offboarding script generated by the organization. The script notifies the MDE client that the machine should be offboarded. Only after the client verifies the validity (signature, expiration) of the offboarding blob will the device successfully offboard. These validity checks are critical, because otherwise initial access malware on a device could simply begin its attack by offboarding the device before further elevating privileges, moving laterally, and attacking the rest of the organization.
Unfortunately for ex-employees without access to their former-company’s IT helpdesk (to allow them to run the offboarding script), the easiest way to resolve this situation is to back up your files and then reinstall Windows.
This document contains some thoughts about the storage and comparison of URLs, common operations crucial to the correct function of security software like Microsoft SmartScreen.
Importantly, URLs are also displayed on security surfaces to enable the user to make a decision based on their contents. Eight years ago, I wrote the Guidelines for URL Display.
Background
The Web allows linking and retrieval of various resources via an address known as a URL. A URL (Uniform Resource Locator) is an identifier used to locate a resource on the Internet. (Note: “URI” stands for Uniform Resource Identifier, and in the real world, the two terms are used interchangeably).
It’s tempting to think of URLs as plain strings because almost all clients (like web browsers) and servers accept URL input as strings. However, despite the existence of various standards for the representation of URLs, there is considerable variation in the handling of URLs that creates inconsistency and variability in the interpretation of URL values. Additional complexity arises because URLs can include seldom-used components that must be understood to properly interpret the URL, and the URL syntax varies between different URL protocol schemes (e.g. HTTPS/HTTP/FTP/mailto/blob/data, etc).
The complexity of URLs is often exploited by attackers, either to socially-engineer users (as in phishing attacks) or to bypass security checks in code.
URL Encoding as Strings
In their canonical form, HTTPS/HTTP URLs are meant to contain only a subset of US-ASCII characters, where characters outside of that subset (e.g. emojis; international character sets like Cyrillic, Hiragana, Katakana, Kanji, etc; and ASCII characters like :@/\?# that are reserved for use to delimit URL components) are meant to be escaped. Escaping is a system whereby a character’s UTF-8 octet (byte) representation is serialized to ASCII by preceding each octet’s value with a % character. For example, the URL:
In this example, the 자 (Hangul Syllable Ja) character has the UTF-8 encoding 0xEC 0x9E 0x90, which is escaped in the URL to the sequence %EC%9E%90, while the Փ (Armenian Capital Letter Piwr) character with the UTF-8 encoding 0xD5 0x93 is escaped to %D5%93.
The hostname of the URL does not use %-escaping, instead relying on a much more complicated escaping mechanism (Punycode), wherein each DNS label component containing non-ASCII characters is prefixed by xn-- followed by ASCII text that encodes any non-ASCII characters. For instance, a URL containing hostname characters from Thai and Latin character sets:
However, not all clients properly support standards-based encoding behaviors– some clients aim to maintain legacy compatibility with behaviors that existed before the current standards were written.
In particular, Microsoft’s MSHTML (and the WinINET network stack beneath it) used by Internet Explorer, Web Browser Controls (WebOCs) and other common Windows platform features, only implements the standards-based behavior when certain flags are set. Otherwise, MSHTML can put raw UTF-8 octets in the hostname component, and put ANSI codepaged (ACP) octets in the path, query, and fragment components.
The snowman emoji is properly UTF-8 escaped in the path component, and it is thunked down to a question mark ? in the query component (as the target character doesn’t exist in the system codepage). However, if we instead pick a character in the system codepage, we see the path component is UTF-8 escaped, but the query string’s raw ACP octet is put out on the wire:
Furthermore, this path-escaping is sensitive to the checkbox in the Internet Control panel, such that even the path may be sent without escaping if the “Send URL path as UTF-8” checkbox is unticked:
The Default URLMon codepath sets those options inside based on conditional logic, while the EdgeHTML fork of URLMon more simply sets them. Computation of the options is complex: for instance, Edge Legacy checks not only the Zone but also allows a Microsoft-delivered CompatView list to weigh in on the proper encoding.
Beyond the WinINET behavior, it is believed that many other HTTP stacks do not properly handle corner-cases (e.g. being passed URLs that do not follow standards-based escaping rules, contain octet sequences that cannot be validly represented in Unicode, utilize overlong encodings, etc.
URL Components
A URL is made up of a sequence of components. For example, an absolute URL containing all available components for the HTTPS protocol scheme might look like this:
In contrast, a relative URL, as seen within a web page, might be as simple as /file.html.
Making URLs Absolute: Combine
Relative URLs are rarely usable on their own; typically, the first thing that code must do before operating on a relative URL is convert it to an absolute URL by performing a combine operation on the relative URL with the absolute URL of its context (e.g. the web page in which it appears) to generate a new absolute URL. For example, combining https://user:pass@sub.example.com:8080/path?query#fragment with /file.html results in an absolute URL of https://user:pass@sub.example.com:8080/file.html. In this combination operation, the context URL’s path is overwritten, and its query and fragment components are dropped.
Deep Dive: Components
Let’s look at each of the URL components and explore how attackers might attempt to confuse code or humans with each component.
Component: Scheme
The scheme component of the URL designates what underlying protocol should be used to retrieve the information, as well as dictating the rules for interpreting the rest of the URL, including whether it uses the standard hierarchical syntax (e.g. HTTP/HTTPS/FTP/FILE) or the opaquegeneric syntax (e.g. mailto/data/blob).
Clients only support a limited set of URL schemes. Adding new schemes to browsers to retrieve resources or to open external applications generally requires installing native code; a web-platform mechanism allows adding schemes from JavaScript, but when invoked the custom-scheme URL is simply translated into a HTTPS URL for further use.
Security Considerations: Supported Scope
Each security-sensitive client must consider how it handles less common URL schemes; many clients will block all URLs except those using popular schemes (HTTP/HTTPS/mailto), but some clients (e.g. browsers) must support invocation of arbitrary URLs.
Because uncommon schemes are a common vector of security compromise, the decision of how to handle such schemes is an important one. If a security mechanism deems certain schemes out-of-scope, then the user could be exploited by those schemes. For example, mailto: links can be used in phishing:
Security Considerations: Parsing changes by Scheme
Each feature that attempts to analyze a URL for security purposes must understand the scheme of the URL and the rules by which it is parsed. For example, a mailto URL uses the format mailto:user@host.com?subject=messagecontent whereby the Internet address appears in the middle of the URL alongside other optional field content.
Security Considerations: Not All Schemes are Routable
Some URL schemes do not refer to a server on the internet; the most broadly supported and commonly used of these are the data and blob schemes.
A data: schemed URL contains the entire resource to which it refers. For example, if a client fetches the URL data:text/html;base64,PGgxPkhlbGxvIHdvcmxkPC9oMT4=, the result is the string <h1>Hello world</h1>, the base-64 decoding of the substring beginning PGg and ending with 4=.
A blob schemed URL refers to a resource which exists only in the memory of the JavaScript context that generated the URL via the createObjectURL() JavaScript method. JavaScript running inside a web page at https://webdbg.com/test/data.htm might generate a blob URL that looks like this:
When fetched from JavaScript inside the originating webpage, that blob URL will return an object (anything from an image to a file download to a HTML document). However, attempting to fetch that same blob URL from any other device (or even another browser window on the same device) will not return any content, because the blob scheme is not globally routable.
Security Considerations: The FILE Scheme is Weird
The file scheme allows routing a request to a file on either the local file system or the filesystem of a remote server. A URL like file:///C:/test.html refers to a file on C: drive of the system where the fetch retrieval occurs, while file://serverhostname/docs/test.html refers to a file on the share named docs on the server named serverhostname.
Fortunately, use of the file scheme in modern browsers is somewhat restricted because retrieving file URLs can result in assorted security and privacy badness, including fingerprinting the apps on the user’s computer or leaking the user’s Windows password hash to a remote servers.
Component: Authority
The Authority component of the URL consists of three subcomponents: userinfo, the fully-qualified hostname, and the port.
Component: UserInfo
The userinfo subcomponent of a URL specifies a username and password that the client should use when authenticating to a server. This subcomponent is only defined for certain URL schemes (e.g. FTP), while it is officially invalid for others (e.g. HTTP and HTTPS) but nevertheless supported (e.g. Firefox and Chromium allow userinfo for HTTP and HTTPS URLs).
Security Considerations: UserInfo UI Spoofing
Way back in Internet Explorer 6, IE started forbidding HTTP/HTTPS URLs containing userinfo because this obscure subcomponent’s primary real-world use was to confuse the user as a part of phishing attacks. Because the UserInfo is typically not present in URLs, a user looking at the URL https://victim.com:80@random.text.evil.com/ will often assume that they are looking at content from victim.com rather than from random.text.evil.com.
This threat vector is not terribly common today: Chromium hides the UserInfo component in its address bar, while Firefox explicitly warns the user about this threat:
The most security-relevant part of a URL is the fully-qualified hostname of the URL. The hostname is registered by an individual or business (e.g. PayPal, Inc. owns Paypal.com) with the relevant DNS registrar of the top-Level domain (e.g. Verisign controls the .com top-level domain).
Security Considerations: Transport Security
If the URL’s scheme is secure (e.g. HTTPS), content delivered from a given hostname is deemed to be under the control of the entity that registered the hostname (modulo compromised infrastructure, etc). However, if the scheme refers to a non-secure protocol like HTTP, and especially if the protocol traverses an untrusted network, the registrable domain information may not accurately describe the true source of the content because the content may have been modified by a man-in-the-middle on the network. The port number only needs to be specified if it is not the default for the scheme (e.g., 80 for HTTP, 443 for HTTPS).
Security Considerations: Parsing of IPv6 Literals
When the URL’s hostname is an IPv6 literal, the address is wrapped in square brackets, e.g. http://[::1]:8080/file.html is a reference to a file hosted on port 8080 of the current device’s IPv6 loopback interface. The fact that a colon character can appear before the colon delimiter representing the start of the Authority’s Port subcomponent can confuse a parser unfamiliar with IPv6 addresses.
While there’s no standard for including an IPv6 scope id within a URL, WinINET allows specification of the scope by %-encoding the % delimiter character, e.g. https://[::1%253]/
Security Considerations: Interpretation of IPv4 Literals non-canonical syntax
Most technically savvy users are familiar with IPv4 literal addresses in dotted decimal format, like http://127.0.0.2/. However, dotted decimal is not the only format; you can also express the same address by dropping the 0. components, like http://127.2, with extra 0s like http://127.000000002/, or in decimal notation http://2130706434/, octal notation http://0177.0.0.2/, or hexadecimal notation http://0x7f000002/.
The various serializations of addresses could be used to evade matching logic. Historically, we’ve also seen some code that assumes that any hostname lacking a dot must not be globally routable and belongs to the (more trustworthy) Intranet zone (leading to an MSRC case for Windows/IE in the early 2000s).
Security Considerations: IDN and PunyCode
Support for non-Unicode characters in URLs can lead to spoofing attacks.
Usually, the security context that the user cares about is the registrable domain of the top-level page’s URL’s origin, even when a given page is made up of components from many different origins. The registrable domain typically consists of a subdomain of an entry on the Public Suffix list. For instance, bbc.co.uk is a registrable domain under the co.uk public suffix. The fully-qualified hostname consists of a registrable domain, and optionally one or more subdomain labels.
The port component of the URL indicates which TCP/IP port should be contacted to send the request.
Security Considerations: Shared Servers and “Well-Known” Ports
In general, a server operator is deemed to be in control of all ports on the server, although notably some systems (e.g. Unix) allow low-permissioned users to perform TCP/IP listen operations only on certain ports (>1024) while requiring administrative permissions to listen to “low ports” (<1024) which are the default ports used by popular services (HTTP/HTTPS/FTP).
Security Considerations: Canonicalization Drops Default Port
When canonicalizing a URL, if the specified target port is the default port for the scheme, it should be removed from the URL entirely. For example,
The Web Platform security model uses a term called “Origin” which is comprised of the triplet scheme+fullyQualifiedHostname+port.
Challenges and Threats
Malicious websites are motivated to misrepresent their provenance in order to trick visitors into performing an unsafe action (e.g., phishing, malware install) or to otherwise grant unwarranted trust in the information provided by the site (e.g., “fake news”).
Other components of the URL (subdomain, userinfo, path, query, and fragment) are completely under the control of the website and may be crafted in an attempt to spoof the user by misrepresenting the registrable domain.
URL Comparison
A critical thing security software needs is a consistent function that turns “a pile of octets that some client is treating as a URL” into “a string that the security software considers to be the canonical / common form that we will use in all subsequent matching logic, even if that string would not be accepted by a real server.”
That function will need to handle things like bare-ACP octets appearing anywhere in the string, invalid UTF-16 sequences, raw (non-encoded) UTF-16 codepoints, and anything else we devise.
Canonicalization and Normalization
We’ll also need comparison functions that work correctly both with and without the most common forms of canonicalization/normalization performed by servers (e.g. https://example.com/blah/..///thisisafile.htm matches https://example.com/thisisafile.htm. Similarly, there are many ways to represent equivalent IPv6 literals, and so on.
Matching and “Rollups”
Beyond that, the software needs to decide how closely two URLs must match to be considered equivalent.
For example, http://example.com/PaTH and http://example.com/path are technically different URLs, but in actual practice, they will return the same content on from most servers.
While the port component in the URL is technically a part of the web origin, in actual practice, it is very uncommon for an arbitrary port to be controlled by a different entity than the default port, and there’s no evidence to suggest that any human being will make a different security decision based on the target port number. As such, security software will often ignore the port when comparing URLs. In Microsoft SmartScreen, for example, a rule set to block https://x.com (implicitly port 443 due to the HTTPS scheme) will also block requests to http://x.com:12345.
Any URL matching function needs to decide how closely two URLs must match; in SmartScreen, we call this roll-up, meaning “Will a block for X.com block X.com/something? What about sub.x.com/anything?” We call these “path rollup” and “domain rollup”, respectively.
Challenge: Supporting URL “Scrubbing”
URLs often contain sensitive information, ranging from PII to document titles, to security nonces that are intended to be available only for a single client computer. In an attempt to limit the privacy impact of URL transmission/telemetry, software may attempt to “scrub” the URL, replacing private data with a replacement character (e.g. https://phone.com/425-830-6600/call is masked as https://phone.com/XXX-XXX-XXXX/call).
Because scrubbing logic is, at best, based on imprecise heuristics, it is subject to false negatives (sensitive values not masked) and false positives (PII-looking values that are e.g. actually just meaningless numbers formatted as if they were telephone numbers).
Any attempt at scrubbing creates a mechanism by which an attacker can easily interfere with URL matching logic. The attacker can “cloak” their endpoint such that it responds only with innocuous content when a “PII-looking” value is omitted from the request URL (e.g. by a server-side detonator that is using a scrubbed URL).
Any attempt at using scrubbed values in threat intel feeds creates problems if the components creating and consuming the feed do not recognize how sensitive data is masked within that data, potentially leading to URL matching ambiguities.
Challenge: Inconsistent Length Limitations
URLs do not have a consistent length limit. Chromium limits URLs to 2MB for navigation (and 32kb for display, in some contexts), while various components of Windows use 2083 characters, and some servers and security software have URL length limits that are longer and shorter.
Some security software imposes a lower limit on URL length, returning error pages if the URL length exceeds their limit (e.g. 4096 characters):
