ShellExecute Doesn’t

My oldest supported Windows application is a launcher app named SlickRun, and it’s ~24 years old this year. I haven’t done much to maintain it over the last few years, although it’s now available in 64-bit and runs great on Windows 10. (Thanks go to Embarcadero, who now offer a free “Community” edition of Delphi, the language/platform I ported SlickRun to circa 1994).

I still fix bugs in SlickRun from time to time, and as I was playing with Rust a few days ago I was reminded of one of the oldest limitations in my code– if you update your system’s %PATH% variable, those changes aren’t seen by applications/consoles spawned by SlickRun (even after the change) until you restart SlickRun. This is particularly annoying because it’s so unexpected– users expect that command consoles launched by Win+R,cmd.exe,Enter will behave the same way as Win+Q,cmd,Enter, but the former consoles have the updated %PATH% while the latter do not.

While ShellExecute() sounds like it’s an API that causes the shell (aka Explorer) to execute something, in fact it does nothing of the sort.

Updating the Environment Block

The root cause of the “outdated path” problem is that processes launched via ShellExecute inherit the environment variables of their spawning process, and those environment variables (typically) are assigned as the process launches and never touched again. Because SlickRun starts with Windows, the %PATH% when it starts is the %PATH% that every process it launches inherits. (You can easily view a process’ environment block using the Properties > Environment tab in Process Explorer).

So, how does Explorer detect the change? That part I figured out ages ago– after updating an environment variable, the System Properties > Environment Variables Control Panel UI (or the SetX.exe console tool) broadcasts a WM_SETTINGCHANGE message to all top-level windows with a lparam containing the string “Environment”. I could easily add code to SlickRun to detect that the variables had changed, but for decades I didn’t really know what to do next… I didn’t know how to read the updated variables (without doing something hacky like restarting the process) nor ensure that they were passed to the applications spawned by ShellExecute.

Yesterday, I got fed up and started Googling. A few posts on StackOverflow mentioned a promising-sounding function, RegenerateUserEnvironment. And while that function appears to be undocumented, there’s an amazing issue filed in an open-source tracker that explains exactly how Windows Explorer uses this function– basically, just wait for the WM_SETTINGCHANGE event, then call the API. The RegenerateUserEnvironment will replace the calling process’ current environment block with the latest values.


// Add to the Private section of your main form's type declaration.
Procedure WMSettingChange(Var MSG: TMessage); MESSAGE WM_SETTINGCHANGE;
// If the system PATH environment variable changes, we need to call an
// undocumented Windows Shell function to rebuild our own Environment
// block such that new consoles/apps we spawn will see the new PATH.
Procedure TMain.WMSettingChange(Var MSG: TMessage);
var hLib: THandle;
pfnRegenerate: Function (oldEnv: Pointer; regenCurrent: BOOL): BOOL; StdCall;
pNil: Pointer;
Begin
if (MSG.LParam = 0) then Exit;
OutputDebugString(PChar(Format('WM_SettingChange in area "%s"', [PChar(MSG.LParam)])));
if (0 <> StrComp(PChar('Environment'), PChar(MSG.LParam))) then Exit;
hLib := LoadLibraryEx('shell32.dll', 0, 0);
if (hLib = 0) then Exit;
pfnRegenerate := GetProcAddress(hLib, 'RegenerateUserEnvironment');
if (Assigned(pfnRegenerate)) then
Begin
pNil := nil;
pfnRegenerate (@pNil, true);
OutputDebugString('SlickRun Process Environment Block updated.');
End;
FreeLibrary(hLib);
End;

Launching at Medium Integrity

While we’re on the topic of executing applications “like the shell”, another scenario came up twelve years ago when Windows Vista was first introduced. The SlickRun installer, written in NSIS, launches SlickRun when installation completes. Unfortunately, the installer runs with Admin rights (High integrity), which means that, by default, all of the programs it launches inherit that integrity. For SlickRun, this is especially bad because it means that any programs that it, in turn, launches during that first session (e.g. your browser!) will run at High integrity too. Not good.

While you can easily use the Runas verb to ShellExecute to launch a High integrity application from a Medium integrity application, there (depressingly) isn’t a way to do the opposite. For years, the official recommendation was to do some fancy coding to clone Explorer’s tokens and use those. Unfortunately, these approaches are quite complicated to implement, especially within a NSIS script.

As it turns out, however, there’s a trivial workaround which works quite well– while ShellExecute doesn’t run things as the shell, applications can easily get Explorer to launch anything they like at Explorer’s integrity. The trick is to simply invoke explorer.exe and pass the filename to be executed as the first command line argument:


; Run what we installed. Use a trick on Vista+ to run as non-Admin
GetDLLVersion "Kernel32.dll" $R0 $R1
IntOp $R2 $R0 >> 16
IntOp $R2 $R2 & 0x0000FFFF ; $R2 now contains major version
IntCmp $R2 6 is6 lessthan6 morethan6
is6:
morethan6:
exec '"$WINDIR\explorer.exe" "$INSTDIR\sr.exe"' ; We use Explorer to launch it to get it to run non-elevated
goto RanIt
lessthan6:
exec '"$INSTDIR\sr.exe"' ; No UAC on XP, but Authenticode prompts if we try to use Explorer to launch app
RanIt:
StrCpy $9 "Success"

While this approach isn’t technically supported, I expect it is likely to continue to work for the foreseeable future.

It’s depressing that together these tricks have taken me almost twenty years to discover, but I’m happy that I have. I hope they help you out.

-Eric

Published by ericlaw

Impatient optimist. Dad. Author/speaker. Created Fiddler & SlickRun. PM @ Microsoft 2001-2012, and 2018-, working on Office, IE, and Edge. Now a GPM for Microsoft Defender. My words are my own, I do not speak for any other entity.

5 thoughts on “ShellExecute Doesn’t

  1. Here’s a tip for NSIS. Change the script to use set “RequestExecutionLevel” to “user”.
    Then change the install path to install to the user’s AppData\Local instead (Microsofts recommendation for per user actually).
    If a admin really need to run a installer as admin they can right click and “Run as administrator” anyway, and a admin is able to manually copy the files anyway.
    You can do fancy things and detect elevated mode and present different paths (i.e. program files folder instead of local) if detecting a elevated script. Let me know if you want to steal ideas from a NSIS script I use.

    As to programs and ENV try to use CreateProcess to launch processes.
    https://docs.microsoft.com/en-us/windows/desktop/api/processthreadsapi/nf-processthreadsapi-createprocessa
    It lets you specify the environment block or use NULL (in witch case the parent process ENV is used).

    Use GetEnvironmentStrings (and FreeEnvironmentStrings when done with it obviously) to get the block.
    This should be a copy of the current env block for the process. (this also means your launch program should re-read as well if it put anything into variables).

    AFAIK this is the only “proper” way to do it (other than reading the env vars from the registry where user and system env stuff are in different places AFAIK)

    While I see no direct mention of it after a WM_SETTINGCHANGE the GetEnvironmentStrings call should return a updated block.
    A test should be easy enough, call the API, don’t free it, wait for the window message then call the API again. (BTW! it’s possible there may be fired multiple WM_SETTINGCHANGE messages).

    The wording in the description for GetEnvironmentStrings is kinda vague, it could mean the block is is the block given to the current process at launch, but it could also mean the block currently available to the current process (which I believe to be true, I have not tested this myself yet).

    ShellEx for urls should be fine as the browsers probably handles paths/env fetching itself, I’ve at least never seen the system default browser have issues with that.

  2. Yes, changing to a per-user install will mean that you get per-user behavior. I don’t want a per-user install.

    Yes, using CreateProcess gives you the ability to pass an environment block, but no, constructing that block is not trivial, and CreateProcess differs from ShellExecute in fundamental and important ways.

    No, GetEnvironmentStrings isn’t sensitive to changes in the system; it simply returns the current process’ (startup) environment block. Hence, this article.

  3. “GetEnvironmentStrings isn’t sensitive to changes in the system”.
    Ah man that really sucks.

    So is the only way to peek into the registry then? (HKLM + HKCU) and get the path variables from there, and I can’t recall if ENV vars are in the same keys as the paths.

    What about CMD START then? By looking at it it has a /I option that you can use to make it inherit the ENV but by default it does not (so it should use the current system ENV).
    Not the most elegant way as you’d be running CMD and then use start. (which itself uses ShellExecuteEx I think).
    Not sure if you can call start directly via ShellEx though. I also have no clue what magic is done by START to get the current env rather than use the parent one, maybe similar to Explorer? If I remember correctly START makes the launched process a child of Explorer rather than the CMD used for the START command (which is itself the child of your SlickRun program).

    Oh btw! It is possible for a program to re-launch itself and prompt for elevation. It’s a tad clunky but possible (I think it was tied to RunAs, but this was years ago so memory is fuzzy). I made a installer that ran as a normal user then gave a choice of user or admin install, relaunched if admin was selected and caused a UAC prompt to display. But I also seem to recall having made a launch wrapper tool too (which had a manifest that asked for administrative privileges) which was used to launch programs as admin (as I said my memory is fuzzy).

Leave a comment