Saturday, August 30, 2014

Fixing Textadept's console window flashing issue (with API hooking in Free Pascal)

Textadept is a lovely little text editor I have been using alongside Emacs for some time. There is one small annoyance about it: on Windows, a black console window pops up then disappears every time it runs a sub-process to perform some task.

The author is well aware of this issue. But according to him, "there is  no way around this" (  He obviously knows how to create sub-process on Windows using CreateProcess but the problem is that the rest of the application code all use FILE* for file I/O. FILE is an opaque type, there is no easy way to convert a Win32 pipe HANDLE to a FILE*.

Today, I decided to give this a shot. Firstly, I need to find out why is the console window popping up at all. I know whatever Textadept uses to create sub-process (glib or lua?), it must eventually call the Win32 CreateProcess function to do the job. So let's fire up our debugger (I used x64_dbg and set a breakpoint at kernel32!CreateProcessA to see what went wrong.

When the debugger breaks in, the stack looks like this:

The first item is the return address, then the 10 arguments for the CreateProcess function. The most interesting one is the 9th argument 'lpStartupInfo' so let's check out what is inside:

In order to hide a sub-process's window, the parent process needs to set the flag STARTF_USESHOWWINDOW in dwFlags, and set the show window command in wShowWindow to SW_HIDE. Looking at the screenshot, the caller obviously didn't set them up correctly (dwFlags and wShowWindow are the 2 words at the end of the 3rd row and the beginning of the 4th row). This is why we are seeing the black console window popping up.

So how do we fix this? I really don't feel like to download the source code and all its dependencies then spend half a day trying to figure out how to build the application from source, especially when its author has explicitly declared "Compiling Textadept on Windows is no longer supported. The preferred way to compile for Windows is cross-compiling from Linux".

But wait, isn't Textadept mostly written in Lua? Lua is well known for being extremely easy to extend. How about a Lua dll extension that dynamically patches the CreateProcess call and supply it with correct arguments?

It didn't take me much time to decided I want to write this extension dll in Free Pascal. Why? because Free Pascal ships with Lua units out of the box.

Putting together a minimal extension in Pascal is just a matter of minutes:

library fixpopen;
{$mode objfpc}{$H+}

  Classes, lua, lauxlib, lualib, windows;

function patch(L:Plua_State):integer;cdecl;

function unpatch(L:Plua_State):integer;cdecl;

function libinit(L:Plua_State):integer;cdecl;export;
  lua_register(L, 'fix_popen_patch', @patch);
  lua_register(L, 'fix_popen_unpatch', @unpatch);



Compile this code to a DLL, put it in the same directory as Textadept, then add these lines to ~/.textadept/init.lua

mylibinit = package.loadlib("fixpopen.dll", "libinit")

Open DbgView and run Textadept. The tracing messages appeared in DbgView as you'd' expected.

Now it's time to add the API hooking. The term "API hooking" sounds intimidating but the principle is really simple. We put a small piece of code at the entry point of the function we want to hook. This piece of code, we call it a 'thunk', will catapult us to our own function, there, we need to restore the first few bytes of the original function, call the orignal function, then put the thunk back so that we can intercept the next API call.

This is the definition of the thunk type:

  TThunk = record
    jmp   : byte;
    offset: longword;

It is just one assembly instruction. The first byte is the op code, which should be $e9, or JMP. Followed by a 4-byte offset.

We need 2 of these thunks, one to store our JMP instruction, another one to store the original content which will be overwritten by our thunk so that we can restore it later.

Some initialization is needed when the DLL is loaded into the Textadept process:
  thunk : TThunk = (jmp:$e9; offset:$0);
  save  : TThunk = (jmp:$0;  offset:$0);
  w32CreateProcess : TCreateProcess = nil;

  if w32CreateProcess = nil then
     // save the API call address
     w32CreateProcess := TCreateProcess(GetProcAddress(GetModuleHandle('kernel32.dll'), 'CreateProcessA'));
     // save API function prelude
     CopyMemory(@save, w32CreateProcess, sizeof(TThunk));
     // fill in the thunk
     thunk.offset:= pointer(@myCreateProcess) - pointer(w32CreateProcess) - 5;

Here we save the address of the CreateProcess function, and its first 5 bytes in 'save'. We also fill in the thunk with the jump offset, which is the relative distance between the next instruction after the JMP instruction (that's where the -5 comes from) to the target address.

Now the fun part, patching the Windows API code.

function patch(L:Plua_State):integer;cdecl;
  bret: BOOL;
  bret:=VirtualProtect(w32CreateProcess, sizeof(TThunk), PAGE_EXECUTE_READWRITE, @protect);
  CopyMemory(w32CreateProcess, @thunk, sizeof(TThunk));
  VirtualProtect(w32CreateProcess, sizeof(TThunk), protect, nil);

Because the code segment is write protected by default, we first need to turn off the protection. Then we just copy the thunk over CreateProcess's code. We already have a saved copy of that memory so we can easily restore that when we need the original code back.

The unpatch function is almost identical, except it copies the saved stuff back.

  bret: BOOL;
  VirtualProtect(w32CreateProcess, sizeof(TThunk), PAGE_EXECUTE_READWRITE, @protect);
  CopyMemory(w32CreateProcess, @save, sizeof(TThunk));
  VirtualProtect(w32CreateProcess, sizeof(TThunk), protect, nil);

Finally, the redirected CreateProcess function:

function myCreateProcess(...
  lpStartupInfo^.dwFlags:=lpStartupInfo^.dwFlags or STARTF_USESHOWWINDOW;
  result := w32CreateProcess(lpApplicationName,lpCommandLine,lpProcessAttributes,

It has exactly the same signature as CreateProcessA. Once we get in here, we first fix the original function so that it can be called. Then we modify the StartupInfo argument to hide the console window. Lastly, call the windows CreateProcessA function and put the thunk back.

And this perfectly solves the black console window flashing issue in Textadept.

The complete source code can be found here


Michal Kottman said...

I'm not sure which version of lspawn you are using, but the lspawn is settings the flags as needed for something like 5 months:

Maybe the blinking console is caused by running os.execute() somewhere in the code, in which case your runtime patch helps.

finalpatch said...

The black console is from the textredux module's use of io.popen().

Arseny Kapoulkine said...

Since you're just patching the existing functions, you don't really need to expose additional entrypoints to Lua - you can patch in DllMain. That means you don't have to use Lua in the DLL at all.

Lachlan said...

Weird coincidence: This is the second time I've seen code for Thunk Patching. The first time I saw it was also written in Pascal - Cheat Engine implements a speedhack by overriding GetTickCount and other time functions.

Is there something about Pascal that encourages people to write Thunk Patchers? It looks like that code could be translated to C or C++, line for line, and get the same result with no extra complexity.