The author is well aware of this issue. But according to him, "there is no way around this" (http://foicica.com/wiki/lspawn). He obviously knows how to create sub-process on Windows using CreateProcess but the problem is that the rest of the application code all use FILE* for file I/O. FILE is an opaque type, there is no easy way to convert a Win32 pipe HANDLE to a FILE*.
Today, I decided to give this a shot. Firstly, I need to find out why is the console window popping up at all. I know whatever Textadept uses to create sub-process (glib or lua?), it must eventually call the Win32 CreateProcess function to do the job. So let's fire up our debugger (I used x64_dbg http://x64dbg.com) and set a breakpoint at kernel32!CreateProcessA to see what went wrong.
When the debugger breaks in, the stack looks like this:
The first item is the return address, then the 10 arguments for the CreateProcess function. The most interesting one is the 9th argument 'lpStartupInfo' so let's check out what is inside:
In order to hide a sub-process's window, the parent process needs to set the flag STARTF_USESHOWWINDOW in dwFlags, and set the show window command in wShowWindow to SW_HIDE. Looking at the screenshot, the caller obviously didn't set them up correctly (dwFlags and wShowWindow are the 2 words at the end of the 3rd row and the beginning of the 4th row). This is why we are seeing the black console window popping up.
So how do we fix this? I really don't feel like to download the source code and all its dependencies then spend half a day trying to figure out how to build the application from source, especially when its author has explicitly declared "Compiling Textadept on Windows is no longer supported. The preferred way to compile for Windows is cross-compiling from Linux".
But wait, isn't Textadept mostly written in Lua? Lua is well known for being extremely easy to extend. How about a Lua dll extension that dynamically patches the CreateProcess call and supply it with correct arguments?
It didn't take me much time to decided I want to write this extension dll in Free Pascal. Why? because Free Pascal ships with Lua units out of the box.
Putting together a minimal extension in Pascal is just a matter of minutes:
library fixpopen;
{$mode objfpc}{$H+}
uses
Classes, lua, lauxlib, lualib, windows;
function patch(L:Plua_State):integer;cdecl;
begin
OutputDebugString('patching');
result:=1;
end;
function unpatch(L:Plua_State):integer;cdecl;
begin
OutputDebugString('unpatching');
result:=1;
end;
function libinit(L:Plua_State):integer;cdecl;export;
begin
lua_register(L, 'fix_popen_patch', @patch);
lua_register(L, 'fix_popen_unpatch', @unpatch);
result:=0;
end;
exports
libinit;
initialization
end.
Compile this code to a DLL, put it in the same directory as Textadept, then add these lines to ~/.textadept/init.lua
mylibinit = package.loadlib("fixpopen.dll", "libinit")
mylibinit()
fix_popen_patch()
Open DbgView and run Textadept. The tracing messages appeared in DbgView as you'd' expected.
Now it's time to add the API hooking. The term "API hooking" sounds intimidating but the principle is really simple. We put a small piece of code at the entry point of the function we want to hook. This piece of code, we call it a 'thunk', will catapult us to our own function, there, we need to restore the first few bytes of the original function, call the orignal function, then put the thunk back so that we can intercept the next API call.
This is the definition of the thunk type:
{$PACKRECORDS 1}
TThunk = record
jmp : byte;
offset: longword;
end;
{$PACKRECORDS DEFAULT}
It is just one assembly instruction. The first byte is the op code, which should be $e9, or JMP. Followed by a 4-byte offset.
We need 2 of these thunks, one to store our JMP instruction, another one to store the original content which will be overwritten by our thunk so that we can restore it later.
Some initialization is needed when the DLL is loaded into the Textadept process:
var
thunk : TThunk = (jmp:$e9; offset:$0);
save : TThunk = (jmp:$0; offset:$0);
w32CreateProcess : TCreateProcess = nil;
initialization
if w32CreateProcess = nil then
begin
// save the API call address
w32CreateProcess := TCreateProcess(GetProcAddress(GetModuleHandle('kernel32.dll'), 'CreateProcessA'));
// save API function prelude
CopyMemory(@save, w32CreateProcess, sizeof(TThunk));
// fill in the thunk
thunk.offset:= pointer(@myCreateProcess) - pointer(w32CreateProcess) - 5;
end
end.
Here we save the address of the CreateProcess function, and its first 5 bytes in 'save'. We also fill in the thunk with the jump offset, which is the relative distance between the next instruction after the JMP instruction (that's where the -5 comes from) to the target address.
Now the fun part, patching the Windows API code.
function patch(L:Plua_State):integer;cdecl;
var
bret: BOOL;
begin
bret:=VirtualProtect(w32CreateProcess, sizeof(TThunk), PAGE_EXECUTE_READWRITE, @protect);
CopyMemory(w32CreateProcess, @thunk, sizeof(TThunk));
VirtualProtect(w32CreateProcess, sizeof(TThunk), protect, nil);
result:=1;
end;
Because the code segment is write protected by default, we first need to turn off the protection. Then we just copy the thunk over CreateProcess's code. We already have a saved copy of that memory so we can easily restore that when we need the original code back.
The unpatch function is almost identical, except it copies the saved stuff back.
unpatch(L:Plua_State):integer;cdecl;
var
bret: BOOL;
begin
VirtualProtect(w32CreateProcess, sizeof(TThunk), PAGE_EXECUTE_READWRITE, @protect);
CopyMemory(w32CreateProcess, @save, sizeof(TThunk));
VirtualProtect(w32CreateProcess, sizeof(TThunk), protect, nil);
result:=1;
end;
Finally, the redirected CreateProcess function:
function myCreateProcess(...
lpStartupInfo:LPSTARTUPINFO;
...):WINBOOL;stdcall;
begin
unpatch(nil);
lpStartupInfo^.dwFlags:=lpStartupInfo^.dwFlags or STARTF_USESHOWWINDOW;
lpStartupInfo^.wShowWindow:=SW_HIDE;
result := w32CreateProcess(lpApplicationName,lpCommandLine,lpProcessAttributes,
lpThreadAttributes,bInheritHandles,dwCreationFlags,lpEnvironment,
lpCurrentDirectory,lpStartupInfo,lpProcessInformation);
patch(nil);
end;
It has exactly the same signature as CreateProcessA. Once we get in here, we first fix the original function so that it can be called. Then we modify the StartupInfo argument to hide the console window. Lastly, call the windows CreateProcessA function and put the thunk back.
And this perfectly solves the black console window flashing issue in Textadept.
The complete source code can be found here http://pastebin.com/zpyHrtRN
