In this Blog I will explain my approach for solving one of the exercises from Practical Reverse Engineering Book, which is enumerating kernel/user Asynchronous Procedure Calls (APC) of a process.

So I started Reverse Engineering Some of the APC related functions, creating windows kernel driver that uses APC in different situation and use Windbg to add breakpoint on the APC callback functions to see how it is dispatched, plus Reading lots of blogs and documentation on how APC is working and how it is dispatched

APC Internal:

Now I will give a short introduction for APC and its internal from kernel mode perspective.

Basically, APCs allow user programs and system components to execute code in the context of a particular thread and, therefore, within the address space of a particular process.

the windows kernel uses APC to complete I/O operations initiated asynchronously, thread suspension,… and it is used by malware author for injecting code inside other process (APC Injection 1, 2).

there is two types of APC user-mode and kernel-mode APC, user-mode APC execute in user space in the target thread’s process context and it requires that the target thread to be in an alterable wait state for being successfully delivered, kernel-APC execute in kernel space and can be classified as regular or special, both kernel/User APC has three functions:

KernelRoutine: this function will be executed in kernel space (at IRQL= PASSIVE_LEVEL in case normal kernel APC and user APC, and at IRQL = APC_LEVEL in case special kernel APC)
RundownRoutine: this function will be called in the kernel space in case the thread is terminated before delivering the APC.
NormalRoutine: this function will be called in kernel space in case kernel-mode APC, and will be called in user space in case user-mode APC

Each Thread has two members of type _KAPC_STATE, named ApcState and SavedApcState in its _KTHREAD Data structure:

ApcState: used regardless of whether the thread is attached to its own process or another process
SavedApcState: used to store APCs for process context which is not the current context and that must wait (for example when a thread attaches to another process, and the APC is queued for its own process).

_KAPC_STATE structure has member called ApcListHead which is two LIST_ENTRY structure and treated as the list head for kernel-APC and user-APC, and will be used to queue APCs for a thread.

1: kd> dt nt!_KTHREAD
............ //Truncated for visibility
   +0x098 ApcState         : _KAPC_STATE       /*APCs that will be dispatched regarding the current process context*/
   +0x098 ApcStateFill     : [43] UChar
   +0x0c3 Priority         : Char
   +0x0c4 UserIdealProcessor : Uint4B
   +0x0c8 WaitStatus       : Int8B
............ //Truncated for visibility
   +0x186 WaitIrql         : UChar
   +0x187 WaitMode         : Char
   +0x140 WaitBlockFill6   : [116] UChar
   +0x1b4 WaitTime         : Uint4B
   +0x140 WaitBlockFill7   : [164] UChar
   +0x1e4 KernelApcDisable : Int2B
   +0x1e6 SpecialApcDisable : Int2B
   +0x1e4 CombinedApcDisable : Uint4B
   +0x140 WaitBlockFill8   : [40] UChar
   +0x168 ThreadCounters   : Ptr64 _KTHREAD_COUNTERS
............ //Truncated for visibility
   +0x240 AffinityFill     : [10] UChar
   +0x24a ApcStateIndex    : UChar
   +0x24b WaitBlockCount   : UChar
   +0x24c IdealProcessor   : Uint4B
   +0x250 NpxState         : Uint8B
   +0x258 SavedApcState    : _KAPC_STATE        /*APCs that will be saved as it can not be dispatched in the current context*/
   +0x258 SavedApcStateFill : [43] UChar
   +0x283 WaitReason       : UChar
   +0x284 SuspendCount     : Char
   +0x285 Saturation       : Char

1: kd> dt nt!_KAPC_STATE
   +0x000 ApcListHead      : [2] _LIST_ENTRY      /*Kernel\user APC queue list head*/
   +0x020 Process          : Ptr64 _KPROCESS
   +0x028 InProgressFlags  : UChar
   +0x028 KernelApcInProgress : Pos 0, 1 Bit
   +0x028 SpecialApcInProgress : Pos 1, 1 Bit
   +0x029 KernelApcPending : UChar
   +0x02a UserApcPendingAll : UChar
   +0x02a SpecialUserApcPending : Pos 0, 1 Bit
   +0x02a UserApcPending   : Pos 1, 1 Bit

APC Enumeration:

First Approach

The first idea i thought of to start enumerating APC queues for all thread inside a process, is by parsing the process thread list from _KPROCESS structure the target member is ThreadListHead _LIST_ENTRY structure, then on each thread I parse the _KTHREAD structure to start to get ApcState _KAPC_STATE structure then parses the kernel-APC and user-APC.

the problem with this approach is that it is based on parsing undocumented structure like _KPROCESS and _KTHREAD structure to get to the required member element, and those structure changes with different version of windows and even on different build number.

0: kd> dt nt!_KPROCESS
   +0x000 Header           : _DISPATCHER_HEADER
   +0x018 ProfileListHead  : _LIST_ENTRY
   +0x028 DirectoryTableBase : Uint8B
   +0x030 ThreadListHead   : _LIST_ENTRY        /*process threads list head*/
   +0x040 ProcessLock      : Uint4B
   +0x044 ProcessTimerDelay : Uint4B
   +0x048 DeepFreezeStartTime : Uint8B
   +0x050 Affinity         : _KAFFINITY_EX
   +0x0f8 AffinityPadding  : [12] Uint8B
   +0x158 ReadyListHead    : _LIST_ENTRY
............ //Truncated for visibility

Second Approach

The first approach was not suitable solution for me as it might lead to system crash, and i wanted to make an implementation that will be stable on all version of windows.

What I thought of is that when you initialize an APC using KeInitializeApc then queue it using KeInsertQueueApc the APC structure will be inserted inside the appropriate queue (Kernel/User APC queue) so why don’t I just initialize and insert my own APC then use this APC structure that I inserted to traverse the entire queue, So after some experimenting and checking how the Kernel/user APCs are dispatched and checking all the problems that i will face to enumerate Kernel/User APCs queue i found the following:

A thread might be created/terminated when enumerating APC will need to stop thread creation/termination in the process (at least the local created/terminated thread not the remote one)
will need to get all threads ID in the target process in a stable and documented way.
when traversing the APC queue, it might be accessed at the same time (for queuing APC or dispatching it) which might lead to crash or inaccurate result.
will need a valid user-mode address for the NormalRoutine in the user-mode APC otherwise the target process might crash.

So I will try to solve those issue each one at a time, the first issue is that a thread might be created/ terminated during our enumeration process, to solve it i decided to suspend the process before the APC enumeration then Resume it after finishing using the NtSuspendProcess and NtResumeProcess API .

the Second issue is that I will need to get all the threads ID in the target process, to do that there is two solution: the first is to use ZwQuerySystemInformation from the kernel with SystemProcessInformation as the SystemInformationClass argument, the second one is enumerating it from user mode using documented Method from windows (CreateToolhelp32Snapshot -> Thread32First -> Thread32Next)

so i decided use the second method which is enumerating thread ID from user-mode process

 hThreadSnap = CreateToolhelp32Snapshot(TH32CS_SNAPTHREAD, 0);
    if (hThreadSnap == INVALID_HANDLE_VALUE)
        return(FALSE);

    // Fill in the size of the structure before using it. 
    te32.dwSize = sizeof(THREADENTRY32);

    if (!Thread32First(hThreadSnap, &te32)) {
        //Error calling Thread32First
        CloseHandle(hThreadSnap);     // Must clean up the snapshot object!
        return(FALSE);
    }

    do
    {
        if (te32.th32OwnerProcessID == dwOwnerPID)
        {
            _tprintf(TEXT("\n     THREAD ID      = 0x%08X"), te32.th32ThreadID);
            _tprintf(TEXT("\n     base priority  = %d"), te32.tpBasePri);
            _tprintf(TEXT("\n     delta priority = %d"), te32.tpDeltaPri);

            Threadarray[counter]= te32.th32ThreadID;    // Move all Target process ID to global array to be passed to the Driver
            counter++;
        }
    } while (Thread32Next(hThreadSnap, &te32));

the Third issue is that the APC list might be accessed at the same time of the APC enumeration which might lead to crash or inaccurate result, what I thought of at start is that there must be some type of synchronization that is being done by windows kernel to lock the list, So i start searching for this synchronization method and I faced the same issue as in my first approach which is that I will need to parse undocumented structure _KTHREAD for each thread (1) so the idea died from the start.

my second approach I saw in the book rootkit arsenal it works by suspending all other CPU on the system by creating threads with the number on processor-1 and each thread raise the IRQL to DISPATCH_LEVEL then raise the IRQL on the current processor to DISPATCH_LEVEL also, this way i will not be interrupted by the windows kernel or any other driver, and as the APC is dispatched at APC_LEVEL or PASSIVE_LEVEL I will guarantee that the APC will not change during the APC enumeration.

NTSTATUS DeviceControl(PDEVICE_OBJECT, PIRP Irp) {

.................. //Truncated for visibility
    
        HookAllCPU();

		//Raise IRQL to DISPATCH_LEVEL on current proccessor,
		KIRQL oldIrql = 0;
		if (DISPATCH_LEVEL >= KeGetCurrentIrql()) {
			KeRaiseIrql(DISPATCH_LEVEL, &oldIrql);
		}

		while (ThreadIDArray[counter] != 0) {

			DbgPrint("APC Enumeration for TID: %d\n", ThreadIDArray[counter]);
			PETHREAD Ethread;
			NTSTATUS status = PsLookupThreadByThreadId((HANDLE)ThreadIDArray[counter], &Ethread);
			if (NT_SUCCESS(status)) {
				DbgPrint("Start Enumerating Kernel APC\n");
				EnumerateKernelAPC(Ethread);

				DbgPrint("Start Enumerating User APC\n");
				EnumerateNormalAPC(Ethread);

				ObDereferenceObject(Ethread);
			}

			counter++;
		}

		//lower the IRQL on current processor
		KeLowerIrql(oldIrql);

		//Signal other threads on other processor to stop and release proseccor
		InterlockedDecrement(&ReleaseFlag);

		//loop until all other threads is stoped
		while (TotalCpuNumberHooked) {}
.................. //Truncated for visibility
}

VOID HookAllCPU() {

	DbgPrint(("Attempting Hooking All CPUs Except the current processor\n"));
	DWORD64 nProcessor = KeNumberProcessors;
	NumberOfCPUToHook = (LONG)nProcessor - 1;
	HANDLE ThreadHandel = NULL;
	OBJECT_ATTRIBUTES inizializedattributes;
	InitializeObjectAttributes(&inizializedattributes, NULL, 0, NULL, NULL);
	for (int i = 0; i < nProcessor - 1; i++) {
		DbgPrint("CPU %d is beining hooked\n", i);
		PsCreateSystemThread(&ThreadHandel, THREAD_ALL_ACCESS,
			NULL, NtCurrentProcess(), NULL,
			(PKSTART_ROUTINE)HookCPU, NULL);

		//Close The Handle
		ZwClose(ThreadHandel);
	}

	//loop untile the other CPU are Hooked
	while (NumberOfCPUToHook != TotalCpuNumberHooked) {}

	return;
}

VOID HookCPU(PVOID) {

	//Raise IRQL
	KIRQL oldIrql = 0;
	if (DISPATCH_LEVEL >= KeGetCurrentIrql()) {
		KeRaiseIrql(DISPATCH_LEVEL, &oldIrql);
	}
	//NT_ASSERT(KeGetCurrentIrql() == DISPATCH_LEVEL);

	InterlockedIncrement(&TotalCpuNumberHooked);
	
	//Loop Untile Released, the release happen after the enumeration
	while (ReleaseFlag) {}

	//Lower IRQL
	KeLowerIrql(oldIrql);

	InterlockedDecrement(&TotalCpuNumberHooked);
	//terminate itself
	PsTerminateSystemThread(STATUS_SUCCESS);
}

the last issue that i will need to solve before writing the final code is that i will need a valid user-mode address to the user-mode APC (any user-mode address will work but if it not valid it will crash the target process), so i will need to parse normal windows API to use as NormalRoutine in user-mode APC, so instead of parsing the API address in the kernel mode (no easy way to do it) i will let the user-mode part of my code use windows API GetProcAddress to get the API i will use LoadLibraryA API in my case.

APC Enumeration Testing:

Now I am ready to test my Approach and see if it will face any issue, the complete code can be found on my GitHub ( https://github.com/MahmoudZohdy/Practical_Reverse_Engineering/tree/main/APCEnumeration ).

It worked just fine, although there is an issue that I parse the APC list head as KAPC structure (I cannot think of a way to distinguish the APC that it is queued and the APC list head)

00000007	1.72731602	Attempting Hooking All CPUs Except the current processor	
00000008	1.72731876	CPU 0 is beining hooked	
00000009	1.72758520	APC Enumeration for TID: 7308	
00000010	1.72758675	Start Enumerating Kernel APC	
00000011	1.72760320	KernelRoutine: 0xFFFFF806203989F0  RundownRoutine: 0xFFFFF806203989F0  NormalRoutine: 0xFFFFF806202DCA00	
00000012	1.72760475	KernelRoutine: 0xFFFFCC8B3CB86128  RundownRoutine: 0xFFFFCC8B3CB86128  NormalRoutine: 0xFFFFCC8B3D744080	
00000013	1.72760642	KernelRoutine: 0xFFFFF806390B1000  RundownRoutine: 0x0000000000000000  NormalRoutine: 0x0000000000000000	   // my Kernel APC
00000014	1.72760701	Start Enumerating User APC	
00000015	1.72761536	KernelRoutine: 0xFFFFCC8B3D744080  RundownRoutine: 0x0000000009000100  NormalRoutine: 0x0000000000000100	   //  APC List Head, not valid function address   
00000016	1.72761679	KernelRoutine: 0xFFFFF806390B1000  RundownRoutine: 0x0000000000000000  NormalRoutine: 0x00007FFE5AC604F0     //	LoadLibraryA address    
00000017	1.72761762	APC Enumeration for TID: 7312	
00000018	1.72761846	Start Enumerating Kernel APC	
00000019	1.72762036	KernelRoutine: 0xFFFFCC8B3D63B128  RundownRoutine: 0xFFFFCC8B3D63B128  NormalRoutine: 0xFFFFCC8B3D744080	
00000020	1.72762203	KernelRoutine: 0xFFFFF806390B1000  RundownRoutine: 0x0000000000000000  NormalRoutine: 0x0000000000000000	
00000021	1.72762263	Start Enumerating User APC	
00000022	1.72762883	KernelRoutine: 0xFFFFCC8B3D744080  RundownRoutine: 0x0000000108000101  NormalRoutine: 0x0000000000000100	
00000023	1.72763026	KernelRoutine: 0xFFFFF806390B1000  RundownRoutine: 0x0000000000000000  NormalRoutine: 0x00007FFE5AC604F0	
00000024	1.72763085	APC Enumeration for TID: 7316	
00000025	1.72763157	Start Enumerating Kernel APC	
00000026	1.72763395	KernelRoutine: 0xFFFFCC8B3D6D6128  RundownRoutine: 0xFFFFCC8B3D6D6128  NormalRoutine: 0xFFFFCC8B3D744080	
00000027	1.72764325	KernelRoutine: 0xFFFFF806390B1000  RundownRoutine: 0x0000000000000000  NormalRoutine: 0x0000000000000000	
00000028	1.72764397	Start Enumerating User APC	
00000029	1.72765243	KernelRoutine: 0xFFFFCC8B3D744080  RundownRoutine: 0x0000000008000101  NormalRoutine: 0x0000000000000100	
00000030	1.72765398	KernelRoutine: 0xFFFFF806390B1000  RundownRoutine: 0x0000000000000000  NormalRoutine: 0x00007FFE5AC604F0	
00000031	1.72765481	APC Enumeration for TID: 7332	
00000032	1.72765565	Start Enumerating Kernel APC	
00000033	1.72765803	KernelRoutine: 0xFFFFCC8B3D647128  RundownRoutine: 0xFFFFCC8B3D647128  NormalRoutine: 0xFFFFCC8B3D744080	
00000034	1.72765923	KernelRoutine: 0xFFFFF806390B1000  RundownRoutine: 0x0000000000000000  NormalRoutine: 0x0000000000000000	
00000035	1.72765994	Start Enumerating User APC	
00000036	1.72766721	KernelRoutine: 0xFFFFCC8B3D744080  RundownRoutine: 0x0000000108000101  NormalRoutine: 0x0000000000000100	
00000037	1.72766840	KernelRoutine: 0xFFFFF806390B1000  RundownRoutine: 0x0000000000000000  NormalRoutine: 0x00007FFE5AC604F0	
00000038	1.72766924	APC Enumeration for TID: 7336	
00000039	1.72766995	Start Enumerating Kernel APC	
00000040	1.72767246	KernelRoutine: 0xFFFFF806203989F0  RundownRoutine: 0xFFFFF806203989F0  NormalRoutine: 0xFFFFF806202DCA00	
00000041	1.72767377	KernelRoutine: 0xFFFFCC8B3D646128  RundownRoutine: 0xFFFFCC8B3D646128  NormalRoutine: 0xFFFFCC8B3D744080	
00000042	1.72767520	KernelRoutine: 0xFFFFF806390B1000  RundownRoutine: 0x0000000000000000  NormalRoutine: 0x0000000000000000	
00000043	1.72767580	Start Enumerating User APC	
00000044	1.72768116	KernelRoutine: 0xFFFFCC8B3D744080  RundownRoutine: 0x0000000008000100  NormalRoutine: 0x0000000000000100	
00000045	1.72768235	KernelRoutine: 0xFFFFF806390B1000  RundownRoutine: 0x0000000000000000  NormalRoutine: 0x00007FFE5AC604F0	
00000046	1.72768319	APC Enumeration for TID: 7340	
00000047	1.72768378	Start Enumerating Kernel APC	
00000048	1.72768605	KernelRoutine: 0xFFFFCC8B3D645128  RundownRoutine: 0xFFFFCC8B3D645128  NormalRoutine: 0xFFFFCC8B3D744080	
00000049	1.72768760	KernelRoutine: 0xFFFFF806390B1000  RundownRoutine: 0x0000000000000000  NormalRoutine: 0x0000000000000000	
00000050	1.72768819	Start Enumerating User APC	
00000051	1.72769403	KernelRoutine: 0xFFFFCC8B3D744080  RundownRoutine: 0x0000000108000101  NormalRoutine: 0x0000000000000100	
00000052	1.72769558	KernelRoutine: 0xFFFFF806390B1000  RundownRoutine: 0x0000000000000000  NormalRoutine: 0x00007FFE5AC604F0	
00000054	1.72774374	Finished UnHooking other CPUs

Security Blog

Asynchronous Procedure Calls (APC) Enumeration

APC Internal:

APC Enumeration:

First Approach

Second Approach

APC Enumeration Testing:

Refreneces: