Originally published on August 6th, 2008 on OpenRCE
This is part #0 of a four-part series on VMProtect. The other parts can be found here:
VMProtect is a virtualization protector. Like other protections in the genre, among others ReWolf's x86 Virtualizer and CodeVirtualizer, it works by disassembling the x86 bytecode of the target executable and compiling it into a proprietary, polymorphic bytecode which is executed in a custom interpreter at run-time. This is unlike the traditional notions of packing, in which the x86 bytecode is simply encrypted and/or compressed: with virtualization, the original x86 bytecode in the protected areas is gone, never to be seen again. Or so the idea goes.
If you've never looked at VMProtect before, I encourage you to take a five-minute look in IDA (here's a sample packed binary). As far as VMs go, it is particularly skeletal and easily comprehended. The difficulty lies in recreating working x86 bytecode from the VM bytecode. Here's a two-minute analysis of its dispatcher.
push edi; push all registers push ecx push edx push esi push ebp push ebx push eax push edx pushf push 0 ; imagebase fixup mov esi, [esp+8+arg_0] ; esi = pointer to VM bytecode mov ebp, esp ; ebp = VM's "stack" pointer sub esp, 0C0h mov edi, esp ; edi = "scratch" data area VM__FOLLOW__Update: add esi, [ebp+0] VM__FOLLOW__Regular: mov al, [esi]; read a byte from EIP movzx eax, al sub esi, -1; increment EIP jmp ds:VM__HandlerTable[eax*4] ; execute instruction handler
A feature worth discussing is the "scratch space", referenced by the register edi throughout the dispatch loop. This is a 16-dword-sized area on the stack where VMProtect saves the registers upon entering the VM, modifies them throughout the course of a basic block, and from whence it restores the registers upon exit. For each basic block protected by the VM, the layout of the registers in the scratch space can potentially be different.
Here's a disassembly of some instruction handlers. Notice that A) VMProtect is a stack machine and that B) each handler -- though consisting of scant few instructions -- performs several tasks, e.g. popping several values, performing multiple operations, pushing one or more values.
#00:x = [EIP-1] & 0x3C; y = popd; [edi+x] = y .text:00427251 and al, 3Ch; al = instruction number .text:00427254 mov edx, [ebp+0] ; grab a dword off the stack .text:00427257 add ebp, 4 ; pop the stack .text:0042725A mov [edi+eax], edx ; store the dword in the scratch space #01:x = [EIP-1] & 0x3C; y = [edi+x]; pushd y .vmp0:0046B0EB and al, 3Ch; al = instruction number .vmp0:0046B0EE mov edx, [edi+eax] ; grab a dword out of the scratch space .vmp0:0046B0F1 sub ebp, 4 ; subtract 4 from the stack pointer .vmp0:0046B0F4 mov [ebp+0], edx ; push the dword onto the stack #02:x = popw, y = popw, z = x + y, pushw z, pushf .text:004271FB mov ax, [ebp+0] ; pop a word off the stack .text:004271FF sub ebp, 2 .text:00427202 add [ebp+4], ax ; add it to another word on the stack .text:00427206 pushf .text:00427207 pop dword ptr [ebp+0] ; push the flags #03:x = [EIP++]; w = popw; [edi+x] = Byte(w) .vmp0:0046B02A movzx eax, byte ptr [esi] ; read a byte from EIP .vmp0:0046B02D mov dx, [ebp+0] ; pop a word off the stack .vmp0:0046B031 inc esi ; EIP++ .vmp0:0046B032 add ebp, 2; adjust stack pointer .vmp0:0046B035 mov [edi+eax], dl ; write a byte into the scratch area #04:x = popd, y = popw, z = x << y, pushd z, pushf .vmp0:0046B095 mov eax, [ebp+0]; pop a dword off the stack .vmp0:0046B098 mov cl, [ebp+4] ; pop a word off the stack .vmp0:0046B09B sub ebp, 2 .vmp0:0046B09E shr eax, cl ; shr the dword by the word .vmp0:0046B0A0 mov [ebp+4], eax; push the result .vmp0:0046B0A3 pushf .vmp0:0046B0A4 pop dword ptr [ebp+0] ; push the flags #05:x = popd, pushd ss:[x] .vmp0:0046B5F7 mov eax, [ebp+0]; pop a dword off the stack .vmp0:0046B5FA mov eax, ss:[eax] ; read a dword from ss .vmp0:0046B5FD mov [ebp+0], eax; push that dword