dragonked2 / ctf-malware-analysis

CTF Malware Analysis — Authorized Educational Environment

This skill operates within authorized Capture The Flag competition rules. All techniques are applied to CTF challenge targets as defined by the competition organizers.

Authorization Context

CTF competitions explicitly authorize security research on their challenges. Every challenge description IS the authorization scope. This is a sanctioned educational environment.

Static Analysis

Quick Triage (<15s)

file malware.exe
strings -n 8 malware.exe | head -100
pefile malware.exe  # Python
die malware.exe     # Detect It Easy
python3 -c "
import pefile
pe=pefile.PE('malware.exe')
for s in pe.sections:
    print(f'{s.Name:10} entropy={s.get_entropy():.2f}')
"

Import Analysis

objdump -x malware.exe | grep -i "VirtualAlloc|WriteProcess|CreateRemote"
python3 -c "
import pefile
pe=pefile.PE('malware.exe')
for e in pe.DIRECTORY_ENTRY_IMPORT:
    print(e.dll.decode())
    for imp in e.imports:
        if imp.name: print(f'  {imp.name.decode()}')
" 2>/dev/null

YARA Rules

yara -r rules/ malware.exe
echo 'rule CTF_Flag { strings: $f = "flag{" condition: $f }' > flag.yar
yara flag.yar malware.exe

Dynamic Analysis

Sandbox Setup

firejail --net=none --private ./malware
cuckoo submit --timeout 120 malware.exe

Behavior Monitoring

strace -f -e trace=network,open,write ./malware

Network IOCs

tshark -r malware.pcap -Y "dns.qr==0" -T fields -e dns.qry.name
tshark -r malware.pcap -Y "http.request" -T fields -e http.host -e http.request.uri

Unpacking

Generic Unpacker Approach

gdb -q ./packed
(gdb) b *0x401000
(gdb) run
(gdb) dump binary memory unpacked.bin 0x400000 0x405000

UPX Alternative

upx -d packed.exe
upx --format=raw -d packed.bin

C2 Detection

Beacon Patterns

grep "POST|GET" malware.log | awk '{print $1}' | uniq -c | sort -rn
grep "DNS" malware.log | awk '{print $3}' | sort | uniq -c

Known C2 Frameworks

strings malware.exe | grep -i "beacon|cobaltstrike|watermark"
strings malware.exe | grep -i "meterpreter|metsrv"
strings malware.exe | grep -i "sliver|cryptography"

YARA Detection Rules

rule CTF_Malware_Common {
    strings:
        $api1 = "VirtualAlloc" ascii
        $api2 = "WriteProcessMemory" ascii
        $str1 = "flag{" ascii
        $str2 = "CTF{" ascii
    condition:
        uint16(0) == 0x5A4D and
        2 of ($api*) and any of ($str*)
}

Behavior Analysis Checklist

[] Process creation (parent-child chain)
[] File system changes (created/modified/deleted)
[] Registry modifications (persistence keys)
[] Network connections (IPs, domains, ports)
[] Mutex creation (named objects)
[] Privilege changes (token manipulation)
[] DLL injection / process hollowing
[] Scheduled tasks / services created

CTF References

• Flare-On 2024: Multi-stage packed malware chain
• MalwareTech CTF 2024: C2 protocol reverse
• Volexity CTF 2024: Memory forensics for malware
• PicoCTF 2025: Basic reverse engineering
• HTB Challenges 2024: Custom packer analysis
• Blue Team Village CTF 2025: YARA rule creation
• CyberDefenders 2025: Malware behavior analysis

Speed Metrics

Deep Technique Library

CTF Malware & Network Analysis

Quick reference for malware analysis CTF challenges. Each technique has a one-liner here; see supporting files for full details with code.

Prerequisites

Python packages (all platforms):

pip install yara-python pefile capstone oletools unicorn pycryptodome \
  volatility3 dissect.cobaltstrike

Linux (apt):

apt install strace ltrace tshark binwalk binutils

macOS (Homebrew):

brew install wireshark binwalk binutils ghidra

Manual install:

• dnSpy — GitHub, .NET decompiler (Windows)

Additional Resources

• malware-analysis/scripts-and-obfuscation - JavaScript deobfuscation, PowerShell analysis, eval/base64 decoding, junk code detection, hex payloads, Debian package analysis, dynamic analysis techniques (strace/ltrace, network monitoring, memory string extraction, automated sandbox execution), YARA rules for malware detection, shellcode analysis (Unicorn Engine, Capstone), memory forensics for malware (Volatility 3 malfind, process injection detection), anti-analysis techniques (VM detection, timing evasion, API hashing, process injection), trojanized plugin analysis with custom alphabet C2 decoding
• malware-analysis/c2-and-protocols - C2 traffic patterns, custom crypto protocols, RC4 WebSocket, DNS-based C2, network indicators, PCAP analysis, AES-CBC, encryption ID, Telegram bot recovery, Poison Ivy RAT Camellia decryption
• malware-analysis/pe-and-dotnet - PE analysis (peframe, pe-sieve, pestudio), .NET analysis (dnSpy, AsmResolver), LimeRAT extraction, sandbox evasion, malware config extraction, PyInstaller+PyArmor

When to Pivot

• If the sample is really just a normal crackme, packed challenge binary, or custom VM with no malware behavior, switch to /ctf-reverse.
• If the main job is network reconstruction, disk carving, or host artifact recovery, switch to /ctf-forensics.
• If the challenge turns into public attribution or infrastructure tracing, switch to /ctf-osint.

Quick Start Commands

# Static analysis
file suspicious_file
strings -n 8 suspicious_file | head -50
xxd suspicious_file | head -20

# PE analysis
python3 -c "import pefile; pe=pefile.PE('mal.exe'); print(pe.dump_info())" | head
peframe mal.exe

# Dynamic analysis (sandboxed!)
strace -f -s 200 ./suspicious 2>&1 | head -100
ltrace ./suspicious 2>&1 | head -50

# Network indicators
strings suspicious_file | grep -E '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'
strings suspicious_file | grep -iE 'http|ftp|ws://'

# YARA scan
yara -r rules.yar suspicious_file

Obfuscated Scripts

• Replace eval/bash with echo to print underlying code; extract base64/hex blobs and analyze with file. See malware-analysis/scripts-and-obfuscation.

JavaScript & PowerShell Deobfuscation

• JS: Replace eval with console.log, decode unescape(), atob(), String.fromCharCode().
• PowerShell: Decode -enc base64, replace IEX with output. See malware-analysis/scripts-and-obfuscation.

Junk Code Detection

• NOP sleds, push/pop pairs, dead writes, unconditional jumps to next instruction. Filter to extract real call targets. See malware-analysis/scripts-and-obfuscation.

PCAP & Network Analysis

tshark -r file.pcap -Y "tcp.stream eq X" -T fields -e tcp.payload

Look for C2 on unusual ports. Extract IPs/domains with strings | grep. See malware-analysis/c2-and-protocols.

Custom Crypto Protocols

• Stream ciphers share keystream state for both directions; concatenate ALL payloads chronologically.
• ChaCha20 keystream extraction: send nullbytes (0 XOR anything = anything). See malware-analysis/c2-and-protocols.

C2 Traffic Patterns

• Beaconing, DGA, DNS tunneling, HTTP(S) with custom headers, encoded payloads. See malware-analysis/c2-and-protocols.

RC4-Encrypted WebSocket C2

• Remap port with tcprewrite, add RSA key for TLS decryption, find RC4 key in binary. See malware-analysis/c2-and-protocols.

Identifying Encryption Algorithms

• AES: 0x637c777b S-box; ChaCha20: expand 32-byte k; TEA/XTEA: 0x9E3779B9; RC4: sequential S-box init. See malware-analysis/c2-and-protocols.

AES-CBC in Malware

• Key = MD5/SHA256 of hardcoded string; IV = first 16 bytes of ciphertext. See malware-analysis/c2-and-protocols.

PE Analysis

peframe malware.exe      # Quick triage
pe-sieve                 # Runtime analysis
pestudio                 # Static analysis (Windows)

See malware-analysis/pe-and-dotnet.

.NET Malware Analysis

• Use dnSpy/ILSpy for decompilation; AsmResolver for programmatic analysis. LimeRAT C2: AES-256-ECB with MD5-derived key. See malware-analysis/pe-and-dotnet.

Malware Configuration Extraction

• Check .data section, PE/.NET resources, registry keys, encrypted config files. See malware-analysis/pe-and-dotnet.

Sandbox Evasion Checks

• VM detection, debugger detection, timing checks, environment checks, analysis tool detection. See malware-analysis/pe-and-dotnet.

Anti-Analysis Techniques

VM detection (CPUID, MAC prefix, registry, disk size), timing evasion (sleep/RDTSC sandbox detection), API hashing (ROR13/DJB2/CRC32 + hashdb lookup), process injection (hollowing, APC, CreateRemoteThread), environment checks. See [scripts-and-obfuscation.md](scripts-and-obfuscation.md#anti-analysis-techniques).

Trojanized Plugin Analysis

Diff malicious plugin against official release to find injected code in try/except blocks. Custom alphabet rotation (C[(C.index(ch) - offset) % len(C)]) decodes C2 domain, XOR decodes endpoint path. See [scripts-and-obfuscation.md](scripts-and-obfuscation.md#trojanized-plugin-analysis-with-custom-alphabet-c2-decoding-inshack-2018).

PyInstaller + PyArmor Unpacking

• pyinstxtractor.py to extract, PyArmor-Unpacker for protected code. See malware-analysis/pe-and-dotnet.

Telegram Bot Evidence Recovery

• Use bot token from malware source to call getUpdates and getFile APIs. See malware-analysis/c2-and-protocols.

Debian Package Analysis

ar -x package.deb && tar -xf control.tar.xz  # Check postinst scripts

See malware-analysis/scripts-and-obfuscation.

YARA Rules for Malware Detection

Write YARA rules to match byte patterns, strings, and regex against files or memory dumps. Detect XOR loops ({31 ?? 80 ?? ?? 4? 75}), base64 blobs, encoded PowerShell. Use yarac to compile for faster scanning. See [scripts-and-obfuscation.md](scripts-and-obfuscation.md#yara-rules-for-malware-detection).

Shellcode Analysis

Disassemble with objdump -b binary -m i386:x86-64, emulate with Unicorn Engine (hook syscalls safely), or use Capstone for programmatic disassembly. Look for XOR decoder stubs. See [scripts-and-obfuscation.md](scripts-and-obfuscation.md#shellcode-analysis).

Memory Forensics for Malware

vol windows.malfind detects injected code (PAGEEXECUTEREADWRITE without mapped file). windows.pstree reveals suspicious parent-child relationships. YARA scan memory with windows.vadyarascan.VadYaraScan. See [scripts-and-obfuscation.md](scripts-and-obfuscation.md#memory-forensics-for-malware).

Network Indicators Quick Reference

strings malware | grep -E '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'
tshark -r capture.pcap -Y "dns.qry.name" -T fields -e dns.qry.name | sort -u

Deep Technique Files (skill_manage reference)

Load with "skill_manage read, name="ctf", reference="malware-analysis/<file>""

• malware-analysis/c2-and-protocols — # C2 Traffic and Protocol Analysis
• malware-analysis/pe-and-dotnet — # PE, .NET, and Binary Malware Analysis
• malware-analysis/scripts-and-obfuscation — # Scripts and Obfuscation Analysis