VM-Based Virtualization for Lua and Luau

VM-based obfuscation is the most powerful technique available for protecting Lua and Luau scripts. Instead of distributing standard Lua source or bytecode, your code is compiled to a custom instruction set that no existing decompiler understands. A Lua interpreter written in Lua (the “VM”) executes these custom instructions at runtime. Even if an analyst obtains your obfuscated file, they see only the VM interpreter and an opaque stream of bytecode. They must first reverse engineer the VM to understand the bytecode format, then reverse the bytecode to recover your original logic. This two-layer reversal is significantly harder than decompiling standard Lua bytecode. This page covers the full virtualization pipeline, how to design a custom instruction set, how to write the VM interpreter, how to encode and compile custom bytecode, and how to obfuscate the VM itself.

How VM Obfuscation Works

The full pipeline transforms your original Lua source into a self-contained obfuscated file that embeds both the custom bytecode and the VM interpreter.

Write Original Source

You write your Lua or Luau code normally. This is the logic you want to protect, such as license checks, decryption routines, or proprietary algorithms.

Compile to Custom Bytecode

A custom compiler translates your Lua source into a proprietary instruction set. This is your “bytecode” — a sequence of bytes or numbers that only your VM understands.

Encode the Bytecode

The custom bytecode is encoded as a Lua string, a table of bytes, or an encrypted blob. It is embedded directly into the output file alongside the VM.

Embed the VM Interpreter

The VM interpreter is written in Lua or Luau and embedded in the output file. It reads and executes your custom bytecode at runtime.

Runtime Execution

At runtime, the VM reads each custom instruction, decodes it, and performs the corresponding operation. The original logic never appears in standard Lua bytecode or source form.

Attackers who intercept your file see only the VM interpreter code plus an opaque bytecode payload. They cannot use standard Lua decompilers because the file does not contain standard Lua bytecode. They must reverse engineer your VM interpreter to understand the custom instruction encoding, then analyze the bytecode stream to reconstruct your logic.

Designing a Custom Instruction Set

The first step in building a VM is defining the instruction set. Your custom ISA must be capable of expressing all Lua operations that your original code uses.

Stack-Based vs. Register-Based VMs

You can choose between two fundamental VM architectures: Stack-based VMs use a value stack. Instructions push operands onto the stack, perform operations on the top elements, and push results back. They are simpler to implement but slightly slower due to stack manipulation overhead.

PUSH_CONST 5      -- stack: [5]
PUSH_CONST 3      -- stack: [5, 3]
ADD               -- stack: [8]  (pops two, pushes sum)

Register-based VMs specify source and destination registers explicitly, like Lua’s own VM. They are more complex to implement but more efficient because they avoid stack overhead.

LOAD_CONST R0, 5  -- R0 = 5
LOAD_CONST R1, 3  -- R1 = 3
ADD R2, R0, R1    -- R2 = R0 + R1

For Lua obfuscation, register-based VMs are preferred because they more closely match Lua’s semantics and produce more compact bytecode.

Minimum Viable Instruction Set

A register-based VM for Lua needs at least these instructions to be Turing-complete and Lua-compatible:

Opcode	Name	Description
0	LOAD_CONST	Load a constant into a register
1	LOAD_NIL	Load nil into a register
2	LOAD_BOOL	Load true or false into a register
3	MOVE	Copy value between registers
4	GET_GLOBAL	Read a global variable into a register
5	SET_GLOBAL	Write a register to a global variable
6	GET_TABLE	Read a table field into a register
7	SET_TABLE	Write a register to a table field
8	ADD	Add two registers/constants
9	SUB	Subtract two registers/constants
10	MUL	Multiply two registers/constants
11	DIV	Divide two registers/constants
12	MOD	Modulo two registers/constants
13	UNM	Unary minus
14	NOT	Logical not
15	CONCAT	Concatenate registers
16	EQ	Equality comparison
17	LT	Less-than comparison
18	LE	Less-than-or-equal comparison
19	JMP	Unconditional jump
20	CALL	Function call
21	RETURN	Return from function
22	CLOSURE	Create a closure

Instruction Encoding

Each instruction can be encoded as 4 bytes:

Byte 0: opcode (1 byte, 0-255)
Byte 1: operand A (1 byte, register index)
Byte 2: operand B (1 byte, register or constant index)
Byte 3: operand C (1 byte, register or constant index, or jump offset)

For instructions that need larger constants, you can use a two-instruction sequence (load constant index into a register, then use that register) or expand the encoding to 8 bytes.

The VM Interpreter Loop

The VM interpreter is the core of the obfuscation. It fetches instructions, decodes them, and executes the corresponding operations.

Complete Minimal Register-Based VM

Here is a complete minimal register-based VM in Lua. It supports LOAD_CONST, MOVE, ADD, SUB, JMP, CALL, and RETURN:

local function run_vm(bytecode, constants, globals)
    local ip = 1              -- instruction pointer
    local registers = {}      -- register file
    local call_stack = {}     -- call stack for nested calls

    while ip <= #bytecode do
        local opcode = bytecode[ip]
        local a = bytecode[ip + 1]
        local b = bytecode[ip + 2]
        local c = bytecode[ip + 3]
        ip = ip + 4

        if opcode == 0 then       -- LOAD_CONST
            registers[a] = constants[b]

        elseif opcode == 3 then   -- MOVE
            registers[a] = registers[b]

        elseif opcode == 8 then   -- ADD
            registers[a] = registers[b] + registers[c]

        elseif opcode == 9 then   -- SUB
            registers[a] = registers[b] - registers[c]

        elseif opcode == 19 then  -- JMP
            ip = ip + (b * 256 + c) - 32768  -- signed 16-bit offset

        elseif opcode == 20 then  -- CALL
            local fn = registers[a]
            local args = {}
            for i = 1, b do
                table.insert(args, registers[a + i])
            end
            local results = {fn(table.unpack(args))}
            for i = 1, c do
                registers[a + i - 1] = results[i]
            end

        elseif opcode == 21 then  -- RETURN
            local results = {}
            for i = 1, b do
                table.insert(results, registers[a + i - 1])
            end
            return table.unpack(results)

        else
            error("Unknown opcode: " .. opcode)
        end
    end
end

VM Components Explained

Instruction Pointer (ip): Tracks the current position in the bytecode. After fetching a 4-byte instruction, ip advances by 4. Register File (registers): A Lua table that acts as the VM’s register bank. Registers are indexed numerically starting from 0. Constant Pool (constants): A table of literal values (numbers, strings, booleans, nil) referenced by index. The LOAD_CONST instruction copies constants into registers. Call Stack (call_stack): Stores return addresses and register states for nested function calls. A production VM needs this for closures and recursive calls.

Encoding Custom Bytecode

You need a way to represent your custom instructions as data that can be embedded in a Lua file.

Binary String Encoding

Use string.char() to pack each instruction into 4 bytes:

local function encode_instruction(opcode, a, b, c)
    return string.char(opcode, a, b, c)
end

local bytecode = ""
bytecode = bytecode .. encode_instruction(0, 0, 0, 0)   -- LOAD_CONST R0, const[0]
bytecode = bytecode .. encode_instruction(0, 1, 1, 0)   -- LOAD_CONST R1, const[1]
bytecode = bytecode .. encode_instruction(8, 2, 0, 1)   -- ADD R2, R0, R1
bytecode = bytecode .. encode_instruction(21, 2, 1, 0)  -- RETURN R2, 1 result

Runtime Decoding

At runtime, extract bytes from the string:

local function run_vm_string(bytecode_str, constants, globals)
    local ip = 1
    local registers = {}

    while ip <= #bytecode_str do
        local opcode = string.byte(bytecode_str, ip)
        local a = string.byte(bytecode_str, ip + 1)
        local b = string.byte(bytecode_str, ip + 2)
        local c = string.byte(bytecode_str, ip + 3)
        ip = ip + 4

        -- execute opcode...
    end
end

Flat Table Encoding

For simpler indexing, store instructions as a flat Lua table of numbers:

local bytecode = {
    0, 0, 0, 0,   -- LOAD_CONST R0, const[0]
    0, 1, 1, 0,   -- LOAD_CONST R1, const[1]
    8, 2, 0, 1,   -- ADD R2, R0, R1
    21, 2, 1, 0,  -- RETURN R2, 1 result
}

This avoids string packing overhead and makes the VM code slightly simpler.

Compiling Lua to Your Custom Bytecode

You need a compiler that translates Lua source into your custom instruction set. There are two approaches.

Option A: Custom Compiler from AST

Write a compiler that parses Lua source into an AST and emits your instruction set. You can use an existing Lua parser library or write a recursive descent parser:

-- Pseudocode for AST-to-custom-bytecode compiler
local function compile_node(node, bytecode, constants)
    if node.type == "number_literal" then
        local const_idx = add_constant(constants, node.value)
        emit(bytecode, LOAD_CONST, node.target_reg, const_idx, 0)

    elseif node.type == "binary_op" and node.op == "+" then
        compile_node(node.left, bytecode, constants)
        compile_node(node.right, bytecode, constants)
        emit(bytecode, ADD, node.target_reg, node.left.target_reg, node.right.target_reg)

    elseif node.type == "function_call" then
        compile_node(node.func, bytecode, constants)
        for i, arg in ipairs(node.args) do
            compile_node(arg, bytecode, constants)
        end
        emit(bytecode, CALL, node.func.target_reg, #node.args, node.num_results)

    -- ... handle all node types
    end
end

Option B: Bytecode-to-ByteCode Translation

Reuse the standard Lua compiler but translate standard Lua bytecode to your custom bytecode. This is easier because the input format is well-defined:

-- Pseudocode for standard-to-custom bytecode translator
local function translate_luac_to_custom(standard_bytecode)
    local custom = {}
    local constants = {}

    for _, instr in ipairs(parse_standard_bytecode(standard_bytecode)) do
        local custom_op = opcode_map[instr.opcode]

        if instr.opcode == OP_LOADK then
            local new_const_idx = add_constant(constants, instr.constant)
            emit(custom, OP_LOAD_CONST, instr.a, new_const_idx, 0)

        elseif instr.opcode == OP_ADD then
            emit(custom, OP_ADD, instr.a, instr.b, instr.c)

        elseif instr.opcode == OP_JMP then
            local offset = instr.sbx
            emit(custom, OP_JMP, 0, math.floor(offset / 256), offset % 256)

        -- ... map all standard opcodes to custom equivalents
        end
    end

    return custom, constants
end

Bytecode-to-bytecode translation is the approach used by many commercial Lua obfuscators because it avoids writing a full parser.

Obfuscating the VM Interpreter Itself

The VM interpreter is now the most valuable target for an analyst. You must obfuscate it just as aggressively as the original code.

Apply Source-Level Obfuscation

Use the techniques from other sections of this guide on the VM interpreter:

Variable renaming: rename registers to _, ip to __, opcode to ___
String encoding: encode error messages and opcode names
Control flow flattening: replace the if/elseif chain with a flattened dispatcher

Obfuscate Opcode Constants

The VM’s opcode numbers should not appear as plain literals. Use constant obfuscation:

-- Instead of:
if opcode == 8 then  -- ADD

-- Use:
local _a = 2 * 4
local _b = 2 ^ 3
if opcode == (_a + _b - 8) then  -- still 8, but not obvious

Function Dispatch Table

Replace the if/elseif chain with a function dispatch table. This is harder to analyze than a flat conditional chain:

local ops = {}

ops[0] = function(a, b, c)  -- LOAD_CONST
    registers[a] = constants[b]
end

ops[3] = function(a, b, c)  -- MOVE
    registers[a] = registers[b]
end

ops[8] = function(a, b, c)  -- ADD
    registers[a] = registers[b] + registers[c]
end

-- In the main loop:
local op_fn = ops[opcode]
if op_fn then
    op_fn(a, b, c)
else
    error("?")
end

Flattened Dispatcher Version

For even stronger obfuscation, flatten the dispatcher so there is no obvious table lookup:

local _ops = {}
for _i = 1, 23 do
    _ops[_i] = function() end  -- dummy entries
end

_ops[0x00] = function(_a, _b) registers[_a] = constants[_b] end
_ops[0x03] = function(_a, _b) registers[_a] = registers[_b] end
_ops[0x08] = function(_a, _b, _c) registers[_a] = registers[_b] + registers[_c] end
-- ...

local _d = _ops[opcode]
if _d then _d(a, b, c) end

Real-World Lua Virtualizers

Several open-source projects demonstrate production-quality Lua VM obfuscation.

Ironbrew2

Ironbrew2 is a Lua 5.1 obfuscator focused on the Roblox ecosystem. It implements a register-based VM with the following features:

Custom instruction set with encrypted opcodes
String encryption using XOR and dynamic keys
Control flow obfuscation within the VM loop
Anti-tamper checks that detect debugging attempts

Ironbrew2 is open source and serves as an educational reference for how VM obfuscation works in practice.

Prometheus

Prometheus is a modular Lua 5.1 obfuscator with pluggable transformation steps. Its pipeline includes:

Lexical obfuscation (variable renaming, string encryption)
Control flow flattening
VM wrapping (converting functions to custom bytecode + VM)
Anti-debug and anti-tamper layers

Prometheus demonstrates how multiple obfuscation techniques can be layered, with VM wrapping as the strongest final step.

VM overhead is significant. A VM-obfuscated script typically runs 5 to 20 times slower than the original. Only apply VM obfuscation to code that is not called in tight loops or performance-critical paths.

Apply the VM only to the most sensitive parts of your codebase, such as license checks, decryption routines, or proprietary algorithms. Leave performance-critical code as standard (but otherwise obfuscated) Lua. This hybrid approach gives you strong protection where it matters without sacrificing overall performance.

Getting Started

Introduction

Source-Level Techniques

AST Manipulation

Bytecode and VM Techniques

Runtime and Environment

Advanced and Combined

How VM Obfuscation Works

Designing a Custom Instruction Set

Stack-Based vs. Register-Based VMs

Minimum Viable Instruction Set

Instruction Encoding

The VM Interpreter Loop

Complete Minimal Register-Based VM

VM Components Explained

Encoding Custom Bytecode

Binary String Encoding

Runtime Decoding

Flat Table Encoding

Compiling Lua to Your Custom Bytecode

Option A: Custom Compiler from AST

Option B: Bytecode-to-ByteCode Translation

Obfuscating the VM Interpreter Itself

Apply Source-Level Obfuscation

Obfuscate Opcode Constants

Function Dispatch Table

Flattened Dispatcher Version

Real-World Lua Virtualizers

Ironbrew2

Prometheus

​How VM Obfuscation Works

​Designing a Custom Instruction Set

​Stack-Based vs. Register-Based VMs

​Minimum Viable Instruction Set

​Instruction Encoding

​The VM Interpreter Loop

​Complete Minimal Register-Based VM

​VM Components Explained

​Encoding Custom Bytecode

​Binary String Encoding

​Runtime Decoding

​Flat Table Encoding

​Compiling Lua to Your Custom Bytecode

​Option A: Custom Compiler from AST

​Option B: Bytecode-to-ByteCode Translation

​Obfuscating the VM Interpreter Itself

​Apply Source-Level Obfuscation

​Obfuscate Opcode Constants

​Function Dispatch Table

​Flattened Dispatcher Version

​Real-World Lua Virtualizers

​Ironbrew2

​Prometheus

How VM Obfuscation Works

Designing a Custom Instruction Set

Stack-Based vs. Register-Based VMs

Minimum Viable Instruction Set

Instruction Encoding

The VM Interpreter Loop

Complete Minimal Register-Based VM

VM Components Explained

Encoding Custom Bytecode

Binary String Encoding

Runtime Decoding

Flat Table Encoding

Compiling Lua to Your Custom Bytecode

Option A: Custom Compiler from AST

Option B: Bytecode-to-ByteCode Translation

Obfuscating the VM Interpreter Itself

Apply Source-Level Obfuscation

Obfuscate Opcode Constants

Function Dispatch Table

Flattened Dispatcher Version

Real-World Lua Virtualizers

Ironbrew2

Prometheus