> ## Documentation Index
> Fetch the complete documentation index at: https://starcryptdocs.starsolenterprise.com/lua-docs/llms.txt
> Use this file to discover all available pages before exploring further.

# VM-Based Virtualization for Lua and Luau

> Implement a custom virtual machine in Lua or Luau that executes a proprietary instruction set, providing the strongest possible source and bytecode protection.

VM-based obfuscation is the most powerful technique available for protecting Lua and Luau scripts. Instead of distributing standard Lua source or bytecode, your code is compiled to a custom instruction set that no existing decompiler understands. A Lua interpreter written in Lua (the "VM") executes these custom instructions at runtime. Even if an analyst obtains your obfuscated file, they see only the VM interpreter and an opaque stream of bytecode. They must first reverse engineer the VM to understand the bytecode format, then reverse the bytecode to recover your original logic. This two-layer reversal is significantly harder than decompiling standard Lua bytecode.

This page covers the full virtualization pipeline, how to design a custom instruction set, how to write the VM interpreter, how to encode and compile custom bytecode, and how to obfuscate the VM itself.

## How VM Obfuscation Works

The full pipeline transforms your original Lua source into a self-contained obfuscated file that embeds both the custom bytecode and the VM interpreter.

**1. Write Original Source**

You write your Lua or Luau code normally. This is the logic you want to protect, such as license checks, decryption routines, or proprietary algorithms.

**2. Compile to Custom Bytecode**

A custom compiler translates your Lua source into a proprietary instruction set. This is your "bytecode" — a sequence of bytes or numbers that only your VM understands.

**3. Encode the Bytecode**

The custom bytecode is encoded as a Lua string, a table of bytes, or an encrypted blob. It is embedded directly into the output file alongside the VM.

**4. Embed the VM Interpreter**

The VM interpreter is written in Lua or Luau and embedded in the output file. It reads and executes your custom bytecode at runtime.

**5. Runtime Execution**

At runtime, the VM reads each custom instruction, decodes it, and performs the corresponding operation. The original logic never appears in standard Lua bytecode or source form.

Attackers who intercept your file see only the VM interpreter code plus an opaque bytecode payload. They cannot use standard Lua decompilers because the file does not contain standard Lua bytecode. They must reverse engineer your VM interpreter to understand the custom instruction encoding, then analyze the bytecode stream to reconstruct your logic.

## Designing a Custom Instruction Set

The first step in building a VM is defining the instruction set. Your custom ISA must be capable of expressing all Lua operations that your original code uses.

### Stack-Based vs. Register-Based VMs

You can choose between two fundamental VM architectures:

**Stack-based VMs** use a value stack. Instructions push operands onto the stack, perform operations on the top elements, and push results back. They are simpler to implement but slightly slower due to stack manipulation overhead.

```text theme={"theme":{"light":"ayu-dark","dark":"night-owl"}}
PUSH_CONST 5      -- stack: [5]
PUSH_CONST 3      -- stack: [5, 3]
ADD               -- stack: [8]  (pops two, pushes sum)
```

**Register-based VMs** specify source and destination registers explicitly, like Lua's own VM. They are more complex to implement but more efficient because they avoid stack overhead.

```text theme={"theme":{"light":"ayu-dark","dark":"night-owl"}}
LOAD_CONST R0, 5  -- R0 = 5
LOAD_CONST R1, 3  -- R1 = 3
ADD R2, R0, R1    -- R2 = R0 + R1
```

For Lua obfuscation, register-based VMs are preferred because they more closely match Lua's semantics and produce more compact bytecode.

### Minimum Viable Instruction Set

A register-based VM for Lua needs at least these instructions to be Turing-complete and Lua-compatible:

| Opcode | Name | Description |
| - | - | - |
| 0 | LOAD\_CONST | Load a constant into a register |
| 1 | LOAD\_NIL | Load nil into a register |
| 2 | LOAD\_BOOL | Load true or false into a register |
| 3 | MOVE | Copy value between registers |
| 4 | GET\_GLOBAL | Read a global variable into a register |
| 5 | SET\_GLOBAL | Write a register to a global variable |
| 6 | GET\_TABLE | Read a table field into a register |
| 7 | SET\_TABLE | Write a register to a table field |
| 8 | ADD | Add two registers/constants |
| 9 | SUB | Subtract two registers/constants |
| 10 | MUL | Multiply two registers/constants |
| 11 | DIV | Divide two registers/constants |
| 12 | MOD | Modulo two registers/constants |
| 13 | UNM | Unary minus |
| 14 | NOT | Logical not |
| 15 | CONCAT | Concatenate registers |
| 16 | EQ | Equality comparison |
| 17 | LT | Less-than comparison |
| 18 | LE | Less-than-or-equal comparison |
| 19 | JMP | Unconditional jump |
| 20 | CALL | Function call |
| 21 | RETURN | Return from function |
| 22 | CLOSURE | Create a closure |

### Instruction Encoding

Each instruction can be encoded as 4 bytes:

```text theme={"theme":{"light":"ayu-dark","dark":"night-owl"}}
Byte 0: opcode (1 byte, 0-255)
Byte 1: operand A (1 byte, register index)
Byte 2: operand B (1 byte, register or constant index)
Byte 3: operand C (1 byte, register or constant index, or jump offset)
```

For instructions that need larger constants, you can use a two-instruction sequence (load constant index into a register, then use that register) or expand the encoding to 8 bytes.

## The VM Interpreter Loop

The VM interpreter is the core of the obfuscation. It fetches instructions, decodes them, and executes the corresponding operations.

### Complete Minimal Register-Based VM

Here is a complete minimal register-based VM in Lua. It supports LOAD\_CONST, MOVE, ADD, SUB, JMP, CALL, and RETURN:

```lua theme={"theme":{"light":"ayu-dark","dark":"night-owl"}}
local function run_vm(bytecode, constants, globals)
    local ip = 1              -- instruction pointer
    local registers = {}      -- register file
    local call_stack = {}     -- call stack for nested calls

    while ip <= #bytecode do
        local opcode = bytecode[ip]
        local a = bytecode[ip + 1]
        local b = bytecode[ip + 2]
        local c = bytecode[ip + 3]
        ip = ip + 4

        if opcode == 0 then       -- LOAD_CONST
            registers[a] = constants[b]

        elseif opcode == 3 then   -- MOVE
            registers[a] = registers[b]

        elseif opcode == 8 then   -- ADD
            registers[a] = registers[b] + registers[c]

        elseif opcode == 9 then   -- SUB
            registers[a] = registers[b] - registers[c]

        elseif opcode == 19 then  -- JMP
            ip = ip + (b * 256 + c) - 32768  -- signed 16-bit offset

        elseif opcode == 20 then  -- CALL
            local fn = registers[a]
            local args = {}
            for i = 1, b do
                table.insert(args, registers[a + i])
            end
            local results = {fn(table.unpack(args))}
            for i = 1, c do
                registers[a + i - 1] = results[i]
            end

        elseif opcode == 21 then  -- RETURN
            local results = {}
            for i = 1, b do
                table.insert(results, registers[a + i - 1])
            end
            return table.unpack(results)

        else
            error("Unknown opcode: " .. opcode)
        end
    end
end
```

### VM Components Explained

**Instruction Pointer (ip)**: Tracks the current position in the bytecode. After fetching a 4-byte instruction, `ip` advances by 4.

**Register File (registers)**: A Lua table that acts as the VM's register bank. Registers are indexed numerically starting from 0.

**Constant Pool (constants)**: A table of literal values (numbers, strings, booleans, nil) referenced by index. The LOAD\_CONST instruction copies constants into registers.

**Call Stack (call\_stack)**: Stores return addresses and register states for nested function calls. A production VM needs this for closures and recursive calls.

## Encoding Custom Bytecode

You need a way to represent your custom instructions as data that can be embedded in a Lua file.

### Binary String Encoding

Use `string.char()` to pack each instruction into 4 bytes:

```lua theme={"theme":{"light":"ayu-dark","dark":"night-owl"}}
local function encode_instruction(opcode, a, b, c)
    return string.char(opcode, a, b, c)
end

local bytecode = ""
bytecode = bytecode .. encode_instruction(0, 0, 0, 0)   -- LOAD_CONST R0, const[0]
bytecode = bytecode .. encode_instruction(0, 1, 1, 0)   -- LOAD_CONST R1, const[1]
bytecode = bytecode .. encode_instruction(8, 2, 0, 1)   -- ADD R2, R0, R1
bytecode = bytecode .. encode_instruction(21, 2, 1, 0)  -- RETURN R2, 1 result
```

### Runtime Decoding

At runtime, extract bytes from the string:

```lua theme={"theme":{"light":"ayu-dark","dark":"night-owl"}}
local function run_vm_string(bytecode_str, constants, globals)
    local ip = 1
    local registers = {}

    while ip <= #bytecode_str do
        local opcode = string.byte(bytecode_str, ip)
        local a = string.byte(bytecode_str, ip + 1)
        local b = string.byte(bytecode_str, ip + 2)
        local c = string.byte(bytecode_str, ip + 3)
        ip = ip + 4

        -- execute opcode...
    end
end
```

### Flat Table Encoding

For simpler indexing, store instructions as a flat Lua table of numbers:

```lua theme={"theme":{"light":"ayu-dark","dark":"night-owl"}}
local bytecode = {
    0, 0, 0, 0,   -- LOAD_CONST R0, const[0]
    0, 1, 1, 0,   -- LOAD_CONST R1, const[1]
    8, 2, 0, 1,   -- ADD R2, R0, R1
    21, 2, 1, 0,  -- RETURN R2, 1 result
}
```

This avoids string packing overhead and makes the VM code slightly simpler.

## Compiling Lua to Your Custom Bytecode

You need a compiler that translates Lua source into your custom instruction set. There are two approaches.

### Option A: Custom Compiler from AST

Write a compiler that parses Lua source into an AST and emits your instruction set. You can use an existing Lua parser library or write a recursive descent parser:

```lua theme={"theme":{"light":"ayu-dark","dark":"night-owl"}}
-- Pseudocode for AST-to-custom-bytecode compiler
local function compile_node(node, bytecode, constants)
    if node.type == "number_literal" then
        local const_idx = add_constant(constants, node.value)
        emit(bytecode, LOAD_CONST, node.target_reg, const_idx, 0)

    elseif node.type == "binary_op" and node.op == "+" then
        compile_node(node.left, bytecode, constants)
        compile_node(node.right, bytecode, constants)
        emit(bytecode, ADD, node.target_reg, node.left.target_reg, node.right.target_reg)

    elseif node.type == "function_call" then
        compile_node(node.func, bytecode, constants)
        for i, arg in ipairs(node.args) do
            compile_node(arg, bytecode, constants)
        end
        emit(bytecode, CALL, node.func.target_reg, #node.args, node.num_results)

    -- ... handle all node types
    end
end
```

### Option B: Bytecode-to-ByteCode Translation

Reuse the standard Lua compiler but translate standard Lua bytecode to your custom bytecode. This is easier because the input format is well-defined:

```lua theme={"theme":{"light":"ayu-dark","dark":"night-owl"}}
-- Pseudocode for standard-to-custom bytecode translator
local function translate_luac_to_custom(standard_bytecode)
    local custom = {}
    local constants = {}

    for _, instr in ipairs(parse_standard_bytecode(standard_bytecode)) do
        local custom_op = opcode_map[instr.opcode]

        if instr.opcode == OP_LOADK then
            local new_const_idx = add_constant(constants, instr.constant)
            emit(custom, OP_LOAD_CONST, instr.a, new_const_idx, 0)

        elseif instr.opcode == OP_ADD then
            emit(custom, OP_ADD, instr.a, instr.b, instr.c)

        elseif instr.opcode == OP_JMP then
            local offset = instr.sbx
            emit(custom, OP_JMP, 0, math.floor(offset / 256), offset % 256)

        -- ... map all standard opcodes to custom equivalents
        end
    end

    return custom, constants
end
```

Bytecode-to-bytecode translation is the approach used by many commercial Lua obfuscators because it avoids writing a full parser.

## Obfuscating the VM Interpreter Itself

The VM interpreter is now the most valuable target for an analyst. You must obfuscate it just as aggressively as the original code.

### Apply Source-Level Obfuscation

Use the techniques from other sections of this guide on the VM interpreter:

* **Variable renaming**: rename `registers` to `_`, `ip` to `__`, `opcode` to `___`
* **String encoding**: encode error messages and opcode names
* **Control flow flattening**: replace the if/elseif chain with a flattened dispatcher

### Obfuscate Opcode Constants

The VM's opcode numbers should not appear as plain literals. Use constant obfuscation:

```lua theme={"theme":{"light":"ayu-dark","dark":"night-owl"}}
-- Instead of:
if opcode == 8 then  -- ADD

-- Use:
local _a = 2 * 4
local _b = 2 ^ 3
if opcode == (_a + _b - 8) then  -- still 8, but not obvious
```

### Function Dispatch Table

Replace the if/elseif chain with a function dispatch table. This is harder to analyze than a flat conditional chain:

```lua theme={"theme":{"light":"ayu-dark","dark":"night-owl"}}
local ops = {}

ops[0] = function(a, b, c)  -- LOAD_CONST
    registers[a] = constants[b]
end

ops[3] = function(a, b, c)  -- MOVE
    registers[a] = registers[b]
end

ops[8] = function(a, b, c)  -- ADD
    registers[a] = registers[b] + registers[c]
end

-- In the main loop:
local op_fn = ops[opcode]
if op_fn then
    op_fn(a, b, c)
else
    error("?")
end
```

### Flattened Dispatcher Version

For even stronger obfuscation, flatten the dispatcher so there is no obvious table lookup:

```lua theme={"theme":{"light":"ayu-dark","dark":"night-owl"}}
local _ops = {}
for _i = 1, 23 do
    _ops[_i] = function() end  -- dummy entries
end

_ops[0x00] = function(_a, _b) registers[_a] = constants[_b] end
_ops[0x03] = function(_a, _b) registers[_a] = registers[_b] end
_ops[0x08] = function(_a, _b, _c) registers[_a] = registers[_b] + registers[_c] end
-- ...

local _d = _ops[opcode]
if _d then _d(a, b, c) end
```

## Real-World Lua Virtualizers

Several open-source projects demonstrate production-quality Lua VM obfuscation.

### Ironbrew2

Ironbrew2 is a Lua 5.1 obfuscator focused on the Roblox ecosystem. It implements a register-based VM with the following features:

* Custom instruction set with encrypted opcodes
* String encryption using XOR and dynamic keys
* Control flow obfuscation within the VM loop
* Anti-tamper checks that detect debugging attempts

Ironbrew2 is open source and serves as an educational reference for how VM obfuscation works in practice.

### Prometheus

Prometheus is a modular Lua 5.1 obfuscator with pluggable transformation steps. Its pipeline includes:

* Lexical obfuscation (variable renaming, string encryption)
* Control flow flattening
* VM wrapping (converting functions to custom bytecode + VM)
* Anti-debug and anti-tamper layers

Prometheus demonstrates how multiple obfuscation techniques can be layered, with VM wrapping as the strongest final step.

> [!WARNING]
>
> VM overhead is significant. A VM-obfuscated script typically runs 5 to 20 times slower than the original. Only apply VM obfuscation to code that is not called in tight loops or performance-critical paths.

> [!TIP]
>
> Apply the VM only to the most sensitive parts of your codebase, such as license checks, decryption routines, or proprietary algorithms. Leave performance-critical code as standard (but otherwise obfuscated) Lua. This hybrid approach gives you strong protection where it matters without sacrificing overall performance.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.