Opcode Remapping and Instruction Set Obfuscation

Standard Lua bytecode uses a fixed opcode numbering scheme that every disassembler and decompiler knows by heart. OP_MOVE is opcode 0 in Lua 5.1, OP_LOADK is opcode 1, and so on. Because these numbers are public and unchanging, any tool that reads Lua bytecode can immediately interpret the instruction stream. By remapping opcodes to different values and patching the Lua virtual machine to understand your custom numbering, you produce bytecode that standard tools cannot decode without first reverse engineering your VM modifications. This page explains how Lua opcodes work, how to remap them by modifying the Lua interpreter source, how to shuffle operand fields, and how to distribute a custom Lua runtime alongside your protected bytecode.

How Lua Opcodes Work

Lua instructions are 32-bit fixed-size values in most Lua versions. The bits are divided into an opcode field and operand fields that vary by instruction type.

Instruction Layout

In Lua 5.1, the instruction format uses the following bit layout:

Bits	Field	Description
0–5	opcode	6-bit opcode identifier
6–13	A	8-bit operand
14–23	B	9-bit operand
24–31	C	9-bit operand

For instructions that need larger constants, the B and C fields can be combined into a single 18-bit Bx field, or a signed sBx field for jump offsets.

Standard Lua 5.1 Opcode Table

Lua 5.1 defines 38 opcodes numbered 0 through 37:

Opcode	Name	Description
0	OP_MOVE	Copy value between registers
1	OP_LOADK	Load constant into register
2	OP_LOADBOOL	Load boolean into register
3	OP_LOADNIL	Load nil into registers
4	OP_GETUPVAL	Read upvalue into register
5	OP_GETGLOBAL	Read global variable into register
6	OP_GETTABLE	Read table field into register
7	OP_SETGLOBAL	Write register to global variable
8	OP_SETUPVAL	Write register to upvalue
9	OP_SETTABLE	Write register to table field
10	OP_NEWTABLE	Create new table
11	OP_SELF	Prepare for method call
12	OP_ADD	Add two registers/constants
13	OP_SUB	Subtract two registers/constants
14	OP_MUL	Multiply two registers/constants
15	OP_DIV	Divide two registers/constants
16	OP_MOD	Modulo two registers/constants
17	OP_POW	Power two registers/constants
18	OP_UNM	Unary minus
19	OP_NOT	Logical not
20	OP_LEN	Length operator
21	OP_CONCAT	Concatenate registers
22	OP_JMP	Unconditional jump
23	OP_EQ	Equality test
24	OP_LT	Less-than test
25	OP_LE	Less-than-or-equal test
26	OP_TEST	Logical test (no jump)
27	OP_TESTSET	Logical test with assignment
28	OP_CALL	Function call
29	OP_TAILCALL	Tail call
30	OP_RETURN	Return from function
31	OP_FORLOOP	Numeric for loop
32	OP_FORPREP	Numeric for loop setup
33	OP_TFORLOOP	Iterator for loop
34	OP_SETLIST	Set array elements
35	OP_CLOSE	Close variables
36	OP_CLOSURE	Create closure
37	OP_VARARG	Vararg operator

Lua 5.4 expands this to 83 opcodes with new instructions for integer operations, constant table improvements, and more efficient closures.

The Remapping Technique

Opcode remapping requires modifying the Lua interpreter source code, recompiling both the compiler (luac) and the runtime (lua), and then using your custom toolchain to produce and execute bytecode.

Step 1: Modify the Opcode Enum

The opcode definitions live in lopcodes.h. In Lua 5.1, the enum looks like this:

/* lopcodes.h - Lua 5.1 standard opcodes */
enum OpCode {
    OP_MOVE = 0,
    OP_LOADK = 1,
    OP_LOADBOOL = 2,
    OP_LOADNIL = 3,
    OP_GETUPVAL = 4,
    OP_GETGLOBAL = 5,
    OP_GETTABLE = 6,
    OP_SETGLOBAL = 7,
    OP_SETUPVAL = 8,
    OP_SETTABLE = 9,
    OP_NEWTABLE = 10,
    /* ... continues through OP_VARARG = 37 */
};

To remap opcodes, shuffle the assigned values arbitrarily:

/* lopcodes.h - Remapped opcodes */
enum OpCode {
    OP_MOVE = 17,
    OP_LOADK = 3,
    OP_LOADBOOL = 29,
    OP_LOADNIL = 8,
    OP_GETUPVAL = 12,
    OP_GETGLOBAL = 35,
    OP_GETTABLE = 1,
    OP_SETGLOBAL = 22,
    OP_SETUPVAL = 7,
    OP_SETTABLE = 14,
    OP_NEWTABLE = 31,
    /* ... all 38 opcodes must be assigned unique values 0-37 */
};

Every opcode must have a unique value in the valid range. If two opcodes share the same number, the VM will execute the wrong instruction and crash unpredictably.

Step 2: Recompile the Toolchain

After modifying lopcodes.h, rebuild both the compiler and the interpreter:

# Standard Lua build process
cd lua-5.1.5
make clean
make linux  # or make macosx, make mingw

# This produces:
# - lua (the interpreter)
# - luac (the compiler)

Your modified luac now emits bytecode with your custom opcode values. Your modified lua interpreter knows how to execute them. Standard luac and lua binaries will not be able to read or run your bytecode.

Step 3: Compile Scripts with Your Custom luac

# Use your custom compiler
./luac -o protected.luac script.lua

# Verify with standard luac (should fail or show garbage)
# /usr/bin/luac -l protected.luac  -- produces nonsense

# Run with your custom interpreter
./lua -e 'loadfile("protected.luac")()'

When a standard decompiler like unluac reads your protected.luac, it interprets opcode 17 as OP_MOVE (which is correct in your mapping but wrong in standard Lua). Since standard tools expect opcode 0 for OP_MOVE, they misinterpret the entire instruction stream and produce garbage output.

Shuffle vs. Encrypt the Dispatch Table

A more aggressive variant goes beyond static remapping and adds runtime decoding to the VM’s instruction dispatch loop.

Static Remapping vs. Dynamic Decoding

Static remapping (the technique above) assigns fixed new numbers to each opcode. Dynamic decoding adds a translation step inside the VM loop:

/* lvm.c - Standard dispatch */
vmcase(OP_MOVE):
    setobjs2s(L, ra, RB(i));
    vmbreak;

/* lvm.c - Dynamic decoding */
vmcase(OP_MOVE):
    {
        int real_opcode = decode_table[GET_OPCODE(i)];
        /* dispatch based on real_opcode */
    }

Runtime-Keyed Decode Tables

You can key the decode table by a hash of the script content, so different scripts use different opcode mappings:

/* Pseudocode for runtime-keyed decoding */
static int decode_table[38];

void init_decode_table(const char* script_hash) {
    /* Derive a permutation from the hash */
    for (int i = 0; i < 38; i++) {
        decode_table[i] = permute(i, script_hash);
    }
}

/* In the VM loop */
OpCode op = decode_table[GET_OPCODE(i)];
switch (op) {
    case OP_MOVE: /* ... */
    case OP_LOADK: /* ... */
    /* ... */
}

This means an analyst cannot create a universal decompiler for your bytecode. Each script requires its own decode table, which is derived from the script itself at load time.

Operand Field Shuffling

In addition to opcode remapping, you can rearrange the bit fields within the 32-bit instruction word.

Standard vs. Shuffled Bit Layout

Standard Lua 5.1 layout:

| C (9 bits) | B (9 bits) | A (8 bits) | opcode (6 bits) |
| 31      23 | 22      14 | 13       6 | 5             0 |

Shuffled layout (example: opcode in high bits):

| opcode (6 bits) | C (9 bits) | B (9 bits) | A (8 bits) |
| 31           26 | 25      17 | 16       8 | 7        0 |

Modifying the Bit Extraction Macros

The bit manipulation macros in lopcodes.h control how the compiler encodes and the VM decodes instructions:

/* Standard Lua 5.1 macros */
#define GET_OPCODE(i)   ((OpCode)((i) & MASK1(SIZE_OP, 0)))
#define SET_OPCODE(i,o) ((i) = (((i)&MASK0(SIZE_OP, 0)) \
        | ((cast(lu_int32, o)&MASK1(SIZE_OP, 0)))))

#define GETARG_A(i)     ((int)(((i) >> POS_A) & MASK1(SIZE_A, 0)))
#define SETARG_A(i,u)   ((i) = (((i)&MASK0(SIZE_A, POS_A)) \
        | ((cast(lu_int32, u)&MASK1(SIZE_A, 0)) << POS_A)))

To shuffle operand positions, redefine POS_A, POS_B, POS_C, and POS_OP:

/* Shuffled positions: opcode in bits 26-31 */
#define POS_OP      26
#define POS_A       0
#define POS_B       8
#define POS_C       17

Both luac (the compiler) and lvm.c (the VM) must use identical macros. If they differ by even one bit position, every instruction will be decoded incorrectly and the VM will crash.

Operand field shuffling requires modifying every instruction encoder and decoder in the Lua source. A single wrong bit field offset crashes every script. Test every instruction type thoroughly.

Distribution Implications

Opcode remapping requires distributing a custom Lua VM alongside your bytecode. This has important practical implications.

Standalone Lua Applications

For desktop or server applications, package your modified lua executable or liblua.so with your application:

# Linux deployment
mkdir my_app/
cp custom_lua my_app/
cp protected.luac my_app/
cp loader.lua my_app/

# The loader uses your custom interpreter
# ./my_app/custom_lua loader.lua

Embedded Lua in Game Engines

Many game engines embed Lua for scripting. To use opcode remapping, you must modify the embedded Lua source and recompile the entire engine:

Locate the Lua source inside the engine (often ThirdParty/lua/ or external/lua/)
Apply your lopcodes.h and lvm.c modifications
Rebuild the engine
Compile game scripts with your custom luac

Roblox Luau Limitations

In Roblox Luau, you cannot replace the VM. The Luau interpreter is built into the Roblox client and server, and developers have no access to modify it. Opcode remapping is not possible in Roblox. For Roblox, use VM-based obfuscation implemented in Luau (see the Virtualization page) instead.

Partial Opcode Remapping Without Recompiling the VM

If you cannot replace the VM, you can still achieve opcode remapping by implementing a custom VM interpreter in Lua itself. Your custom bytecode uses your own opcode numbers, and a Lua-written VM decodes and executes them. This approach is what VM-based obfuscation does at its core. Instead of patching the C VM, you write a Lua VM in Lua:

-- Simplified partial remapping without C source changes
local custom_opcodes = {
    [0] = "MOVE",      -- your OP_MOVE
    [1] = "LOADK",     -- your OP_LOADK
    [2] = "ADD",       -- your OP_ADD
    -- ...
}

local standard_opcodes = {
    MOVE = 0,    -- standard Lua 5.1 OP_MOVE
    LOADK = 1,   -- standard Lua 5.1 OP_LOADK
    ADD = 12,    -- standard Lua 5.1 OP_ADD
    -- ...
}

-- Translate custom bytecode to standard bytecode at runtime
local function translate_instruction(custom_op, a, b, c)
    local op_name = custom_opcodes[custom_op]
    local standard_op = standard_opcodes[op_name]
    return encode_standard_instruction(standard_op, a, b, c)
end

This translation step is the foundation of the virtualization technique covered in detail on the Virtualization page.

Opcode remapping requires deep knowledge of Lua internals and careful testing across all instruction types. A single wrong bit field offset or opcode assignment crashes every script. Always test with comprehensive scripts that exercise every opcode.

This technique is what serious commercial game protection tools do under the hood. It is the most robust protection technique that does not require building a full custom VM from scratch.

Getting Started

Introduction

Source-Level Techniques

AST Manipulation

Bytecode and VM Techniques

Runtime and Environment

Advanced and Combined

Opcode Remapping and Instruction Set Obfuscation

How Lua Opcodes Work

Instruction Layout

Standard Lua 5.1 Opcode Table

The Remapping Technique

Step 1: Modify the Opcode Enum

Step 2: Recompile the Toolchain

Step 3: Compile Scripts with Your Custom luac

Shuffle vs. Encrypt the Dispatch Table

Static Remapping vs. Dynamic Decoding

Runtime-Keyed Decode Tables

Operand Field Shuffling

Standard vs. Shuffled Bit Layout

Modifying the Bit Extraction Macros

Distribution Implications

Standalone Lua Applications

Embedded Lua in Game Engines

Roblox Luau Limitations

Partial Opcode Remapping Without Recompiling the VM

​How Lua Opcodes Work

​Instruction Layout

​Standard Lua 5.1 Opcode Table

​The Remapping Technique

​Step 1: Modify the Opcode Enum

​Step 2: Recompile the Toolchain

​Step 3: Compile Scripts with Your Custom luac

​Shuffle vs. Encrypt the Dispatch Table

​Static Remapping vs. Dynamic Decoding

​Runtime-Keyed Decode Tables

​Operand Field Shuffling

​Standard vs. Shuffled Bit Layout

​Modifying the Bit Extraction Macros

​Distribution Implications

​Standalone Lua Applications

​Embedded Lua in Game Engines

​Roblox Luau Limitations

​Partial Opcode Remapping Without Recompiling the VM

How Lua Opcodes Work

Instruction Layout

Standard Lua 5.1 Opcode Table

The Remapping Technique

Step 1: Modify the Opcode Enum

Step 2: Recompile the Toolchain

Step 3: Compile Scripts with Your Custom luac

Shuffle vs. Encrypt the Dispatch Table

Static Remapping vs. Dynamic Decoding

Runtime-Keyed Decode Tables

Operand Field Shuffling

Standard vs. Shuffled Bit Layout

Modifying the Bit Extraction Macros

Distribution Implications

Standalone Lua Applications

Embedded Lua in Game Engines

Roblox Luau Limitations

Partial Opcode Remapping Without Recompiling the VM