Sandbox and Deobfuscator Detection in Lua/Luau

Automated deobfuscators often execute your script in a sandboxed Lua VM and observe its behavior to reconstruct the original logic. Detecting these environments at runtime lets you disable obfuscated functionality, produce misleading output, or waste the attacker’s computational resources. This page covers library probing, behavioral anomaly detection, coroutine and metatable checks, known deobfuscator fingerprints, and safe response strategies.

Detecting Sandboxed Standard Libraries

Sandboxes frequently strip or stub out standard library functions to limit what obfuscated scripts can do. Probing for the presence and behavior of these functions reveals whether your code is running in a restricted environment.

Library Presence Checks

Check whether functions that are normally available in a full Lua VM are missing or non-functional:

local function check_library_integrity()
    local checks = {
        { name = "os.exit",      fn = os.exit },
        { name = "io.open",      fn = io and io.open },
        { name = "package.loadlib", fn = package and package.loadlib },
        { name = "debug.getinfo", fn = debug and debug.getinfo },
        { name = "collectgarbage", fn = collectgarbage },
        { name = "loadfile",     fn = loadfile },
    }

    local missing = 0
    for _, check in ipairs(checks) do
        if type(check.fn) ~= "function" then
            missing = missing + 1
        end
    end

    return missing, #checks
end

A high missing count suggests a sandboxed or emulated environment. os.exit and io are particularly common targets for removal because they allow scripts to affect the host system.

Scoring Function

Convert the results into a suspicion score that you can threshold against:

local function sandbox_suspicion_score()
    local missing, total = check_library_integrity()
    local score = (missing / total) * 100

    -- Additional checks
    if os and os.execute then
        local ok = pcall(os.execute, "exit 0")
        if not ok then
            score = score + 20
        end
    end

    return score
end

If the score exceeds a threshold you define (for example, 50), treat the environment as suspicious.

Detecting Behavioral Anomalies

Even when libraries are present, sandboxes and emulators often behave differently from a real Lua VM. Detect these anomalies by probing deterministic behavior.

math.random Seed Anomalies

In a real Lua VM, math.random without an explicit seed uses a default seed derived from the system time. In some sandboxes, the RNG is seeded to a fixed value for reproducibility. Check whether the sequence matches an expected non-seeded pattern:

local function is_rng_deterministic()
    local samples = {}
    for i = 1, 10 do
        samples[i] = math.random(1, 1000000)
    end

    -- In a sandbox with fixed seed, the same sequence may repeat
    -- across runs. Compare against a known reference if available.
    -- Here we just detect suspiciously low variance.
    local sum = 0
    for _, v in ipairs(samples) do
        sum = sum + v
    end
    local avg = sum / #samples

    -- All samples near the average suggests poor randomness
    local variance = 0
    for _, v in ipairs(samples) do
        variance = variance + (v - avg) ^ 2
    end
    variance = variance / #samples

    return variance < 1000 -- suspiciously low
end

Time Anomalies

Sandboxes sometimes return a fixed or stepped time from os.time(). Check whether time advances between two calls separated by a busy-wait loop:

local function is_time_frozen()
    local t1 = os.time()

    -- Busy wait for a short period
    local start = os.clock()
    while os.clock() - start < 0.1 do end

    local t2 = os.time()
    return t1 == t2
end

If os.time() does not advance, the environment may be emulated or time is being controlled.

Function Address Anomalies

In a real Lua VM, tostring of a function includes a process-specific address. In some emulated VMs, function addresses are always 0x0 or a fixed value:

local function is_function_address_suspicious()
    local repr = tostring(print)
    return string.match(repr, "0x0+%s*$") ~= nil
end

A 0x0 address suggests the VM is not allocating real function objects.

Coroutine Behavior Checks

Some sandboxes implement coroutines incorrectly. A specific yield and resume pattern may produce wrong status values or fail to preserve values across yields.

Coroutine Canary Test

local function coroutine_canary()
    local co = coroutine.create(function(a, b)
        coroutine.yield(a + b)
        return a * b
    end)

    local ok1, sum = coroutine.resume(co, 3, 4)
    local status1 = coroutine.status(co)

    local ok2, product = coroutine.resume(co)
    local status2 = coroutine.status(co)

    return ok1 and ok2
        and sum == 7
        and product == 12
        and status1 == "suspended"
        and status2 == "dead"
end

If any of these assertions fail, the coroutine implementation is non-standard.

Metatable Behavior Checks

Metamethods are complex to emulate correctly. A sandbox with a simplified object model may fail to invoke __index and __newindex under all circumstances.

Metatable Canary Test

local function metatable_canary()
    local index_called = false
    local newindex_called = false

    local t = {}
    setmetatable(t, {
        __index = function(_, k)
            index_called = true
            return k .. "_value"
        end,
        __newindex = function(_, k, v)
            newindex_called = true
            rawset(t, k, v)
        end,
    })

    local _ = t.foo -- triggers __index
    t.bar = 42      -- triggers __newindex

    return index_called and newindex_called and t.bar == 42
end

If index_called or newindex_called is false, the sandbox does not implement metamethods faithfully.

Detecting Known Deobfuscators

Some deobfuscators leave fingerprints in the global environment or modify standard functions in recognizable ways.

Global Name Fingerprints

Check for global names that deobfuscators inject to mark or hook your code:

local function check_deobfuscator_globals()
    local suspicious_names = {
        "_DEOB",
        "_HOOK",
        "_UNPACK",
        "_DECOMP",
        "_TRACE",
    }

    for _, name in ipairs(suspicious_names) do
        if _G[name] ~= nil then
            return true, name
        end
    end

    return false, nil
end

pcall Wrapper Detection

Deobfuscators often wrap your script in pcall to catch errors during analysis. You can detect this by raising an error and inspecting the stack depth:

local function is_inside_pcall_wrapper()
    local level = 0
    local ok, err = pcall(function()
        error("canary")
    end)

    -- If we are inside a pcall wrapper, the error is caught
    -- and the stack trace is shallower than expected
    if ok then
        -- Error was swallowed: definitely wrapped
        return true
    end

    -- Count stack frames in the error string
    for _ in string.gmatch(err, "\n") do
        level = level + 1
    end

    return level < 3 -- suspiciously shallow
end

Responding to Sandbox Detection

When you detect a sandbox, avoid crashing loudly. A sudden error tells the analyst exactly which check triggered and helps them patch it.

Recommended Responses

Response	Description	Effect on Analyst
Misleading Output	Return plausible but wrong results	Wastes time analyzing false logic
Infinite Loop	Enter a loop with no side effects	Consumes resources, no progress
Long Sleep	Sleep for hours or days	Blocks automated pipelines
Dummy Data	Return hardcoded dummy values	Produces useless output

Misleading Output Pattern

If a sandbox is detected, return plausible but wrong results for the first several calls, then stop entirely:

local call_count = 0

local function protected_function(x)
    if is_sandbox_detected() then
        call_count = call_count + 1
        if call_count <= 10 then
            return x * 0.99 -- slightly wrong, plausible
        else
            return nil -- silent failure
        end
    end

    -- Real logic
    return x * 2
end

This pattern misleads an analyst who is tracing outputs to reverse engineer your algorithm. The first ten results look almost correct, which delays discovery of the sandbox detection.

Sandbox detection is an arms race. A determined analyst can patch your detection checks, replace your is_sandbox_detected function, or run your script in a fully faithful VM. Use sandbox detection to slow down automated tools and raise the cost of analysis, not as a definitive protection layer.

Getting Started

Introduction

Source-Level Techniques

AST Manipulation

Bytecode and VM Techniques

Runtime and Environment

Advanced and Combined

Sandbox and Deobfuscator Detection in Lua/Luau

Detecting Sandboxed Standard Libraries

Library Presence Checks

Scoring Function

Detecting Behavioral Anomalies

math.random Seed Anomalies

Time Anomalies

Function Address Anomalies

Coroutine Behavior Checks

Coroutine Canary Test

Metatable Behavior Checks

Metatable Canary Test

Detecting Known Deobfuscators

Global Name Fingerprints

pcall Wrapper Detection

Responding to Sandbox Detection

Recommended Responses

Misleading Output Pattern

​Detecting Sandboxed Standard Libraries

​Library Presence Checks

​Scoring Function

​Detecting Behavioral Anomalies

​math.random Seed Anomalies

​Time Anomalies

​Function Address Anomalies

​Coroutine Behavior Checks

​Coroutine Canary Test

​Metatable Behavior Checks

​Metatable Canary Test

​Detecting Known Deobfuscators

​Global Name Fingerprints

​pcall Wrapper Detection

​Responding to Sandbox Detection

​Recommended Responses

​Misleading Output Pattern

Detecting Sandboxed Standard Libraries

Library Presence Checks

Scoring Function

Detecting Behavioral Anomalies

math.random Seed Anomalies

Time Anomalies

Function Address Anomalies

Coroutine Behavior Checks

Coroutine Canary Test

Metatable Behavior Checks

Metatable Canary Test

Detecting Known Deobfuscators

Global Name Fingerprints

pcall Wrapper Detection

Responding to Sandbox Detection

Recommended Responses

Misleading Output Pattern