Add script execution debugger

[febyeji]

Hi, I hope to add a script execution debugger to rust-bitcoinkernel, building on the proof-of-concept in the script_debug branch. It exposes Bitcoin Script interpreter state at each opcode step to Rust callers via a callback. Please feel free to comment on this!

Design

Hooks into EvalScript via a DEBUG_SCRIPT macro in the C++ subtree. At each step, the callback receives the main stack, altstack, raw script bytes, opcode position, conditional execution state (fExec), and script error on failure. On the Rust side this is exposed via a ScriptDebugger type and a trace_verify entry point.

Work Plan (Not concrete though)

Phase 1: Core infrastructure

1. C++ EvalScript hook
2. Rust opcode/stack types
3. ScriptDebugger and trace_verify
4. Expose script error via thread-local
5. Expose fExec (conditional branch state)
6. Serde feature + std::error::Error

Phase 2: Testing and tooling

7. Integration tests
8. Add script_debugger examples
9. Script verification coverage tests (hope to address Feat: Improve script verification coverage #132 here)

[pzafonte]

Hi @febyeji Looking forward to working together on this issue. Great start, here are some of my thoughts to possibly sharpen the scope and fill in the gaps.

A few things worth noting as we formalize the design:

The script_debug branch already has the skeleton in place (ScriptDebugCallbackHolder, ScriptDebugger, script_debug_wrapper, and a working example in examples/script_debug.rs), so Phase 1 could be re-scoped around what is still missing rather than what already exists, specifically the three items below jump out to me:

fExec isn't in the current callback. The current signature is Fn(stack, script, pos, altstack). Without executing: bool, you can't tell if an opcode ran or was skipped inside a false OP_IF branch. A simple test case that would catch this: OP_0 OP_IF OP_1 OP_ELSE OP_2 OP_ENDIF the OP_1 step should arrive with executing = false.

Error state could reuse what's already there. Rather than a raw code, adding Option<ScriptVerifyError> as a final parameter (None during execution, Some(err) on failure) keeps it consistent with what verify() already returns.

It's worth noting that kernel_register_script_debug_cb is a global, so tests using the debugger need cargo test -- --test-threads=1 or they'll interfere with each other.

For Phase 2:
A struct would make the API easier to extend. Instead of five positional args, a ScriptDebugStep { stack, altstack, script, opcode_pos, executing, error } means adding fields later won't break existing callers, and the closure becomes just Fn(&ScriptDebugStep).

[febyeji]

@pzafonte Thanks for the thorough review! Regarding your suggestion on error state, I'm leaning toward adding a separate ScriptExecError enum. ScriptVerifyError covers config-level issues (bad flags, missing spent outputs) while interpreter failures (EVAL_FALSE, SIG_NULLFAIL, CLEANSTACK, etc.) are a different concern. To keep verify() consistent, we could nest it:

pub enum ScriptVerifyError {
    TxInputIndex,
    InvalidFlags,
    InvalidFlagsCombination,
    SpentOutputsRequired,
    Invalid(ScriptExecError), // was just `Invalid`
}

What do you think? (I am also happy with flattening all error variants though)

[pzafonte]

@febyeji Thanks for that clarification and I think adding a ScriptExecError enum is right. And, I was looking more closely at interpreter.cpp:

for (; pc < pend; ++opcode_pos) {
    DEBUG_SCRIPT(stack, script, opcode_pos, altstack);  // <- fires here
    bool fExec = vfExec.all_true();                     // <- fExec computed
    ...
    if (!script.GetOp(pc, opcode, vchPushValue))
        return set_error(serror, SCRIPT_ERR_BAD_OPCODE); // <- error set after

DEBUG_SCRIPT fires at the start of each iteration, before the opcode executes and before set_error is called. So serror at the moment of the hook is always the previous step's error. The failing step's error gets set after the hook fires and the function returns. To capture the error on the failing step, I think we need a second call after set_error, rather than just passing serror to the existing hook.

So I think we might need two separate hooks. One per-step and one on-failure.

// Per-step hook fires before each opcode
DEBUG_SCRIPT(stack, script, opcode_pos, altstack, fExec);

// Separate failure hook fires once on failure
DEBUG_SCRIPT_ERROR(*serror);

// The Rust Callback could get two closures
pub struct ScriptDebugCallbackHolder {
    pub on_step:    Box<dyn Fn(&ScriptDebugStep)>,
    pub on_failure: Box<dyn Fn(ScriptExecError)>,
}

interpreter.cpp might then look something like this:

for (; pc < pend; ++opcode_pos) {
    bool fExec = vfExec.all_true();
    DEBUG_SCRIPT(stack, script, opcode_pos, altstack, fExec);  // <- adds fExec
    ...
}
// single point where EvalScript returns false
if (*serror != SCRIPT_ERR_OK) {
    DEBUG_SCRIPT_ERROR(*serror);  // <- fires once on failure
}
return false;

So on second thought, in this case adding Option<ScriptVerifyError> as a final parameter to on_step would be redundant. on_step only fires during successful execution, and when failure does occur on_failure would already then carry it directly as ScriptExecError. So two flat enums, ScriptExecError on ScriptDebugStep might be a good design choice here. What are your thoughts?

[febyeji]

Thanks for the detailed analysis @pzafonte! I've started addressing some of this in PR #139:

• fExec is now computed before DEBUG_SCRIPT fires, and passed through to the Rust callback
• The callback uses a struct (ScriptDebugFrame) rather than positional args

For the error hook, I agree that a separate DEBUG_SCRIPT_ERROR makes more sense than trying to put everything on the per-step callback. I'll leave that for a follow-up.