Locksmith Loop: An Agentic Method for Deterministic Validation of Legacy COBOL-to-Java Migration

Migrating legacy COBOL to Java is a nightmare of corner cases and missing test data. A new arXiv paper proposes an agentic test-synthesis method called the 'Locksmith Loop' that aims to make validation deterministic rather than probabilistic.

The approach starts by setting up two runtime environments: the COBOL source and the generated Java target, both instrumented with mocks and executed off-mainframe on commodity hardware. An iterative agentic loop then performs 'Witness Search' over input mocks to penetrate program branches, followed by parity-preserving mutations. When routing boundaries are reached, an analyzer identifies a 'Locked Paragraph' — a condition preventing deeper exploration.

Across three COBOL-Java case studies (two open-source programs and one internal production-like COBOL program, ranging from 430 to 4,114 source lines), Locksmith consistently improved coverage beyond input-search plateaus. It reached nearly complete coverage on the two open-source programs and 91.90% branch coverage on the internal production-like COBOL program. Crucially, the generated Java matched the COBOL reference under deterministic parity checks in all accepted test cases.

The paper's key contribution is using a deterministic oracle to validate agentic coding output — a shift from relying on statistical likelihood to provable equivalence. The authors claim this is a novel approach to validating AI-generated code migrations.

For engineers dealing with legacy modernization, this offers a concrete path to verifying that an LLM-generated Java port behaves identically to the original COBOL, without needing exhaustive pre-existing test suites.