Every benchmark result is suspect until it survives contact with reality. Ashish Sinha recently ran his tool, schemagate, against a newly released SIGMOD'27 public benchmark for Text-to-SQL under Role-Based Access Control (RBAC). The results are stark: while the paper’s best configuration allowed models to answer 27% of questions they should have refused, schemagate reduced unauthorized answers to 5.4% on the BIRD dataset, all while guaranteeing zero queries touched forbidden data.

The Hallucination Problem in RBAC

The core finding of the SIGMOD'27 paper is uncomfortable for developers relying on simple schema masking. When restricted columns are merely hidden from the context window, large language models do not stop. Instead, they improvise. They patch together answers using visible columns, generating SQL that looks plausible but violates access policies. The authors’ best 'hide the schema' setup on the BIRD benchmark still failed to refuse 27% of the queries that required restricted data. This is not a minor edge case; it is a fundamental failure mode of probabilistic reasoning when the ground truth is removed.

Schemagate’s Architectural Advantage

Schemagate tackles this by treating policy as visibility, not just filtering. The model never sees the name of a hidden table or column. Crucially, it receives a one-line notice stating its view is partial without specifying what is missing, prompting refusal over improvisation. The real teeth, however, lie in the post-generation scope check. Every generated query is validated against the policy before execution. If the SQL touches anything outside the allowed scope, it is refused. This transforms access control from a probabilistic likelihood into a deterministic guarantee.

Benchmark Results and Ablation Studies

Using Claude Sonnet 4.5, the same model as the paper’s best runs, schemagate achieved an AC-F1 score of 90.1 on BIRD, beating the paper’s best score of 88.1. On Spider, it tied at 93.0. The ablation studies reveal the cost of each component: removing the scope check doubled the rate of bad answers on BIRD from 5.4% to 11.3%. Interestingly, providing rich column descriptions actually hurt performance, helping models find workarounds for questions they should have refused. The one-line notice reduced bad answers from 16% to 5% in a 100-question test.

Key Takeaways

  • Hiding schema elements is insufficient; LLMs will hallucinate answers using visible data, leading to 27% error rates in standard benchmarks.
  • Deterministic scope checking is critical. Schemagate’s approach ensures 0% of queries read forbidden data, unlike probabilistic methods.
  • Over-contextualizing can hurt access control. Adding column descriptions to visible fields increased error rates by helping models improvise around restrictions.
  • The new benchmark is public, and schemagate 1.6.0 now ships with the optimized refusal notice by default.

The Bottom Line

If your text-to-SQL system relies on the model 'understanding' what it can't see, you are playing Russian roulette with your data privacy. Hard scope checks are the only way to turn access control from a suggestion into a guarantee.