Public sampler
Run ten synthetic, English language GOSI style reconciliation tasks with reference solutions and final state checks.
See exactly what is included →Executable environments · GCC workflows
We build RL environments where agents are scored on what actually changed in the database.
Not on what they report. Whether the right records were updated, whether anything else was damaged, and whether the agent correctly did nothing when nothing should have changed.
One method. Three ways to engage.
Start with inspectable public evidence, define a limited evaluation around your question, or scope an environment for a specific workflow.
Scoped evaluation pilot
Agree on a workflow, target agent interface, pass conditions, exclusions and evidence package before any build or evaluation.
See a pilot outline →Custom environment work
Model a specific administrative workflow with synthetic records, resettable state and verifiers shaped to the buyer’s use case.
Understand the environment →Why saved state matters
Completion claims are not completion.
An agent can report success after updating the wrong record, leaving a prerequisite unresolved, or changing an unrelated account. A verifier reads the underlying state instead of trusting the final message.
How verification works →Example task
Find the records that meet a date and eligibility rule, complete only the qualifying renewals, resolve their blockers, and leave every other record unchanged. If the qualifying count is wrong, make no changes.
Tasks use synthetic people and records. They do not touch a live government system.
What we've built
Our first environment covers employment and residency administration, one of the harder regulatory domains to model: eligibility conditions spread across three separate systems, social insurance contributions with cohort branching and caps, dual calendar systems, and rules that change by effective period.
It is built, calibrated, and partly public. Ten tasks are published under Apache-2.0 with reference solutions, deterministic checks, and a documented trust boundary. Running two frontier models through eighty attempts, we found agents completing part of a multi-step job and reporting it finished, updating the correct record while altering an unrelated one, and selecting a genuine published rate from the wrong effective period. None of it produced an error.
Every rule we model is traced to an official published source or explicitly labelled as our own construction, and that provenance table ships with the environment.
Where it applies
The machinery is the same for any workflow where an agent changing the wrong record costs money. The interface, the checker, the adversarial test set, the isolation architecture and the provenance method all carry over. What changes is the domain content.
Current work includes manufacturing procurement, purchase orders matched to invoices, and customer service agents that update records after a call. If you are deploying agents on consequential workflows and want to know whether they finish the job before they touch live data, the published sampler is the fastest way to see how we score.
Have a workflow to evaluate?
Send the workflow, agent interface and the decision the evidence needs to support.
Email talal@maydanlabs.comMaydan Visuals / Research and evaluation
How well does visual AI represent Middle Eastern markets?
Explore regional research insights, an illustrative comparison, and concrete briefs for custom evaluations and commissioned datasets.
Explore Maydan Visuals →