Expand file tree
team-solve
Directory actions
More options
Directory actions
More options
Latest commit
History
team-solve
Folders and files
parent directory
README.md
team-solve
Five small models on one programming problem: a planner, two ideators, a solver, a manager and a repair worker, each from a fresh context, talking only through files on disk; a script that runs the samples decides success. The method of GVS5H (Lee, 2026) as a Memdoor workflow: eight YAML files, no code, no model named.
The result
The authors' harness on the same model, Qwen3.8-27B, with their prompts, loop and 128K reply cap. The first ten problems of their LiveCodeBench Hard list, five passes each, graded on every hidden test with their corrected grader:
The authors' columns: the regraded result files in their repository, the numbers in their paper. Ours: bench/results/parity-5pass/ , all 50 runs, each with every request and reply, the program, the grader's output and the cost. A median of 40 minutes and $0.39 a run; $35.79 for the 50.
Sample: ten problems, five passes, against the authors' 100. 9.8 and 9.6 are within noise. The claim is parity with their harness.
Contamination: the contests date from spring 2025; the model may have seen them. Not ruled out for any column.
Fable 5: one call per problem, from the authors' files, not run in this harness.
On these ten, the authors' own five-pass data: Fable 5 at 80, their harness at 96. The workflow's 98 reproduces that gap. Across their full 100: 92.4 and 90.4 (their patched_verdicts.json ).
The loop sees the public samples; the score is the hidden tests.
Host: OpenRouter, not a local machine.
The lost problem: the manager invented a wrong counterexample and the team rewrote a correct program. Twice elsewhere a correct program was overwritten and then recovered. Both are gaps in the authors' loop, kept as is.
Parity check (every request byte for byte against their code), hosts, the changes it took, the first run's findings: NOTES.md . The result on one page: memdoor.ai/docs/team-solve .
Run it
curl -fsSL https://memdoor.ai/install.sh | bash memdoor connect openrouter # or anthropic, openai, gemini, chatgpt # or no key: memdoor connect local (Ollama, llama.cpp, vLLM, LM Studio; Ollama needs a 32K window, not its 4K default) git clone --depth 1 https://github.com/guregodevo/memdoor mkdir -p ~/.memdoor/workflows cp -r memdoor/examples/workflows/team-solve ~/.memdoor/workflows/ mkdir my-problem && cd my-problem cp -r ../memdoor/examples/workflows/team-solve/sample/* . # task.md + tests/1.in … 5.out memdoor workflow run team-solve --wait # exit 0 when every sample passes
Your problem: task.md and paired tests/N.in / tests/N.out . Program: solution.py , stdin in, compared to the samples with diff . Needs Python 3 and bash. /workflow in memdoor tui draws the run live.
The steps name no model; the /model pin answers, a local one included ( how to connect one ). A role is its prompt in steps/ .
How it works
plan ──┬── ideate-a ──┬── solve ── test ── review ── repair ── verify └── ideate-b ──┘
Every agent prompt: "fresh context, read the files, do not rely on chat"; the paper credits that ledger discipline for the gain. One YAML file per step in steps/ , in the format of mario , the engine underneath. requires: is the DAG.