Knowledge-backed agent experiment
Neil tried to take a design agent to school. The pilot could not show that the extra context helped.
Completed 15-session pilot. Curated design knowledge produced no measurable gain over a strong role prompt.
01 / Why it exists
Could an AI agent develop real design judgement, or was it only following a sophisticated prompt? The usual assumption is that extensive design knowledge and a disciplined process will produce better work. Neil tested that belief instead of turning it into a claim.
An earlier 48-session evaluation had shown structured knowledge improving clearly on no context at all. This pilot set a Swiss graphic design problem and asked a harder question: would a curated design vault add anything to a strong role prompt?
He ran 15 fresh sessions across five briefs under three conditions. A separate tool-free judge scored anonymous pairs without seeing which condition produced each written design direction.
02 / Evidence in the work
The pilot did not show a useful gain.
Screens and artefacts from the project itself. Each caption explains what it shows.
03 / The result
The result
The strong role prompt averaged 23.0 out of 24. Curated design principles averaged 23.2. Adding workflow, diagnostics, decisions and standards also averaged 23.2.
The automated blind assessment found no consistent improvement from curated knowledge and no incremental signal from the process layer. The baseline was already near the ceiling, leaving little room for the pilot to discriminate between conditions.
04 / What Neil learned
What Neil learned
This version of the vault did not show a measurable advantage over a disciplined role prompt for these tasks. The result was disappointing, and that was useful. It exposed the difference between producing plausible design directions and demonstrating real design expertise. Adding more documents would not answer that question.
The next test needs to evaluate actual rendered work with independent human reviewers. Until then, the vault is useful as a process and provenance resource, not a proven performance layer.
05 / Limits
Limits
This was five briefs, one response per condition and an evaluator from the same broad model family. It assessed written specifications rather than visual hierarchy, crop, spacing or final poster quality.
It is a useful null result for this near-ceiling pilot. It does not prove that knowledge-backed agents fail, or that task-specific private knowledge cannot help.
06 / What this demonstrates
The value is judgement: Neil designed a blind comparison, retained a null result and identified the stronger visual and human-evaluated test needed next.
Next step