Evaluation matrix and tool design, and the matrix that answers the wrong questions.
An evaluation matrix is the operational backbone of an evaluation. It defines the questions being asked, the sub-questions that operationalise them, the data sources that will answer them, the methods of collection, and the analytical approaches that will be applied. From the matrix flows the instrument suite. The KII guides. The survey schedules. The FGD guides. The observation protocols. Every question asked in the field should route back to a specific sub-question in the matrix. Every sub-question in the matrix should be addressed by at least one instrument.
Where a generated matrix quietly goes wrong.
The most common failure when AI is used to generate an evaluation matrix is standard-framing bias. The model defaults to the OECD-DAC criteria, relevance, coherence, effectiveness, efficiency, impact, sustainability, even when the ToR is asking a different set of questions. The result is a matrix that looks methodologically clean but does not actually address the evaluation's real terms of reference. The evaluation then proceeds to produce evidence on the wrong questions.
The second failure is sub-question sprawl. The model generates fifteen sub-questions for every high-level evaluation question, producing a matrix that is exhaustive on paper and unworkable in practice. Sample sizes get stretched thin across too many sub-questions. Data collection becomes unfocused. The evaluator ends up quietly ignoring half the matrix and the client discovers this only when the report cannot substantiate claims on the ignored sub-questions.
Four structural controls, each answering a specific failure mode.
ToR-anchored generation answers standard-framing bias. The AI is required to derive the question structure from the ToR and the programme documents, not from the model's training on standard evaluation frameworks. Where the ToR asks for something that does not map cleanly to OECD-DAC criteria, the matrix reflects that rather than smoothing it into a standard shape. Every high-level evaluation question in the matrix is required to cite the specific ToR paragraph it derives from. Standard framings are used where the ToR calls for them and rejected where it does not.
Bounded generation answers sub-question sprawl. The workflow sets defined sub-question limits per high-level question, tuned to the evaluation's budget, timeline, and sample sizes. The matrix is built for the evaluation the client is commissioning, not for a hypothetical unlimited-resource version. Where the sub-question count is constrained, the workflow requires the evaluator to make explicit choices about what is prioritised and what is left out, with the reasoning recorded rather than hidden.
An evaluation matrix, an instrument suite, a traceability report, an assumptions register, a disclosure.
Every Meridian matrix and instrument design assignment produces five artefacts. The evaluation matrix itself, in the client's preferred format, with every high-level question cited to the ToR paragraph it derives from and every sub-question mapped to data sources, methods, and analysis types. The instrument suite, KII guides, survey schedules, FGD guides, observation protocols as required, with every question tagged to the specific sub-question it addresses and the specific respondent group it is intended for. The traceability report, showing the mapping in both directions and confirming that every sub-question has instrument coverage and every instrument question has a matrix anchor. The assumptions register, distinguishing assumptions the evaluation intends to test from those it is accepting as given. And the disclosure text, a short methodology annex drafted for inclusion in the inception report.
The five artefacts together mean the matrix and instrument suite are inspectable. When a client asks whether the ToR is being addressed, the answer is a specific mapping. When a client asks whether the instrument suite is coherent, the answer is a bidirectional traceability check. When a client asks what assumptions the evaluation is inheriting from the programme documents, the answer is the assumptions register.
The differentiation is the discipline.
Evaluation matrices and instruments sit upstream of every subsequent stage. A matrix defaulted to standard framings sends data collection after the wrong questions, and no downstream discipline can recover from that. ToR-anchoring and bidirectional traceability exist because the cost of getting this stage wrong is paid in the cost of the whole evaluation.