{
  "title": "RAG Evaluation Plan",
  "description": "Builds an evaluation plan for a retrieval-augmented generation system, covering retrieval quality and answer faithfulness separately.",
  "category": "RAG and Retrieval",
  "tags": [
    "evaluation",
    "rag",
    "retrieval"
  ],
  "templateContent": "You are a RAG evaluation specialist. The system below retrieves from: {{corpus_description}}\nand answers questions about: {{use_case}}\n\nDesign an evaluation plan that separately measures:\n1. Retrieval quality (e.g. recall@k, precision@k, whether the gold passage is retrieved at all).\n2. Answer faithfulness to retrieved context (does the answer only use what was retrieved?).\n3. Answer relevance to the user's question.\n\nFor each metric, state how to compute it and what data it needs (labeled gold passages, reference answers, etc.).\nThen produce {{question_count}} example evaluation questions spanning easy lookups, multi-passage synthesis, and questions with no good answer in the corpus (should trigger an \"I don't know\").",
  "variables": [
    {
      "name": "corpus_description",
      "label": "What the corpus contains",
      "type": "multiline",
      "required": true,
      "default": null,
      "allowedValues": null
    },
    {
      "name": "use_case",
      "label": "What questions it answers",
      "type": "text",
      "required": true,
      "default": null,
      "allowedValues": null
    },
    {
      "name": "question_count",
      "label": "Number of example questions",
      "type": "number",
      "required": true,
      "default": 9,
      "allowedValues": null
    }
  ],
  "usageInstructions": "Describe your corpus and use case, then use the plan to build a small labeled gold set before trusting any retrieval metric.",
  "limitations": "Faithfulness and relevance scoring here assume a human or LLM-judge grader; the plan does not itself implement automated scoring code."
}