Testing, Validation, and Troubleshooting

253. Two candidate model configurations must be compared on quality and cost before one is adopted. Which solution meets these requirements?

1. A Compare the two configurations' published benchmark scores.
2. B Deploy both configurations and compare user complaint volumes.
3. C Compare the configurations' parameter counts and context window sizes.
4. D Run an Amazon Bedrock model evaluation over a representative dataset and compare quality alongside measured token cost and latency.

254. A quality gate must prevent a GenAI deployment when output quality falls below the current production standard. Which solution meets these requirements?

1. A Publish the quality standard in the team's release checklist.
2. B Deploy the change and compare quality metrics over the following week.
3. C Run a regression evaluation against a curated set in the pipeline and fail the deployment when the score falls below the production baseline.
4. D Review a sample of outputs manually before each release.

255. An agent's performance must be measured on whether it completes assigned tasks correctly rather than on individual response quality. Which solution meets these requirements?

1. A Measure task completion rate and tool usage effectiveness across a set of representative tasks.
2. B Measure the average number of tokens the agent consumes per task.
3. C Measure the fluency of the agent's individual responses.
4. D Measure the agent's response latency per turn.

256. Retrieval quality must be measured separately from generation quality in a RAG application. Which approach is appropriate?

1. A Measure retrieval with relevance scoring against labelled query and passage pairs, and measure generation separately given correct context.
2. B Measure the number of passages retrieved for each query.
3. C Measure the retrieval latency and the generation latency separately.
4. D Measure end-to-end answer accuracy and attribute failures proportionally between the stages.

257. User feedback must be collected continuously and fed back into evaluation of a GenAI application. Which solution meets these requirements?

1. A Provide a rating interface on each response, store the ratings with the interaction identifier, and use them to calibrate automated evaluation.
2. B Infer satisfaction from whether the user submits a follow-up question.
3. C Ask users to email the team when a response is unsatisfactory.
4. D Collect feedback through a quarterly user survey.

258. An application intermittently fails with an error indicating the input exceeded the model's context window. Which solution meets these requirements?

1. A Increase the maximum output tokens so the model has more room to respond.
2. B Reduce the model's temperature so it uses the context more efficiently.
3. C Retry the failed request with the same input.
4. D Measure the token count before invocation and trim or summarize the least relevant context so the request fits within the window.

259. A RAG application returns an irrelevant answer, and the team must determine whether retrieval or generation failed. Which step should be taken first?

1. A Increase the size of the generation model and retest the query.
2. B Inspect the passages that were retrieved for the query to establish whether the answer was even available in the context.
3. C Re-embed the entire corpus and retest the query.
4. D Lower the generation temperature and retest the query.

260. Responses have become inconsistent after a prompt template change, and the team must establish which revision caused it. Which solution meets these requirements?

1. A Increase the temperature so the inconsistency is less noticeable.
2. B Revert to the original prompt and accept the loss of the intended improvement.
3. C Compare the outputs of each prompt version against the same test inputs to identify the revision where behaviour changed.
4. D Rewrite the prompt from scratch using the current requirements.

261. Retrieval quality degraded after the embedding model was upgraded, although the corpus is unchanged. Which cause is most likely?

1. A The similarity threshold was reset to its default value.
2. B The corpus documents were modified during the upgrade.
3. C The vector store's storage capacity was exceeded during the upgrade.
4. D Part of the index still holds vectors produced by the previous model, and vectors from different models are not comparable.

262. An evaluation framework must assess factual accuracy, relevance, and fluency separately rather than as one score. Which rationale applies?

1. A A response can be fluent and relevant while being factually wrong, so a single score conceals which dimension failed.
2. B A combined score cannot be calculated for generated text.
3. C Separate scores are required by evaluation tooling.
4. D Separate scores are easier to compute than a combined score.

263. A canary evaluation must validate a model update against real traffic before full rollout. Which design is appropriate?

1. A Route a small share of traffic to the new version and compare quality and business metrics against the current version over the same period.
2. B Route all traffic to the new version during a low-traffic period.
3. C Route a small share of traffic to the new version and confirm it returns responses without error.
4. D Compare the new version's offline evaluation scores against the current version's.

264. A user feedback mechanism must produce data useful for improving a GenAI application. Which design is appropriate?

1. A Capture a rating without linking it to the interaction.
2. B Capture free-text feedback through a separate support channel.
3. C Capture aggregate satisfaction through a monthly survey.
4. D Capture the rating together with the interaction identifier so the feedback can be traced to the exact prompt and response.

265. A quality gate must prevent a deployment whose hallucination rate exceeds an agreed level. Which approach is appropriate?

1. A Record the rate in the release notes for the reviewer.
2. B Measure the rate on a labelled evaluation set in the pipeline and block promotion when it exceeds the threshold.
3. C Review a sample of outputs manually before each release.
4. D Measure the rate in production and roll back if it exceeds the threshold.

266. An agent evaluation must establish whether multi-step reasoning is sound rather than only whether the final answer is correct. Which approach is appropriate?

1. A Measure the total time taken per task.
2. B Examine the reasoning trace and tool calls for each task, since a correct answer can follow from faulty reasoning.
3. C Measure the proportion of tasks with a correct final answer.
4. D Measure the average number of steps per task.

267. A stakeholder report must communicate GenAI evaluation results to a non-technical audience. Which content is most appropriate?

1. A The models and configurations that were compared.
2. B What the system does well, where it fails, and what those failures mean for the business.
3. C The full metric tables for every evaluation dimension.
4. D The evaluation dataset's size and composition.

268. An application intermittently returns truncated answers, and the token usage shows responses hitting the configured limit. Which action is appropriate?

1. A Raise the output token limit for the affected request class, or instruct the model to produce a more concise answer.
2. B Increase the number of retrieved passages so the answer is better informed.
3. C Increase the request timeout so generation has more time.
4. D Raise the model's temperature so responses vary in length.

269. A model integration fails intermittently with errors that name a validation problem in the request. Which step should be taken first?

1. A Retry the failing calls with a longer backoff.
2. B Switch to a different foundation model.
3. C Increase the client's timeout.
4. D Log the exact request payload for the failing calls and compare it against the API's expected schema.

270. Retrieval quality has degraded, and the team must determine whether the cause is the embeddings or the chunking. Which approach is appropriate?

1. A Hold one variable constant and re-evaluate, testing the current chunks with a different embedding model and the current embeddings with different chunking.
2. B Increase the number of candidates retrieved and re-evaluate.
3. C Change both the chunking and the embedding model and re-evaluate.
4. D Re-embed the corpus with the same model and re-evaluate.

271. A prompt performs well in testing but poorly in production, and the inputs differ in ways the test set did not represent. Which action is appropriate?

1. A Extend the test set with examples drawn from production traffic so it represents what the application actually receives.
2. B Increase the size of the existing test set with more examples of the same kind.
3. C Revert the prompt to its previous version.
4. D Raise the temperature so the model handles varied input more flexibly.

272. An evaluation must measure relevance, factual accuracy, consistency, and fluency of GenAI outputs. Which approach is appropriate?

1. A Define a metric for each dimension, score a representative set, and report each dimension separately.
2. B Measure only fluency.
3. C Use a single overall satisfaction score.
4. D Measure only factual accuracy.

273. A multi-model evaluation must compare cost-performance across candidates. Which metric is appropriate?

1. A Quality score per unit cost, computed from measured quality and measured token spend on the same evaluation set.
2. B Model parameter count.
3. C Raw quality score alone.
4. D List price per token alone.

274. An annotation workflow must produce reliable quality labels for GenAI outputs. Which design is appropriate?

1. A Annotators using their own judgement without a rubric.
2. B Automated labelling only.
3. C Use multiple annotators with a shared rubric, agreement measurement, and adjudication of disagreements.
4. D Assign one annotator per output.

275. A continuous evaluation workflow must run automatically as the application changes. Which trigger is appropriate?

1. A Only on a monthly schedule.
2. B Any change to the prompt, model, retrieval configuration, or guardrail, since each can change behaviour.
3. C Only when a user reports a problem.
4. D Only model version changes.

276. A RAG evaluation must measure whether the retrieved context was actually used in the answer. Which metric applies?

1. A Context utilization or faithfulness, measuring how much of the answer is grounded in the retrieved passages.
2. B Retrieval latency.
3. C Answer length.
4. D Number of passages retrieved.

277. A retrieval quality test must measure whether relevant passages are ranked highly, not just retrieved. Which metric applies?

1. A Retrieval latency.
2. B Number of documents in the index.
3. C Mean reciprocal rank or precision at k, which account for the position of relevant results.
4. D Total recall regardless of position.

278. Deployment validation must detect semantic drift in a GenAI application after an update. Which approach is appropriate?

1. A Run synthetic user workflows against the updated deployment and compare outputs to the pre-update baseline for meaning change.
2. B Check the error rate.
3. C Check that the deployment completed.
4. D Check that the endpoint responds.

279. An evaluation report for stakeholders must support a decision between two model configurations. Which presentation is appropriate?

1. A A side-by-side comparison on the same dimensions with cost, highlighting where each is stronger and the recommended choice.
2. B Raw metric tables for each configuration.
3. C The full evaluation dataset.
4. D The engineering team's preference.

280. A GenAI application returns truncated responses only for certain inputs. Which cause should be investigated first?

1. A Whether the network is slow.
2. B Whether the API key has expired.
3. C Whether those inputs produce prompts near the context limit, leaving insufficient room for the response.
4. D Whether the model is overloaded.

281. A prompt template produces inconsistent output formats despite instructions. Which troubleshooting step is appropriate?

1. A Increase the temperature.
2. B Test the template systematically across input variations to identify which inputs break the format, then constrain the output with a schema.
3. C Switch to a different model without testing.
4. D Remove the format instructions.

282. Embedding quality is suspected of degrading retrieval. Which diagnostic is appropriate?

1. A Rebuild the index with the same model.
2. B Inspect the raw embedding vectors visually.
3. C Increase the number of retrieved results.
4. D Measure retrieval recall on a labelled query set with the current embeddings, and compare against a re-embedded sample with an alternative model.

283. A prompt maintenance process must detect when a model update changes how the prompt is interpreted. Which approach is appropriate?

1. A Monitor complaints.
2. B Rewrite the prompt after each model update.
3. C Assume the prompt continues to work.
4. D Version the prompt with its expected behaviours and re-run the behaviour tests after each model update.

284. A retrieval issue must be traced to whether the vectorization step or the chunking step is responsible. Which approach is appropriate?

1. A Re-embed everything and see.
2. B Increase the chunk size blindly.
3. C Replace the vector store.
4. D Inspect the chunks for the affected documents to see whether the relevant content was split poorly, then check whether the correct chunk embeds near the query.

285. An evaluation dataset must represent the application's real usage. Which approach is appropriate?

1. A Use a public benchmark dataset.
2. B Use examples written by the development team.
3. C Use the examples from the prompt.
4. D Build it from sampled production traffic covering the range of request types and difficulty.

286. An automated evaluator's scores must be trusted before being relied upon. Which approach is appropriate?

1. A Assume agreement because the evaluator uses a capable model.
2. B Calibrate it against human ratings on a sample and measure the agreement.
3. C Compare its scores against the application's error rate.
4. D Use it only for high-scoring responses.

287. An evaluation must detect that a change improved one request class while degrading another. Which approach is appropriate?

1. A Report the total number of requests evaluated.
2. B Report the aggregate score before and after.
3. C Report the change's effect on the largest class.
4. D Report results per request class rather than as a single aggregate.

288. An evaluation must establish whether a retrieval change improved the final answer. Which approach is appropriate?

1. A Measure the end-to-end latency.
2. B Measure both retrieval metrics and end-to-end answer quality, since better retrieval does not always improve the answer.
3. C Measure the answer quality only.
4. D Measure retrieval recall only.

289. An evaluation harness must produce comparable results across runs. Which approach is appropriate?

1. A Adjust the metrics as the application evolves.
2. B Sample a new dataset for each run.
3. C Fix the dataset, the metrics, and the model settings so only the variable under test changes.
4. D Run the evaluation on whatever traffic is current.

290. A human review process must scale to a large volume of generated output. Which approach is appropriate?

1. A Review a fixed random sample.
2. B Review only outputs users complained about.
3. C Sample for review using a risk-weighted selection so consequential outputs are reviewed more heavily.
4. D Review every output.

291. An evaluation must assess whether an agent completed a multi-step task correctly. Which approach is appropriate?

1. A Define the expected end state and intermediate checkpoints, and verify both.
2. B Count the number of steps the agent took.
3. C Measure the time the agent took.
4. D Check whether the agent returned a response.

292. Evaluation results must be communicated so a product owner can decide whether to ship. Which approach is appropriate?

1. A Present the full metric tables.
2. B Present the quality change per request class, the cost implication, and the residual risks in business terms.
3. C Present the evaluation methodology.
4. D Present the engineering team's recommendation alone.

293. An agent intermittently fails to complete tasks it previously handled correctly. Which approach is appropriate?

1. A Switch the agent to a larger model.
2. B Restart the agent's execution environment.
3. C Compare the current reasoning traces against those from successful runs to identify where the behaviour diverges.
4. D Increase the agent's step limit.

294. A GenAI application returns correct answers in testing but poor answers for a subset of production users. Which approach is appropriate?

1. A Increase the number of retrieved passages.
2. B Compare the characteristics of the failing requests against the test set to identify what the test set did not represent.
3. C Retrain the model on more data.
4. D Increase the model's temperature.