Increasing the maximum token ceiling to twenty-four thousand proved necessary to prevent hidden thought processes from exhausting the budget before the final answer could be articulated. This discovery occurred during a large-scale evaluation of Large Language Models (LLMs) tasked with generating project risk registers based on real-world documentation in 2026. The research team utilized a dataset of 21 unique projects, including documents from the World Bank and the United Kingdom government, to test the descriptive capabilities of models such as Gemini, GPT-OSS-120B, and Llama-3.3-70B-Instruct. As the automated scoring system processed the results across 392 separate generations, a significant anomaly appeared in the data grid. While most configurations performed within expected parameters, one specific model variant using a structured prompting strategy returned a 100% failure rate across 42 consecutive runs. In a scientific context, a perfect zero or a perfect hundred often signals an issue with the measurement tool rather than the subject of the experiment itself. Consequently, the team paused the evaluation to look beyond the summary tables and into the raw output files. This deep dive revealed that the automated parser was rejecting valid content due to minor formatting discrepancies, such as missing project IDs that the system already possessed. By examining these failures manually, the investigators could distinguish between a model’s inability to reason and a software harness’s inability to interpret diverse response formats.
1. Investigating Extreme Results and Categorizing Failures
When a benchmark produces a result that indicates absolute failure, it is a standard professional practice in 2026 to treat that outcome as a bug until proven otherwise. A model that fails every single attempt out of dozens of trials usually suggests a mismatch between the model’s output format and the parser’s expectations. The research team moved away from the high-level scoring tables and began a granular review of the raw text generated by the LLMs. This process involved checking whether the model had actually failed the task or if the scoring script had simply failed to find the answer. In this specific case, the “failed to parse” error messages were the first clue. Many of the failures occurred because the parser was looking for a specific field that the model deemed redundant based on the context of the prompt. By shifting focus from the numerical scores to the raw data, the researchers identified that the model was often providing correct risk assessments that were being discarded by an overly rigid extraction script. This manual audit is a critical step in maintaining research integrity, as it prevents the publication of misleading performance data that could be cited by others in the field.
The next logical step was to categorize every unsuccessful run by its root cause to understand the specific patterns of failure. Out of the 88 total parse failures identified across the entire testing grid, the researchers broke them down into “format failures” and “content failures.” A format failure occurs when the model identifies the correct information but wraps it in a way the code does not recognize, such as listing risk objects without an enclosing array. Conversely, a content failure happens when the model provides incorrect data or violates the semantic requirements of the task. For the Llama-3.3-70B model, 30 of its 42 failures happened simply because it did not repeat a project identification number that was already present in the metadata. Only a small fraction of the errors were genuine formatting problems like invalid JSON structures. By isolating these causes, the team could see that the model was not “collapsing” but was instead falling victim to a brittle regex-based extraction method. This distinction allowed the team to plan a targeted fix that would improve the accuracy of the benchmark without artificially inflating the performance scores through manual intervention or leading prompt adjustments.
2. Establishing Remediation Rules and Refining Parser Logic
To ensure the integrity of the research, the investigators established a strict set of rules before any code was modified. One of the primary risks when fixing a scorer after seeing the results is the temptation to “over-fix” until the numbers look favorable. To prevent this bias, the team committed to never regenerating any content and never editing the raw outputs stored on disk. The goal was only to re-derive the results using a more sophisticated parser while keeping every content-related failure intact. If a model provided an incorrect category for a risk, it remained a failure regardless of how well it was formatted. These pre-established rules acted as a safeguard, ensuring that the updated results would still reflect the model’s actual performance rather than the researchers’ desire for a cleaner dataset. By documenting these rules, the team maintained a clear audit trail that demonstrated their commitment to objective evaluation. This transparency is vital in the 2026 research landscape, where the complexity of model interactions makes it easy to accidentally introduce human bias into automated scoring systems.
The refinement of the parsing logic involved moving away from simple regular expressions to more robust decoding methods. The updated parser utilized a recursive decoding approach that could collect all sequential JSON values and handle various structural variations, such as bare arrays or minor case-sensitivity issues in category labels. For example, if a model labeled a risk as “External” instead of the required “external,” the new logic would accept it as long as the content matched the predefined taxonomy. Additionally, the code was updated to automatically fill in metadata, such as project IDs, from the run’s own metadata instead of requiring the model to repeat it. This shift toward a more resilient parsing strategy ensured that the evaluation measured the model’s intelligence and reasoning rather than its ability to perfectly mimic a rigid and often arbitrary data structure.
3. Maintaining Criteria and Implementing Automated Safeguards
While the parsing logic was made more flexible, the core evaluation criteria remained strictly unchanged to preserve the validity of the experiment. A model’s response was still considered a failure if it categorized a risk outside of the eight-category taxonomy, such as using “legal” or “market” when those terms were not part of the allowed set. These types of errors were classified as content failures because they represented a failure of the model to follow specific task instructions rather than a mere formatting quirk. The investigators were careful to ensure that the new parser did not “paper over” these genuine errors. By maintaining a strict boundary between format flexibility and content accuracy, the research team ensured that the final recall scores were a true reflection of the models’ utility in a professional setting. This balance is essential for benchmarks designed to test real-world applications where following specific taxonomies and requirements is just as important as the underlying reasoning. The decision to keep these failures visible helped to highlight the specific areas where models still struggle with adherence to professional standards.
To prevent future regressions, the team implemented a series of automated safeguards and unit tests designed to lock the new rules in place. These tests included specific cases where invalid data was intentionally fed to the parser to ensure it was still correctly rejected. One such test, named to verify that invalid categories were never accepted, ensured that any future updates to the parsing code would not accidentally lower the bar for quality. These regression tests provided a programmatic way to enforce the research standards across different versions of the code. By treating the scoring harness with the same level of rigor as the models themselves, the researchers created a reproducible and stable environment for their work. This approach aligns with the best practices of 2026, where the reliability of the evaluation pipeline is considered just as important as the data itself. The implementation of these tests allowed the team to move forward with confidence, knowing that their improvements to the parser were both beneficial and safe from unintended side effects that could compromise the final research findings.
4. Addressing Performance Bottlenecks and Reporting Findings
During the troubleshooting process, a secondary and unrelated issue was discovered regarding how current models utilize their output token budget. In 2026, many high-performing models, such as those from Google and OpenAI, utilize internal “reasoning” or “thinking” tokens that are generated before the visible answer appears. These tokens often consume the maximum token limit set by the API, causing the response to be truncated before the actual JSON content is fully articulated. The researchers noticed that several models were returning incomplete answers with a finish reason of “length,” which was being misinterpreted as a failure of the model to generate a valid response. To resolve this, the team increased the maximum token ceiling to over twenty-four thousand for all calls. This adjustment provided ample space for the model’s internal thought processes without penalizing the final output. This finding underscored the importance of logging API metadata, such as completion token counts and finish reasons, to differentiate between a model that has reached its reasoning limit and one that has simply provided a poor answer.
The final phase of the project involved the full disclosure of the bug, the fix, and the resulting changes in the data. After re-running the improved parser on the saved outputs, the total number of parse failures across the grid dropped from 88 to 49. The Llama model variant that previously showed a zero percent success rate recovered to a recall score of approximately 0.43, which was consistent with its performance on other prompts. The 49 remaining failures were confirmed to be genuine errors, such as field-length violations or invalid categories. By documenting the entire correction process and making the updated code and raw outputs public, the researchers provided a model for transparency in the AI research community. The investigation concluded that identifying and correcting harness bugs is a necessary part of modern LLM evaluation, as it ensures that the published results are a true reflection of model capability. The research team determined that sharing these mistakes was more valuable than presenting a perfect, uncorrected dataset, as it provided actionable insights into the common pitfalls of automated benchmarking. The adjustments ensured that the final conclusions were supported by accurate data and a resilient evaluation methodology.