> ## Content Index > Fetch the complete content index at: https://piotrmechlinski.com/llms.txt > Use this file to discover other available public pages before exploring further. # AI's Jagged Frontier: Genius Here, Foolish Next Door - URL: https://piotrmechlinski.com/writing/ai-jagged-frontier/ - Published: 2026-04-18T22:00:00.000Z - Updated: 2026-09-08T07:53:17.000Z - Description: AI can excel at one task and fail on the next. Test the exact workflow, contain the miss and monitor the boundary in production. - Author: Piotr Mechlinski - Tags: Judgment One of Google's leading AI systems won a gold medal at the 2025 International Mathematical Olympiad. The best model on another benchmark read analogue clocks correctly 50.6% of the time. Humans scored 90.1%. Both results appear in the [Stanford AI Index 2026](https://hai.stanford.edu/ai-index/2026-ai-index-report/technical-performance?ref=piotrmechlinski.com). Gold-medal mathematics. A coin flip on the clock. This is the trap behind most executive AI demos. Exceptional performance on one task creates confidence that spreads into the next task. The capability does not spread with it. AI has a jagged frontier. Two tasks can look equally difficult to you and sit on opposite sides of what the system can do reliably. A benchmark shows one point on that frontier under one set of conditions. Your workflow lives somewhere else. ## The same tool can improve work and damage it The phrase "jagged technological frontier" comes from a field experiment with 758 management consultants. The researchers gave participants realistic knowledge-work tasks. For 18 tasks selected inside the AI capability frontier, people using GPT-4 completed 12.2% more tasks and worked 25.1% faster on average. Their output quality also improved. Then came a complex managerial task selected outside the frontier. Participants using AI were [19% less likely to produce the correct solution](https://www.hbs.edu/ris/Publication%20Files/dell-acqua-et-al-2026-navigating-the-jagged-technological-frontier%5F5c589c8c-fbb5-458f-b285-c944746cd717.pdf?ref=piotrmechlinski.com). Experienced professionals had access to a capable system. They did not have to follow its answer. Performance still deteriorated. That result belongs to one experiment, one organization and one generation of model. It does not give you a universal failure rate. It proves the management problem: adjacent tasks within the same workflow can respond differently to the same AI assistance, and professionals may not know which side of the boundary they occupy. The frontier also moves. A model update can pull yesterday's failure inside. A new input, language, customer segment or exception can push today's success outside. ## Your unit of testing is the workflow "How smart is the model?" is too broad to guide an investment. Ask a narrower question: which exact task, under which conditions, meets what threshold, and what happens when it fails? Imagine a workflow called "review a supplier contract." That label hides several jobs. Extract renewal dates. Compare a clause with your standard. Detect an unusual obligation. Judge commercial exposure. Recommend a negotiating position. One system may be excellent at the first task and unreliable at the fourth. A polished summary can make that difference harder to see. Testing the workflow as one block returns an average. Averages hide the edge that can hurt you. Map the boundary in five moves: 1. **Split the workflow into decisions.** Name the input, required output, user and consequence for each task. 2. **Define acceptable performance.** Use ground truth where it exists. Set separate thresholds for ordinary errors and high-consequence errors. 3. **Test real conditions.** Include expected volume, messy inputs, rare cases, different user groups and non-optimised prompts. Record the model and configuration. 4. **Contain the miss.** Route uncertain or high-consequence cases to a named person, safe fallback or stopped process. Give that person enough evidence and time to make a real judgment. 5. **Monitor the boundary.** Track failures, overrides and changes in inputs after deployment. Reopen the decision when the model, prompt, data or surrounding process changes. The [NIST AI Risk Management Framework playbook](https://airc.nist.gov/airmf-resources/playbook/measure/?ref=piotrmechlinski.com) makes the same operating distinction. Performance estimates from controlled tests do not fully represent the deployment context. NIST recommends measuring in conditions similar to expected use, documenting differences and monitoring against ground truth. That last step matters because pre-launch evaluation has a hard limit. A [2026 NIST report](https://www.nist.gov/publications/challenges-monitoring-deployed-ai-systems-center-ai-standards-and-innovation?ref=piotrmechlinski.com) says post-deployment monitoring is needed to verify real-world reliability, find unforeseen outputs and see unexpected consequences. It also says the methods are still developing. Monitoring is a management obligation, not a solved product feature. ## Human review needs a job description "Keep a human in the loop" sounds responsible. It often means one tired employee clicks approve after the system has framed the answer. A useful checkpoint names the reason work stops, the evidence the reviewer receives, the authority they hold and the time available. It measures how often people override the system and whether those overrides are correct. Some tasks deserve full automation. Some need sampling. Some need approval when confidence drops or consequences rise. Some should stay human until the evidence changes. The design follows the task. The press release does not decide it. ## A demo earns permission to test The gold medal is real. So is the clock. You do not need to choose between AI optimism and AI fear. Both are lazy when the capability boundary can be measured. Take the next workflow on your roadmap. Break it into decisions. Test the ordinary cases and the ugly ones. Set the failure threshold before the pilot produces an answer everyone wants to believe. A spectacular demo earns permission to test. Production trust comes later. Your move.