When a consultant reached out regarding a discussion with a client evaluating use of AI for mainframes, I did the usual diligence of performing a landscape check using an AI model. Not a single prompt, but a structured, multi-step process: web search for current sources, an explicit instruction to separate vendor claims from independently verifiable value, and a self-critique pass where the model checked its own output for gaps. That is roughly what a diligent person does with these tools today, short of building a dedicated research pipeline. The output that came back certainly looked the part: sourced, structured, appropriately hedged. Then I checked it against primary sources myself.
The gap between the first draft and the verified version is the actual subject of this piece. It is a small case study in why "AI produced a well-cited summary" and "AI produced a defensible summary" are not the same claim.
What the first pass got right
The core framework held up. Value in AI for mainframes is genuinely asymmetric: strong in operational assist and code comprehension, weak and unproven in full automated code translation.
Operations and onboarding
This is the most defensible category. The best-documented case, IBM's own CIO organization deployment of watsonx Assistant for Z, reported an 8% reduction in onboarding time for early-tenure staff, 10% faster incident resolution, and a 50% cut in Db2 patching time. Those numbers are real in the sense that they are published and specific. They are also from a single internal deployment of roughly 300 users, measured against IBM's own internal records, with IBM's own disclaimer that results elsewhere may vary. That is not a knock on the number, it is a note on what kind of evidence it is. "Best-documented vendor case" and "proven industry benchmark" are different epistemic categories, and the original summary did not distinguish them.
Code comprehension and dependency mapping
Understanding what decades-old, undocumented code actually does before touching it is where AI's pattern-matching strengths line up cleanly with the problem, and where the risk of a subtle error is lowest, since you are not yet writing production logic.
Full code translation
COBOL to Java is where the hedging in the first draft was earned. AI can generate plausible-looking translated code fast. Verifying it against mission-critical transaction logic is where the promised savings tend to evaporate, sometimes described as the "missing junior loop": AI removes the entry-level grunt work, but senior engineers now spend their time auditing AI output instead of writing code, which is not obviously less expensive. The junior loop that used to build institutional knowledge, novices learning a codebase by writing it, breaking it, and fixing it under supervision, does not get automated away so much as replaced by a different, more expensive form of labor: senior people doing entry-level verification work at senior rates.
What it missed
Here is the part that should give anyone pause about treating a rigorous AI research cycle as a substitute for due diligence: the research entirely missed a five-month-old event that had already reshaped the exact landscape it claimed to be summarizing.
In February 2026, Anthropic announced that Claude Code could automate the dependency-mapping and documentation phase of COBOL modernization, the "understand what this code does" work that traditionally required teams of consultants and months of billable time. The market read it as a direct threat to a large, historically high-margin part of IBM's consulting business, and IBM's stock fell roughly 13% in a single day, its worst drop since October 2000. Accenture and Cognizant moved on the same news. No single-day move in a diversified, dividend-anchored stock has one clean cause, but multiple outlets named the announcement as that day's trigger, and the size of the reaction is itself a data point about how the market is pricing this category.
IBM's public response is worth repeating almost verbatim, because it is the sharpest available statement of where the real difficulty sits. Rob Thomas, IBM's Senior Vice President of Software and Chief Commercial Officer, published a rebuttal the same day that did not name Anthropic but was clearly aimed at the announcement:
That is IBM conceding the discovery layer to AI while defending execution as the actual moat, a more precise version of the "verification bottleneck" the original research gestured at but did not actually locate.
This is the real tension in the story, and it is worth sitting with rather than resolving: the market moved on perceived threat, a 13% single-day repricing, while IBM's rebuttal points at technical reality, that dependency-mapping was never the hard part of modernization and execution and transaction integrity are. The AI research reported both in the same flat, confident tone, with no signal that one is sentiment and the other is substance. That gap is the actual failure mode this piece is about, and it survives even a rigorous, multi-step prompting process, which is the uncomfortable part.
There is a nuance buried in the coverage worth surfacing too, and this is where the first draft got sloppy. Thomas's rebuttal claims nearly 40% of COBOL does not run on mainframes at all. That figure does not appear to be IBM's own internal number. It traces most plausibly to a recurring COBOL survey series Micro Focus, now OpenText, has commissioned from Vanson Bourne since 2020, one round of which found 43% of COBOL running on z/OS and 31% on Windows, meaning most COBOL by that survey's count runs somewhere other than the mainframe. Forty-three percent on the mainframe lines up comfortably with "nearly 40%" not on it, even with no citation attached to Thomas's rebuttal.
What matters more than whether the number is right is whose number it is. IBM's SVP is citing a competing vendor's commissioned research, mid-rebuttal to a stock selloff, in a category where that vendor has its own commercial interest in COBOL looking under-served by the mainframe. That is a more tangled provenance than either "IBM's own data" or "an independent audit," and exactly the kind of nuance a flat, confident AI summary collapses into a single unqualified statistic. What is being reported as "AI disrupts the mainframe" may really be "AI disrupts COBOL," a broader claim with its own chain of vendor interests worth tracing before it gets repeated as settled fact.
What it could not tell apart
The bigger issue was not a missing fact. It is a known failure mode, that a language model reports claims of different evidentiary weight in the same confident register regardless of source, showing up in a concrete, checkable place: this research, on this topic, that day.
The research surfaced three genuinely different evidentiary weights and flattened all three: an industry survey run by a company that sells mainframe modernization services, claiming up to 225% first-year ROI and roughly $12B in collective annual client savings; IBM's internally measured 8% onboarding improvement, from a single 300-person deployment; and an "80% code accuracy" claim from a migration-tool vendor with no stated baseline or independent audit. Self-reported industry survey, single-company internal case study, unaudited vendor benchmark: same confident tone, three different amounts of trust each one has actually earned.
Days after the original research, Gartner published a genuinely independent data point on this topic: a prediction that more than 70% of mainframe exit projects initiated in 2026 will fail to produce their intended benefits, specifically due to overestimating generative AI's capabilities, and that by 2030, 75% of vendors in the "mainframe exit" category will pivot or disappear. Gartner's reasoning is structural, not sentimental: the volume and interconnection of decades of mainframe-resident data makes wholesale migration "physically and financially impossible" for most large enterprises, and investor pressure is pushing vendors to embed AI into offerings regardless of whether it meaningfully improves outcomes.
That is a genuinely useful data point, the kind of independent counterweight that is rare in this space. It was not in the first draft, because the first draft was built before it existed and never got refreshed against a later source. But "independent" and "final word" are not the same thing either, which is worth addressing directly before this piece makes the same mistake it is critiquing.
What no report captures
Landscape checks are good at framing and at surfacing a stat that reorients a conversation. They are not a substitute for having actually run one of these programs, and it shows fast the moment you push past the summary into execution. Mainframe environments are not just COBOL: it is JCL, CICS, IMS, VSAM, batch scheduling dependencies, and decades of undocumented coupling between hardware and software that took thirty years to accumulate one exception at a time. A report that only talks about "code translation" is quietly assuming the easy fraction of the problem represents the whole thing.
That gap is also where vendor hype does its damage. Glossy case studies and confident migration timelines deserve the same scrutiny as an AI-generated summary, not less, and the only real defense is going past the reference slide: talk to the actual client named in the case study, ask what the timeline looked like versus what was promised, and find out what got quietly descoped along the way before any budget gets committed. No report, human or AI-assisted, replaces that legwork.
The actual takeaway
None of this makes AI research untrustworthy. It makes a plausible, well-cited AI summary a first draft, not a deliverable, and the gap between the two is not about whether the AI hallucinated. It did not. Every claim in the original research was traceable to a real source. The failure mode was subtler: treating claims of different evidentiary strength identically, and missing that the landscape had already shifted since whatever data the model was drawing on was current.
For anyone using AI to scope a client conversation, a board deck, or a go or no-go decision on mainframe strategy: the fluent, well-hedged paragraph is not the hard part anymore, even when it came from a genuinely rigorous prompting process. Knowing which sentences in it would survive being checked against a primary source is.
If the client conversation behind this piece goes somewhere, the operative fork is not "AIOps versus migration," it is narrower and more consequential: is the goal to exit the mainframe, or to keep it and make it more AI-legible? Gartner's segment-by-segment framing, modernize in place for large or complex environments, case-by-case exits only for medium ones, mainframe-as-a-service for small ones, is a reasonable starting frame, and a rigorous AI research cycle taken at face value would have missed it entirely. But it is a starting frame, not an answer. The actual answer, in a mainframe environment specifically, is more likely to come from someone who has spent enough time in one to know which parts are genuinely load-bearing, which vendor timelines are fiction, and which battles are worth picking first. That is not a case for secrecy or gatekeeping, it is an acknowledgment that landscape checks, whoever or whatever produces them, are starting points, not substitutes for deep engagement. That is true whether the summary in front of you came from an AI, an analyst firm, or a vendor deck. Fluency and rigor are not the same thing, and neither one substitutes for having actually been there.