The re-grade was a verdict, not a record

An AI agent re-graded 9,967 HIGH-severity cloud findings and changed 75.1% of them — and the headline that will outrun the study, that AI shrinks your security backlog, is the one the data does not support. 58.5% went down. 16.6% went up. The same research group's published benchmark puts an agent's verdict at 94% and the answer underneath it at 78%, which tells you precisely what you have bought: an ordering for your queue, built on a record you may not be able to reconstruct when someone asks why a HIGH was quietly made a LOW.

At a glance

Two pieces of work from the same team, published nine months apart, and they are only useful read together. The first is a peer-reviewed-style benchmark on arXiv; the second is vendor research reaching the trade press today, with no primary paper I can find. That asymmetry is worth holding on to.

What was measured	Scope	Result	What it licenses
Re-grading HIGH cloud findings	9,967 HIGH findings, two commercial platforms, eight production environments	75.1% changed grade — 58.5% down, 16.6% up	Re-ordering a queue
Why findings moved down	Largest single pattern across the downgrades	32.3% were inert — attached to nothing, or orphaned	Deleting resources, not dismissing rules
Cross-vendor identity reasoning	50 data-grounded tasks over eight integrated platforms	78% answer correctness at full context; complete failures down to 4%	Calibrating how much to trust the above
Verdict vs. the answer behind it	Same benchmark, strongest configuration	94% verdicts against 78% answer correctness	Nothing — it is the warning label

Three quarters of the findings moved. The bar nobody will quote is the third one.

The downgrades are the story being told; the upgrades are the story

Read the 58.5% as a product claim and it says: buy this and your HIGH queue halves. Read the 16.6% and the claim collapses, because a tool that only ever moved findings down would be indistinguishable from a tool that had learned to say "probably fine" — and that is the single most likely failure mode for a language model pointed at a backlog someone wants shortened. The upgrades are what make the result an argument about measurement rather than an argument about volume.

They also change who should care. If the agent only sorted, this would be a productivity story for a security team. Because it raises as well as lowers, the honest reading is that the severity field in your scanner was never a measurement of your risk — it was a property of the rule that fired. Sola reports that each vendor rule it examined issued a single severity across every failing instance, on the order of hundreds of rules covering tens of thousands of findings. One number per rule, applied to every environment that rule has ever matched. That is a constant with a severity-shaped name.

Which is not a scandal; it is the only thing a rule evaluated against one resource can do. Severity in the CVSS lineage is a property of a vulnerability class, and the step from "this class is severe" to "this instance is severe here" requires facts the rule never had: whether the thing is reachable, whether anything is attached to it, whether a control sits in front of it. The agent's contribution is not better judgement. It is the join.

The asymmetric evidence rule is the best part of the design

The join is the product. The asymmetry in the middle box is what keeps it from becoming a dismissal machine.

The reported evidence rules are worth copying whatever you think of the headline. To lower a grade, the agent had to observe a property that offsets the exploit path — the resource is unreachable from the internet, no live service answers, the source range is narrow, an account-level control blocks it. To raise one, it needed evidence that the condition is worse in this environment and a check that no guardrail such as a firewall or allowlist sits in front of it. Anything else keeps the vendor grade.

That asymmetry encodes the thing most triage automation gets backwards. The default is not "decide"; the default is "defer to the existing rating", and both directions have to earn their move. It is also why the 24.9% that kept their grade is a sign of health rather than of weakness — a re-grader that moved everything would be a re-grader with no threshold.

Note what the dominant downgrade reason actually is. 32.3% of cases involved inert resources: things attached to nothing, or orphaned. That finding is not "this HIGH was wrong". It is "you are paying a scanner to generate HIGHs about infrastructure you forgot to delete", and the correct response is a deletion sprint, not a severity change. The failure taxonomy hiding inside a third of your backlog turns out to be an asset-lifecycle problem wearing a security label.

94% verdicts, 78% answers: the gap is the whole caveat

Column one is a label with three possible values. Column two is what an auditor reads.

The same group's Cross-Vendor Sola ISPM Benchmark — 50 data-grounded tasks requiring multi-hop entity resolution across eight integrated platforms including AWS, Okta, Azure AD and Google Workspace — reports 78% answer correctness for its strongest full-context configuration, with complete failures down to 4%. The vendor's own write-up of that work cites 94% verdicts against that 78%. Hold those two numbers next to each other and you get the most useful sentence in either study: the agent is substantially better at reaching the right conclusion than at holding the right reasons for it.

That is not a quirk of this product. It is the generator–verifier gap showing up in the direction people forget — a verdict is one token from a three-way choice, and a chance-corrected three-way choice starts at 33%, so a high verdict rate is cheap to reach and expensive to interpret. The evidence chain is the expensive part, and it is the part that degrades.

The benchmark also tells you why. Adding structured relational context improved answer correctness by roughly 34% in relative terms and cut exploration queries by about 70%, with the largest gains attributed to cross-vendor graph topology. In other words the agent's accuracy is mostly a function of whether the relationships between your accounts, identities and resources were handed to it — which is an argument for a graph over your estate and against expecting a model to rediscover it by poking at APIs. The same finding is the one that makes graph-shaped retrieval worth its complexity in this domain specifically.

So what may you actually do with a re-grade?

The distinction that matters is between ordering work and closing work, and it maps onto the two numbers above almost exactly. A verdict you trust at 94% is good enough to decide what a human looks at first. A record you trust at 78% is not good enough to be the reason a finding stopped existing.

Use	Needs	Verdict is enough?	Why
Ordering today's queue	A relative ranking	Yes	A wrong order costs minutes; the finding is still there tomorrow
Suppressing a class of alert	A rule you can state	No	Suppression is permanent and silent; it needs the reason, not the label
Closing a finding	An auditable record	No	This is the 78% column, and the one an assessor reads back to you
Raising a severity	A reproducible exposure check	Yes, with the evidence	Over-escalation is self-correcting; the 16.6% is the cheap direction to be wrong in

Four things follow for anyone wiring this into a real pipeline.

Store the evidence as a first-class artefact, not as a rationale string. If roughly half the verdicts rest on evidence gathered beyond the flagged resource, then the resource-scoped record your scanner writes cannot hold the reason. Persist the specific observations — which API, which account, which timestamp — joined to the finding ID, which is what turns this into an audit trail about a decision rather than a log of one. A downgrade whose justification was a paragraph of prose is a downgrade you will re-litigate from scratch.

Re-grade, never suppress. The failure you are protecting against is not a wrong number; it is a wrong number that stops the finding from ever being looked at again. Keep the vendor severity in its own field, write the agent's grade in a second field, and sort on the second — so that reverting the whole experiment is a query change rather than a recovery.

Treat evidence staleness as part of the verdict. Every downgrade reason in the list is a mutable property. "Not reachable from the internet" is true until someone attaches a load balancer, and the finding's grade does not know that happened. A re-grade needs an expiry, and the environments where this matters most are exactly the ones changing fastest.

Measure your own agreement before you adopt anyone's numbers. Pull 200 findings, have two engineers grade them independently, then compare both against the agent. If your own humans disagree with each other more than the agent disagrees with them, the interesting defect is in your rubric. That exercise is the judge-calibration step, and skipping it is how a 75.1% headline becomes a procurement decision.

One more caution on the evidence standard for the research itself. The benchmark is on arXiv and reproducible in shape; the re-grading study reached the press as vendor research, and I could not find a primary paper or an independent replication. Both may be entirely sound. But a study whose own conclusion is "verdicts outrun records" is a study to read with its own standard applied, and the honest status of the 75.1% today is a striking result from an interested party, not yet replicated.

FAQ

Does this mean cloud security vendors have been inflating severities?

No — it means a rule evaluated against one resource cannot know anything about the environment around it, so it has to emit a single rating. The severity field is a property of the rule, not a measurement of your risk, and that was always true.

Why does the 16.6% of upgrades matter more than the 58.5% of downgrades?

Because it is evidence the agent has a threshold rather than a bias. A triage tool that only ever moves findings down produces the same output as one that has learned to say "probably fine", and the two are indistinguishable from the downgrade rate alone.

Can I use an agent's re-grade to close findings?

Not on these numbers. The verdict is reported at 94% while the answer behind it is at 78%, and closure is the use that depends on the record rather than the label. Use it to order a queue and to decide what a human reads first.

What is the single highest-value change this suggests?

Hand the agent your estate's relationships instead of making it discover them. Structured relational context improved answer correctness by about 34% relatively and cut exploration queries by about 70% in the benchmark, which is a larger effect than any model choice reported in it.

Is 32.3% of downgrades really just orphaned resources?

That is the largest reported single pattern, and it reframes a third of the backlog as an asset-lifecycle problem rather than a security-triage one. The right response is deleting the resources, which removes the findings permanently and for a reason anyone can audit.