GPT-5.6 Sol edges GPT-6 Astra on confirmed bug finds in 50-PR benchmark

A benchmark posted to r/ChatGPTCoding puts GPT-5.6 Sol ahead of GPT-6 Astra on bug detection, with Sol finding 107 confirmed bugs against Astra's 91 across 50 real pull requests. The runs covered changes from five open-source projects: Cal, Sentry, Discourse, Keycloak and Grafana. The poster, u/entelligenceai17, reports that Astra scored higher on precision and returned results with lower latency, so the two models trade places depending on which measure you read. The account asks readers for feedback on the methodology before it runs the next comparison.

Coding benchmarks built on real pull requests are meant to approximate the work a reviewer does, reading a diff against a live codebase rather than solving an isolated puzzle. Pulling 50 PRs from five established projects gives the comparison a spread of languages and review cultures, and the poster says every finding was independently verified rather than accepted as reported by the model. The post frames itself as an evaluation in progress and asks for methodological criticism; the numbers arrive with a chart image and a link promised in the comments.

The headline counts measure how many bugs each model surfaced and had confirmed, which makes Sol the leader on recall-style volume. Astra's edge on precision means a larger share of the findings it reported were judged genuine, and its lower latency means it produced those findings faster. The post does not give the precision percentage, the latency figures, the number of findings each model returned before confirmation, or how the 50 PRs were sampled.

Verification is the least documented part. The post says findings were independently verified but does not name the verifier, describe the adjudication rules, or state whether one reviewer judged both models' output. It also does not report how many runs were performed, so it is unclear whether 107 and 91 are single-run totals or averages. The author invites that scrutiny, saying feedback is wanted ahead of the next evaluation.

The comparison is one entry in a rotating series rather than a standalone study. The same poster says Fable versus Opus runs next week, which means the model lineup changes while the harness and PR set may not. Whether the next test reuses the same 50 pull requests, the same five repositories and the same prompt format is not stated, and that matters for reading the results side by side.

Supporting material sits outside the post text. The benchmark link is promised in the comments rather than included in the body, and the only artifact attached directly is a chart image. The posting account's handle contains a company name; the post itself does not say who ran the benchmark, whether either model's maker was involved, or what the runs cost.

Nothing in the post says whether raw outputs will be released, so the 107-to-91 gap cannot be checked by anyone outside the group that produced it. The next comparison, Fable against Opus, will show whether the methodology changes in response to the feedback the poster asked for. Until per-repository numbers and the verification process are published, the counts stand as a self-reported result from a single account on one subreddit.