Evmbench

evmbench solves a problem that has nagged AI security evaluation for years: nobody agrees on what 'good at finding smart contract bugs' means, so every vendor demo picks its own examples. By publishing an open benchmark built with OpenAI and Paradigm, evmbench gives auditors and researchers a fixed set of realistic contract tasks and a shared way to report results. Two evaluators running the same model get comparable numbers, which is what makes the output usable in a procurement or model-selection conversation. The design detail that matters most is the split between the full benchmark and the hosted interface. The benchmark itself spans detection, patch generation, and exploitation. The web page you actually touch at paradigm.xyz does one thing: you drag a contract folder or ZIP into the upload area, choose a model from the selector, agree to the Terms of Service and Privacy Policy, and start an analysis that reports only high-severity findings. That is a sensible default, since low and medium noise buries the signal when you are scanning a batch of contracts, but it also means the UI cannot tell you whether the model wrote a patch that compiles or an exploit that executes. For those questions you go to the repository and the blog post. Strengths: the provenance is unusually strong for a free tool, the benchmark targets EVM contract code specifically rather than general software bugs, and the folder/ZIP upload means you can point it at real contracts rather than synthetic puzzles. The results are reproducible and open, which is the honest counter to a vendor's cherry-picked demo. Weaknesses: it is an evaluation instrument, not an audit. A run tells you what a model flagged at high severity, and interpreting those flags still takes smart contract security knowledge, so it is not beginner-facing. The benchmark's EVM focus makes it irrelevant if you run on other chains. The hosted interface has no repository-connected or continuous scanning mode described; each run starts from an uploaded folder. Where it fits: choosing between candidate models before wiring one into a review workflow, stress-testing a vendor's claim that its AI assistant catches critical bugs, comparing two model versions on the same contract set, and AI capability research where exploitation ability is itself the thing you want to measure. Where it does not: it will not replace an audit engagement, it will not monitor your contracts over time, and if your stack is not EVM you are outside its scope entirely.