I recently wrote about the many API tools that support OpenAPI today, from Linters to Documentation and from Editors to API Gateways.
The only gap remaining is in testing.
While many test automation tools leverage OpenAPI, for the most part, they act as test generators. They take OpenAPI as input and produce something else. And you’ll have to maintain that something else yourself.
For example, Postman can generate collections from OpenAPI. And ReadyAPI can import OpenAPI to generate tests.
But today, we can also use LLMs to generate tests. Just jump into OpenAI’s o3, upload your OpenAPI file, and type:
“Generate a full test suite for the OpenAPI file attached.”
You’ll get code you can run along with instructions on how to do that. And you can get way more fancy with the prompt.
But what if we had a simple CLI tool that takes OpenAPI as input and actually runs tests? It would then exit with 0 or 1, depending on whether there were failures.
You could easily use such a tool in CI/CD, and it would help prevent many bugs and issues.
A tool like that exists, but hasn’t been maintained - it’s called Dredd and does precisely what I described:
$ dredd
But unfortunately, it won’t work with recent versions of OpenAPI.
One particularly valuable use of a tool like that would be to repeatedly test the output of a vibe coding session in a deterministic way, and then feed the output back to the codegen tool used. You’d be assured the code matches the OpenAPI document.
I think building such a testing tool would be an excellent opportunity for a company like SmartBear, Postman, Zuplo, Scalar, Treblle, Theneo, Traefik, and many others to create such a tool, open source it, build a community, and also a product on top of it! Who’s up for it?