OpenAI did something similar with their o1 model. Ran a coding problem through o1 thousands or maybe millions of times and then checked if the solution was correct.

I can imagine a great pipeline for performance optimization: 1. have an AI generate millions of tests for your existing code 2. have another AI generate faster code that still makes the tests pass

So I guess all I want for Christmas is a massive compute cluster and infinite OpenAI credits :P