> We use a host of quantitative and qualitative measures. None of them show any positive improvements. These include the basics like roadmap reviews, demo sessions, feature cycle time, etc as well as fairly comprehensive business metrics.
Those are very high level. If there's no movement on those, I'd guess there are other things bottlenecking the teams. They can code as fast as possible and things still move at the same pace overall. Nice thing to know.
If you want to really test the hypothesis that Copilot and ChatGPT have no impact on coding speed, look at more granular metrics to do with just coding. The average time from the moment a developer picks up a work item to the time it gets merged (assuming code reviews happen in a timely fashion). Hopefully you have historical pre-AI data on that metric to compare to.
Edit: and average number of defects discovered from that work after merge
> look at more granular metrics to do with just coding. The average time from the moment a developer picks up a work item to the time it gets merged (assuming code reviews happen in a timely fashion)
We do collect this data.
I personally don't put a lot of stock in these kinds of metrics because they depend far too much on the way specific teams operate.
For example perhaps Copilot helps developers understand the codebase better so they don't need to break up the tasks into such small units. Time to PR merge goes up but total coding time could easily go down.
Or perhaps Copilot works well with very small problem sizes (IMO it does) so developers start breaking the work into tiny chunks Copilot works well with. Time to PR merge goes way down but total code time for a feature stays the same.
For what it is worth I do not believe there have been any significant changes with these code level metrics either at the org level.
Those are very high level. If there's no movement on those, I'd guess there are other things bottlenecking the teams. They can code as fast as possible and things still move at the same pace overall. Nice thing to know.
If you want to really test the hypothesis that Copilot and ChatGPT have no impact on coding speed, look at more granular metrics to do with just coding. The average time from the moment a developer picks up a work item to the time it gets merged (assuming code reviews happen in a timely fashion). Hopefully you have historical pre-AI data on that metric to compare to.
Edit: and average number of defects discovered from that work after merge