When you remark on improvements, up is generally better and down is generally worse. So saying "response times will be higher" gives an immediate sentiment of improvement. But, obviously, a moments thinking helps you re-orient and realize it's better. This is why plots often have "lower is better" in the legend, to help readers understand.
I often use 'slower' and 'faster' as a native speaker to help reinforce the meaning of the direction.
It makes more sense if the direction is split from the unit in the sentence.
Consider: "after investigating response times and doing and doing a bunch of work, my PR causes a 30% reduction" - if you're busy doing lots of things and already have a lot of cognitive load, this could sound bad because the important phrase "30% reduction" is in there. You are a detail oriented person who is immune to this effect, but there are people who need help reducing their cognitive load in this type of thing.
You can help people reduce cognitive load by replacing "reduction" with "improvement" so they immediately understand that it moved in the right direction.
"Response times will be higher" sounds very confusing as a way of saying we'll take less time to respond, right? So why should "response times will be lower" mean we'll take more time if the opposite construct is confusing?
Far better to just use the comparative forms that we already have for time specifically to make it perfectly clear.
I often use 'slower' and 'faster' as a native speaker to help reinforce the meaning of the direction.