To account for such errors the limiter should probably set the limit to the highest of any road within ~50 m, with a possible exception for school zones that aren't immediately adjacent to highways.