Most of the time, you convert the image from gamma space to linear space, you do whatever you want to do, and then convert back.