It certainly shouldn't. Your aim is just to keep bringing your attention back to the object of attention. Each time you do that is a success, so feel pleased with yourself at that point.

Fighting is a bad habit that will send you down a blind alleyway in meditation. There should be no striving, no effort, just gentle persistence.