It's just math, not even a very difficult one. You need detailed data about trajectories and properties of celestial bodies, and then it's just Newtonian dynamics.
Play a little bit of Kerbal Space Program if you want to get a feel of it :).
Just a thought: will Newtonian dynamics not introduce a significant error over these speeds and distances? I believe that Newton's physics are just a simplification of Einsteins Relativity. I am obviously not a physicist, so I hope somebody can straighten me out.
Also I am not sure that we know all the weights of the celestial bodies with enough precision not the introduce an error while performing a 'slingshot'. Rosetta has propulsion and my guess it that it can be used to actively correct small mistakes in positioning.
Any error due to using Newtonian dynamics instead of relativity is almost certainly swamped by guidance error and such. These things are never perfect, and corrections are always needed during the trip. They are really good, and the errors are small, but they're there, and errors due to relativity are smaller still.
To give you an idea of the magnitudes involved, consider that GPS satellites do need to be aware of relativity. GPS depends on extremely accurate clocks, to the extent that relativistic time dilation due to the altitude of the satellites becomes important.
However, the necessary correction for relativity is only 38 microseconds per day. That's about 0.00000004%. It matters for GPS, but that's a special case.
For another example: the orbital precession of Mercury was one of the first indications that there might be something beyond Newtonian gravity. Newtonian gravity would have Mercury following a steady ellipse, with small changes due to the gravitational influence of other bodies and the fact that the Sun is not a perfect sphere. However, Mercury's orbit precesses (which is to say, the ellipse itself rotates slowly) more than could be explained by influences from the Sun and known planets. This was initially thought to be due to an unseen planet called Vulcan, but none could be found. Eventually it turned out that general relativity explained the discrepancy perfectly.
The total precession of Mercury's orbit is about 574 arc seconds per century. The amount of that due to relativity is about 43 arc seconds per century. That's about 0.12 arc seconds per year, and it means that if you used pure Newtonian mechanics to predict Mercury's position one year from now, relativity would cause your prediction to be off by (very) roughly 17km. Not much, and trivially corrected for using the spacecraft's thrusters while on course. That's equivalent to a velocity error of 0.5 millimeters per second on the spacecraft's part, and I don't think rockets are that accurate to begin with.
You know the masses of celestial bodies relatively well from their observed trajectories and it is not too difficult to include relativistic corrections in your calculations (though active corrections using thruster will still be extremely helpful).
I imagine finding the best trajectory in the first place to be the more difficult and interesting problem.
(Not an ESA guy, just wanna-be-condensed matter physicist.)
There are people from ESA here, so I hope they can write about this in more detail. From what I know, Newtonian dynamics is good enough to move around the Solar System at the speeds we commonly employ.
We've been doing gravity assists for some time now, so I guess this topic is mostly figured out, though probably some active corrections are required. Again, maybe the ESA guys will chime in. Or InclinedPlane ;).
Play a little bit of Kerbal Space Program if you want to get a feel of it :).