I've had some success by simply putting a small amount ant powder around and under the pot regularly ... it seems cut all activity out within a week or so. The ant bait boxes are also quite good on the top of larger pots.

It's best to repot as soon as possible though (assuming you've got them a bit under control).

i had this trouble with my thyme a few weeks back .
I took the thyme plant out of the pot , sprinkled ant powder round the base of the plant , (where the roots are) , rinsed out the large plastic pot and have had no trouble since

Once you've got rid of them, raise the pot off the ground with pot feet or bits of broken slate or something to give clearance underneath. This discourages the ants.