Just use some ant powder over places where you see them.....make sure your containers are big enough for your fruit plants to flourish too. If you put a ring of ant powder all round the stem of the tree and they are looking for aphids to get the sweet fluid they excrete, they may go elswhere.....lots of other places for them in the garden.

If you mean the ants have nested inside the tubs, then yes, you need to get rid of them, they sometimes eat through the roots of the plant to give themselves more room for their nest. You will have to turn out the tubs, let the ants disperse and remove any signs of a nest, then repot in fresh compost and this time, stand the pots on something which raises them off the ground by an inch or so - pot feet, old roofing slates, anything you can use that will keep it level but raised and leave the drainage holes clear.
If the ants are running over the stems and foliage, this is usually because of aphid infestation - get rid of the aphids, the ants stop coming. The ants themselves do not damage the leaves and stems at all.

Would nt be too sure about the damage they do,(root damage would make sense) but they" farm " aphids ,so it could be a sign aphids are about,like is said