does anyone know the best way to remove stumps from a small garden?

A short while ago, Bamboo (I think) wrote a blog about removing stumps. See if you can find it.

Yes, I have a blog on tree stump treatments, but in your case, you should probably hire a stump grinder and get it out, or most tree surgeons have someone who you can pay who will come and do the job for you, removing all the major roots as well, and take the debris away.
As for a shrub to plant, need a bit more info - what do you want it to do? Just look good, flower, evergreen screening? Is the spot sunny or shady? Sorry to ask so many questions, but you get the best answer if I have more info.
And if you want to read the blog, click on my icon and when you get to my home page, go to blogs - there's only 4 to choose from.

It depends on how good the access to the stump is, Georgie.
A number of years ago we had a large tree taken down that had been growing on our boundary. The stump was huge so I got a neighbour to come along with a JCB. He positioned himself outside our fence and removed the whole stump in about five minutes! Very impressive. The alternative would have been to hire a stump grinder. This would have been expensive and taken several days.

Add lots of rotted manure or garden compost once the stump is gone as the soil will be quite poor. How about a Hydrangea quercifolia ( for slight shade) or Hydrangea paniculata ( for sunny spot). Or Cornus elegantissima for winter twig interest, or a Choiysia for an evergreen.