Cut the stems as near to the ground as possible and apply an ivy killer to the roots. If the ivy is buried in the brickwork you will need to cut them either side of the root and remove it carefully as the brick may well need to be replaced although it is more likely that with correct pointing the brick can be saved.

Not easy, as I expect you've discovered, if you've got live ivy growing, do what Scotsgran said, and then to remove the growths on the brickwork, a wallpaper scraper is probably the best tool to use, but you may find some of the brick comes with it if the brickwork and pointing are not in the best condition.