Although you mention trees Mahonia is usually a large a shrub, so I will suggest a few of those that might work, once its dug out and the soil replenished. Photinia 'Red Robin', Viburnum 'tinus', Laurels - Prunus laurocerasus 'Rotundifolia': or an Escallonia such as 'Apple Blossom' and Holly tree but prickly?,

Yew will form a dense hedge given time. It is slow growing, but the flip side to this coin is that you only need prune once a year around September to keep it tidy.