Can I use the soil?

Cannot see any reason why there is going to be anything wrong with the compost. Any spores of almost anything will have been destroyed in that length of time.

If you made your compost properly the heat generated as the organic material rots down is enough to destroy the mildew spores. And as Owdboggy says after 4 years most harmful spores and bacteria will have been dealt with.

Thanks very much for your replies, the compost I've been making in my garden looks great and I want to use it on my allotment....I've been worried about spreading the mildew to plants there.
Daisy