That's the other risk with leaving them in the ground over winter - if we have a cold spring, which we did this year, right up to June apart from a week in April, the plants are very slow to get going. If you'd started them off indoors or the greenhouse in pots, the cold wouldn't have affected them, or if the spring had been warmer and brighter.

Unless the soil is well-drained and sheltered; not inclined to get frozen, I would be inclined to dig them up each year. The tubers do tend to rot and there might only be a few remnants of the original ones left so therefor late and feeble plants.