Unfortunately that sounds like late frost damage. Next year if possible, try covering the plant if frost is forcast to protect the flower buds.

Also give plant a nice feed with ericacious feed or sequesterine (not sure of spellings here ! ) if your plant gets the morning sun then you are going to always have trouble unless you protect on cold mornings protect with a fleece.