I presume it's in the ground?

Two things perhaps are happening here. First, this plant is a classic sulker. It can take three to four years before it "takes off". The second is that at the base of a wall, it can be very dry and reduce it's initial growth rate.

Mine has been the same......3 years old, but I have 3 lovely flower heads this year..and it has grown to about 4ft now. It is at the base of a north facing wall. I did get a bit dispirited with mine but after reading that they like plenty of moisture..I watered it regularly which seems to have got it growing........