Are you sure that you have Castor Oil Plant? They are not very hardy in the UK. There is another plant called False castor Oil Plant/Fatsia japonica. Google those and come back as I think you have a Fatsia as they are hardy and relatively common in the UK.

you can if kildermorie is right cut them back quit brutaly .

I think Kildermorie must be right, in which case, so is NP. My Fatsia is in full glory, the winter does not harm it at all, even the big freeze of Dec. 2010, but my Ricinus have just come out from the greenhouse and are grown as an annual in all parts of the UK. In fact it is so sensitive to frost that I am leaving a couple, about 6 inches high at the moment, in their pots under glass for another few weeks to be sure.

Assuming its Fatsia (which I'm sure it is), these reach about 8 feet by 6 feet. Pruning isn't required, however, do it, if you must, during May, taking out growth you don't want, and reshape, but trying to keep it at 4 feet isn't a good idea.