if they are unsightly then yes you can. I'd probably wait a few more weeks as there are bound to be a few more frosts and the damaged leaves will help protect the new leaves.

welcome to GoY too:o).

If it's got leaves on it now then it is not a castor oil plant.

Ricinus communis, castor oil plant, is an annual in the UK. The first frost every autumn kills it off overnight.

I think what you have is Fatsia japonica, false castor oil plant. It's an important difference