Not by any means sure but could it be a sucker from a Kerria japonica - do you know if next door has one?

It's in a pot, I think I may have planted it or it may be a weed. It's about 4' tall and I don't really want to leave it in if it's a weed.

I don't think its a normal weed - its obviously not herbaceous anyway, Looks more like a tall shrub or young tree - sorry no further ideas.

Silver Birch (betula pendula) seedling?
Can take up to 15 to 20 years to get the typical white bark or it may never turn white.
Are you prepared to wait?
Silver Birch seeds are wind blown, so you don't even need one very close to you.

Funny, looking at this again today Silver Birch was my first thought rather than Kerria. I think you're probably right.