Well you are, really - the only way you can manage it now is to dig very big hole, and remove the rootball in its entirety from the pot and place in the hole intact. But unless there were other plants in the pot with roots, it's unlikely to hold together without breakage to the lily roots.

do what I do, sink the pot into the garden where you want them, up to the rim of the pot. This way you can place them where you have a gap and lift them later in the year and replace with other plants like dahlias. They wont need any extra watering either.

I'm going to do that too -thanks. x