Where are you growing them, Tonmar? Inside, in pots? Heated or unheated? Welcome to GoY, by the way!
I'm not sure about orange trees, but lemon trees need to be kept above freezing, but below 30°c, have lots of light and preferably a slightly dampish environment.

Both overwatering and underwatering can cause flower and fruit drop. I would let the top cm of the potting compost dry out between soakings. A strong feeding could also cause that, so I would avoid feeding--or give very light feeding--from bloom until the fruit is the size of a marble.