I'd guess that the plant just isn't getting enough water. This often happens for a number of reasons:
1. The plant is still small and isn't quite up to supporting a lot of fruit yet. I always remove the first tiny courgettes to give the plant a chance to get going.
2. The female flowers are not being pollinated properly, which happens in cold weather or when there is a shortage of pollinating insects.
3. The soil is very dry and the fruit shrivels.
If you find the fruit that has fallen off has a brown core, then I think that makes poor pollination most likely.