I don't think there's a particular time - just cut them off carefully.

If you want to save seed (which is quite large), wait until the fruits start to turn brown and crack open. Sow immediately as viability drops by the week, literally! Especially though, don't let the seeds dry out.

If you aren't saving the seeds, the sooner you remove the fruit the better--I just twist off the dead flowers with the baby fruit in passing.