Are you growing outside in a sheltered town garden or inside? They are often difficult, what with yellowing leaves, scale insect and spider mite. The moisture balance and amount of light and heat can be critical. The main reason for the premature fruit drop is stress. First they will drop their fruit and then their leaves. You need to take all these points into consideration and perhaps try a potassium feed to strengthen the flowers and fruit.

Thanks, I'll try that next year. Think I'm too late for this year!