Maybe you are overwatering it when you bring it in, I hardly water mine it is very neglected but looks beautiful and I have to keep hacking branches off to keep it tame ! Mine is kept indoors all the time here and I have noticed if I give it too much water at one time (when I remember) it loses a few leaves.

Some leaf loss is natural, but if you lose quite a few, it's probably just the plant adjusting to its new conditions - these plants don't like changes in habitat, draughts, etc., and often react by dropping their leaves, so its the transition from outside to inside that probably causes this. I have a different variety of this, and I find it does drop leaves more than usual when autumn arrives, even though it's always in the house. the exception would be if the leaves are going soggy and yellow before dropping off - that would mean over watering.