If the leaves are brown & chewed, you should definitely cut them off and discard. It gives a clean & tidy appearance & combats pathogens which can survive bitterly cold winters.

if you don't like the look of them then yes. the slugs will be hiding in the soil during the day and that's why you cant see them.
I tend to tidy mine up late autumn when most of the leaves have died off ready for them to overwinter.