Pollen beetles, maybe?

Could be aphids if they seem reluctant to actually fly, and they group tightly around the base of the flower buds, where the flower stems join the main stem, and/or on the undersides of the leaves. If so, first try washing them off with a fairly strong spray of plain water. If that doesn't work, spray them with a weak solution of organic soap--it should just feel slippery between the fingers. Many chemical insecticides work on aphids, but frequent use usually causes more problems than they cure.