Whenever we've tried to grow broccoli, it's always been infested by these tiny grey bugs to the point where we can't even wash them off.

I assume they could be the same thing that infested our broccoli a few weeks ago, pip, Flea Beetles. We used the usual soap spray but it was far too late.Nasty little things.

The grey bugs are Bishop bugs, a knd of aphid which adores all kinds of Brassicas. We get them mainly on sprouts. You really do have to get rid of them before they become a big problem. Since we do not spray we normally wash them off the plants with a strong jet of water at the first sign of them and keep on doing it until there are none left.
However, once they get established then you are definitely into spraying with one of the recommended for vegetables, insecticides.
If your veg patch is not well drained then could I suggest that you raise the soil above the water level, somehow. That would make your plants grow stronger and more resistant to insect damage.

Thanks heaps for your advice