I wonder if it was black popular seeds. there is a tree in the village that produces thousands of wind blown seeds in late april. It looked like it had been snowing. Also some of the early willows may have released their downy seeds too.
rape seed pollen is yellow., most pollen is yellowish though there are many exceptions.

Denise, a question was asked recently by Linda 235 about a similar problem. It appears to have been from a neighbour's Goat Willow tree. Anything like that near to your garden?

Willow was going to be my suggestion too