I had the same thing with 2 lupins in pots. Talk about an infestation! I thought all was lost because I don't use chemicals. I squished along the stems and then drenched the plants with washing up water, thinking I'd probably have to bin them. But no, after repeating the exercise three days running they had almost all gone and the plants hadn't suffered. They are now coming into flower and i check them every day squishing the odd one i see.

Whilst congratulating myself on exterminating the wee beasties, I had failed to notice that my annual lupins in pots had succumbed to the same fate. Process repeated and all seems well so far.
I suppose washing-up liquid contains chemicals but not too toxic .

I tend to squish them but any soap will work as a drench.

There seems to be a particular type of lupin aphid, more grey/white & a bit larger than usual? It may be an import from the US so our native ladybirds haven't come to terms with it yet.
I had a debilitating infestation last year but my lupins are all in pots so advice to pick up the pots & blast them daily with a pump-up pressure sprayer with a drip of soap may not work for everyone.
Good news is that this year they seem to be really clear of bugs. Whether that's down to the aphids not surviving over Winter or something else I don't know but don't write the plants off too soon. Give them a bit of care this year & they might pay you back in time?
P.S. I think I posted pics of the offending sapsucker buggers