You don’t swallow all of the milk and thus not all of the virus particles. There’s always some residue that can then start to infect you just as if you inhaled the virus.