Germs will be in your mouth whether you rinsed or not. The bacteria matter, those are the reason you don't want food there in the first place, and those are killed better by not rinsing.

Think of the paste more like something you need to apply and let sit, and less as some kind of fancy soap to wash the teeth with.