No, you don't necessarily have to register as a VAT payer in each country you sell in.
If you're based in the EU, you should register in the same country that your business is located. If you're not based in the EU, we suggest registering in the country where you get the most sales.
After that, you'll have to pay attention to the distance selling threshold for other EU member states. If you reach the distance threshold of another country, you may have to register and remit VAT there too.