If you lose to an opponent, the Toads that came to cheer you on will leave your kingdom. Although it's sad to see them leave, the number of Toads that leave when you lose is less than the number of Toads that join when you win. Plus, even if you lose, you get to keep the coins you collected during the rally.
Keep playing to gradually increase your number of Toads!