I really don't think it matters if the refs affect the outcome or not. Regardless of result, when the refereeing is bad the quality of the game suffers, not to mention all the ammo it gives the conspiracy kooks.
Same way if the Celtics play lousy but win, we don't shrug and say "Well, it didn't affect the outcome." It hurts the quality of the game, and it worries us going forward because we expect it to burn us later.