Obviously there are some special circumstances (See: Donaghy, Tim)
But more often than not, the Refs are like the wind and the rain, there is nothing you can do about them.
They are such a large part of the game, any sort of challenge rule would slow the game. The refs are responsible for so much that they will make mistakes, it happens, human error. Yea some calls are bad, but rarely is it one call that decides, usually it is the other 47 minutes of basketball.
So yes, it is frustrating, but I think too much oversight could be a bad thing.