This is not accurate.
Actually, it is. Sterling broke an explicit rule laid out in the NBA bylaws, which he is contractually obligated to uphold. The scope of the commissioner's ability to investigate, arbitrate, and levy punishment are also clearly laid out in these bylaws. The punishment given to Sterling was within this scope. A judge would throw out the case as soon as he saw it. It is very cut and dry.
He could claim that Silver abused his discretion, and that the commissioner's decision was unfair. ...to me, it is a little unfair.
"Abused his discretion"? What does that mean? Sterling admitted he made the statements and that he knew he was being recorded. There is no abuse. Whether you think it is fair is irrelevant. Hell, it doesn't even matter legally if it is objectively fair. The judgment was legally rendered within the confines of the NBA's bylaws. There is no legal recourse, unless malfeasance can be proven on the part of the NBA (as in, someone paid them off to do it, or something). This is extremely unlikely.
Because Sterling had a private phone call with his girlfriend that she (as I understand California law) illegally leaked, he should lose the team?
California law is irrelevant. The statements were made public, and as part of the NBA's investigation STERLING ADMITTED IT WAS HIM AND THAT HE KNEW HE WAS BEING RECORDED. It doesn't matter at all how or why the recordings were leaked. This isn't a criminal case.
So if you have a private conversation with a friend of yours and say something negative about gay people, and your friend, angry because you are suing him, sends it to your boss - you ought to get fired? It's a histrionic reaction in a "gotcha" culture.
It is a legal precedent, actually. You are not free from the consequences of your words or your actions. Employers have every right to take this kind of action, provided it is within the explicit scope of their contracts with the employees. By working for a company, you explicitly agree to follow their rules. This is extremely cut and dry, and laws of this sort are not new, they have been around for decades. Your vague hand-waivings about "gotcha" culture and unsubstantiated and irrelevant. This is NOT a criminal case, where such concerns might become relevant. Sterling and the NBA had mutually agreed upon rules and regulations. There is nothing Sterling can do about it.
If Sterling went to court, it would be thrown out almost instantly. Further, HE has no reason to go to court. Why would he? He has already admitted that it is him on the tape and that he consented to the recording, which takes away his only possible avenue of complaint against the judgment. Further, he will still make a lot of money for the sale of the Clippers. For a businessman who is notoriously unheeding of public opinion, what do you think he will do? Waste money on court fees for a case he is doomed to lose (and make no mistake, his lawyers are telling him that yesterday) or just cash out his chips, sell the team, and continue with his real estate business?