Stern has done a lot of good things for the league, and continues to do so. He's brought a ton of revenue in, and the league is more popular worldwide than maybe ever before. He has done well with the collective bargaining agreement, including the rookie salary scale and (in my opinion) the minimum age requirements.
However, I agree that he should go. Domestic television ratings are way down in general, and have been declining steadily for awhile now. Rather than being the second most popular sport domestically, it has again fallen behind baseball, and there are concerns it will be surpassed in popularity by Nascar. The last round of expansion was a failure, and the way Stern handled the Seattle relocation was terrible.
But, it all starts and ends with the officiating, and it's a mess. The public perception is that the NBA is cheating, and that goes far beyond Tim Donaghy. In no other sport does the home team get nearly the number of calls that NBA teams get on their home court. Multiple refs have been investigated by the FBI, the NBA Finals have been officiated atrociously (despite these refs supposedly being the best of the best), you have NBA spokesman saying the game should be called differently at different times, etc., etc.
Changes need to be made, and to restore public confidence, Stern can't be the one to do it. Perhaps that's unfair, but his credibility is shot. The league needs to revamp its officiating system, and to start fresh with new leadership.