First, I'm a Cal alum so I watched a lot of brown last year. For many of the reasons already stated, I would not draft brown and three. And I disagree that he is an insanely good athlete. A good one, yes. Insanely good, no.
That jump shot will limit him quite a bit. And there is a question of how well he can pplay without the ball.
Now on the plus side, brown has a rather slippery way about him. I mean, he seems to slither his way to the basketball with relative ease. He's a pretty good finisher but that will be more difficult in the NBA.
Bottom line: I'd pick him from 6-8 but not at three.