The thing about Brown is that everybody assumes that because he's a great athlete with good size+length, he'll automatically be a great NBA defender. Hence the comparisons to Iggy, Kawhi, TA, even Lebron. This is far from guaranteed. Plenty of great athletes never become better than average defenders (see: Green, Jeff), and Brown has not done anything defensively that shows unusual aptitude over and above his physical tools.
I don't have a great sense of how good Brown will be on D, but that is probably where most of his potential values, and if we're taking him #3 it's because Ainge feels he has a good chance of becoming an elite defender. Right now I don't see it.