Yeah, this has been noted at various points over the last coupe years. It's a combination of two things:
1. Defensive statistics are inaccurate. 90% of defensive plays do not result in a steal or block, and team-based metrics (plus-minus) are heavily affected by lineups - for example, the +/- might just be telling you that Smart+ET are better on defense than AB+IT, which is very different from saying that AB is worse than average. These types of biases don't necessarily improve when you have a large sample size.
2. The eye test is also inaccurate. As others have noted, AB's on-ball defense is excellent, and that's mostly what casual fans see when they watch people playing defense; people are less good at appreciating good team defense, which is something that AB is not nearly as good at.
Overall, I think AB is a very good defender, but I think that Crowder is at least as good as him and Smart is significantly better.