I value Smart's defense over Avery's because while Avery is better as a pure on-ball defender at the point (one of the best in the league, maybe the best), Smart has much more versatility. I trust Smart to cover bigger twos and switch onto threes and even small fours. I don't think Avery is really in his element defending bigger players.
Smart is also a good rebounder for his size. Avery is average at best in that regard.
Finally, Smart's steal rate is much better than Avery's.
That said, it's fair to wonder how Smart's defense and rebounding would change if his offensive burden were closer to Avery's. Similarly, Avery's defense might stand out more (like it did his first couple years in the league) if his offensive burden were closer to the ideal (10-12 FGA per 36 as opposed to 14-15).
For my money, I'd rather have a two guard like Smart than Avery. AB's offense is no-question much better than Smart's right now, but Smart is a more useful all-around defender and rebounder, plus he can actually handle the ball and create for others a bit, which I still do not trust AB to do. Avery seems to have plateaued as a guy who takes as many mid rangers as three pointers, and who doesn't often get to the rim or draws fouls.
Smart has the potential to develop into a guy who can get to the rim a bit and still be a decent spot up option from three. If he can do that, I think he'll easily eclipse AB's overall value while taking fewer shots on offense.
To me, Avery would be best on a team where he can defend the point full time and play off the ball close to full time on offense, probably next to a dominant scorer / creator on the wing e.g. James Harden, Kevin Durant, Demar Derozan, Paul George etc. Basically, I'd want him to be a better version of Patrick Beverley.
I don't like him so much playing next to a tiny point like Isaiah who (1) has to be hidden on the weaker backcourt player as often as possible, meaning AB sometimes has to cover big twos and switch onto pure threes and (2) often likes to operate off the ball when the defense keys in on his driving too much.