his defensive impact outweighs any of those things you mention. He is already an elite role player under rookie contract. We are now hoping he takes the leap from role player to star- which he has plenty of time to do.
Smart is a good defender but his shooting needs to improve. There are two ends of the court, you should not stink at one of them and he does right now. I honestly, think if he does not Ainge could trade him. I would remind you that Tony Allen had many of the same strengths and weaknesses that Smart has and Ainge traded him. Allen could drive too. That alone should tell you he is not untouchable.
Hopefully, his shot improves and this will not be an issue. That would be the best case scenario.