I think the fundamental problem in terms of expectations is that he was the product of a tanking year. In the 2013-2014 season most of us were rooting for the team to lose (let's face it) for a shot at Wiggins, Parker, Embiid, Exum, etc. We got the consolation prize in the 6th pick, which became Smart. But I think since we all went through the pain of losing big time for that year, our expectation was that we'd be rewarded with an all-star.
Well, 6th picks don't typically become all-stars. They might, and Smart certainly has a chance to. But right now you have to be happy for what you have: one of the best defensive guards in the league (already), and a guy who's clearly working his tail off to improve his offensive game.