Think if Magic really want Smart over Exum (or Parker, Wiggins, Embiid) than they'll try to swap the pick for either Jazz's two picks (5 and 23) or Celtics (6 and 17). Doesn't make sense for them to draft him there and not try to trade down first.
They might try, but it's possible (or even likely) that the Celtics and Jazz don't see a major talent drop-off between Exum, Smart, Vonleh, and Gordon/Randle (or at least not enough one to justify such a trade). In other words, the Magic will have to take the player they think is the best unless either Boston or Utah strongly believes that player to be much, much better than the remaining players. There's no reason to think that Orlando is the only team that rates Smart and Exum similarly.
The Magic could call Danny and say "we're leaning Exum but willing to move the pick" and Danny could say: "Go right ahead. We're perfectly happy with either Smart or Gordon (etc.)." Even if Danny thinks Exum is on another level he might be willing to call Orlando's bluff, especially if he thinks the Jazz are locked in on Vonleh or Randle.