I'd generally rather trade for an undersized player of proven effectiveness than a taller player who has not shown anywhere near an equal level of production given ample opportunity, despite having superior athleticism.
Kenneth Faried is about the same size as Millsap. Maybe not as heavy, but otherwise about the same. Would you avoid trading for him, too?