I'd really like him, too, but due to his age and contract, I wouldn't give up anything substantial for him. If Horford leaves and they look for a salary dump, I'd love him, but I really wouldn't offer up anything more than like Amir, salary filler, and number 16 for him.
He actually would be a perfect 4 in between Durant and Horford. If we could convince those two to sign first, I'd consider overpaying a bit for Milsap, though not all that much.