I don't think anybody questions his fit at all. The entire concern is due to his age. This is the last time we will have cap space for awhile and people want a player who is not just ready to contribute now but will be in the next couple years when LeBron is older. Millsap doesn't work with that timetable.
The other thing is, due to his age he would be less tradeable through the course of his contract. If we signed someone younger like Hayward, at the very least we'd always be able to deal him for something coming back (remember, we are going to be over the cap soon). With Millsap you aren't going to get anything back, this is his last big deal. When his next contract expires he'll probably be close to retirement age.
He is basically the same age (within 40 days) as LeBron. If you think we are ready to beat the Cavs and Warriors the next couple seasons, then signing Millsap is a good move. Otherwise, it's best to sign someone who will still be in their peak 3, 4, 5 seasons down the line when LeBron will be 35, 36, and 37 years old.