Mainly poor effort and poor natural instincts. I think his foot speed is fine.
One thing I like about him is that he doesn't foul. In fact, he probably could challenge shots more aggressively than he does -- that might be one reason he is easy to score against.
He got a bad reputation in his first two NBA stops, and then his third team was terrible. In Utah, he played next to another true big (Favors or Gobert) right around the small ball revolution. He was a young player who could have put more effort into defense (which is pretty normal), but he was also in a terrible situation where he was consistently taken advantage of by smaller players.
In his second NBA stop, he played for OKC in the infamous playoffs when Billy Donovan mouthed the words on live TV, "Can't play Kanter." Again, in OKC he either played next to Ibaka or Adams during the small ball revolution when he was isolated and taken advantage of.
In his third NBA stop he played for one of the worst defensive teams in the NBA.
I'm not saying he is Ben Wallace or anything, but I think his reputation as a defender is worse than his actual on court play.