Generally, I don't think a comment like this should be taken offense to no matter who says it whether he be black or white. However, in Payton's case, I don't think it was a joke. While I'm not offended, I think this is reflective of his ignorant mindset and I have little to no time for him.
In the name of consistency, I'd like to see him reprimanded for this. Wont happen, and I wont lose sleep over it, but certainly should happen.