Maybe it is just the people I hang around with, but I have found that the word "retarded" (when used in a negative way) is not nearly as acceptable as you make it sound. I really think it has become a word that will incite a very negative reaction, if used in a negative way.
Now, I think it is unfair to compare it to a word like the N word of F word, because they have very different histories. The N and F word both became taboos because they became known as words of hate towards those groups of people.
Retarded never was really used like that (at least to the level the other words were). It was actually the proper term for the mentally challenged (is that even the correct term these days?). People didn't call mentally challenged people "retards" because they hated them, the way people used the N and F word. Kids (and immature adults) would just use the word to make fun of other kids who they thought were acting stupid. It's offensive, but on a completely different level.
I think a better comparison would actually be to the word "Gay". Again, this was (and still is) a proper term for homosexuals, and really does not have a history of being used as a term of hate towards Gay people. However, it has been used by kids to make fun of other kids who are effeminate (or something like that). So again, offensive, but on a different level from the F word and N word.
So to sum it up, generally, the words that are really over the line to most people are the ones that have been used as slurs to direct hate towards the people they refer to. Words misused to mock other people are just offensive...but you can ussually still get away with in public without being labelled a bigot...just immature.
Edit: and just to be clear, I agree that no one should be using the word "retarded" to make fun of people...I am just trying to explain the reason why it is not in the same category as the F and N word.