Ok, I was reading this one earlier today and couldn't comment due to technical difficulties, but.....
I used to feel exactly like the OP and would never say this particular phrase period due to the implications upon others. However, after years of censorship, one of my best friends in the world and one of the most morally and ethically sound people I have ever known personally (who had a mentally challenged, physically handicapped sister, R.I.P as of last year BTW) got me used to the phrase, because he used it regularly on a daily basis and there wasn't such a derogatory connotation to it. It was more of a general term. To him it simply meant "expletive" just like the FU word can be used in many contexts.
Now this does not make it right, nor does giving an abnormal meaning to a normally offensive term make it right (calling your non-gay friend gay, no reason etc.), but using that term within your own culture may become acceptable for one reason or another, and we have to accept that instead of judging from afar with no experience with the people that you feel have been so oppressed.
Even though I felt that it was politically incorrect, I accepted it because of his influence on me and his culture rubbing off on me.
Now personally I still struggled with the concept, tried not to say it, but now I say it all the time and can't help it. And honestly I still feel bad or correct myself in the process, but truth be known, it's a part of my regular day life now and it once was not.
I guess what my point is, is that we are all influenced by our surroundings and our culture, so whatever is cool in your company is cool in general regardless of your beliefs as long as no one gets hurt in the process. If we we're all good enough friends to talk crap about one another, nobody would be up in arms about liberal rights.
My advice is to get to know that person you see in a wheelchair, or mentally challenged adult, and once you're comfortable enough with them, you can ask them if it's offensive to them. Otherwise you have no idea how they feel about it.
PS- Edgar- No disrespect to OP, but We need basketball!