I disagree that Ray is a liability on defense, at least consistently so. He struggles against Joe Johnson, that much is clear. However, opposing SGs have a 12.7 PER against the Celtics when Ray is on the court, and have an eFG% of .428. Last season, SGs had a 12.3 PER, and the year before, a 12.4 PER. These are remarkably consistent and excellent numbers.
There are problems with defensive statistics, but you generally don't see consistency like the above for three straight years unless somebody is a pretty good defender. As mentioned, there's the occasional game where Ray will struggle keeping up with an elite player, but overall he's pretty good on the defensive end. If he was as bad as some suggest, there would be a clear track record of SGs consistently exploding for big nights on a regular basis, rather than only on rare occasions.