Even when discarding feelings influenced by the sports rivalries, I think NYC and LA are two of the best AND worst places to visit. Vegas is a distant second in this category.
I love visiting Boston, Portland (ME and OR), Miami, and Seattle.
When I was in the Army, I did some training in places I would never want to return, but for the most part, that was due to the training grounds, not the region. I can't really think of a place I hate to go. Maybe I need to get out more.