I was a rugger (well, sort of...winger here

) in college and grad school, and I'd agree with those who say rugby is overall a rougher sport that requires greater endurance, but football is significantly more dangerous in terms of severe injuries. Demo derby vs NASCAR race is a good analogy. Rugby tends toward chronic injuries accumulated over the game and over time, and football tends more toward severe, one-time injuries.
A key difference is that rugby is much less of a collision sport - typically a guy that hits you in a rugby game started coming at you from less than 10 feet away, whereas in football someone might sprint the width of the field and launch themselves at you. Rugby tackling is focused more on wrapping up and pulling down, and football tackling is more about hurling yourself like a missile to blow the guy up. The hits are much more frequent in rugby though, and you don't have time to collect yourself (or get off the field) between plays.
Oh, and rugby does have the unique danger of all sorts of fun illnesses brought on by shooting the boot and the like. But I'll take a rugby social over a football postgame celebration any day.
EDIT:
Where did you go to school?
University of Rochester for undergrad, UNC for grad (played for the B-school team, though I wasn't a B-schooler)