I played football from age 10 to age 18, and I've played Rugby since (18-25). I've played at the collegiate and men's club levels, and I've both here in the states and in Canada.
Football is the riskier sport. In rugby, because there are no pads, people tackle and run differently. Its like the difference between boxing and judo. Football is a game of inches, and Rugby is a game of possession. In rugby unless you're at the goalline you never hit like you do in football. You don't care if they fall forward an extra 2 feet, you're more worried about your defense resetting and making sure your guys are rucking properly.
Football is played 4 downs and 10 yards at a time. Football at a pro level is played by people who are freakishly athletic and fearless. Football imbibes its players with a false sense of immortality because they have these hard plastic pads on.
So when football players tackle, its like two cars crashing head on.
After every Rugby game, you're left feeling a little broken, beat up and worn down. The adrenaline wore off sometime in the second half. There is a formation of small round bruises rapidly turning purple on your back. There is a neat square at one end, then a cluster of about 6 on the other end, forming a neat semi circle. You laugh when you realize its a boot mark. You're beat, and you want a beer. When I played football I always thought there should be an extra quarter because I wasn't tired.
However in my football career, which was pretty good IMHO, I broke both feet twice a piece, I broke 6 or 7 toes, suffered countless stingers and neck strains, and worst of all, completely blew out one knee on a chop block.
In rugby I've broken one ankle on a wet muddy day that was just a freak accident. Nothing else. Men play well into their 40's. I've never been named player of the week or all conference, but I'd had awesome times and I learned what a boat race is.
I would say football is a sport that is much more violent, higher stakes. Rugby is a rougher sport with lower stakes.
The final part: If American Football players played Rugby we'd be world champions. THat is virtually indisputable.
Football is a gentleman's game played by hooligans. Rugby is a hooligan's game played by gentlemen.