This year's Patriots Defense was set to win the SuperBowl. I watched them yesterday in the 4th quarter, playing without Tommy Kelly and Vince Wilfork, and they gave the offense every chance to win the game. I hope Tommy Kelly can play at some point, and Armstead is able to give something after week 7, but this D is really elite.
As for the Offense, it's not only one factor. Brady isn't having his best year, and yesterday the O-line wasn't very good. McDaniels playcalling has been questioned by a lot of fans in Pats Pulpit, and while losing Welker and Hernandez hurts, Edelman replaces Welker at a high level. The biggest problem is injury bug: Gronk, Amendola, Vereen, Ridley, and even the rookies. It seems like they can't field a healthy lineup, so the chemistry is hard to build. Brady will trust the new guys and eventually will trust himself to play like the best QB in the game. It's a long season.
Coming into week 5, the Patriots had 13 rookies on their roster. After cutting Sudfeld, I guess it's 12, but that's still a lot to overcome. It's a long season, Pats are 4-1, they have an Elite Defense for the first time in some years, so they just need their injured players to get healthy.