The exaggeration on this forum is a bit too much. Neither Towns nor Irving are "abominations" defensively. That's narrative based, not stats or eye-test based.
Towns' weakness is probably exploited more because he's a big man. In one sense, that makes him a "worse" defender. What I like about Irving is that, when it matters, he definitely knows how to pick his spots defensively. He made Curry work in the finals when he played for Cleveland, including a key block late in one game. And remember the Rockets game last year? That game started to turn on multiple defensive plays that Irving made. Towns hasn't proven yet that he can rise to that level. In fact, he was outplayed on both ends of the court by Capella in the playoffs.
Neither are high level defenders, but neither are terrible. They aren't Jamal Crawford, or Isaiah Thomas, or Deangelo Russell, or Nick Young, or CJ McCullom, or Greg Monroe, or Ryan Anderson, or Frank Kaminsky, or Jabari Parker.
I still don't think, given Irving's track record of high level basketball against the best players in the most critical situations, that you could say Towns is better (although he might prove to be). I still think Irving has another level to go to as a basketball player.
Towns' upward trajectory was plateaued the last few seasons. His efficiency has risen, but he defense has stayed relatively the same, he hasn't improved as a playmaker for others, and he hasn't improved his intangibles. He still could get better, and it's reasonable to think he will, but it's also possible that he never gets better. It's also possible this is who is he is. IF (big if) this is who he is, I think one could argue that Brown is a better asset than Towns.
That's the reason I suggested you could argue the Wolves get the best two players in the trade Moranis proposed earlier.