Of course not. First, the need for injury insurance is being significantly understated - Irving is averaging ~60 games/season, limps into the playoffs, missed the 2015 Finals and 2018 Playoffs, and is unwilling to play through discomfort.
Second, Rozier averaged 17 pts, 5 rebs, 6 asts, 1 steal in the 2018 playoffs with 48/35/83 splits and elite perimeter defense. This at 24 years old with a $3m salary this season.
He has also had less 10 games to adjust from starting PG on a near-Finals team. Every one needs to chill out. We'll be in the Finals this year, Rozier will be instrumental in key wins (like last season), and Brown will come around.