I don't know about "worried", but there are some legitimate reasons for concern.
1) KG is coming off a major injury for the first time in his career. We don't know the extent of that injury, or how it will heal.
2) Powe is gone until at least the all-star break, and may not be back with the team at all.
3) BBD can leave in free agency, as can House (if he opts out), Starbury, and Mikki (good riddance).
4) Pierce, Ray, and KG are all getting older, and all showed signs of aging this season.
5) The team is already over the luxury tax, before signing BBD or any outside free agents.
6) We don't have a lot of depth on the team.
7) Even with KG, the Cavs were neck-and-neck with the Celts all season long.
So, it's not like there's nothing to be concerned with. We'll see how Danny reacts to mitigate / alleviate some of the above, but as of right now, there are serious questions about whether the team coming back next year will be good enough to contend.