I don't think time is close to running out, yet. If he has another full (hopefully uninterrupted) season under his belt after this one and he's still a fringe rotation guy, then it's time to be worried.
He's still very raw and young, and circumstances have conspired to keep him off the floor at the NBA level. He has shown flashes of talent which seem really promising to me.
My biggest concern is whether he can stay healthy. If he can and next season isn't too drastically derailed by COVID, I'm hopeful he can become a regular 15-20 minute per game contributor, which would be a significant jump.