I mean Xavier McDaniel wore #31 after Maxwell and before the number was retired, so it is not entirely impossible that this could be re-visited. Still, X-Man was never thought to have the same potential that Hayward has when each signed with the Cs.
Heck, #20 may get retired based on Hayward's play. I guess it all depends on how the future plays out.