I think he gets the "yips" during games, some guys do. You can be the best shooter in the world and it does not matter if you can't do it in a game...
Getting the yips is extremely rare in basketball, especially in the pros. Baseball and golf are more known for it. In the last 30 years the only player in the NBA that I can remember having a case of the yips is Markelle Fultz.
It's most likely he is struggling with an aspect of the system and severely pressing when given his very small amount of playing time.
Thankfully, this year, there will be tons of practice time to work things out. Look at Grant, for instance. Started his career 0 for 25 and has been a pretty great three point shooter since then.
The season is very young and so is Nesmith. I am confident he will be making his threes and be a positive addition before we know it. He showed late last year over an extended stretch that he is definitely capable of it.