It's the #1 factor, for sure. The thing is, it's a bit of a funnel. The most physically gifted kids play more sports, get more training, etc and it self-selects for physical specimens.
There are also different types of physical gifts. Certainly being tall, jumping high and having long arms is great for basketball, but don't be fooled into thinking Larry Bird or Steve Nash didn't have physical gifts. Just different ones.