I haven't really been following Gordon that closely, so I don't feel equipped to weigh in on how he's performed thus far or on his upside. But as someone already mentioned, the fact that he's got 2 fewer years left on his rookie deal is important. This is also the reason why you don't trade a lot to get someone like Nerlens Noel, unless your plan is to pay him the max and prevent anyone from stealing him when he gets to restricted free agency.
On Brown, I do think his potential upside is tremendous. I suppose he could bottom out as a sort of Jeff Green type in a worst-case scenario. But already he has a unique combination of ball handling, knack for getting to the hoop, and physical ability. Defensively, he could develop into a Kawhi Leonard-type defender who could guard multiple positions, though I guess Gordon also fits that mold. In any case, it would take a lot for me to feel comfortable not seeing that through.