To be fair, we are just brains in meat suits. Brains adjust their physical structures to integrate with their tools efficiently. This is proven in imaging studies of those skilled in use of a particular tool (ex fencers) and likely extends to the abstract (ex feral children having underdevelopment of some brain structures).

So, it makes sense that something with large potential social impact like gender only makes sense to impact structure.