Could. But it’s not all that simple. The motive behind allowing freedom of movement is to create a common market. Work has to be able to move freely for something to be a common market. The same has to be true for capital and goods. All that isn’t as simple as abolishing customs. Regulations can, for example, create powerful barriers in the market (for example when France and Spain have different and mutually exclusive regulations for mechanical wine harvesters) and a big part of what the EU does (and for what it is often ridiculed) is to harmonize regulation.

It’s possible that freedom of movement could be disadvantageous for one country if a common market isn’t also created and that’s were the complications start, were it’s necessary to have some sort of common political structure (and not just a treaty signed by the respective heads of government) in place.