The European Union is an idea based on an ideology, not on some fundamental inalienable truth of nature that cannot be broken. As an ideology, it is a belief not a fact.
The EU argument is that you cannot have free movement of goods without free movement of people. Economic theory has nothing to prove this imperative. The idea of the free movement of goods among countries is that you do not have to have free movement of people in order to have full employment among trading nations. You can have investments in your own countries and then trade.
The free movement of people is another thing altogether. People had always been free to move around the world since time immemorial until the idea of nations cropped up which was as recent as the breakup of the British Empire in the last fifty years or so. What makes the current free movement of people so unmanageable is the age of information and mass travel so that instead of a trickling effect we get an avalanche which puts pressures on local councils to provide the necessary services.
(Malaysia has chosen to ignore the problems of immigration because they consider them to be foreign workers here for a temporary period only, when in fact they are all here for one whole generation with families and children.)
The mounting social pressures put an enormous strain on the government of the day especially when the economy is weak and unemployment is rising among the locals.
The free movement of people is a political ideal aimed at the creation of a unified broader federation of nations, as a political bloc so to speak. It has less force from an economic point of view. The central issue may simply be record-keeping of the movement of people. The solution may simply be administrative.
There is currently great vehemence against Brexit in the EU because there is real danger that the EU experiment may fail. Nations with successful cities are under tremendous social pressures. With the rise of unemployment, nations with good welfare systems may have to discriminate against newcomers so that the systems do not break down. There will be budgetary pressures and this is when the free movement of people becomes a real economic issue.