There would also be questions over whether the UK would lose its "substantial influence" over the development of EU financial regulations and would retain the flexibility enshrined in the renegotiation settlement secured by David Cameron.
Asked whether uncertainties of this kind might lead companies to relocate business activities away from the City in the event of Brexit, Mr Carney said: "One would expect some activity to move. Certainly, there is a logic to that.
"There are views that have been expressed publicly and privately by a number of institutions that they would look at it. I would say a number of institutions are contingency planning for that possibility - major institutions, foreign headquartered, which have their European headquarters here.
"There would be an impact. I can't give you a precise number in terms of institutions or jobs or activity, because we don't know where we would be on that continuum between full mutual recognition or pure third-country access."
Asked if some degree of loss of business could be expected if full mutual recognition was not retained, Mr Carney said: "Without question."
He told the MPs: "It is reasonable to expect that certain firms would take a view in terms of relocation."
Mr Carney added: "Mutual recognition arrangements are possible to achieve, but in general they take a very long time to achieve and the challenge is the degree of freedom one retains in setting one's own path, rules, approach, and maintaining that mutual recognition."
Britain would have to think through the benefits of seeking to preserve as much as possible of the City's existing business at the cost of "in effect ceding sovereignty over this aspect of our authorities" and losing flexibilities in the realm of prudential and macro-prudential regulation and supervision, which have just been reinforced by Mr Cameron's renegotiation, he said.
In a letter to the committee released as he appeared to give evidence, Mr Carney said the renegotiation deal - which will come into effect only if the UK votes to remain in the EU - "delivers a number of protections and additional tools that will help safeguard the Bank's ability to continue to achieve its statutory objectives".