My understanding is that it wasn’t considered such a big deal largely for those reasons, although anti-Arab racism and Islamophobia would likely have also been a factor.
‘Population transfer’ itself was not a taboo proposal in that era, it was considered a potentially suitable means to ‘solve’ the problem of ethnic minorities living in the new nation-states emerging from the ruins of empire. The Norwegian Fridtjof Nansen was awarded the Nobel Peace Prize in the early 20s in part for brokering the famous ‘population exchange’ between Greece and Turkey, which involved many more people than were made refugees by the 1948 war (one of my very
favorite books concerns this episode). This example was specifically cited in the Peel Commission on Palestine (1937) as a possible means by which to make a partition of Palestine into Jewish and Arabs states feasible.