Rabbi Natan Slifkin has an extensive presentation of all the different sides of the kezayis discussion here

Rabbi Natan Slifkin has an extensive presentation of all the different sides of the kezayis discussion here. He brings many opinions that actually do hold that the correct size of the kezayis is the actual size of an olive:

R. Chaim of Volozhin (1749-1821) is widely revered as the father of the yeshivah world. Less known and certainly less popular in the yeshivah world is his view as to the size of the matzah that one is obligated to eat on Pesach. R. Chaim was of the view that this kezayis is actually the size of an olive—around three or four cubic centimeters.

As far as why the size seems to have increased he says:

It is in Ashkenaz that we find the olive beginning to evolve with the statements of the Rishonim themselves (as opposed to with later mistaken inferences regarding the Rishonim). The Rishonim of Ashkenaz translated the size of an olive into a proportion of an egg, but they gave different quantities. This was based on differing resolutions of various passages in the Talmud. In one place, the Talmud states that a person can swallow food up to the size of two olives: The Sages evaluated that the throat cannot hold more than two olives. (Talmud, Krisus 14a) Elsewhere, the Talmud states that a person can swallow food up to the size of an egg: The Sages evaluated that the throat cannot hold more than a chicken’s egg. (Talmud, Yoma 80a) These passages indicate that an olive is half the size of an egg. However, in a third place, a different conclusion emerges. The Talmud (Eruvin 82b) discusses the amount of food required for an eruv. Two of the views cited express their opinion in terms of kabin, which in turn can be expressed in quantities of eggs (since 1 kav is 24 eggs):  Rabbi Shimon: Two meals are 2/9 of a kav, which is 5 1/3 eggs.  Rabbi Yochanan ben Beruka: Two meals are ¼ of a kav, which is 6 eggs. Elsewhere, the Talmud states that two meals are equal to 18 dried figs. Now, as we saw earlier, an olive is known to be smaller than a dried fig. This results in the following calculation:  Rabbi Shimon: Two meals = 5 1/3 eggs = 18 dried figs; thus 1 olive is less than 3/10 of an egg  Rabbi Yochanan ben Beruka: Two meals = 6 eggs = 18 dried figs; thus 1 olive is less than 1/3 of an egg How are all these sources to be reconciled?