The Holy Ark was 2 1/2 Cubits, by 1 1/2 Cubits, by 1 1/2 Cubits (Shemot 25:10).
The Talmud discusses (Baba Basra 14A-B) how much space the Luchot occupied in the Ark. The Talmud says that each tablet was 6 handbreadths, by 6 handbreadths, by 3 handbreadths. The Talmud goes on to discuss exactly how much room the Luchot occupied in the Ark, depending on whether the Cubits used to measure the Ark were made of 5 or 6 handbreadths.
The Talmud goes say there was some empty space left over and discusses what was in the empty space.
According to the opinion that the Whole Luchot and Broken Luchot were placed in the Ark (Baba Basra 14B), how did they both fit?
I'm assuming that both sets of Luchot were the same size, since G-d tells Moshe to hew a new set of Luchot like the first set (Shemot 34:1). No matter how you calculate the Handbreadths/Cubit ratio used in the Ark, there isn't enough room for a second set of Luchos.
There are opinions that Moshe made a second Ark which were eventually used for the broken Tablets (see for example Rashi to Devarim 10:1), but:
Others (such as the Ramban 10:5) Disagree with this.
The opinion in the Talmud that says both sets of Tablets were placed in the Ark seems to be saying that they were both placed in the same Ark.
Tosafot (Eruvin 63B) is of the opinion that there were two Arks, but says that when King Shlomo built the Beit Hamikdash both Luchot were placed in one Ark (according to the opinion that the broken Luchot were also in the Ark).