Rav Yaakov Kamenetsky (http://hebrewbooks.org/pdfpager.aspx?req=47587&st=&pgnum=179) addresses the above question. Here is an organized summary of his points:
- Tes and Pei don't seem to look alike
- In these cases, the words are written incorrectly in the Gemara. What should be פאין is spelled פיפין. (Also, what should be גמלין are גמין. Rav Yaakov argues that צדין should be צדיקין as well, but let's leave that aside.)
The Rambam quotes this Gemara in Hilchos Tefillin 1:19 as follows:
אין כותבין תפילין ומזוזה אלא בכתב אשורית. והתירו בספרים לכתוב אף ביוני בלבד. וכבר נשקע יוני מן העולם ונשתבש ואבד לפיכך אין כותבין היום שלשתן אלא אשורית. וצריך להזהר בכתיבתן כדי שלא תדבק אות באות שכל אות שאין העור מקיף לה מארבע רוחותיה פסול. וכל אות שאין התינוק שאינו לא חכם ולא סכל יכול לקרותה פסול. לפיכך צריך להזהר בצורת האותיות שלא תדמה היו"ד לוא"ו ולא וא"ו ליו"ד ולא כ"ף לבי"ת ולא בי"ת לכ"ף ולא דל"ת לרי"ש ולא רי"ש לדל"ת וכן כל כיוצא בזה עד שירוץ כל הקורא בהן:
Of note is that Rambam quotes this Halacha along with two other related Halachos:
- Writing S"T, Tefillin and Mezuzos in Ashuris script, not any other language (including writing Sifrei Torah in Greek, which was originally allowed, but the Rambam notes that since Greek is now "lost", we cannot use it anymore).
- Making sure letters do not stick to each other.
Why would Rambam include point 1 in this Halacha, it seems to be a totally seperate topic? Also, note that Rambam chooses only ו/י, כ/ב, ד/ר, but not the others, and expects us to know from those examples. Rav Yaakov feels that based on Rambam's style he should have quoted ג/צ as well (but not א/ע because those don't look the same, only sound the same).
Based on all of the above, he comes to the following conclusion (as hinted to by the Rambam). The Hebrew letters Tes and Pei are not what the Gemara is referring to! The Gemara is referring to the Greek letters Theta and Phi, which do looked very similar back then, and are still quite similar even in today's version of Greek letters (they are basically a circle with some line through it, as shown in this chart).
(He also suggests that Gamin and Tzadin are Greek letters, but again, this question focuses on Tes and Pei.)
Therefore, the Rambam begins by talking about that Greek is lost, and as a result does not quote the Greek letters that were in the Gemara. The word פיפין must be used because a Hebrew word always begins with a Dagesh, so in order to signify the soft "PH" sound, it doubles the Pei. (Similar answers are given for Gamin and Tzadin as well. See the actual piece inside for more details, as well as other proofs, notes, and attempted answers.)
ראוי לאמרו, ושפתיים ישק!
For the sake of Emes, I will be posting another answer below with my understanding of the Rambam (what I believe to be the simple Peshat), as well as some other points worth considering.