The Gemara Sotah 10a explained by Rashi says, Tamar wasn't even related to Yehuda anymore once Er and Onan died and this was not an act of Yibbum. This is because she had been a young orphan married by her mother which is not a kiddushin Mideoraisa (valid biblical marriage), and when Yehuda told her she couldn't marry Sheila she did Miun (retro-active rejection of rabbinic marriage) once she realised Yehuda never intended to marry her of to Sheila.* So when Yehuda sought her, he asked her if she was an erva (i.e daughter in-law) to which Tamar responded correctly no, because she had not able to marry properly as her father was already dead and her marriage was now a Kiddushei taus (mistaken marriage after having rejected it):
יתומה אני - והייתי קטנה והשיאוני אמי ואחיי ואין נישואין לער ואונן כלום לאסור עליו משום כלתו ואין זכות לאם ולאחים להשיא קטנה אלא האב זוכה בה
So since a father in law can't do Yibbum, Yehuda married her from scratch as she was never considered married to Er or Onan, Peretz was Yehuda's son not Er's son.
*If she would have been married by her father a Kiddushin deoraisa Yehuda would have been forbidden to marry Tamar as stated in Vayikra 18,15: ערות כלתך לא תגלה (do not sleep with your daughter inlaw)