The implication of the Rashbam seems to be that he thought that the brothers may have been fooled into thinking the person in front of them was their brother.
Rashbam to Bereishis 45:27
את כל דברי יוסף - שאר דברים שדיבר אליהם הכתובים למעלה, שבכה על צואריהם והכירו בודאי שהוא אחיהם
The words of Yosef - the rest of what he said, which was written above, that he cried on their necks and they recognized with certainty that he was their brother
In their follow up response, they confirmed that they knew it was Yosef. The implication is that until they confirmed that they were certain it was their brother, Yaakov did not disbelieve them that someone had said he was their brother (after all, they had told this fellow that they had a brother who went missing (Bereishis 42:13)), but he doubted that it actually was their brother.