My impression (since childhood) of the incident in the garden is that the serpent persuaded Chava to eat, and then she took some fruit to Adam and he ate it. This question asks if he knew where the fruit came from when he ate it, which presupposes that he weren't there. This answer brings different views about his knowledge, but none of the sources there assert that he was a direct witness.
The plain text, however, is much more ambiguous. The end of Bereishit 3:6 says:
... וַתִּתֵּן גַּם-לְאִישָׁהּ עִמָּהּ, וַיֹּאכַל.
and she gave also unto her husband with her, and he did eat.
This is in the same verse where she herself eats -- she saw it was good, ate, and gave to her husband who was with her. Rashi addresses her reasons but says nothing about the "with her" part or the timing.
How do Chazal get from this text to the understanding that he wasn't there at the time and she brought him the fruit elsewhere? Is it that we know he wouldn't directly transgress so he must not have been there, or or is my impression wrong and it's possible he was there the whole time (which raises questions about why he didn't intervene)?