Yevamot 34b explains it thus:
מיתיבי כל עשרים וארבעה חדש דש מבפנים וזורה מבחוץ דברי ר' אליעזר א"ל הללו אינו אלא כמעשה ער ואונן כמעשה ער ואונן ולא כמעשה ער ואונן כמעשה ער ואונן דכתיב והיה אם בא אל אשת אחיו ושחת ארצה ולא כמעשה ער ואונן דאילו התם שלא כדרכה והכא כדרכה בשלמא אונן דכתיב ביה ושחת ארצה
Soncino translation, my notes in braces:
An objection was raised: During all the twenty-four months {after a birth, when a mother is nursing} one may thresh within and winnow without {see below}; these are the words of R. Eliezer. The others said to him: Such actions are only like the practice of Er and Onan! -Like the practice of Er and Onan, and yet not [exactly] like the practice of Er and Onan: ‘Like the practice of Er and Onan’, for it is written in Scripture, And it came to pass, when he went in unto his brother's wife, that he spilt it on the ground; and ‘not [exactly] like the practice of Er and Onan’, for whereas there it was an unnatural act, here it is done in the natural way.
[The source for] Onan's [guilt] may well be traced, for it is written in Scripture, That he spilt it on the ground
From the footnotes in the Soncino translation: "thresh within and winnow without: Euphemism. This would prevent possible conception which might deprive the young child of the breast feeding of his mother."
Now, the reason Onan was spilling his seed on the ground is that he didn't want to father a child with Tamar (38:9). In other words, he performed yibum but then denied the purpose of yibum -- he sabotaged it to Er's and Tamar's detriment. It would have been better for him to refuse the marriage in that case. So we could say that he was punished for this. The sequence, in any case, is: decide not to fulfill the purpose of yibum -> spill his seed during the act -> punishment.