Rama is Rabbi Moshe Isserless. He wrote annotations on Shulchan aruch. Shulchan Aruch was written by Rabbi Yosef Karo. Shulchan Aruch is divided in 4 parts (following the 4 parts of the 4 Turim) The first part is called Orach Chayim (= OC). 603 is the num. of the Chapter, 1 is the paragraph. This is in Halachot Rosh Hashana. Rama said: Doubt about sin is stringent than sin. this is a mixture of personal words of Rama mixed with words of Rabenu Yona on Brachot. You can find the quoted passage in Wikitext here.
וספק עבירה צריך יותר תשובה מעבירה ודאי כי יותר מתחרט כשיודע שעשה משאינו יודע ולכן קרבן אשם תלוי הוצרך להיות יותר ביוקר מחטאת (דברי עצמו ורבינו יונה ריש ברכות):
A doubt about sin need more repentance (a greater effort to reach repentance) than a sure sin, because one who know that he have erred (naturally) repents more than one who doesn't know if he have erred. Therefore the korban Asham Talui is more expensive than the korban Chat'at (to buy it needs more financial effort).
The Gaon Mivilna reported the source as a Gemara in Kretot 22b (the Gemara needs a special drasha for the minimum value of Asham talui, without this Drasha we would think that the value of 1/6 dinar is good enough as chat'at) and Avot Derabbi Nathan. This Drasha stresses that tshuva for doubt... as RAMA wrote.
The Ramban is on Parashat Vayikra, 5, 15:
וטעם אשם תלוי מפני שבעליו סבור שאין עליו עונש כי לא נודע שחטא מפני זה החמיר עליו הכתוב בספקו יותר מודאו
The reason ... because of the tendency to think that there is no punishment when it is not certain that he made an error. This is the reason of the stringency for this offering which comes after a doubt... (there is above a large explanation of the word asham)