I know how you're feeling. I visited my first synagogue with my church confirmation class in high school, then visited a reform synagogue where I went to college, and then went on to conservative and eventually orthodox synagogues where I finally converted. (I had a conservative conversion earlier, but I don't count that.) And although the synagogue your visiting is more like a "museum" than an actual synagogue, if your desire to convert is sincere, then your next step will be to an active "shul" (Yiddish for synagogue). So my answer will mostly address what happens after your visit to the "museum synagogue," but also has application to your current issue.
The first visit to a synagogue, the first call to a rabbi, and even the second or third call to a rabbi, can be very stressful for someone wanting to convert. There is no question that you can go to the synagogue -- just dress appropriately and wear a yarmulke (assuming you're male) -- most synagogues (but not all) have them available near the door for visitors. A female non-Jewish visitor need not cover her hair, but -- especially in Orthodox synagogues -- should minimize skin exposure by wearing longer sleeves, knee-length or longer skirts/dresses, and a modestly-cut collar. It would be best if you can find a Jewish member to go with you to help guide you. If you don't know anyone, don't be shy ... ask someone who looks like they know what they're doing. Orthodox services are in Hebrew almost exclusively, conservative congregations use less and announce pages, but still it may be hard to find your page. If offered an honor, such as taking out the Torah, or getting an aliyah (an honor at the Torah reading), gently turn down the offer by explaining you're not Jewish and not entitled to the honor until you complete conversion. Also, don't put on a prayer shawl -- while there's no real law against it, a non-Jew doesn't have the commandment to wear one, so its a bit tacky IMHO. Some rabbis endorse wearing a tallis for their married conversion candidates before the formal conversion ceremony. Consult your local rabbi to be sure.
As to your prayers -- by all means, say any prayer from your heart that is directly to God, but I would encourage you to follow along in the prayer book to get an idea of how Jews pray. Much of our prayer book contains entire chapters or verses of many psalms.
Introduce yourself to the rabbi. Tell him that for now you're trying to decide if conversion is for you. You don't need to start learning with him (or his designee) until you're ready. But ask him if there are general adult education classes you can attend.
As for books, I hope you've started collecting some. To Be a Jew by Donin, and
Becoming a Jew by Maurice Lamm are helpful. Rabbi Lamm's book has a chapter with a personal story about a first visit to a synagogue by a prospective convert. An excerpt of my personal story is in that book, also, at page 38. Good luck.