This is an excellent and very, very, important question for a website like this, one that I've been thinking about from the beginning.
First of all, let me identify with the answers provided by Shalom and Alex and reiterate that mi.yodeya does not provide professional (particularly rabbinic) advice.(1) We try to alert users to this with
red writing at the top of each page, a bold "Consult your Rabbi" suggestion on the "Ask a Question" page,(2) a disclaimer that comes up in red on the sidebar for new users and in bold in the the FAQ. In addition, I encourage answer-writers to stick in some humble language like "Here's my understanding, but ask your Rabbi before taking action." as necessary.
The remaining question is: What if people ignore the warnings and use mi.yodeya information as if it was professional rabbinic advice? I don't have an ironclad answer, but I think that on balance, we're doing OK. People give classes on Jewish law and practice all the time, and there's always the danger that attendees (or archive-listeners) will act on what they hear in the class or in an associated Q&A session without asking their Rabbi first.(3) The danger here is similar and, I think, similarly tolerable, with the advantage that we're explicitly warning against such practice at every turn.
Hopefully, more people will be encouraged to ask their Rabbi questions that otherwise wouldn't have than the other way around, thanks to our questions and answers spreading awareness of the issues and our disclaimers reminding people to go talk to their Rabbi.
All that said, I do need to upgrade my efforts (non-zero but not successful) to get Rabbinic guidance myself on this very point. And, of course, I'm very interested to hear what community members think, either in comments to this post or in email to firstname.lastname@example.org.
UPDATE: I spoke to an internationally-respected rabbi who has experience and interest in Internet issues. He generally approved of the approach we're already taking, provided that we make sure that the disclaimers are prominent enough to be seen by everyone, and that we make it clear that while discussions of Jewish life naturally include Halachic content, people should go to a rabbi, not a website, for actual Halachic advice (my paraphrasing). I think we're pretty much fulfilling his advice already, but I'm going to review our disclaimers and consider beefing them up. Also, though the rabbi didn't bring it up, I'm seriously considering implementing some sort of flagging mechanism like the one Barry suggested. (See also my comment there.)
UPDATE 2 (2011 June 19 - after the migration to SE 2.0): When we migrated from the SE 1.0 site mi.yodeya, which I could edit more of, to this SE 2.0 site, the disclaimer went away, since it's not a standard part of an SE 2.0 site. We asked to have it put back, and SEI added it to the sidebar for new users. I also added it to the FAQ.
1) If you're wondering what it's good for, given that, please see my blog post on the topic.
2) See Update 2, above.
3) Recall the Blue Fringe lyric "I just heard a half hour halacha shiur / And decide to change the way I've lived for 18 years."