The opinion of the Shulchan Aruch, Orah Haim, Siman 124, S'if 4 is that 9 men answering is required, but the Mishnah Berurah (S'if Katan 19) brings down from Magen Avraham and the Maharil (Ashkenazic authorities) that the common custom is to use even those who are not listening to complete the minyan (which would mean 6 listening and 3 not listening). The Mishnah B'rurah suggests the Shaliah Tzibbur make a condition that if there are not 9 listening it should be a T'filas N'davah in order to take the Bais Yosef's opinion into consideration. This would imply that one should wait until there are 9 men who can respond, but Rav Feurst told me that Reb Moshe Feinstein told him that 6 men was enough (of course all of this is provided that there are a total of ten men there).
http://www.hebrewbooks.org/pdfpager.aspx?req=14170&st=&pgnum=294&hilite=