There are three aspects to your question:
- Where do we learn that it is necessary to pray in a minyan ("quorum")?
- How do we know that the number of people in a minyan is ten?
- From where do we derive the features of these people (age, gender, etc)?
To take these questions in order, the oldest source that testifies to the requirement for praying in a quorum is the mishna in Megillah 4:3. It reads as follows:
אין פורסין על שמע, ואין עוברין לפני התבה, ואין נושאין את כפיהם, ואין
קורין בתורה, ואין מפטירין בנביא, ואין עושין מעמד ומושב, ואין אומרים
ברכת אבלים ותנחומי אבלים וברכת חתנים, ואין מזמנים בשם - פחות מעשרה.
ובקרקעות - תשעה וכהן, ואדם כיוצא בהן
Not everything in the above mishna is easily understood. The following is my own rough translation:
One may not recite the Shema' responsively with its blessings¹, nor
may one go before the ark [to lead the congregation - Rav Ovadiah of
B.], nor [may the kohanim] raise their hands, nor may one read from
the Torah, nor read the haftorah from the Prophets, nor do the
standing and sitting², nor may one recite the blessing of mourners,
the consolation of mourners nor the blessing of the bridegroom, nor
invite [others at the meal to recite birkat hammazon] with the
name [of God: "נברך לא_להינו"] - with fewer than ten people. Likewise,
when it comes to [redeeming consecrated] land: nine people and the
kohen [cf: Leviticus 27:14ff]. When it comes to [assessing the value
of] a person, this is as before [ie: nine people and a kohen - cf:
¹ Acc. to the Ran, in the name of the Geonim, and to the Rambam,
Hilkhot Tefillah 8:5.
² Acc. to all authorities, this was a mourning practice that is no
longer in effect. There is some debate, however, as to when it
occurred and how, but it is also mentioned in Ketubot 2:10, Bava Batra
6:7 and Ohalot 18:4.
There is nothing within this source that can tell us what the origins of this practice were, nor anything that might further delimit the identities of the ten people involved. For sociological explanations that account for the need to pray within a community, see:
Eric M. Meyers, "Jewish Culture in Greco-Roman Palestine" in Cultures of the Jews: A New History (ed. David Biale; New York: Schocken Books, 2002), 135-179;
Shaye Cohen, The Beginnings of Jewishness: Boundaries, Varieties, Uncertainties (Berkeley: University of California Press, 1999),58-59;
Lee Levine, The Ancient Synagogue: The First Thousand Years (2nd ed; Yale University Press, 2005), 135ff.
In an attempt to explain the origins of this law, the rabbis suggested that its roots lay in Leviticus 22:32 - ונקדשתי בתוך בני ישראל, "that I may be sanctified in the midst of the Israelites". But if sanctification in the people's midst is the source of this principle, from where is the number of people who are involved derived?
In Megillah 23b of the Babylonian Talmud, the following midrash is presented:
מנא הני מילי? אמר רבי חייא בר אבא אמר רבי יוחנן: דאמר קרא "ונקדשתי
בתוך בני ישראל" - כל דבר שבקדושה לא יהא פחות מעשרה. מאי משמע? דתני רבי
חייא: אתיא "תוך" "תוך". כתיב הכא "ונקדשתי בתוך בני ישראל" וכתיב התם
"הבדלו מתוך העדה". ואתיא "עדה" "עדה". דכתיב התם "עד מתי לעדה הרעה
הזאת". מה להלן עשרה אף כאן עשרה
From where are these matters [discussed in the mishna above] derived?
Rabbi Hiyya bar Abba says in the name of Rabbi Yohanan: since the
verse says, "that I may be sanctified in the midst of the Israelites"
[Leviticus 22:32], nothing that has sanctity can be with fewer than
How do we know this? Rabbi Hiyya taught: [the word] "midst"
(תוך) appears twice. It says here, "that I may be sanctified in the midst of the Israelites" and it says there, "separate
yourselves from the midst of this congregation" [Numbers
16:21]. [Also, the word] "congregation" (עדה) appears twice. [It says
here, "separate yourselves from the midst of this congregation"
and] it says there, "How much longer [must I tolerate] this wicked
congregation..." [Numbers 14:27]. Just as there [with the
spies] it was ten, so too here [in Leviticus 22:32] it is ten.
This style of argument, employed by Rabbi Hiyya in the gemara, is known as a gezeira shavah: literally "an equal decree", it constitutes a textual analogy made on the basis of the reappearance of a key word, regardless of context. It can be found within Hillel's seven principles (Yerushalmi, Pesachim 6.1), Ishmael's thirteen principles (Sifra, intro.) and Rabbi Eliezer's thirty-two principles (Midrash Agur). Like so many other passages in which this principle is employed, the foregoing midrash is an asmakhta': an attempt at grounding a rabbinic practice within scripture, though not representative of how that practice was originally derived.
You will have noted that none of the above sources specify any details as regards the identities of the congregants. So far as the process by which poskim arrived at the conclusion that a minyan may only be constituted of ten adult men, the following online articles will no doubt be of interest:
Egalitarianism, Tefillah and Halakhah, by Rabbi Ethan Tucker and Rabbi Micha'el Rosenberg: an article (in PDF), in which they review the historical process and attempt to justify egalitarian minyanim;
Women and Minyan, by Rabbi Gil Student: an article published at Torah Musings, in which R' Student appraises and rejects the article by Rabbis Tucker and Rosenberg;
Women and Minyan, by Rabbi Aryeh A. Frimer: originally published in Tradition 23:4 (1988), this article needs to be resized as you go through it, but constitutes a useful overview of the process by which the halakha developed, in defence of an Orthodox perspective;
Women and Minyan II, by Rabbi Gil Student: a useful summary of the opinions of differing Orthodox (and pre-Orthodox) authorities;
Shira Hadasha, a post-denominational community in Israel and around the world, here links to a variety of online resources that provide further information on non-Orthodox approaches to egalitarian minyanim. Unfortunately, some of the links are dead, and Rabbi Daniel Sperber's Congregational Dignity and Human Dignity appears to be one of them. I've included it nonetheless, in the hope that it might return.