When God gives the Torah to the Jews, He uses the following preamble (Ex 19:3):
וּמֹשֶׁה עָלָה, אֶל-הָאֱלֹקים; וַיִּקְרָא אֵלָיו ה', מִן-הָהָר לֵאמֹר, כֹּה תֹאמַר לְבֵית יַעֲקֹב, וְתַגֵּיד לִבְנֵי יִשְׂרָאֵל.
And Moses went up to God, and the Lord called to him from the mountain, saying: “Thus shall you say to the house of Jacob, and declare to the children of Israel...”
Why the repetition? Nothing in Torah is superfluous. The Mishna says: His ‘house’ means his ‘wife’. [Yoma 2a] The Midrash [Pirke de-Rabbi Eliezer 41; also Mekhilta de-Rabbi Ishmael] says:
Rabbi Pinchas said: On the eve of Shabbat, the Israelites stood at Mount Sinai, with the men and the women apart. [God] said to Moses, “Go, speak to the daughters of Israel [and ask them] whether they wish to receive the Torah.” Why were the women asked [first]? Because it is the way of men to follow the opinions of women, as it is said, “Thus shall you say to the House of Jacob” -- these are the women -- “and declare to the Children of Israel” -- these are the men. They all replied as with one mouth, “All that the Lord has spoken we will do and we will listen.” [Ex. 24:7].
OK, but a search of the Sources reveals thousands of occurrences of the phrase "the house of [an individual]" and in most cases it definitely does not mean the women. Why is this one singled out for that meaning? Is there a rule for telling which is which?