"What might Yiftach have had in mind when he talked about something coming out of his house?"
I believe he had in mind that a male member of his family, i.e. one of his sons, would come to greet him. Notice his wording "וְהָיָה לַיהוָה וְהַעֲלִיתִיהוּ עֹלָה", which the commentators translate "It will be for God or I will bring it as an olah offering (if appropriate)". Yiftach's initial statement was that the first greeter from his home would be dedicated to God. If it would be one of his sons, then that son would be dedicated to the service of God, similar to what Chana did with her son Shmuel. He would become like a kohen or levi. This is what Ralbag suggests here.
Only in his second part of the "or statement" does Yiftach acknowledge that what crosses the threshold of his home first may actually not be human, but of his collection of livestock; perhaps not likely, as you suggested, but possible. In this case, he says, he will bring it as an offering to God. [I cannot be sure what he is referring to with "אֲשֶׁר יֵצֵא מִדַּלְתֵי בֵיתִי לִקְרָאתִי", but perhaps he was thinking that he may chance upon one of his animals upon approaching his property (unlikely though it is that it would be "coming to greet" him).]
Yiftach's mistake was that it didn't seem to cross his mind that his daughter might come out to greet him first. Perhaps he was thinking it more in the nature of sons to come out and greet their father, while daughters (at least in those times) tend to stay inside and wait for the father to arrive. But alas, his daughter came to greet him first, and therefore had to be "dedicated to God", which in a woman's case involves celibacy, as a marriage is seen as being "in service" of her husband which would undermine her dedication to the service of God.
