Excellent question.
Back in an agricultural society, people had animals around. So he was thinking it would be a cow, sheep, or goat. Nonetheless, the Talmud said he should not have taken that oath -- what if it was a horse or donkey? (Which can't be used as a sacrifice.)
Some Christians took this story as a message "oh, always fulfill your oaths." The traditional Jewish interpretation is disgusted by that possibility! If I swear to commit murder, or eat a cheeseburger, the oath is null and void because I can't override the word of God!
So Jephthah messed up in taking the oath. (He could have pledged "the first unblemished cow, sheep, or goat that would exit the yard.") But the far greater mistake was in carrying it out.
Jewish tradition has it that if someone made an oath with faulty information, and regrets ever having made it to begin with, he can have it annulled by going to a panel of three knowledgeable Jewish men, or one great rabbi. Our understanding is that Jephthah said "hey I'm the big military leader, the rabbi should be knocking on my door to annul my oath!" Meanwhile the rabbi said, "I'm the spiritual leader, that ignoramus should be knocking on my door, then I'll annul his oath!" (Recall that Jephthah had grown up as an outcast, and was finally the big man on top of the pyramid. For him to go ask someone for help would mean not being it anymore, which he couldn't face.) So the stubborn men each refused to swallow their pride and give an inch, and something terrible happened as a result. (You'll see all sorts of problems going on in the book of Judges. It was written by Samuel, who is framing the past few centuries of history as an explanation of why the people now needed a monarch.) Reminds me a lot of Dr. Seuss' "The Zax."
While the simple reading of the verse is that he did actually kill her, some commentaries suggest instead that she was forced to live out her life as a hermit, effectively rendering her "good as dead" from the perspective of human society. (Hey we're social animals.) But one way or the other, the story is intended as tragedy -- just because a Judge was a good national leader, he could still be a terrible person.