The whole point of that story was that young David, without armor, without a sword felled the big champion well-armed dangerous giant with only a slingshot and the power of G-d. Which is what this passage presage in the Middle East right now. Israel has always been outnumbered and outsized and has prevailed with the help of G-d.
- The 'giant' was a giant because it suffered from a birth defect of excessive insulin growth factors, acromegaly or gigantism, a hormonal imbalance. http://emedicine.medscape.com/article/925446-overview
2 Samuel also says he had 6 fingers and 6 toes, 24 digits total.
http://www.mechon-mamre.org/p/pt/pt08b21.htm
So, one might ask "why philistines have so many birth defects?"
Can you see the false assumption? philistines were not larger than Israelites, the giant and his family was a genetic anomaly. Not all philistines had acromegaly.
The chapter you cite shows the gigantism was in this Gittite family, the father Goliath was slain by Elhanan the son of Jaare and his son became a champion of the philistines, whom David slew many according to the Mechon Mamre link above, pasukim 19-22.
The generation of Israel coming in from the desert may have been smaller because they were almost exclusively vegetarian, eating maan, and they do tend to be shorter at least if limited zinc/protein/iron is available before puberty like the Persian and Egyptian dwarves. http://www.ifm.net/industry/zinc2.htm Once in the land, eating meat, they were the same size as everyone else in the region. If you actually read the accounts of the philistines in 1 and 2 Samuel, you will see the whole POINT of the story is that a tiny group of people, outnumbered in every way, overcame the odds with the help of G-d.
Here it is:
מב וַיַּבֵּט הַפְּלִשְׁתִּי וַיִּרְאֶה אֶת-דָּוִד, וַיִּבְזֵהוּ: כִּי-הָיָה נַעַר, וְאַדְמֹנִי עִם-יְפֵה מַרְאֶה. 42 And when the Philistine looked about, and saw David, he disdained him; for he was but a youth, and ruddy, and withal of a fair countenance.
מג וַיֹּאמֶר הַפְּלִשְׁתִּי, אֶל-דָּוִד, הֲכֶלֶב אָנֹכִי, כִּי-אַתָּה בָא-אֵלַי בַּמַּקְלוֹת; וַיְקַלֵּל הַפְּלִשְׁתִּי אֶת-דָּוִד, בֵּאלֹהָיו. 43 And the Philistine said unto David: 'Am I a dog, that thou comest to me with staves?' And the Philistine cursed David by his god.
מד וַיֹּאמֶר הַפְּלִשְׁתִּי, אֶל-דָּוִד: לְכָה אֵלַי--וְאֶתְּנָה אֶת-בְּשָׂרְךָ, לְעוֹף הַשָּׁמַיִם וּלְבֶהֱמַת הַשָּׂדֶה. {ס} 44 And the Philistine said to David: 'Come to me, and I will give thy flesh unto the fowls of the air, and to the beasts of the field.'
מה וַיֹּאמֶר דָּוִד, אֶל-הַפְּלִשְׁתִּי, אַתָּה בָּא אֵלַי, בְּחֶרֶב וּבַחֲנִית וּבְכִידוֹן; וְאָנֹכִי בָא-אֵלֶיךָ, בְּשֵׁם יְהוָה צְבָאוֹת, אֱלֹהֵי מַעַרְכוֹת יִשְׂרָאֵל, אֲשֶׁר חֵרַפְתָּ. 45 Then said David to the Philistine: 'Thou comest to me with a sword, and with a spear, and with a javelin; but I come to thee in the name of the LORD of hosts, the God of the armies of Israel, whom thou hast taunted.
מו הַיּוֹם הַזֶּה יְסַגֶּרְךָ יְהוָה בְּיָדִי וְהִכִּיתִךָ, וַהֲסִרֹתִי אֶת-רֹאשְׁךָ מֵעָלֶיךָ, וְנָתַתִּי פֶּגֶר מַחֲנֵה פְלִשְׁתִּים הַיּוֹם הַזֶּה, לְעוֹף הַשָּׁמַיִם וּלְחַיַּת הָאָרֶץ; וְיֵדְעוּ, כָּל-הָאָרֶץ, כִּי יֵשׁ אֱלֹהִים, לְיִשְׂרָאֵל. 46 This day will the LORD deliver thee into my hand; and I will smite thee, and take thy head from off thee; and I will give the carcasses of the host of the Philistines this day unto the fowls of the air, and to the wild beasts of the earth; that all the earth may know that there is a God in Israel;
מז וְיֵדְעוּ כָּל-הַקָּהָל הַזֶּה, כִּי-לֹא בְּחֶרֶב וּבַחֲנִית יְהוֹשִׁיעַ יְהוָה: כִּי לַיהוָה הַמִּלְחָמָה, וְנָתַן אֶתְכֶם בְּיָדֵנוּ. {ס} 47 and that all this assembly may know that the LORD saveth not with sword and spear; for the battle is the LORD'S, and He will give you into our hand.' {S}
מח וְהָיָה כִּי-קָם הַפְּלִשְׁתִּי, וַיֵּלֶךְ וַיִּקְרַב לִקְרַאת דָּוִד; וַיְמַהֵר דָּוִד, וַיָּרָץ הַמַּעֲרָכָה לִקְרַאת הַפְּלִשְׁתִּי. 48 And it came to pass, when the Philistine arose, and came and drew nigh to meet David, that David hastened, and ran toward the army to meet the Philistine.
מט וַיִּשְׁלַח דָּוִד אֶת-יָדוֹ אֶל-הַכֶּלִי, וַיִּקַּח מִשָּׁם אֶבֶן וַיְקַלַּע, וַיַּךְ אֶת-הַפְּלִשְׁתִּי, אֶל-מִצְחוֹ; וַתִּטְבַּע הָאֶבֶן בְּמִצְחוֹ, וַיִּפֹּל עַל-פָּנָיו אָרְצָה. 49 And David put his hand in his bag, and took thence a stone, and slung it, and smote the Philistine in his forehead; and the stone sank into his forehead, and he fell upon his face to the earth.
נ וַיֶּחֱזַק דָּוִד מִן-הַפְּלִשְׁתִּי בַּקֶּלַע וּבָאֶבֶן, וַיַּךְ אֶת-הַפְּלִשְׁתִּי וַיְמִתֵהוּ; וְחֶרֶב, אֵין בְּיַד-דָּוִד. 50 So David prevailed over the Philistine with a sling and with a stone, and smote the Philistine, and slew him; but there was no sword in the hand of David.
Source: http://www.mechon-mamre.org/p/pt/pt08a17.htm