Kanaf applies distinctly to the edge of clothing because its nature is that of the outer, lighter part of a bird's wing. And it is precisely the [mobility and] coverability of that body part that defines it, in contrast to ever/evra, which is the whole of the wing, or agaf, which is the forewing right where it attaches to the body. Malbim puts a fine point on "kanaf" in T'hilim 91:4.
בְּאֶבְרָתוֹ יָסֶךְ לָךְ וְתַחַת כְּנָפָיו תֶּחְסֶה
With his wing he surrounds you and under his wingfeathers you huddle
ויש הבדל בין אבר ובין כנף, שהכנף הם הנוצות הארוכות המחוברות בקצה האבר המעופף שנקרא אבר, והעוף יסך ויגן ע"י האבר בצדדי אפרוחיו ויפרוש הכנפים שהם הנוצות למעלה למחסה עליהם
And there is a difference between "ever" and "kanaf", for the "kanaf" is the long feathers attached to the end of the flying limb which is called "ever", and the bird surrounds and protects with the wing to the sides of its chicks and spreads out its k'nafayim, which are the feathers, to cover them from above.
For more on wing terms in Tana"ch see באור שמות הרדפים שבתנ"כ pp. 9-11.
This shows why this term wouldn't be fitting to carry over to a face or land, but it doesn't explain why pe'a couldn't be used for clothes.
Klein interestingly lists several cognates for פאה, concluding with the assertion that "All these words prob. meant originally ‘part cut off’". If it were so it would fit perfectly with the facial and fieldial examples you gave, both of which deal with (not) cutting off the edge of the object under consideration.