Since your comment says that you do want the Rabbinic reason why fur is allowed, the answer appears to be the Rashi on that pasuk. This prohibition applies only to the Festivals when the people enter the Temple and have to bring karbonos. We also see from the answer to Acceptable to receive transplant of organs grown inside pigs? that one is only forbidden to sell (pig as an example) that leads to it being eaten. Thus, fur for a hat is like pig skin for a football (see quote at the end of this post). Another example is Can I wear pigskin shoes? which also uses Rashi to show that it is allowed.
Also we see
Now, the above is true regarding the flesh of the animal. Animal hide,
however, does not carry impurities, especially when it is tanned.
This also includes the fur. Thus there is no impurity in the hat.
Rashi on Leviticus 11:8
and you shall not touch their carcasses: One might think that Israelites are prohibited to touch a carcass. Scripture, however,
says, “Say to the kohanim …[(a kohen) shall not defile himself for a
(dead) person among his people]” (Lev. 21:1); thus, kohanim are
prohibited [from defiling themselves by human corpses], but ordinary
Israelites are not prohibited. Now a kal vachomer can be made: Since
in the more stringent case of defilement by a human corpse, only
kohanim are prohibited, then in the more lenient case of defilement by
animal carcasses, how much more so [should only kohanim be prohibited!
If so,] what does Scripture mean by, “you shall not touch their
carcasses”? [It means that Israelites may not touch animal carcasses]
on the Festivals [since at those times they deal with holy sacrifices
and enter the Temple]. This is what [the Sages] said: A person is
obligated to cleanse himself on Festivals. - [R.H. 16b, Torath Kohanim
11:74]
As an example from Acceptable to receive transplant of organs grown inside pigs?
[Rashba][1] (Responsa: 3:223) writes that there is no universal
prohibition to benefit from pigs; only a rabbinic injunction against
engaging in commerce with them.
אם החזיר מותר בהנאה, במה שהתרת ליקח אותו בחובו? חדא: שהוא בעצמו אינו אסור בהנאה, דבר תורה, ואפילו לעשות בו סחורה, אלא מדבריהם
According to most authorities this limit on commerce is a rabbinic injunction. (Cf. Bartenura in Sheviit there, and Beit Yosef
YD 117). They specify that this prohibition only applies when the
commerce may lead to consumption of the food. (Cf. [Shach][2] YD
117:2). This would be another reason why the rabbinic injunction
against commerce would not apply to a transplant; the transplant would
not lead to consumption of pig.
Accordingly, even in cases where there is no danger to life, there
would be no prohibition on using pig organs.