In different places, it is explained that lentils are served as "mourners food".
For example, Rashi on Genesis 25:30 writes (citing the Gemara in Bava Batra 16b):
Jacob was boiling lentils to provide the customary first meal for the immediate mourners. Why should lentils be the mourner’s food? Because they are round like a wheel and mourning (sorrow) is a wheel that revolves in the world (it touches everyone sooner or later as a revolving wheel touches every spot in turn) (Bava Batra 16b). And a further reason is: just as lentils have no mouth — (the word mouth פה is used in Hebrew of a serrated edge) — so, too, mourners have no mouth (appear dumb), for speech (greeting others) is forbidden to them.
Furthermore, the Gemara (Talmud) (Eruvin 81a) explains Ezekiel 4:9 as "However, bread prepared from lentils alone is edible".
Lentils, in combination with for example bread, was used as a proof that a certain sale had taken place. This can be found in the commentary of the Chizkuni on the story of Yaakov and Esav, like you mentioned:
'ויעקב נתן לעשו וגו, meanwhile Yaakov had already given to Esau, etc.;” at the same time when Esau had paid him money for the birthright, Yaakov had already fed him bread and a dish of lentils as proof that the sale had taken place. This was a normal procedure when commercial transactions took place in those days. We find an additional example of this in Genesis 31,46, when Yaakov and Lavan conclude a peace treaty by the stone monument.
This can also be found in the commentary of the Radak:
and that the dish of lentils and the bread he gave him and which they both ate together was only to seal the bargain between them, a custom to which we have already referred
A more kabbalistic approach on the choice of using lentils in the dish, can be found in the commentary of the Zohar on this verse:
And the heart replies, 'Give me the first and choicest of whatever you swallow, give me your birthright,' this is the meaning of "Sell me this day your birthright" (Ibid. 31). That is, swear by your desire. As the heart contemplates food, the liver swallows. If it were not for the heart, longing and thinking about food, the liver and other organs would not be able to swallow. As Rabbi Yosi said, This is the way of slaves, who do not to eat before their master. Rabbi Yosi said, It is later written, "Then Jacob gave Esau bread and pottage of lentils" (Beresheet 25:34). What are these lentils? They are round as a circle, and as the circle which revolves around the world does not deviate from its path, so man in that time will never deviate from his. Although there will yet be all that is good and precious and perfect, with all that, the worldly habit of eating and drinking will not change.
In Berachos 40a, it is said that lentils can prevent diphtheria from afflicting a house. However, this dish should not be eaten every single day:
Rav Mari said that Rabbi Yoḥanan said: One who is accustomed to eat lentils once in thirty days prevents diphtheria from afflicting his house. The Gemara comments: However, one should not eat lentils every day. What is the reason? Because it is deleterious in that it causes bad breath.