In Sanhedrin 92a, there is some discussion on the value of giving out bread to others and leaving behind bread prior to saying the blessings after meals:
ואמר רבי אלעזר: כל הנותן פיתו למי שאין בו דעה - יסורין באין עליו
R. Eleazar said: Whoever gives of his bread to one who lacks knowledge will be assailed by suffering [...]
ואמר רבי אלעזר: כל שאינו משייר פת על שלחנו - אינו רואה סימן ברכה לעולם
R. Eleazar also said: He who leaves no bread on the table [at the end of his meal] will never see a sign of blessing
Shmuel Barzilai asserts that the custom of shirayim during a Shabbat tisch [chassidic gathering with a rebbe] comes from this gemara. In his work, Chassidic Ecstasy in Music, he records the custom and the spiritual significance of it:
The Sages also said: "Whoever does not partake of bread at his [the Tzaddik's] table has never seen a blessing in his lifetime." The Chassidim receive the Tzaddik's shirayim and what is left of the meal is shared among them, as the Tzaddik's act of eating is altruistic, and the Chassidim would almost give up their souls to receive shirayim from the Tzaddik's hands.
The Kabbalah endows mystical importance to these shirayim, as food contains "elevation of the sparks." The Besht said:
Why did the Holy One, Blessed be He, create man with the ability to become hungry and thirsty when He could have created a man able to live without food or drink? For it is said, 'their souls fainted in them' [Ps. 107:5]? Therefore, G-d made man hungry for food and thirsty for drink, in order for him to uplift the sparks, that is, the souls that have fainted in [becoming] the food, which is the meaning of 'their souls have fainted in them'.
The Chassidim viewed the shirayim as a meritorious 'charm' for filling them with heavenly awe, livelihood and good health. In some courts of Tzaddikim, a second table was set as soon as the first was complete. The second 'round' offered various delicacies in honor of the Shabbat. At this second setting, the Tzaddik would discuss issues with his Chassidim and relate Chassidic tales. The sense was one of "brethren sitting together" [Ps. 133:1]. This second setting was usually lengthy, lasting until well into the night, and was one of the most beautiful periods of the whole Shabbat. (Barzilai 89)
For an alternative, simpler explanation, Steven Bayme argues that the origin of shirayim developed from the belief that a tzaddik could perform wonders, so picking up his crumbs could "provide physical closeness and connection to the rebbe" (Understanding Jewish History 248). Similarly, in his Hakira article "Hasidism and the Rebbe/Tzaddik: The Power and Peril of Charismatic Leadership," Elijah Judah Schochet writes that:
Hasidim believed that
such edibles contain magical power as they had been touched and
sanctified by the Rebbe/Tzaddik. Shirayim were believed to be a segula
for the ill, and the verse “He will bless your bread” [Ex. 23:25] was interpreted
to mean that as a result of eating sanctified food “I (G-d) will remove
all disease from your midst.” [cf. Kalonymous Kalman HaLevi Epstein, Meor va’Shemesh, “Mishpatim.”]
And then there are other explanations for the custom, summarized by R. Ari Enkin and noted by user2083. I'm not familiar with the first recorded source of the custom, but regarding the reason for its performance on Shabbat and holidays, it is likely due to the fact that rebbes were particularly accessible to their chassidim during this time.