Based on: On the Mainline, the OU.
The History
To summarize, we see a negative portrayal of swaying during study in the 11th century in a poem of R. Shemuel HaNagid. [i] We find more positive explanations in the early 12th century in the Kuzari. The Zohar also references swaying during study. The controversies regarding dating this work would determine whether or not this is the earliest source.
By the later 12th century, we find Rishonim discussing swaying during prayer as well.
Reasons For Swaying While Learning
The Kuzari writes regarding swaying during study:
The Rabbi: It is said that it is done in order to arouse natural heat. My personal belief is that it stands in connection with the subject under discussion. As it often happened that many persons read at the same time, it was possible that ten or more read from one volume. This is the reason why our books are so large. Each of them was obliged to bend down in his turn in order to read a passage, and to turn back again. This resulted in a continual bending and sitting up, the book lying on the ground. This was one reason. Then it became a habit through constant seeing, observing and imitating, which is in man's nature. (Hartwig Hirschfeld's translation)
He gives two reasons. The first is one which people say, that it is to arouse heat in the body, I guess like a form of exercise. Alternatively, he means to arouse passion, to get into it. Then there is the reason preferred by the author, which is his suspicion that originally it was because people shared books and they moved in and our of the way so they could read and give others a chance to read and ultimately this became the habit and spread
The Zohar (Parashat Pinhas) also discusses swaying during study, and seems to connect it to the deep connection that Jews have to the Torah, and their resultant passion when studying:
We arose and went on our way, the sun becoming stronger and more oppressive. We saw some trees in the wilderness with water underneath, and we sat down in the shade of one of them. I asked him: How is it that of all peoples of the world, only the Jews sway to and fro when they study the Torah, a habit which seems to come natural to them, and they are unable to keep still? He replied: You have reminded me of a very deep idea which very few people know. He pondered for a moment and wept. Then he continued: Alas for mankind who go about like cattle without understanding. This thing alone is sufficient to distinguish the holy souls of Israel from the souls of heathen peoples. The souls of Israel have been hewn from the Holy Lamp, as is written, “The spirit of man is the lamp of the Lord” (Prov. xx, 27). Now once this lamp has been kindled from the supernal Torah, the light upon it never ceases for an instant, like the flame of a wick which is never still for an instant. So when an Israelite has said one word of the Torah, a light is kindled and he cannot keep still but sways to and fro like the flame of a wick. But the souls of heathens are like the burning of stubble, which gives no flame, and therefore they keep still like wood burning without a flame.’ Said R. Jose: ‘That is a good explanation; happy am I to have heard this.’ (Soncino translation.)
Some Rishonim connect it to the verse, “the people saw and trembled” (Ex. 20:15) such as Ba’al ha-turim there who comments, “Therefore we sway during Torah learning (limud ha-Torah), since the Torah was given with fear, terror, and shaking (b-aimah, reses, and zeiah).” Darkhei Moshe, commenting on Tur, (Hilchot Birkhat ha-Shahar 48:1), cites Abudirham,who applies this to the communal Torah reading as well. Cf. Rama, Shulhan Arukh ad loc.
Reason for Swaying While Praying
The Rishonim also discuss swaying during prayer [ii], many such as Sefer Hasidim (12th cent.) (57) explain this based on the verse "kol atzmotai tomarnah Hashem mi kamokha". Shibolei HaLeket, 17 cites Ma’aseh Merkava who writes this as well.
Rivash, quoted by Mekor Hesed on Sefer Hasidim, ibid. gives a parable that when one is drowning and jumping around in the water, people won’t make fun of him. So too, when one is davening and shaking himself, he is attempting to remove distractions. “When a man is drowning in a river and making many movements to remove himself from the water, surely those that are watching him will not laugh at him and his [strange] movements; so too, when one prays (k’she’mitpallel) and makes many [strange] movements, one should not laugh at him.”
R’ Schwab provides his own understanding of shukeling in R’ Schwab on Prayer (page 167):
“There are two ways in which a person can relate to Hakodosh Baruch Hu. One is through ahava (love), in which a person feels very close to Him, and the other is through yirah (awe), in which one is awestruck by His Omnipotence and Omniscience. This may explain the ancient Jewish practice of “shokeling,” swaying forward and backward during teffilah. The forward motion expresses one’s desire to come close to Hakodosh Baruch Hu, but then, upon reflection, one realizes that He is the Ribbono Shel Olam, the Master of the universe, which causes one to reel back in awe. These thoughts are typically evoked during meditation.”
Possible Motivations for Variations in Movement
For swaying while studying, according to the first reason in the Kuzari, the motion is intended to excite oneself, so this would explain more vigorous movement that also incorporates hand movement. The same would apply to the Zohar which also seems to connect it with excitement. This would also make sense according to the Abudirham and others who connect the movement to trembling in awe of the Torah.
For swaying while praying: Sefer Hassidim is explicit that the entire body shakes in accordance with the verse: All my bones, etc. this would include the hands as well as the body.
Interestingly, Regarding A few people squint their eyes and some rub their hands while shuckling
Besides for being explainable based on the aforementioned Kuzari and Zohar for study, and Sefer Hassidim and Shibbolei HaLeket for prayer in general, there is also a history of gesticulations for Shema in particular. See: http://seforim.blogspot.com/2011/08/head-movements-of-shema.html.
The Talmud (Berakhot 13b) states:
Symmachus says: “Whoever prolongs the word ehad (“one”), his days and years are prolonged.
Said Rav Aha bar Ya’akov: “And [specifically] the letter dalet [of ehad].”
Said Rav Ashi: “Provided he does not speed up the letter het [of ehad].”
R. Yirmiyah was sitting before R. [Hiyya bar Abba]. He saw that he was prolonging overly much. He said to him: “Once you have proclaimed Him King above and below, and to the four winds of heaven, you need not any further.
Rabbi Menahem Ha-Me’iri of Perpignan, Provence (1249-1306) finds an added dimension: “The amount of lengthening the letter dalet is that required to picture in the heart that He, blessed be He, rules over heaven and earth and the four winds of the world. And for this reason, it is customary to tilt the head and move it to these sides. Nevertheless, if one prefers not to tilt the head, one need not, because the thing depends not on the tilting of the head and its movements, but rather upon the feeling of the heart.”
These sorts of motions are attributed to R. Hai Gaon as well.
[i] In this poem he is criticizing the degeneration of Torah study...As an example of their lack of orderliness he mentions their shokeling:
והנה רב ותלמידים מנידים לראשיהם כערער בערבה
The teacher and students were bobbing their head like a tamarisk in the wilderness."
This then is the first incidental mention of shokeling, at least in Jewish sources.
[ii] Other sources are in favor of not shaking at all during prayer due to the awe one should have while praying particularly during the amidah:
These include Ba’al Shem Tov, quoted by Mekor Hesed, ibid, Perush Azulai on Sefer Hasidim, ibid who in turn cites Rabbi Yisroel Saruk and his student Ramah, and the Shlah (quote found in Netiv Binah, vol. 1, pg. 96). This was also the practice of the Arizal and German Jewry as noted by Rabbi Dr. Joseph Breuer zt”l, in an essay entitled “Our Way,” (A Unique Perspective, pg. 384). Most later authorities write that one should do whatever is best for his concentration. These include Arukh Hashulhan and Mishnah Berurah