I don’t see it being called Minhag Yisroel but the source for the practice in general is from Yehoshua chapter 2.
The red thread or הוט שני is the Sign of truth which was given to the prostitute Rachav to indicate her home and all contained within it would be safe from the tumult of conquest by the children of Israel in Eretz Yisroel. The practice was to indicate the non-Jew who protects Jews from danger.
The thread was hung in the window of the home to be visible to the outside. In many ways, it is reminiscent of the lambs blood smeared on the doorposts of the Jewish homes in Egypt at the first Passover.
Apparently, there are some Jews who employ this red thread as a similar mark of protection, like Rachav the prostitute. And this is like is explained by the "Aish Rabbi" who says:
As a matter of fact, the red thread (“roiteh bendel” in Yiddish) is one of those Jewish practices which has very little Jewish about it. The peddlers of red threads today claim that they come from a string wrapped around Rachel’s Tomb seven times and that they possess Kabbalistic powers, in particular warding off the Evil Eye. And in fact, there are rabbis who attest that similar customs has been practice going back several generations. (Many tie a red string to a baby’s crib for protection.) Yet there is no known early source which mentions or condones the practice. If it has any Kabbalistic powers, there is no work on Kabbalah which makes any mention of it.
However, it is surprising that the Aish Rabbi fails to recognize the red thread practice mentioned in Sefer Yehoshuah.
This connection between the red thread, Rachav the prostitute, Yehoshuah and several of the prophets of Israel is discussed in Megillah 14b which says:
Rav Naḥman said: Huldah was a descendant of Joshua. An allusion to this is written here: “Huldah the prophetess, the wife of Shallum, the son of Tikvah, the son of Harhas [ḥarḥas]” (II Kings 22:14), and it says elsewhere with regard to Joshua: “And they buried him in the border of his inheritance in Timnath-heres [ḥeres]” (Judges 2:9), therefore intimating that there is a certain connection between them. Rav Eina the Elder raised an objection from a baraita to Rav Naḥman’s teaching. The baraita indicates that Huldah was in fact a descendant of Rahab, and seemingly not of Joshua: Eight prophets, who were also priests, descended from Rahab the prostitute, and they are: Neriah; his son Baruch; Seraiah; Mahseiah; Jeremiah; his father, Hilkiah; Jeremiah’s cousin Hanamel; and Hanamel’s father, Shallum. Rabbi Yehuda said: So too, Huldah the prophetess was a descendant of Rahab the prostitute, as it is written here with regard to Huldah: “The son of Tikvah,” and it is written elsewhere in reference to Rahab’s escape from the destruction of Jericho: “This cord of [tikvat] scarlet thread” (Joshua 2:18). Rav Naḥman responded to Eina the Elder and said to him: Eina the Elder, and some say that he said to him: Blackened pot, i.e., my colleague in Torah, who has toiled and blackened his face in Torah study, from me and from you the matter may be concluded, i.e., the explanation lies in a combination of our two statements. For Rahab converted and married Joshua, and therefore Huldah descended from both Joshua and Rahab. The Gemara raises a difficulty: But did Joshua have any descendants? But isn’t it written in the genealogical list of the tribe of Ephraim: “Nun his son, Joshua his son” (I Chronicles 7:27)? The listing does not continue any further, implying that Joshua had no sons. The Gemara answers: Indeed, he did not have sons, but he did have daughters.
This same theme of the red thread is also mention in Shir HaShirim 4:3 in describing the beauty of G-d’s beloved.
