There is more than one answer to this question. But here's one, albeit a long one.
According to the Sefer Yetzirah (02:03), the letters of the Hebrew alphabet can be grouped in five categories according to the part of the mouth used to form them. Rashi often seems to support a theory something like this one, as do the sages and other commentators, frequently suggesting letter-exchanges consistent with the categories of the Sefer Yetzirah in dozens of places. Some of these are sorted into categories below.
In my opinion a natural but doubtful assumption of the modern reader is that these rabbis were doing some sort of etymology. That is, a modern reader might take word similarities as evidence that at some earlier time the language was different and developed gradually into the language we know. But attributing that assumption to the rabbis seems unjustified. Some of these examples below are more naturalistic, comparing words similar in both meaning and sound as someone might do in any language; but that does not point clearly to an earlier state of the language. And some examples feel more Masoretic, comparing words that do not seem obviously related, apparently intended in a different spirit or predicated on the belief that Hebrew, and in particular the Hebrew of scriptures, is special. So that's a theoretical problem that's hard to deal with.
I use ArtScroll books a lot, and these notes reflect two ArtScroll-isms. First, I use the pound symbol to indicate footnotes in ArtScroll, either the note itself or the place in the body where the note is indicated. Second, I use the longer page numbers of ArtScroll gemara, where 24b2 means the standard page 24b, but page 2 of ArtScrolls printing of that page. If you're not using ArtScroll editions, or if you're using an ArtScroll edition inconsistent in these details, then just ignore the #3-style elements and the digit following the letter in page numbers; the rest of the citation might still be accurate.
In order to illustrate the system I wanted to divide examples by category. But in fact, when the system is itself the point of discussion, examples may be drawn from all categories. For example when the Ramban (Gen-41:47 Ramban) wants to explain this system, he presents many examples of various substitutions, diverse but consistent with the categories. Koomtza and Goomatz mean "pit" although one is spelled with a Koof and one a Gimmel; Mazeg and Mascha mean "mixed", although Zayin-Gimmel and Samech-Chaf; Kana and Gana mean "garden" although one Chaf and one Gimmel; Vayisachroo and Vayisgeroo mean "they closed" although one Chaf and one Gimmel; Mechurasam and Megoiroisayich refer to dwellings although one Chaf and one Gimmel; Koiva and Choiva mean "helmet" although one Koof and one Chaf; Tikein means "establish," and Tachshit means "ornament," whether spelled Koof or Chaf; and Vayatziku and Vayatigu mean "to set or assign" although spelled Koof or Gimmel.
Now. The categories.
1. Letters of the Throat: Aleph, Hei, Ches, Ayin
- Aleph & Hei: The comments of the Ba'al Ha Turim on Koof-Reish-(Aleph or Hei) might belong at (Ex-05:03 BHT+) (Aleph) where ArtScroll printed them, or at (Ex-03:18) (Hei). ArtScroll notes that either word can mean "Happened or happened upon."
- Aleph & Ayin: (Num-14:44 Rashi #4) They were "intransigent" (Vayapilu, with Ayin), related to "Darkness" (Oifel, with Aleph)
- Aleph & Ayin: The gemara (Megillah-24b2 #15} remarks that people who confuse Aleph and Ayin should not lead prayers. Rashi notes that when the gemara (Berachos-32a} comments that R' Eliezer Ben Ya'akov read one of these letters as the other, it means that he did this for the purpose of exposition.
- Aleph & Ayin: (Chullin-060b3 #29} They are called Avim (with an Aleph) because they desired (with an Ayin) many goods.
- Aleph & Ayin: (Aleph or Ayin)-Lamed (for El or Al) Al (Gen-37:35 Rashi); (Ex-40:03 Ramban); (Num-14:14); (Berachos-31b4 to 32a1}. Ayin-Lamed where Aleph-Lamed expected (2 Kings-22:08)
- Hei & Ches: Hespit (with Hei) construed (Berachos-57a1 #4} as Chasu Pedavitu (with Ches)
- Hei & Ches: "Holy" (with Hei) construed as "Deconsecrate" (with Ches) (Berachos-35a1 #13}, (Shabbos-105a1 #5}
- Hei & Ches: (Pesachim-004a Rashi}, (Megillah-12a3 etc}, (Megillah-15a2 #23}, (Sanhedrin-039b2 #19}
- Hei & Ches (Gen-06:16 Rashi), (Num-05:19 Rashi #9), (Num-26:13 Rashi #2)
- Hei & Ches (Avodah Zarah-19a4 #41} Hevel (with Hei, 'vanity') & Chevel (with Ches, 'bundles')
Hei & Ches, Shin & Sin (Sanhedrin-070a4 #44}, (Psalms-104:15}
Ches & Ayin (Deut-02:23 Ramban), cf. Rashbam here. Five or six examples. {MI-Demai-03:02 ArtScroll p95}
2. Letters of the Palate: Gimmel, Yud, Kaf, Chaf, Koof.
- Gimmel & Kaf: Loi Seileich Rachil="You shall not go about spying" (Lev-19:16 Rashi), although Rachil is spelled with a Chaf, is related to expressions of Rageil (with a Gimmel) such as Hoilchim Lerageil (Judges-18:14) or Loi Rageil Al Lashoinoi (Psalms-15:03).
- Gimmel & Kaf: (Isaiah-19:04 Rashi sv V'sikarti)
Gimmel & Kaf & Koof: (Ex-28:31 Ramban #89). Migbas and Mikbas mean "hat," although one is spelled with a Gimmel and one with a Koof.
- (Psalms-63:12 Metzudas Tsion sv Yisochar)
- (Malachi-03:03 Metzudas Tsion sv V'zikat)
3. Letters of the Tongue: Dalet, Tes, Lamed, Nun, Tav.
- Dalet & Lamed: Vayintsloo ("so they emptied") Egypt (Ex-12:36) explained: they made Egypt a Metsudah ("a trap") (Berachos-09b1 #10 Tos}, although Tsadi is followed by Lamed in Vayintsloo and by Dalet in Metsudah. Cf. (Pesachim-0119a Tos sv Kimetsudah}
- Tes & Tav: (Ex-32:15 Rashi)
- Lamed & Nun: (Gen-25:04 Rashi sv Ooletushim), (Chagigah-12b Rashi sv Aliyas agalim}
- Lamed & Nun: Avel for Aven (1 Samuel-06:18 Rashi), Nishkan for Lishkan (Nehemiah-13:07)
- Lamed and Nun: Letushim are so-called because they are Netushim = "scattered," and Lamed and Nun interchange (Gen-25:03 Rashi).
4. Letters of the Teeth: Zayin, Samech, Tzadi, Reish, Sin.
- Zayin & Samech: (Ex-28:28 Ramban) relates Yud-Zayin-Ches here to Yud-Samech-Ches of (Proverbs-15:25}; Ayin-Lamed-Zayin (Psalms-149:05) to Ayin-Lamed-Samech (Proverbs-07:18); Nun-Sav-Tzadi (Deut-12:03) to Nun-Sav-Samech (Job-30:13); and Nun-Samech-Ches (2 Kings-11:06) to Nun-Zayin-Ches (Psalms-109:18 to 19), (Isaiah-23:10)
- Zayin & Tzadi: Rashi explains (Lev-13:37 Rashi sv V'seior shochoir) the word Tzadi-Hei-Vav-Veis (of the previous passage) in terms of Zayin-Hei-Vav-Veis; Cf. (Sotah-36b4 sv V'yafozu}.
- Zayin & Tzadi: (Gen-49:24 #6) relates Vayafoizu (with Zayin) to Vayafotzu (with Tzadi). In the note here Artscroll lists many more examples.
- Samech & Tzadi: Isstinas with a Samech connected to Tsinas with Tzadi. Such substitutions are common. {ZZ-Introduction to the Mishnah Berachos 02:06 = 16b}
Samech & Tzadi: The Ramban (Gen-39:20 Ramban #64) compares Samech-Hei-Reish ("round chamber" of the present verse) to Tzadi-Hei-Reish ("window" of Gen-6:16).
- Samech & Shin: ArtScroll compares Shin-Ches-Tes to Samech-Ches-Tes (Gen-41:08)
- Samech & Sin: (Megillah-13a2 #21}, (Yevamos-121b3 #24}
- Samech & Sin: (Ex-19:04 Rashi), (Deut-33:03 Rashi #7), and Onkelos on both verses, compare Yud-Sin-Aleph to Yud-Samech-Ayin.
- Samech & Sin: (Gen-24:33 Rashi & Targum Yonasan), (Gen-30:14 Rashi); (Ex-25:05 Rashi), (Shabbos-028}; (Ex-25:31}; (Ex-31:10 Rashi) compares a Hebrew word (with Sin) to an Aram word (with Samech); (Lev-26:13 BHT); {ME-Gen-02:21 Rabbah 17:06); (Deut-32:15 Rashi)
Samech & Sin: (Isaiah-55:13) = (Megillah-10b4}
Samech & Sin: (Ex-35:12 Rashi #4), (Deut-26:19 BHT+), (Deut-32:02 Rashi #5), (Deut-32:15 Rashi), (Deut-33:03 Rashi #7), (Ezra-04:05 Rashi)
- Samech & Sin: Connection in "Satan," see (Lev-26:13 BHT+)
- Samech & Sin: (Lev-02:14 Rashi #13) compares Geres (with Sin) to words with Samech at (Lamentations-03:16), (Psalms-119:20). ArtScroll further compares Seitim with Sin (Hosea-05:02) or Samech (Psalms-101:03), Mikas with Sin (Psalms-06:08) or Samech (Job-17:07), Soichrim with Sin (TN-2 Chronicles-24:12) or Samech (Ezra-04:05).
Samech & Sin: Basar (flesh) is explained as an acronym. The second letter is explained first as Sruchah (foul), then as Sheol (grave). Basar has a Sin; Sruchah Samech; Sheol Shin.
- Samech & Sin: Ches-Reish-Samech to Ches-Reish-Sin (Deut-28:27 Rashi)
- Samech & Sin: Geres (Avodah Zarah-19a3}
- Samech & Sin: Reish-Mem-(Samech or Sin), (Gen-01:24 Ramban)
- Samech & Sin: (Gen-26:20 Rashi #1), (Bava Metzia-14a sv Asikin}
- Samech in mnemonic recalls a word actually spelled Sin (Pesachim-42b2 #26}
- Tzadi & Sin: Sin-Fei-Nun connected to Tzadi-Fei-Nun (Deut-33:19 Rashi)
- Tzadi & Sin: (Ex-25:29 Ramban) suggests Kesavos (with Sin) is connected to Ketzavos (with Tzadi), interchanging like Shok & Tzok
- Tzadi & Sin: Tzadi-Ches-Koof to Sin-Ches-Koof (Gen-17:17 Ramban), (Gen-18:15 Ramban), (Lev-26:43 BHT)
- Shin & Sin, Hei & Ches: (Sanhedrin-070a4 #44}, (Psalms-104:15)
Sin (Num-11:08 Rashi).
- Samech & Sin: (Lev-2:14 Rashi #13) compares Geres ("pulverized," with Sin, (Lev-2:14)) to Gorsa ("shattered," with Samech, (Psalms-119:20), cf. (Lamentations-3:16)). ArtScroll further compares Seitim="wayward" with Sin (Hosea-5:02) or Samech (Psalms-101:03), Mikas="from anger" with Sin (Psalms-6:08) or Samech (Job-17:07), Soichrim="they hired" with Sin (TN-2 Chronicles-24:12) or Samech (Ezra-4:05). See also (Avodah Zarah-19a3).
5. Letters of the Lips: Beis, Veis, Vav, Mem, Pei, Fei.
- Beis & Pei: (Daniel-11:24 Rashi sv Yivzor); (Menachos-027b Rashi sv L'miutei derech meshofash}
- Veis and Fey: Hevkeir with a Veis equals Hefker with a Fey. (Eduyos-Mishnah 3 6a7 #1}
- Veis and Vav (Gen-30:20 Ramban), bringing many examples
Veis and Fei (Ex-15:10 Ramban); (Lev-19:20 Ramban); (Deut-07:12 Ramban (at end)).
(Psalms-68:31 Rashi & Radak sv Bizar)
Also associated in grammar, cf. (Ex-23:25 BHT+ #164).
P.S.
@Reuben Bakst, who asked the original question, responded to this answer with a few further questions about the status of Vav, which is pronounced sometimes like Veis and sometimes otherwise.
Today, in looking at the beginning of book two of Tanya, I noticed the following.
The Ba'al Ha Tanya mentions the two sentences (Shema Yisroel and Baruch Shem) said during the recital of the Shema. The editor (of my edition of Tanya) comments there that "va'ed equals echad through the substitution of letters, as stated in the Zohar." That remark bears this footnote:
"Zohar II, 134a. Hebrew grammar classifies the letters of the alphabet according to their syntactic functions, their respective sources in the organs of speech, and so on. Within each group, the letters are interchangeable. The letters aleph and vav both belong to the group of "connective letters" (otiyot hahemshech) and may thus be interchanged. The letters chet and ayin fall into the category of guttural letters (otiot groniyot) and may likewise be interchanged. Hence, echad is the equivalent of va'ed."
So whatever else that may mean, it suggests to me that although the Vav of Va'ed is a consonant, it can be classed with Aleph rather than with Veis.
I've never actually looked at the Zohar; I simply copied the citation. I also notice, looking at this 6-year-old answer at the Stack Exchange, that it now collapses into chaos at the end.
So, as they say, Tzarich Iyun.