In Shulchan Orach Siman 124 Sif 7 the machaber speaks about how bad it is to speak during Chazaras HaShatz (when the Shliach Tzibbur is repeating Shomeneh Esrei out loud.) The Shulchan Aruch HaRav over there in Siman 124 in Sif 10 writes that someone who talks in a shul when the time the tzibbur is "oskin b'shivcho shel makom" (involved in the praises of HaShem) that he makes himself as if he doesn't have "chelek" (portion) in Elokei Yisrael (The G-D of Israel.) His makor is from the Zohar in Parshas Trumah (Daf 131b.) As well the Shulchan Aurch in Orach Chaim Siman 151 Sif 1 writes that "sicha butteleh" (idle speeach) is ossur to speak in a Beis HaKeneses or Beis HaMedresh. This is talking about in general, how much more so when the Tzibbur is davening from what we see above. See too what the M"B writes there in Sif-Katan 2.
With this small introduction my question is as follows. Does it say any where b'farush that it's "ossur" to speak in the middle of Kaddish. Let's say it's a person who is not davening with the tzibbur at all who comes into the shul for a different reason. Or someone who plans on davening with the tzibbur but hasn't started yet and is l'maseh able to speak still but is speaking now during Kaddish. Where does it say that this is ossur or where do the poskim learn it out from? The makoros that I provided above would seemingly suffice I think but I'm looking to see where there is more.
