Why is it that when people make kiddush/havdallah they insist on filling up the cup all the way to the top, sometimes till it overflows just a little? Is this a documented minhag/custom etc...?
The Ram"a brings down in Shulchan Aruch, Orach Chayim 296:1 that two customs exist entailing spilling out wine at havdala:
- Spilling during the b'racha on the wine to symbolize having an abundance of wine that is overflowing onto the floor
- Spilling after havdala to extinguish the flame and "wash [one]'s eyes" to show his love for the mitzva
It could be that overfilling automatically achieves the first one.
The Rema writes (Orach Chaim 183:2) that the cup used for a berachah (the context is talking about the cup of wine for birkas hamazon, but the same applies to kiddush, etc.) should be מלא על כל גדותיו, filled to the brim.
Though Taz (ibid. :4, end) points out that it is (or was, in his times) quite common to leave a space at the top. He questions the propriety of this, but concludes that perhaps it's okay in order to prevent wine from spilling on the food and ruining it.
See the following article, quoting Rav Tuvia Goldstein, "Remember what is minhag and what is Torah Law," with respect to overdoing the wine-spilling thing. Wasting wine unnecessarily is a violation of Bal Tashchit.
This was in reference to the custom of spilling a bit of wine during mention of the ten plagues. I wonder if it is also applicable to the minhag quoted by the OP.