During a recorded shiur I heard recently, the Rav quoted a Chazal saying, "The ותיקי ירושלים wouldn't eat a meal with others unless they knew the other guests so that they wouldn't join in a meal with an individual who wasn't fit to eat with - אינו הגון." After searching all over, I still can't find this statement - where is the source of this Chazal, or anything similar to it?
The source is a beraita quoted in Sanhedrin 23a:
תניא נמי הכי כך היו נקיי הדעת שבירושלים עושין לא היו חותמין על השטר אלא א"כ יודעין מי חותם עמהן ולא היו יושבין בדין אלא א"כ יודעין מי יושב עמהן ולא היו נכנסין בסעודה אלא אם כן יודעין מי מיסב עמהן:
That is also taught: This is what the scrupulous people of Jerusalem would do: They would not sign a document unless they knew who was signing with them, and they would not sit in judgment unless they knew who was sitting with them, and they would not join a meal unless they knew who was reclining with them.
JoelK gave the source specifically mentioning the נקיי הדעת. But as the OP asked for sources, it's worth noting the gemara in brachos 43b which adds a reason to avoid eating with "unfit people":
תָּנוּ רַבָּנַן: שִׁשָּׁה דְּבָרִים גְּנַאי לוֹ לְתַלְמִיד חָכָם...וְאַל יָסֵב בַּחֲבוּרָה שֶׁל עַמֵּי הָאָרֶץ: מַאי טַעְמָא? — דִּילְמָא אָתֵי לְאִמְּשׁוֹכֵי בָּתְרַיְיהוּ
"Six things are a "genai" (disgrace, degradation) for a talmid chacham... He shouldn't have a meal with ignoramuses. What's the reason? Lest he be drawn after them."
This halacha is codified in the Rambam in Hilchos Deos 5:2