Mesilat Yesharim chapter 9.
והכלל להבחין בין שתי היראות הוא מה שחלקו חכמים זכרונם לברכה באמרם (פסחים ח, ב) היכא דשכיח היזיקא שאני. כי מקום שההיזק מצוי ונודע, יש להשמר. אך מקום שאין ההיזק נודע, אין לירא. ועל כיוצא בזה נאמר (חולין נו, ב) ריעותא דלא חזינן לא מחזקינן, ואין לו לחכם אלא מה שעיניו רואות. הוא עצמו ענין הפסוק שהבאנו למעלה "ערום ראה רעה ונסתר", הא אינו מדבר אלא בנסתר מן הרעה אשר רואה לא ממה שיוכל להיות שיהיה אפשרי שיבוא, והוא ממש ענין הפסוק שהזכרתי למעלה, אמר עצל שחל בדרך וגו'.
The general principle to distinguish between the two types of fear is as our sages specified saying: "where harm is likely it is different" (Pesachim 8b). For in a place where harm is likely and foreseeable it is proper to guard oneself. But in a place where there is no known danger one should not fear. On similar to this the sages said: "we do not assume any defect without cause" (Chulin 56b). And "a judge need be guided only by that which his eyes see" (Bava Batra 131a). This itself is the intent of the verse we brought earlier "the clever man sees the evil and hides himself" (Mishlei 22:3), which states only about escaping from the evil that one can see not of the evil which perhaps, possibly, may occur. This is precisely like the verse brought earlier "the lazy man says there is a lion in the way.." (Mishlei 26:13). Our sages, of blessed memory, illustrated, in successive degrees, to what extent unfounded fear can go to cause a person to refrain from good deeds. They said (Midrash Devarim Raba 8:6):
When damages are frequent you need to be afraid, not when it's rare.
The source is in Gemara Pesachim 8b.
But R`Eleazar said: Those sent [t perform] a religious duty do not suffer harm? - Where the injury is probable it is different, for it is said And Samuel said, How can I go?
if Saul hear it, he will kill me.
And the Lord said, Take a heifer with thee, etc.
An other source cited by the Mesilat Yesharim is in Chulin 56b. Bad thing that one doesn't see, are not considered as actual.