R Howard Jachter wrote here a long article on the topic -- find a summary below with light edits. To immediately answer the question of those who rule more strictly he cites R Tzvi Pesach Frank, R Mordechai Yaakov Breish & R Yitzchak Yaakov Weiss and those who recommend that one to be strict if possible: Rav Yosef Henkin and Rav Ovadia Yosef.
Rav Shlomo Zalman Auerbach rules that it is entirely permissible to open a fridge door when the motor is not running. In fact, he writes that it is not right to be strict on this matter, as it will lead one to limit his Oneg Shabbat, enjoyment of Shabbat.
His reasoning is based on the fact that opening up the door will not immediately lead to turning on the motor. The inevitable time delay between the opening of the door and causing the motor to go on leads Rav Shlomo Zalman to classify this as a "Grama" - "Koach Sheini" (indirect action).
Now the Rama (334:22 and see Biur Halacha ad. locum. s.v. D'Gram Kibui) rules that Grama is permitted only in situations of great need. Rav Shlomo Zalman asserts, however, that since one's intention is merely to open the door and not to turn on the refrigerator's motor, Grama would be permissible in all situations even absent any unconventional needs. Moreover, he writes that since he is only causing the motor to go on earlier than it would have gone on without his opening the refrigerator door, (also see Minchat Shlomo 91:10) one may treat the act of opening of the door even more leniently than Grama. Thus, opening the refrigerator door would be permissible in all situations.
Many poskim concur with Rav Shlomo Zalman's lenient approach. Indeed, Rav Aharon Lichtenstein told this author that Rav Soloveitchik agreed with the lenient approach. Rav Moshe Feinstein (Igrot Moshe O.C IV: 74- Bishul -28) seems to fully accept Rav Shlomo Zalman's ruling (also see Igrot Moshe O.C. 2:68)- as does Rav Eliezer Waldenburg (Tzitz Eliezer 8:12 and 12:92). (See Encyclopedia Talmudit 18:663 note 13.)
Many eminent authorities, on the other hand, either rule strictly (Har Zvi O.C. I:151; Chelkat Yaakov 3:179; and Minchat Yitzchak 2:16) or at the least recommend that one to be strict if possible (Rav Yosef Henkin, Eidut L'Yisrael p. 122; Rav Ovadia Yosef, Yabia Omer I:O.C. 27). The problem is that once an action is performed routinely it cannot be classified merely as Grama (see Shabbat 120b and Rabbeinu Chananeil ad. loc. s.v. Rav Ashi and Bava Kama 60a and Rosh Bava Kama 6:11). Rav Shlomo Zalman responds that this applies only when one intends to create the resultant action. When opening the door one does not intend to turn on the motor.