R Howard Jachter wrote [here][1] a long article on the topic -- find a summary below with light edits. To immediately answer the question of those who rule more strictly he cites R Tzvi Pesach Frank, R Mordechai Yaakov Breish & R Yitzchak Yaakov Weiss and those who recommend that one to be strict if possible: Rav Yosef Henkin and Rav Ovadia Yosef.

> Rav Shlomo Zalman Auerbach rules that it is entirely permissible to
> open a fridge door when the motor is not running.  In fact, he writes
> that it is not right to be strict on this matter, as it will lead one
> to limit his Oneg Shabbat, enjoyment of Shabbat.
> 
> His reasoning is based on the fact that opening up the door will not
> immediately lead to turning on the motor. The inevitable time delay
> between the opening of the door and causing the motor to go on leads
> Rav Shlomo Zalman to classify this as a "Grama" - "Koach Sheini"
> (indirect action).
> 
> Now the Rama (334:22 and see Biur Halacha ad. locum. s.v. D'Gram
> Kibui) rules that Grama is permitted only in situations of great need.
> Rav Shlomo Zalman asserts, however, that since one's intention is
> merely to open the door and not to turn on the refrigerator's motor,
> Grama would be permissible in all situations even absent any
> unconventional needs.  Moreover, he writes that since he is only
> causing the motor to go on earlier than it would have gone on without
> his opening the refrigerator door, (also see Minchat Shlomo 91:10) one
> may treat the act of opening of the door even more leniently than
> Grama. Thus, opening the refrigerator door would be permissible in all
> situations.
> 
> Many poskim concur with Rav Shlomo Zalman's lenient approach. Indeed,
> Rav Aharon Lichtenstein told this author that Rav Soloveitchik agreed
> with the lenient approach. Rav Moshe Feinstein (Igrot Moshe O.C IV:
> 74- Bishul -28) seems to fully accept Rav Shlomo Zalman's ruling (also
> see Igrot Moshe O.C. 2:68)- as does Rav Eliezer Waldenburg (Tzitz
> Eliezer 8:12 and 12:92). (See Encyclopedia Talmudit 18:663 note 13.)
> 
> Many eminent authorities, on the other hand, either rule strictly (Har
> Zvi O.C. I:151; Chelkat Yaakov 3:179; and Minchat Yitzchak 2:16) or at
> the least recommend that one to be strict if possible (Rav Yosef
> Henkin, Eidut L'Yisrael p. 122; Rav Ovadia Yosef, Yabia Omer I:O.C.
> 27).  The problem is that once an action is performed routinely it
> cannot be classified merely as  Grama (see Shabbat 120b and Rabbeinu
> Chananeil ad. loc. s.v. Rav Ashi and Bava Kama 60a and Rosh Bava Kama
> 6:11). Rav Shlomo Zalman responds that this applies only when one
> intends to create the resultant action. When opening the door one does
> not intend to turn on the motor.

R Yair Hoffman writes [here][2] that the answer changes over time as refrigerator technology evolves and brings additional views of whose who disagreed with RSZA

>  - Rav Eliyahu Henkin, zt’l, ruled that one can only open the
> refrigerator when the compressor is on and it is forbidden to do so
> when the compressor is off
 - This was also the view of Rav Elyashiv,
> zt’l, as cited in Mitbach K’halacha, page 303
 - The Chazon Ish ruled
> that it is forbidden to open the refrigerator at all, whether the
> compressor is on or off. (Cited in HaPardes, Cheshvan 5719/1958)

He writes that issues are actually becoming more complicated with modern refrigerators and brings additional poskim being concerned

>  - Rav Hershel Schachter holds that recording the information on a computer [as is done in modern fridges] would under many circumstances
> at least be considered ksiva miderabanan. He also feels that there may
> be issues of davar shemiskaven–the person intends for it to happen
> since it is beneficial for him–even on a rabbinic violation.
> 
>  - Rabbi Eli Gersten of the OU stated that Rav Yisroel Belsky pushed to have timers on refrigerators. He told Rabbi Gersten that his
> grandfather, Rav Wilhelm, used to sell timers in his hardware
> store–they called it “the Rav Henkin switch.” Rabbi Gersten also
> stated that Rav Belsky used the language of “chashashos,” concerns, of
> a Torah prohibition. Rabbi Gersten further stated that when one student from Yeshiva Torah
> Vodaath (who accompanied Rav Belsky at the OU on Thursdays when he
> came to rule on questions) expressed grave concerns after purchasing a
> new refrigerator, he said, “Look, until a timer comes out, just open
> it with a shinui. When the timer comes out, purchase it.”

See also [here][3] from Star-K and [here][4] for additional concerns.


  [1]: https://www.koltorah.org/halachah/opening-refrigerators-on-shabbat-by-rabbi-howard-jachter?rq=%22Opening%20Refrigerators%20on%20Shabbat
  [2]: http://www.5tjt.com/is-your-fridge-kosher/
  [3]: https://www.star-k.org/articles/articles/kosher-appliances/464/keeping-your-cool/
  [4]: http://www.shabes.net/shabat-files/alonim/mekarerim-eng.pdf