One 18th century source for this is the Noda BiYehudah (Mahadura Kamma OH 35):
אמנם ילמוד דברי תורה שיש בהם ממש...וג"כ ספרי מוסר חובת הלבבות מן אחר שער היחוד עד גמירא
However, a person should learn Torah which has substance...and also ethical work [such as] Hoveot Halevavot [starting] from after Sha'ar HaYihud, until its end.
The Seer of Lublin (18th century) is quoted here as saying that laypeople shouldn't learn it since they will not understand it..
R. Moshe Tsuriel here cites a saying in the name of the Vilna Gaon, that studying Kuzari is preferable to studying Sha'ar HaYihud.