Assuming for the sake of argument that studying for a stressful Torah test would be a violation of oneg Shabbat I see two reasons why it might nevertheless be permissible.
- It might be akin to fasting on Shabbat in the event that one has a bad dream. Although this detracts from the Shabbat experience, it is permissible (Shabbat 12b, Rambam's Hikhot Taaniyot 1:12). The Sefer HaOrah (Hilkhot Taanit Vol. I:71) indicates that although sub-optimal (according to some, one should fast to make up for fasting on Shabbat), it is the right thing to do in spite of oneg Shabbat, since if one didn't fast one would be (even more) uncomfortable since one is nervous about the dream, and the fast will help:
דקיימא לן כל היושב בתענית קורעין לו גזר דינו של שבעים שנה עונג הוא הוא לו כשמתענה וקורע גזר דינו, שאם אין מתענה יושב ודואג כל היום כולו ואינו נהנה בעונג שבת מפני הפחד
This is similarly indicted by Rashi to Ta'anit (12b s.v. Vaafilu) and by R. Hai Gaon, and "most commentators" referenced by Rashba (Berakhot 31b), and by Tur (OH 288) who write that in this case, it is considered oneg Shabbat for him to fast since it will relieve him.
Similarly, in our case it could be argued that in event that one will be nervous and uncomfortable anyway, given the looming test, and studying will alleviate the tension, that one would be allowed to study. (Whether or not this would apply even if one could've done it earlier, is debatable.)
- Another reason is based on Rif's understanding of a passage in Shabbat (19a) regarding riding on boats on Shabbat. Rif (Shabbat 7b) explains that it is forbidden since the motion ruins one's oneg Shabbat, but the Talmud states that it is permissible for a mitsvah, since the mitsvah supersedes oneg Shabbat. Similarly, in our case, one might suggest that the mitsvah of Torah study supersedes oneg Shabbat. One might claim, however, (cf. Tsits Eliezer 1:21) that Rif would only allow a mitsva instead of oneg Shabbat where one begin's the mitsva before the onset of Shabbat and the the mitsvah of oneg Shabbat, (such as where he get's on the boat before Shabbat). However, perhaps the Rif would not allow one to perform a mitsvah that would preclude oneg Shabbat after the onset of Shabbat.
Regarding מתוך שלא לשמה בא לשמה, that sees totally irrelevant, as why would something forbidden become permissible just because it will later become permissible. It is not even entirely clear that it is relevant. Even if one values Torah study in and of itself, he might still find studying for a test stressful.
As always, consult a competent halakhic authority for practical issues.