I saw a source for this idea in a sefer in the name of the Gra who says
כשם שיש ביטול תורה בכמות יש ביטול תורה באיכות
Just like there is a bittul torah in quantity, there is bittul torah in quality.
It didn't cite a source for this Gra, but I'm assuming if it's something he wrote, it's not something the Rishonim say. I'm not sure if there's an earlier acharon that says the same.
You asked what are the parameters of this idea and where does it come from?
Based on Minchas Asher Vayikra Simman 63/64
The lack of proper quality in talmud torah can be manifested in two ways:
- Quality of the material
- Quality of the act of learning
What are the sources for these two types?
Quality of the material
The gemarra in Bava Metzia 33a teaches us that there are levels in the quality of the material that you learn
הלומד מקרא מדה ואינה מידה, משנה מדה ומקבל עליה שכר, גמרא אין לך מדה גדולה מזו
Someone who learns Tanach, it's a measure and isn't a measure. Mishnah, it's a measure and they receive reward. Gemarra, there's no greater measure than that.
Rashi there explains that the main form of talmud torah is learning that brings you to a complete understanding of the laws of the torah. This is more likely accomplished through learning Gemarra as opposed to Mishnah alone or Tanach alone. (I admit this gemarra isn't a proof there is an issur of learning the lower quality material, but based on the next section I don't see a reason to distinguish between learning with less effort being called bittul and learning lower quality material).
Quality of the act of learning
Rav Chaim Volozhin writes (found in Eitz Chaim § 36):
שתזדרז ללמוד בזריזות וחשק גדול, כי מה שלומדים בעצלות כל היום יכולים ללמוד בזריזות בשמה שעות
A person should exert themselves to learn with alacrity and great passion, because what a person can learn laxly the whole day can be accomplished with exertion in a few hours
Is there an issue with not learning to the best of one's ability?
The Ran in Nedarim 8a writes that even though the gemarra says you fulfill the mitzvah of talmud torah by saying kriyas shema twice a day, you still have an obligation to learn whenever you can with all of your strength
מסתברא לי דלאו דווקה דבהכי מיפטר שהרי חייב כל אדם ללמוד תמיד יום ולילה כפי כחו ואמרינן בפ"ק דקידושין דף ל ע"א ת"ר ושננתם שיהו דברי תורה מחודדין בפיך שאם ישאלך אדם דבר שלא תגמגם ותאמר לו וכו' וק"ש שחרית וערבית לא סגי להכי
It makes sense to me that [the gemarra is] not to be taken literally when it says you exempt [your obligation to learn by saying kriyas shema], because a person is obligated to learn at all times, day and night, according to their strength, and it says in the first chapter of Kiddushin 30a: The Rabbis taught "ושננתם", that the words of Torah should be sharp in your mouth, such that if a person asks you something you don't stutter and respond, and kriyas shema of morning and evening don't satisfy this requirement.
If a person learns not according to strength they have but rather in a lax way, they aren't fulfilling their obligation. (Rav Asher concedes that the obligation to constantly learn isn't the simple understanding of the verse and isn't a Torah obligation, rather Chazal made a drasha on the verse to teach you a worthy endeavor for a Jew to follow. He also points out that the Radvaz Volume 3 Simman 416 feels the Ran is a lone opinion that you don't fulfill your obligation with kriyas shema, but Rav Asher brings many Rishonim that sound like the Ran. He resolves the conflicting statements in his sefer).
The Achronim (Beis Efraim Orach Chaim Simman 68, Avnei Nezer Orach Chaim Volume 2 Simman 517, among others) ask on the gemarra in Megillah 3a that says we are mevatel talmud torah to read the megillah, yet reading the megillah is also talmud torah. You can ask similarly on the gemarra in Shabbos 11a that says we are mevatel talmud torah to say kriyas shema, yet kriyas shema is also talmud torah. Simply put you can say reading the megillah is talmud torah but all the effort required in going to shul and all that goes into that is what the gemarra is referring to (as @mevaqesh said in the name of the Ritvah and other Rishonim), but that doesn't resolve the issue with the gemarra in Shabbos.
The gemarra in Brachos 10a says
גדול הקורא קריאת שמע בעונתה יותר מהקורא בתורה
Greater is one who recites shema at the designated time than one who reads from the Torah
The Rashba asks on this that what is the chiddush? Reading the shema is also reading from the Torah, and besides that you accomplish the mitzvah of reciting the shema so this statement is obvious. He answers that קר"ש גדול מהשונה בתורה, reciting shema is greater than one who is learning torah, not merely reading it. Rav Asher learns two things from this Rashbah:
- Reciting shema has a mitzvah of talmud torah like one who reads from the torah, besides the mitzvah of shema itself
- Someone who learns torah is greater than one who only reads torah
Using this Rav Asher explains the gemarra in Megillah and Shabbos. Even though there's a mitzvah of talmud torah when reciting the megillah and shema, this is just the level of reading torah. However, if someone is learning torah, getting a deep understanding of the laws of the torah, and he interrupts this to recite the megillah and shema, this is considered bittul torah, because learning torah is greater than reading it.
In summary we see varying levels in the quality of one's talmud torah, and there seems to be an obligation to try one's best to maximize that quality, as well as a concept of bittul torah when one doesn't.
Regarding your question about balancing one's talmud torah with other duties (such as chesed), the Rambam, Hilchos Talmud Torah 3:4 rules (based on the gemarra in Moed Katan 9b):
היה לפני עשית מצוה ותלמוד תורה, אם אפשר לצמוה להעשות על ידי אחרים לא יפסיק תלמודו, ואם לאו יעשה המצוה ויחזר לתלמודו
If there is before you a mitzvah opportunity and learning torah, if it's possible for the mitzvah to be done by another then don't interrupt your learning, otherwise do the mitzvah and return to your learning.
It sounds like from the Rambam that it would be considered bittul torah if one needlessly interrupted their learning, regardless of what the mitzvah is (besides megillah and shema, as above). However, if no one else can do the mitzvah, then one must stop their learning (and it shouldn't be considered bittul torah), and the Rambam adds something the gemarra doesn't mention, to return to your learning, which sounds like it would be a problem if one doesn't return to their learning.
For further study on this last topic, see here for various sources regarding interrupting one's learning to do a mitzvah.
