What are the halachik definitions of bittul torah?
I.e. some say that studying secular sciences qualifies as bittul torah while others say that it doesn't. How are they defining the halachik definition of bittul torah differently and on what basis?
The Gemara in Megillah 3a:
מכאן סמכו של בית רבי שמבטלין תלמוד תורה ובאין לשמוע מקרא מגילה
The house of Rebbi was mevatel Torah to come and hear the Megillah Much has been written about why the reading of the Megillah is considered bittul Torah, as the Megillah itself is seemingly a part of Torah.
Some explain that the Mitzvah of reading the Megillah is meant to be an act of listening, not an act of learning. The Beis Ephraim in O.C. #68 (paragraph starting מעתה) explains that it is a qualitatively lower level of learning, and there is such a thing as ביטול תורה באיכות - qualitative bittul Torah. (The Beis Ephraim has further proof to this from Berachos 10b and the Rashba there, among other proofs.) The glosses of the Rashash to that Gemara, based on the Shulchan Aruch, says the same thing. The Pele Yo'etz in Chelek 2, Erech Tehillim, 88b also writes that there is Bittul Torah in qualitatively "lesser" learning. The Chassid Yavetz (15th century) in his commentary to Avos 6:4 writes a similar idea - that the learning of Torah is measured according to the one learning, not the material learned. The Shulchan Aruch HaRav (18th century) in Hilchos Talmud Torah 2:5 writes that anyone who reviews what they have already learned because it is easier (as opposed to in order to remember it) literally is responsible for their own life.
Either way, even a mitzvah (even those who understand the Gemara is about the transit time of going to hear the Megillah will agree that the travel to do a mitzvah is also considered a level of mitzvah and even has the exemption from other mitzvos due to being involved in a current mitzvah) is called bittul Torah and requires justification (See the gemara there, which learns from a kal v'chomer that this particular Bittul Torah is acceptable). It seems that any act of not learning needs justification (I am making the assumption that the threshold of justification is more than "I was having fun;" otherwise the Gemara wouldn't very much need to tell us about reading Megillah being enough of a justification, nor to learn that it is a justification from a kal v'chomer).
NOT REALLY AN ANSWER:
A brief look at the Gemara cited in YeZ's answer sheds some light on the topic although it doesn't directly answer the question. The Gemara records that Rebbi's household derived that they ought to be mevatel Torah to hear the megillah from the fact the a drasha teaches that Kohanim and Leviim cease their Avodah to hear the Megillah.
Now, an obvious question exists; the Megillah ought to also constitute Torah, so why is it bittul Torah? From this, some Acharonim derive the concept of "bittul Torah in eichus (quality)"
However we can answer a question with a question; we always pause Torah to perform mitzvos, so why is a drasha necessary to tell you to stop learning?
Ritva answers that in reality the only question of the Gemara was whether or not time should be taken from Avodah or Torah to GATHER A MINYAN FOR MEGILLAH. We hold that Megillah can be read without a minyan, and a minyan is just a hiddur. Thus the Gemara never pitted reading Megillah against learning, but rather gathering a minyan against learning. Accordingly the whole proof from the Gemara that sub-standard learning constitutes bittul Torah disappears. (whether or not the Ritva happens to agree to the existence is a separate question. Maybe he does, but it wouldn't be because of this gemara. See comments for a dicussion of whether or not this was the Ritva's view)
(See Aruch Hashulchan who assumes that this is the position of Rambam as well and uses this to answer the "bittul Torah question")
Furthermore, whether or not one learns like the Ritva you don't see from this Gemara that were Megillah reading to be "bittul Torah" that it would be prohibited. Perhaps Beis Rebbi had a middas chassidus of learning all day and therefore only stopped because of the drasha. We see no proof of a general prohibition to stop learning and certainly no proof that "lesser study" constitutes bittul Torah.
I haven't found any Rishonim who dispute the Ritva, and haven't found any Acharonim who understand this Gemara as bittul Torah in eichus before the 19th century.