The Shulchan Aruch rules (OC 155:1OC 155:1)
אחר שיצא מבהכ"נ, ילך לבה"מ; ויקבע עת ללמוד, וצריך שאותו עת יהיה קבוע שלא יעבירנו אף אם הוא סבור להרויח הרבה.
After one leaves the synagogue he should go to the study hall and establish a time of study which should be fixed such that it won't be overridden even if he will gain much [money].
The Mishna Berura there (sk 4sk 4) quotes a Yerushalmi (9th Chapter of BrachotBrachot 9:5 רב חלקיה...) that says implies that these fixed times are meant to be minima for Torah study and that one who only learns at fixed times when he has other free time available is as if he rejects God's covenant.