I heard an answer in the name of one Lithuanian rabbi (I forgot the name), it is also partially mentioned here. The reason that Yaakov did not rush was because of Rachel. She was pregnant and had to give birth. Now, when Yaakov told his wives about necessity to go back to Eretz Yisrael, the verse Bereshis 31:14 states:
וַתַּ֤עַן רָחֵל֙ וְלֵאָ֔ה וַתֹּאמַ֖רְנָה ל֑וֹ הַע֥וֹד לָ֛נוּ חֵ֥לֶק וְנַחֲלָ֖ה בְּבֵ֥ית אָבִֽינוּ׃
Then Rachel and Leah answered him, saying, “Have we still a share in the inheritance of our father’s house?
Why does it mention Rachel before Leah, the older sister? Midrash Bereshis Rabba 74:4 states:
“Rachel and Leah answered and said to him: Is there still a share or inheritance for us in our father's house?” (Genesis 31:14).
“Rachel and Leah answered and said to him” – why did Rachel die first? Rabbi Yudan and Rabbi Yosei, Rabbi Yudan said: Because she spoke before her sister. Rabbi Yosei said to him: ‘Have you ever seen a person call Reuben, and Simeon answer him? Did he not call Rachel, and Rachel answered him?’ According to the opinion of Rabbi Yehuda, it works out well. According to the opinion of Rabbi Yosei, she died only from the curse of the elder [Jacob], as it is stated: “With whomever you find your gods, he shall not live” (Genesis 31:32); it was “like an error that emerges from the ruler” (Ecclesiastes 10:5). “Rachel stole [the household idols that were her father’s]” (Genesis 31:19); “Rachel died…” (Genesis 35:19).
We know from Bereshis Rabba 71:5 that Rachel was a master at keeping slilence and letting other people speak:
Rachel adopted the wisdom of silence, and masters of silence were produced from her. Benjamin: “chalcedony [yashefe]” (Exodus 28:20), there is a mouth [yesh peh]; he knew of the sale of Joseph, but did not tell. Saul: “But the matter of the kingship…he did not tell him” (I Samuel 10:16). Esther: “Esther did not disclose her birthplace or her people” (Esther 2:20).
The fact that Rachel spoke first is teaching us something important. The verse Bereshis 31:14 is giving praise to Rachel's self sacrifice. As we know, Yaakov avinu kept all mitzvos in Eretz Israel, but the Torah prohibits a man to marry two sisters, Vayikra 18:18. Coming to Eretz Israel for Rachel was equivalent to death sentence, since she was אֲחֹתָ֖הּ. Therefore, in return Yaakov extended the journey and delayed coming home, so that he could spend extra time with Rachel and she could give birth to Binyamin.