One reason given is that Par'o did not allow them to leave. Note that when they wanted to bury Ya'akov in Mearas Hamachpelah, Yosef had to request permission to do so and push hard. As Rashi says in Vay'chi 50:5
as he adjured you: But were it not for the oath, I would not permit you [to go]. He (Pharaoh) was afraid to tell him (Joseph) to transgress the oath, however, lest he say, “If so, I will transgress the oath that I swore to you that I would not reveal that I understand the holy tongue (Hebrew) in addition to seventy languages of the nations of the world, but you do not understand it (Hebrew),” as is found in Tractate Sotah (36b).
Also, they had to leave their families behind in Egypt in order to ensure that they would return.
This implies that if he was reluctant to let them leave after Yaakov died, he would have been even more reluctant to let them leave (and give up his hold on Yosef) while Yaakov was alive (and possibly holding off the return of the remaining five years of the famine).
Also see below where I point out that Hashem had said that Yaakov would die in Mitzraim with Yosef to tend him and would be brought back to Canaan.
Similarly, they did not dare try to take Yosef out even to be buried.
And Joseph died at the age of one hundred ten years, and they embalmed
him and he was placed into the coffin in Egypt.
Also Hashem had told Yaakov that he would die in Mitzrayim, so they could not go back earlier.
I will go down with you to Egypt, and I will also bring you up, and
Joseph will place his hand on your eyes.
and I will also bring you up: He promised him (Jacob) that he would be
interred in the [Holy] Land. — [from Yerushalmi Sotah 1:10]
Once conditions started turning bad, there are several reasons given for not leaving.
During the days of the Soviet Union, Jews were persecuted but were treated as traitors if they tried to leave. People writing about the parshah would use this as an analogy to the condition of the Jews especially within the Student Struggle for Soviet Jewy (SSSJ). Par'o did not want the Jews but he also did not want to expel them. That is why he enslaved them, and slaves were forbidden to run away from their master. Par'o owned the Jewish slaves. They were not owned by individual masters as Yosef had been owned by Potifar.
Another reason is that by the time the Jews could have left, they had increased so greatly that they could no longer travel as a family (by natural means) and make the trek. It was difficult for the family of 70 to actually get to Egypt and they would not have made it had Yosef not sent wagons and supplies for the trip Vayigash 45:19 and guaranteed support once they had arrived in Egypt. See Vayigash 47
As we see in the discussions about the Valley of the Bones Yechezkel 37:1
and that was full of bones: Our Rabbis said (Sanh. 92b) that they were of the tribe of Ephraim, who left Egypt before the end [of the exile], and the people of Gath who were born in the land slew them, as is stated in (I) Chronicles (7:20ff.)
When a band of Ephraim fought their way out, they all died.
@DanF brought up the bris bain habesorim which decreed 400 years of oppression.
And He said to Abram, "You shall surely know that your seed will be
strangers in a land that is not theirs, and they will enslave them and
oppress them, for four hundred years.
According to those who say that this is counted from the birth of Yitzchak, we see that the troubles in Canaan (such as with Avimelech and the Pelishtim) were counted in this way. Thus, it would not necessarily be a reason for the Bnai Yisrael to decide to stay in Egypt. Additionally, it would not be up to them to decide matters on the basis of this prophecy. Note that Yaakov did not go to Egypt in the first place without having gotten permission from Hashem that this was the right step. As a result, I am not including it in the explanation as to why they did not leave Egypt.
While this may have been part of the underlying reason why Hashem set up matters in this way, it was not the reason that the Bnai Yisrael reacted to events they were living through. The reasons that I gave were the outside and visible events that kept them in Egypt.