Genesis 32:11 quotes yaakov - "כי במקלי עברתי את הירדן הזה..." When did he cross the yarden? Looked back but didn't see it.
Given the location of Padan Aram and Eretz Kenaan, he had to cross the Yarden as part of his journey. He is speaking of the fact that he had to leave without any money or other forms of wealth. "With my staff" would be like the modern idiom "with just the clothes on my back".
Since he is about to recross the Yarden with his family and his flocks, this is a way of contrasting his situation then and now.
Rashi quotes the medrash Tanchuma which implies that he traveled directly (crossing the Yarden) and used the staff to allow him to cross safely.
for with my staff: I had with me neither silver nor gold nor livestock, but only my staff. The Midrash Aggadah states: He placed his staff into the Jordan, and the Jordan split. — [from Tanchuma Buber, Vayetze 3; Aggadath Bereishith 45:2]