IIRC the idea of "to square" addresses two main issues - measuring a minimal Succah and a minimal house for Mezzuzah (I'll focus on the first but the discussion on the later is similar).
IIRC the basic measurement that the Gemmorah does is 7x7 Tfochim (49sq.T). Rambam Succah 4.7 uses the term "כדי לרבע" unexplained and the Achronim (see Taz, Bac"h and others on Shu"A) have a great discussion whether it means:
"of equal footage", like a Succah 1x49 Tfachim (Ta"Z) or
any form that "a square 7x7 Succah can fit in it", like a round hut (Ba"ch).
What is the Talmudic source for both opinions?
EDIT: in Likutim on Rambam I have found the following source:
הלכה א: רחבה אין פחות מז"ט על ז"ט. ולעניין אם ארוך יותר מז' ומשלים לרוחב שהוא פחות מז' עמ"ש בזה בש"י ריש סוכה והוא מחלוקת בין בבלי וירושלימי. (קובץ)
Who is ש"י, where are those Bavli and Yerushalmi and who's Kovetz?