Rabbi Yosef Zvi Rimon of Yeshivat Har Etzion states:
What is considered "rounded?" The Bi'ur Halakha (s.v. Ve-lo) rejects the notion that a token rounding is sufficient to exempt a garment from tzitzit, but remains undecided about the amount of rounding necessary to exempt the garment. There are Acharonim, though, who do venture opinions.
The Maharil Diskin (responsa, end) writes that the radius should be three etzba'ot (fingers or thumb-breadths, with an etzba equaling two centimeters according to R. Chaim Na'eh and 2.4 cm according to the Chazon Ish), while the Arukh Ha-shulchan believes that the length of the arc should be 3 tefachim (i.e., the radius should be 1.9 tefachim).
The most likely opinion, however, appears to be that of the Chazon Ish (in Dinim Ve-hanhagot Me-hachazon Ish, vol. II, 3) who says that the determining factor is simply whether it appears round to the eye, and thus even three fingers is not necessary. [The logic being that the definition of rounded should not be an objective arc, but rather relative to the object concerned - Ed.]