The chapter breaks appear to stem in part from paragraph breaks in the Masoretic Text, which appear in the respective codices (Cairo, Aleppo, and Leningrad). Just like scrolls, these codices were written in columns (usually three per page, since a codex is a "book"). These columns contained paragraphs of verses, but the paragraphs were not indented (like modern English). Instead, the spacing between clumps of verses is what determined whether or not there were major or minor separations of thought, which is how we determine the paragraphs and chapters.
For example, when groups of verses have minimal spacing between them, the ideas have their close connection. In the published form of the Masoretic Text (such as the Biblia Hebraica Stuttgartensia) the letter ס (samekh) will appear before the connecting paragraphs, which is the first letter of the Aramaic passive participle סתומא (determinative form), which means "what is closed."
On the other hand, within the codices of the Masoretic Text, when one clump of verses did not have the close connection with the preceding paragraph(s), but were instead to be read as more independent than the preceding paragraph(s), then the these subsequent paragraphs began on their own line with no attempt to minimize the spacing with the preceding paragraph. Thus in the modern scholarly published form of the Masoretic Text (such as the Biblia Hebraica Stuttgartensia) the letter פ (pei) will appear before the connecting paragraphs, which is the first letter of the Aramaic passive participle פתוחא (determinative form), which means "what is open."
Thus the Masoretic scholars developed a very simple system for paragraph divisions. In English, indentation will mark the paragraph, but there is no way to group major and minor thoughts of paragraphs in English, unless further indentation occurs or by using sub-bullet statements. However, in the codices of the Masoretic Text, it is the spacing that counts.
One excellent example from Megillot will illustrate.
In the Song of Songs, the first paragraph of the text (1:1 - 1:4) is the major paragraph with all subsequent paragraphs in the book "closed" with the first paragraph -- and so the book is one big thought of "close together" paragraphs. However, in 8:11 there is an "open" paragraph break marked by the letter פ (pei), because in the codices of the Masoretic Text there is a distinct separation from the previous paragraph, and so the reader stops and reads the remainder of the book (8:11-8:14) as a separate thought, or coda, to the book. This coda is a shift of narration. That is, the third-person narrator takes over and closes the book with blessings upon Solomon and his beloved.
In closing, the small letters ס (samekh for "close together") and פ (pei for "open or far apart") appear only in modern scholarly editions of the Masoretic Text so that we, the readers, will know when and where the spacing occurred between clumps of verses in the codices of the Masoretic Text, and therefore enable the reader to make the correct assessment as to where the major and minor breaks in paragraphs occurred throughout any given book.
