I have also had this question for some time and I was glad to see someone else asking it. From my research into book binding techniques it is obvious that it comes from various decoration techniques. This style seems to come from the Victorian era, but I am not sure how it got into the Jewish book printing business and seems to have stuck around longer. I assume that it is because seforim are valued as holy books and therefore receive a bit more decoration. However, I also heard that the practice of painting book edges originated with accountants wanting to ensure that pages were not stolen or removed from their financial records.
However, the techniques that I have found online which most closely resemble seforim are marbled edges or spotted edges. The former is much more ornate than seforim usually are and the latter is much less pronounced, being just speckles. The Jewish style seems to have developed its own approach and I would be interested to see what the technique is for applying this decoration.
Here is how they do marbled edges: https://www.youtube.com/watch?v=103pWv-gAek
I think this opens a whole range of topics in Jewish publishing. Thinking about ornate title pages, binding, edge spotting, page layout, etc...