Yes, it has been suggested that the Essenes and the Baitusim (Boethusians) are one and the same sect. Evidence has been advanced, much of it on the assumption (a very big assumption) that the Essenes are the same as the Qumran sect. Here is a paper by Zachi Dvira on the Baitusim, and on pp. 42-43, he provides a good summary of the attempts to identify them with the Essenes. I further summarize as follows:
The first such attempt was by R. Azariah de Rossi and was based essentially on (1) the fact that the Talmud never claims that Baitusim denied the concept of an afterlife like the Saducees (which would have exluded them from being Josephus' Essenes), and (2) the similarity between the names "ביתוסים" and "איסיים" to the extent that that former could have derived from "בית איסיאי" or something similar.
Skip ahead a few centuries and we find several scholars following R. Azariah's lead (see Dvira's paper for references). Some base their identification of Baitusim with Essenes on R. Azarah's reason #2, providing a more detailed analysis of the connection between "ביתוסים" and "בית איסי" (does איסי = יוסי?) Others find similarities between what we know about the beliefs of the Baitusim and what we find about the beliefs of the Qumran sect (given that Qumran = Essene) or similarities between their interpretations of certain halachot.
Professor Adiel Schremer published a paper (available here) arguing against this linguistic association between "ביתוסים" and "איסיים", pointing out several difficulties of the above-suggested derivation; additionally, we only know the Greek name "Essene", but not what the Hebrew form of the name was.
Professor Moshe David Herr also argued against this identification, identifying differences in the known practices and beliefs of the Baitusim and the Essenes and Qumran sect, even though others had pointed out some similarities. Dvira in his paper also provides some fundamental differences between these groups and concludes that no further attention should be given to this suggestion that Baitusim were Essenes.