I think that this answers the question.
BusinessHalacha.com quotes Rav Moshe Feinstein that even if you trim the branches, the fruit still belongs to the original owner. Note that it does not deal with whether or not trimming would damage the tree (and trimming is valid only if not trimming would damage your property).
THE GRAPES OVER THE FENCE
After Mincha, people began walking downstairs to the function room for Seuda Shlishis. Mr. Adler accompanied Rabbi Dayan.
"My wife raised an interesting monetary question this afternoon," Mr. Adler said. "Our neighbor has a grape vine growing on the backyard fence, and some clusters of grapes hang over the fence into our yard. To whom do they belong?"
Rabbi Dayan thought for a moment. "This question was posed to Rav Moshe Feinstein zt"l many years ago. (Igrot Moshe, C.M. vol. I, #43) He ruled that the fruit belong to the owner of the vine. Rav Moshe explains that although the Gemara (B.B. 27b) debates whether bikkurim can be brought from a tree whose roots or branches extend into someone else's property, ownership of the fruit is determined solely by where the stem emanates from the ground. Since the vine's stem emanates from your neighbor's property, even the clusters of grapes over the fence are his. The Rama also rules that the branches follow the base of the tree. (C.M. 167:2)"
"If the grapes are his," asked Mr. Adler, "am I allowed to trim the branches that interfere with my yard?"
"Yes," said Rabbi Dayan. "The Gemara teaches that branches of a tree that interfere with public traffic can be cut down. The same would seem true of branches that interfere with someone else's private property. Nonetheless, despite your right to trim these branches, the fruit belong to their owner."
Rabbi Dayan and Mr. Adler washed and sat down for Seuda Shlishis. They helped themselves to some whitefish salad.
Mr. Adler turned to Rabbi Dayan, "Would it make a difference whether the neighbor is Jewish or not?"
"No," replied Rabbi Dayan. "Rav Moshe writes explicitly that since monetary ownership is determined solely by the stem, there is no difference between a Jew and a non-Jew. Stealing from a non-Jew is also prohibited."
Mr. Adler took a drink and thought for a moment of the grapes that fell to the ground. He turned again to Rabbi Dayan, "What about individual grapes that fall onto the ground?"
"In principle, there is no difference between whole clusters and individual grapes," said Rabbi Dayan. "Both belong to your neighbor. However, in practice, we can assume that he is interested only in the whole clusters, not in individual grapes that fall into your yard. Therefore, you can typically take individual grapes that fall, since he most likely doesn't mind, but you cannot take the clusters without asking his permission."