Well if it itself has anything to do with idolatry, there is in fact an obligation to destroy not only it, but even the place it was taken from,as mentioned in Mishneh Torah, Avodas Kochavim chapter 6:
1
It is a positive commandment to destroy false deities, all their
accessories, and everything that is made for their purposes, as
[Deuteronomy 12:2] states: "You shall surely destroy all the places
[where the gentiles... served their gods]" and, as [implied by
Deuteronomy 7:5]: "Rather, what you should do to them is tear down
their altars."
In Eretz Yisrael, the mitzvah requires us to hunt after idol worship
until it is eradicated from our entire land. In the diaspora, however,
we are not required to hunt after it. Rather, whenever we conquer a
place, we must destroy all the false deities contained within.
[The source for this distinction is Deuteronomy 12:3, which] states:
"And you shall destroy their name from this place," [implying that]
you are obligated to hunt false deities in Eretz Yisrael, but you are
not obligated to do so in the diaspora.
Besides for that, there is a prohibition of benefiting from them:
2
It is forbidden to benefit from false deities, their accessories,
offerings for them, and anything made for them, as [implied by
Deuteronomy 7:26]: "Do not bring an abomination to your home."
Anyone who derives benefit from any of the above receives two measures
of lashes: one because of the prohibition, "Do not bring an
abomination...," and one because of the prohibition, "Let nothing
which is condemned cling to your hand."
so even if it can be argued for some reason that returning it to someone is not benefit, and if (for some reason) the obligation to destroy it is disregarded, but you are at least causing the other person to benefit from it, and there is a prohibition to not put a stumbling block in front of the blind. There are many examples of this law, one that comes to mind is
Hilchos Kilayim chapter 10 halacha 31
31
[The following laws apply when] a person dresses a colleague in
kilayim. If the wearer acted consciously, he is liable for lashes73
and the person who dressed him is liable for "plac[ing] a stumbling
block before the blind."74
and this prohibition (of not worshipping idols) also applies to non-jews, as mentioned in hilchos melachim umilchamoseihem chapter 9 halacha 2:
2
A gentile who worships false gods is liable provided he worships them
in an accepted manner.
A gentile is executed for every type of foreign worship which a Jewish
court would consider worthy of capital punishment. However, a gentile
is not executed for a type of foreign worship which a Jewish court
would not deem worthy of capital punishment. Nevertheless, even though
a gentile will not be executed for these forms of worship, he is
forbidden to engage in all of them.
We should not allow them to erect a monument, or to plant an Asherah,
or to make images and the like even though they are only for the sake
of beauty.
as well as the prohibition of a jew to cause a non-jew to sin, following under the category of not placing a stumbling block in front of the blind, applies as well, as demonstrated from this other example case from hilchos kilayim halacha #2
A Jew is forbidden to give his animal to a gentile to have him mate it
with a forbidden species.8
and it's explained the rational in the note there:
8.
Based on Bava Metzia 90a, the Radbaz and the Kessef Mishneh state that
the difficulty is that it is forbidden to give a gentile instructions
to perform a prohibited activity. According to the Rambam (Hilchot
Melachim 10:6), it is forbidden for a gentile to crossbreed species.
The Rama (loc. cit.:4), however, states that if the act is performed
for the benefit of the gentile, there is no prohibition.