Rabbi Ribiat in 39 Melachos, Volume 2, page 281, Shemiras Shabbas
KeHilchasa 26:10, Rambam Hilchot Shabbat 7:4, 8:3. explain that the
Melacha of Kotzer refers to detaching produce from the ground using a
utensil such as a sickle.
Tolesh which is the Toldah of Kotzer, includes detaching produce from
the ground with one's hands. Uprooting or severing any part of a
living plant is forbidden on Shabbat under the category of Kotzer.
There is a discussion amongst the Poskim, if Kotzer applies to things
which do not grow from the ground.
Minchas Chinuch Kotzeir 2 asserts that the opinion of Rambam is that
Kotzeir is not limited to Gidulei Karka, and this seems to fit with
the Gemara Shabbos 107b that pulling a fetus out of its mother’s womb
is an act of Okeir Davar M'Gidulo. The Talmud Yerushalmi Shabbos 48b
takes this to an extreme and says that pulling a fish out of water is
an act of Kotzeir M'Doraisa.
However the Magen Avraham (340:15) seems to imply that Kotzeir only
applies to things that grow from the ground. While the Magen Avraham
isn't discussing Kotzeir, he does write that all of the Melachos are
learned out from the Mishkan and since we hold that dosh is limited to
Gidulei Karka, the others must be as well.
Ramban Shabbos 107b also insists that Kotzeir is limited to Gidulei
Karka. He rejects the proof from the fetus case, claiming that the
Gemara sometimes uses the phrase “Okeir Davar Migidulo” with regards
to other Melachos, in this case: Netilas Neshama. He further asserts
that the Bavli disagrees with the Yerushalmi about fish. In fact, he
suggests that even Rebbi Yehudah who holds that Dosh applies to things
that don’t grow from the ground would agree that Kotzeir only applies
to Gidulei Karka, since the ground is such an integral part of
Meleches Kotzeir.
Based on the above it is clear that at least some authorities will say that there is no Tolesh for something that is not connected to the ground. Even according to the authorities that say that Kotzer / Tolesh applies to things that are not connected I would venture to say that if one removes a piece of the bread together with the mold most likely there would be no issue of Tlisha.
Thanks to http://halachipedia.com/index.php?title=Kotzer for this answer.