Say you are fully aware that your yetzer harah drove you to do something specific- to visit a certain person on a certain day, or to have a certain kosher meal for dinner. Is that activitity necessarily something one must avoid doing, given its source?
If the activities you are performing are halachically acceptable, I see no formal obligation to stop (but see Ramban's comment on Vayikra 19:2 - naval b'reshut haTorah).
However, if you feel the yetzer hara is driving you to those, you become conscious of the test you are facing: giving in or overcoming it. This is the "job" of the yetzer hara, to test you and allow you to gain credit by overcoming the challenge. See Mesilat Yesharim (starting with ch. 1)
The Holy One, blessed be He, has put man in a place where the factors which distance him from the blessed G-d are numerous. These are the physical lusts which if he is drawn after them, behold, he draws away and goes ever further from the true good. Thus, we see that man is truly placed in the midst of a raging battlefield. For all matters of this world, whether for the good or for the bad, are trials for a man. Poverty from one side versus wealth from the other. [...]
If he will be a man of valor, emerging from the battle victorious on all fronts - he will be the "Adam HaShalem" (whole/perfect man) who will merit to cling to his Creator and will emerge from this corridor to enter into the palace to enlighten in the Light of (eternal) Life. According to the extent that he conquered his inclination and lusts, and distanced from the factors which distance him from the good, and exerted himself to cling to G-d, to that extent will he attain it and rejoice in it.
Rabbi Yehuda taught: In the future, at the end of days, God will bring the evil inclination and slaughter it in the presence of the righteous and in the presence of the wicked. For the righteous the evil inclination appears to them as a high mountain, and for the wicked it appears to them as a mere strand of hair. These weep and those weep. The righteous weep and say: How were we able to overcome so high a mountain? And the wicked weep and say: How were we unable to overcome this strand of hair?
Why does it say "slaughter"? Why not kill or eliminate? He answers that, after a cow is slaughtered, it still has a purpose. It changes its essence and moves from providing milk to providing meat.
Similarly, after God reveals himself to the world, the yetzer hara will still have a purpose. Everyone will understand at that time that its whole purpose was to entice man and see man stand up to the test. It acted only as a means for increasing our reward in the World to Come.