I realize this is a very common question, but the problem of evil is an important one, and as far as I can tell, has not been asked here.
In short, why does G-d allow evil to exist?
I would say that an answer on the simplest level would be to give us בחירה - choice. If there was no evil (and no "evil inclination"), there would not be much meaning to doing "good" and fulfilling the Torah. If everything was good and clear-cut, we would not really have free choice.
On a deeper level, this question could probably fill books (and likely does)
Evil exists because Gd desires for it to exist. Just as gravity exists, planets, entropy, time, space, animals etc. Everything in the world has it's purpose and one can find positive reasons for all things, and negative consequences of all things. For example, if gravity didn't exist, people would not be able to stay on the planet, however because of gravity, people get hurt when they run and fall, and bridge construction causes people to die.
Evil, has its own positive consequences as well.
G-d created good and evil, says both the Torah and Isaiah (Deut. 30:15-20; Isa. 45:7). One reason for having both is explained at Deut. 30:15-20 -- with Free Will, G-d had to give us choices, both good and bad, and the inclination to pursue evil as well as the competing inclination to pursue good. Without these inclinations fighting each other, our election of one over the other would be a foregone conclusion just like the last Egyptian election. Secondly, he created other kinds of evil -- natural disasters, still-born babies, birth defects, diseases, poor people -- to teach us His mida of compassion. Without a communal need that people give of themselves, we would lack community.
Of course, good and evil in this world are completely relative. What is good for the butcher is an evil for the animal to be slaughtered. We see this very much in the institution of karbonot (sacrifices) of which the Torah spends much time discussing even though animal offerings have only been an active part of our religion for a relatively short period (about 1400 years). As I heard brought down from Rabbi Shmuel Goldin, we brought animal offerings for unintentional or unwitnessed sins in order to appreciate that but for certain factors, our own fate might be the same as this poor animal. It is ironic that we should learn compassion through the suffering of others, but how else would we learn it?
For more thoughts on this, see Rabbi Benjamin Blech's book, "If G-d is Good, Why is the World So Bad?", Simcha Press 2003.
The Ramchal in Da'as Tevunos gives several reasons for the creation of evil:
In siman 38, he explains that Hashem's Oneness (the revelation of which is the purpose of creation - siman 40) can only be fully grasped by seeing the facade of what it would mean for Hashem to not be in complete control, and then negating that facade. In siman 40 he writes that Hashem made evil in order to use it as a vehicle for revealing His complete absoluteness.
הרע שברא הוא ית' להודיע אמתת יחודו ולנסות בו את האדם
...the evil that He created in order to make known His true Oneness and to test man with it
I briefly discussed how this works here.
In this point, the Ramchal is arguing that the primary purpose of the existence of evil is not for the purpose of facilitating free-will, but rather for the purpose of completing the appreciation of Hashem's Oneness. Evil, and the notion that there is a reality to something outside of Hashem's will, exist only to facilitate the revelation of Hashem's Oneness when evil is obliterated.
Later (Siman 118), discussing the existence of bad things, e.g. suffering, he says evil was created in order to have a mechanism for punishing the wicked:
ונמצא כלל הרע, אשר הוא ית' השיב אחור ימינו מהנהגת עולמו, וינהגהו בכבדות, תוקף חשך סתרו, ושם בטבע הלז כל הפסדי הטוב, הלא המה כל מדות בית דין, להביא במשפט על כל נעלם ורשעים
(Paraphrased) Evil was created to bring punishment upon the evildoers.
Evil in this context means the existence of negative things. If Hashem is completely good and bad is an "external" creation (siman 104), the existence of negativity needs to be explained.
As for the Evil Inclination, Da'as Tevunos cites the Zohar (Siman 78):
כמאמרם ז"ל (זוהר ח"א, קו ע"ב), לא נברא יצה"ר אלא לנסות בו את בני האד"
The Evil Inclination was only created to test man.
The Tanya at the end of Chapter 9 bring a parable from the zohar illustrating the role of evil.
אך נפש הבהמית שמהקליפה, רצונה להפך ממש
But the animal soul derived from kelipah desires the very opposite; it desires that the body be pervaded with its faculties and its thought, speech and action.
לטובת האדם, שיתגבר עליה וינצחנה, כמשל הזונה שבזהר הקדוש
But the animal soul desires this for man’s benefit, in order that he prevail over her and vanquish her, as in the parable of the harlot [related] in the holy Zohar.19
The parable: A king desired to test the moral strength of his only son. He had a most charming and clever woman brought before him. Explaining to her the purpose of the test, he ordered her to exert every effort to seduce the crown prince. For the test to be valid, the supposed harlot had to use all her charms and guile, without betraying her mission in the slightest way. Any imperfection on her part would mean disobedience, and the failure of her mission. However, while she uses all her seductive powers, she inwardly desires that the prince should not succumb to them.
So too in our case: The kelipah itself desires that man overcome it and not permit himself to be led astray. The entire stratagem is solely for man’s benefit.