A person who damages is obligated for five things: Tzaar (distress), ripui (healing), sheves (idleness), boshes (embarassment), and nezek (damage) (CM 420:3). However, we nowadays only obligate someone to sheves, ripui, and nezek (1:2), and some say only nezek (Rama there).
There is no question that annoying someone doesn't count for nezek or ripui. However, it may count for sheves, since you may be caused to waste time. So according to the Shulchan Aruch, you can take sheves, and according to the Rama you can take nothing.
However, it is nevertheless forbidden to annoy someone — ואהבת לרעך כמוך" "אל תונו איש את אחיו"