Contrary to what the previous answer states, there can be a phonemic difference in many of these instances. If one pronounces an unaspirated /bet/ as a /waw/, for example, it might be mistaken for a conjunctive; if one usually differentiates between a /tav/ and a /sav/, the use of the former in a situation where the previous word sounds as though it concludes with a /ha/ might be mistaken for indicating the presence of a definite article. But just because an individual within the congregation might be confused, that is insufficient reason to make the baal qoreh repeat what he has leyned.
The following summarises my reasoning in this matter:
The most stringent position when it comes to this issue appears to originate in what the Rambam wrote (Hilkhot Tefillah 12:6). There, he rules that if the baal qoreh errs even in the pronunciation of a single letter (אפילו בדקדוק אות אחת), we make him go back and read it again properly. It is difficult to source the origin of this ruling: Hagahot Maimoniyot learns it out from the beginning of the fourth perek in Yerushalmi Megillah, where such stringency is taken as regards the pronunciation of the targum, while the Kesef Mishna suggests that it is something that can be inferred only, and from Yerushalmi Megillah 2:2 instead.
Either way, while the Tur mentions the Rambam's ruling (OC 142), he actually sides in favour of a midrash in Shir haShirim Rabbah 2:13, that even if one were to omit a letter altogether - although without changing the meaning of the word - we do not make him repeat it. Such is not the interpretation of everybody on this midrash (a midrash that, read this way, might also allow us to substitute one word for another during leyning; see, for example, the Arukh haShulchan OC 142:2). It is also not the opinion of the Bet Yosef.
The Shulchan Arukh (OC 142:1) rules like the Rambam: "קרא וטעה אפילו בדקדוק אות אחת מחזירין אותו" in adopting the more stringent position, although without defining what might constitute such an alteration. Another post-Rambam source for this opinion is the Maharam Mintz (Shut 81). Note, however, the stipulation of the Rama:
ודוקא שינוי שמשתנה על-יד-זה הענין אבל אם טעה בנגינת הטעם או בניקוד אין
מחזירין אותו אבל גוערין בו
Specifically this is if the alteration changes the meaning, but if he
erred in the trope or in the vocalisation, we rebuke him [presumably
later, and in private - as per the Tur] but we don't make him repeat
The Magen Avraham (ibid.) explains that an example of vocalisation that changes the meaning would be pronouncing יַעשה (with a patach) as יֵעשה (with a tzere), or חֲלב (with a chataf) as חֵלב (with a tzere), etc. This Magen Avraham is important, since it defines the responses of both the Mishna Berurah and the Arukh haShulchan in their interpretation of the Rama.
The Mishna Berurah (142:6) brings the stringent opinion of the Maharam Mintz to the effect that one should be careful as regards even the placement of stress, but when it comes to the issue of needing to repeat one's words after making an error, he sides with the Rama (142:4) in that this only needs to happen in the event that the alteration changes the meaning, and adds to this the opinion of the Shulchan Atzei Shittim that this is also the case if he errs in the trope and, in so doing, alters the division of the passuk in a meaningful way.
The Arukh haShulchan likewise brings the opinion of the Rama, although adds that this is the case even if one is reading for oneself and not for the congregation. His conclusion (142:5) is important to the specific issue being asked here:
וכל זה בדיעבד, אבל לכתחלה החיוב על הקורא לקרא בטעמים ובדקדוק כדת
וכהלכה שלא יניח הנע ולא יניע הנח וישמור דגושים ורפויים
This is all after the fact, but ab initio the person leyning has an
obligation to read with the [exact] trope and the [exact]
pronunciation, in accordance with custom and law, neither to elide a
vocal shva nor to accentuate a silent shva, and to protect both
aspirated and unaspirated consonants.
This statement of his is sourced in the Bet Yosef (Tur OC 142), in which the Bet Yosef relates stringency in this regard to the supposition that even the eccentricities of the Tiberian masorah have a source in the supernal realms. Importantly, however, the Arukh haShulchan rules that punctiliousness in their pronounciation is an ab initio matter (לכתחלה), and that one need not worry about them after the fact.
From the foregoing it seems apparent to me that there is scope to be lenient and to suggest that there are different types of changes that can be made to the pronunciation of the text.
Elision of letters and the substitution of words: a small minority of poskim (such as the Tur, based on a midrash) might allow this, under certain circumstances, to slide. The overwhelming majority of poskim would demand a reread;
Mispronunciation of vowels: in the event that the mispronunciation doesn't change the meaning of the word, there is scope to be lenient and to merely reprove the baal qoreh afterwards and in private; in the event that it does change the meaning, there is good precedent to demand a reread;
Mispronunciation of trope: in the event that the mispronunciation changes the meaning, there is some small precedent to demand a reread; in the event that it does not change the meaning, there appears to be no reason to do so;
Mispronunciation of a shva or of an aspirant [eg: pronouncing the taf as a saf and vice versa]: there is good precedent to suggest that one needs to careful with these things at the outset, but I have found no precedent in favour of correcting a person after the fact.
The foregoing is based solely on what I have seen in the
above-mentioned sections of the Rambam, the Tur, the Shulchan Arukh,
the Mishna Berurah and the Arukh haShulchan, as well as in a small
selection of their respective commentaries. I have not taken into
consideration laws pertaining to the shaliach tzibbur, the leyning of
the megillah or the voluminous responsa on any of these issues. As in
all situations, CYLR.