The Mishna Berura writes that one needs to honor shabbos with clean clothes. At the same time, in discussing the concept of oneg shabbos in regards to food, he says that while it is a custom to have fish, if one does not like fish he does not have to eat it since it would not bring him pleasure/oneg shabbos.
The Chazon Ish (חוט שני ח״א פ״א ס״ק ב) writes that an article of clothing in which there is no oneg/enjoyment in wearing it does not have to be worn. As an example, he brings a garment which one does not want to wear because of the heat and says that there is no "honoring of shabbos" in wearing it. (anything that is not "oneg shabbos" is not "kavod shabbos," but if you do wear it then you still fulfill "kavod shabbos.")
Additionally, in the book Kitzur Hilchos Shabbos, it says that one should not take off his shabbos clothes immediately upon returning from shul as this would make it appear as though he only wore them for the sake of going to shul and not for the sake of Shabbos.
All this having been said, it seems as though after the meal you are permitted to change into something more comfortable if your other clothes bring you discomfort.