After doing some research I found in the Malbim 41:9-10 which says that Yosef had Ruach Hakodesh to interpret the dreams of the cupbearer and master baker. The cupbearer had a more gross offense since he could have prevented the fly from being in the cup. However the master baker didn't know the stone was in the bread since he didn't bake, rather he just presented it. So the fact that Yosef said that the baker will be hanged proved that his words were divinely inspired.
The Malbim in Parshas va'yeishiv explains that the way it worked was that the cupbearer had servants and the master baker had servants. So in reality based off a court case the cupbearer's offense was a lesser offense if we put guilt on the cupbearer's servants since a fly can jump into the wine at any moment, but if we put the guilt on the cupbearer himself than it is a very great offense since he is holding the cup the whole time. On the other hand, the master baker himself did a small offense since he could not have known about the rock, but his servants did a very serious offense. Based off this information, a regular court would find the cupbearer himself more guilty than the baker since he should have known better. However, the inside information Yosef received from HaShem is that the court will now punish the main ministers (cupbearer and baker) based off the actions of their servants, and the baker therefore is more culpable since his servants put a rock in the bread which is a terrible offence much worse than a fly.
So it seems that Yosef was divinely inspired and explained the dream in a way which seems counter intuitive to the case. However, this itself gave credibility to Yosef' s ability when it came to interpreting Pharaoh's dream.
Since it was done through Ruach Hakodesh Yosef would not change the interpretation.