Where does the Rambam state that the non-Halachic parts of the Gemara - for instance, Agadata, Science etc - aren't to be taken literally or simply aren't factually correct?
I suppose you are referring to Rambam's introduction to Perek Helek. See the English translation of Abelson on WikiSource:
The first class is, as far as I have seen, the largest in point of their numbers and of the numbers of their compositions; and it is of them that I have heard most. The members of this class adopt the words of the Sages literally, and give no kind of interpretation whatsoever. With them all impossibilities are necessary occurrences. This is owing to their being ignorant of science and far away from knowledge. They do not possess that perfection which would spur them on of their own accord, neither have they found any means for rousing their attention. They think that in all their emphatic and precise remarks the Sages only wished to convey the ideas which they themselves comprehend, and that they intended them to be taken in their literalness. And this, in spite of the fact that in their literal significance some of the words of the Sages would savour of absurdity.