In the following article, the author makes a compelling argument for why Judaism doesn't always approach scripture literally. However, although there are many examples throughout the Bible for the use of allegory, some passages are in fact taken literally. For example, Kosher laws are implemented in that manner. How then can you tell the difference? Is there any method or approach for telling the difference between whether allegory or literalism is appropriate? Other questions on Stack Exchange mentioned allegory and literalism, but I couldn't find any that directly address this specific point.
The simple answer is that this determination is not much different than other determination about the meaning of a text, which is made on a case-by-case basis. One has to use logical analyses of the information presented, both inherently (regarding clues in the text itself) and in a broader context, including whether or not there is a relevant and reliable (Mosaic) tradition one way or the other.
For example, in Bava Bathra 15a, a view is presented that the entire story of Job is just an allegory describing events and individuals that never were. This approach is ultimately rejected based on an analysis of the presentation of specific, seemingly irrelevant details.
An "allegory" is essentially a way of describing the patterns of an idea or event with another symmetric idea or event. This complicates the analysis inasmuch as there are often symmetries and "supersymmetries" in history, creation, and commandments.
For example, the commandments themselves are often symbols (i.e. allegories) put into practice. Thus, traditionally, tefillin are literally worn as signs upon the arm and between the eyes. Likewise, maaseh avoth siman l'banim means that actual historical events that occurred to the forefathers, are a sign of events to come that would befall their descendants.
Nonetheless, with regard to more general rules, I would venture that an idea that is frequently described as a "secret" (e.g. sod bereishith) is more likely to include more encoded information. And contradictions may also be a likely clue that at least one of two passages are not literal.
How to really understand the Bible
Good question. For one, anthropomorphic statements about G-d should always be taken figuratively as metaphors. For example, Maimonides understood the "Garden of Eden" story to be a parable. Ralbag understood that the sun did not really stand still for Joshua at Gibeon, this was a feeling Joshua felt "as if" the sun stood still.
In his work called Chelek, Maimonides writes that those who take Midrashim literary are "fools," while those who reject them out of hand are also "fools." Midrashim are imaginative parables, sermons designed to teach moral lessons. People should mine Midrashim for lessons about proper behavior. The same can and should be applied to Torah study.
In his Guide of the Perplexed 2:48, Maimonides explains that whenever the Bible says that G-d did something, it is not that G-d actually did it but what occurred was the result of the laws of nature. Since G-d created the laws of nature, the Bible attributes the event to G-d, since, though not the direct cause, G-d is the ultimate cause.
Be it as it may, even some biblical laws are metaphors. For example, the laws of tefillin, though very ancient, were probably not from Sinai. But they are still sacred and very powerful. But as the Rashbam points out when comparing them to Mishle (Proverbs), the original intent was figurative. The term Haza"I "halakhah le-Moshe mi-Sinai means that they should be related to "AS IF." A slippery slope?
No, Maimonides stressed that the Kosher laws, tefillin, and all biblical and rabbinical enactments should be observed as the rabbis explained them (as per Deuteronomy 17).