My understanding is it all depends on what the yeast grew. If it grew off of wheat or barley or the like, it's a problem.
As I heard from one OU lecture, an interesting ingredient in today's food production is "torula yeast." Paper makers had all this "pine wood soup" from cooking their shredded pulp, which they couldn't just pour down the drain for environmental reasons. So a type of yeast which loves to eat pine soup became popular, breaking down the waste product much better -- and as it turns out, torula yeast is a nice food ingredient, and all it's grown off is pine wood, so there are virtually no kashrut or allergy issues with it! (May still need Passover supervision, though.)