(Bruchos 36a) only the primary-fruit are prohibited, but this is only outside Israel. In Israel, the secondary-fruit is prohibited.
Plants with more than one edible "fruit" (fruit, berries, flowers, shells, leaves, stalks and wood) only have one primary-fruit. That primary-fruit is usually the reason we cultivate the plant, and would be ha'aitz, any secondary-fruits are ha'adamah. Outside Israel, only the primary fruit is prohibited in the first 3 years, and any secondary fruit, like leaves are permitted to eat outside Israel.
Also, (36b), outside Israel, it is even permitted to eat any primary-fruits which are questionable if they are under 3 years. They're only prohibited outside Israel if you know for a fact it is under 3 years - you know when it was planted and picked.
edit: Also, peels and shells that protect the primary-fruit (without the shell it'll die) are prohibited along with the primary-fruit.