In Hilkhoth Lolav 7:5, the Rambam writes as follows:
5 arba'ah minin ellu misswah ahhath hen ume`akvin zeh eth zeh wa-khulan niqra'in misswath lolav wa-ein pohhathin mehen wa-ein mosifin 'aleihen. wa-im lo nimssa ehhadh mehen ein mevi'in tahhtaw min ahher had-domeh lo.
Translation:
These four species together constitute a single misswah and the absence of any one of them prevents the misswah from being performed. And they are all together referred to as 'misswath lolav' and we do not subtract from them and we do not add. And if one of them cannot be found, we do not bring another species in its place that is similar to it.
And in the seventh halakhah there, he writes as follows:
7 kammah nottel me-hen? lolav ehhadh wa-ethrogh ehhadh ushnei baddei `aravah usheloshah baddei hadhas. wa-im rassah le-hosif ba-hadhas kedhei sheyyihyeh agudah gadholah - mosif wa-noy misswah hu. aval she'ar ham-minin ein mosifin 'al minyanam wa-ein gor'in mehen wa-im mosif o gara' - pasal.
Translation:
7 How many of each [of the four species] do we take [to perform the misswah of nettilath lolav]? One lolav and one ethrogh and two willow branches and three myrtle branches. And if one desires to add to the number of myrtle branches so that there will be a large bundle then he may add and it serves as a decoration to the misswah. But with regard to the other species, we do not add to their number and we do not subtract from them, and if one either added or subtracted he invalidates their use to fulfill the misswah.
So, essentially, if each particular species is me-`akev to the other, and if even adding or subtracting from the number of each kosher species causes the agudah to become pasulah, then kol she-ken that a fifth species added to the agudah would cause it to become pasulah and prevent from being yossei yedhei hhovatho.
Le-`aniyuth da'ati, according to the Rambam, to do so would prevent one from fulfilling the misswah of lolav. However, the Ba'alei Tosafoth on Sukkah 31b (dh"m Ho'il) state differently regarding the waving of five species:
de-le-`olam yedhei misswah yassa ella 'over 'al bal tosif
Translation:
[In doing this] a person will always fulfill the misswah [of lolav] but he will nevertheless be in violation of the prohibition of adding to a misswah.
Addendum:
There are those are of the opinion that the text of the Rambam there in Hilkhoth Lolav 7:7 should be emended to read not "pasal" but "lo pasal". This is based largely on a the testimony of Rabbenu Avraham ben HaRambam who says that his father later changed his mind based on a similar argument as brought by the Ra'avadh in his hasaghoth on the Mishneh Torah. Notably, HaRav Yosef Qafihh z"l is of this opinion - which he explains in a comment (#20) which lasts for several pages - and rules that although the textual tradition for the "pasal" reading is strong and consistent, the text should nevertheless be changed to read "lo pasal" and the original reading should be viewed as stemming from a mahadhurah qamma written in the Rambam's youth.
Although the halakhah is strictly speaking of adding or detracting from the number of each of the arba`ah minim and not explicitly regarding the addition of an entirely new species in addition to the four (although 7:5 does address the exchange of kosher species for a similar non-kosher one), I believe that this is due to the fact that while it was apparently a common error to equate species with one another, to add a fifth species may not have even been a hawa mina that was prevalent enough to merit discussion.
Hope this helps. Kol tuv.