The text does not directly tell us what they did wrong. The rabbis offer several interpretations (recorded in various places in the talmud and collected in Vayikra Rabbah, a midrash collection).
On 2:23: Rabbi Akiva sticks close to the text, saying that they died because they offered "strange fire", which he does not define. Rabbi Yose says they died because they drew near to God; they entered the holy of holies at a time other then when service was commanded, as it is written: "When they drew near before the Lord and died" (Lev 16:1).
Rabbi Eleazar agrees that they died for drawing near; further, he says, the "strange fire" was not the fire from the continual offerings but was ordinary "secular" fire. (To use a modern analogy, instead of taking a flame from the continual fire kept for that purpose, they pulled out matches and started their own.) R. Eleazar doesn't bring a source for this.
Bar Kappara, in the name of R. Yirmiyahu b. R. Eleazar, said they died on account of four sins: drawing near, offering (when not commanded), the strange fire (offering the wrong way), and for not seeking counsel of each other (acting on their own initiative without checking). Another interpretation, noted by R. Yishmael among others, is that they were drunk at the time, because immediately after this incident the torah admonishes Aharon to not enter the sanctuary after having drunk wine. This is explained in Vayikra Rabbah 12:1 (as translated here):
This is analogous to a king who had a faithful attendant. [When he found him standing at tavern entrances, he severed his head in silence and appointed another attendant in his place. We would not know why he put the first to death, but for his enjoining the second thus, “You must not enter the doorway of taverns,” from which we know that for such a reason he had put the first one to death. Thus [it is said], “And fire went forth from before the Lord and consumed them, and they died before the Lord.” But we would not know why they [Nadab and Abihu] died, but for His commanding Aaron, “Do not drink wine that will lead to intoxication.” We know from this that they died precisely on account of the wine. For this reason Scripture showed love to Aaron by directing the divine utterance to him alone.
Whatever their error, Vayikra Rabbah 2:23 also notes that God was grieved by their deaths; this wasn't wrathful punishment but unfortunate consequence. This is the meaning of "Through them that are nigh unto Me I will be sanctified" (Lev 10:3).