If someone made a vow, (even though the Gemara generally suggests that we avoid vows) then they get the reward for the mitzvah of keeping a vow.
However, if it is just a chumrah or custom, then it should fall under the category of any good deed that one is not commanded in.
The Gemara in Kidushin 31a tells us the story of a Gentile, Dama ben Nesina, who fulfilled a good deed that he was not commanded to do, yet he received a great reward. Although R' Chanina there states that it is greater to perform a commandment than to volunteer, all agree that volunteering is certainly worthy of reward.
The Ritva on this Gemara quotes the Ramban:
..."However, even such a person (i.e., one who is not obligated) is fit to receive reward; as out of his piety and goodness of his heart, he volunteered to perform a mitzvah."
It may be that there is no obligation, but it is still a worthy deed. G-d allowed room for his creations to innovate and better themselves.
If a custom beautifies a mitzvah (decorating a Torah with silver adornments) it is derived by the Talmud from the verse expounded by Ex. 15:2 "This is my G-d, and I will glorify Him." (see Shabbos 133b and Babba Kama 9b)
If a chumrah is attached to a mitzvah of some sort, then its category derives from that mitzvah. For instance, those who only eat gefilte fish on Shabbos (as opposed to whole fish) because they do not wish to accidentally remove bones on Shabbos, are keeping a custom that is a chumrah which is obviously linked to the greater mitzvah of Shabbos.
Sometimes a Jewish custom is related to "holiness" relating to the mitzvah in VaYikra 19:2 "Speak to the entire congregation of the children of Israel, and say to them, You shall be holy, for I, the L-rd, your G-d, am holy."
On this mitzvah, our Rabbis comment:
"Sanctify yourself with that which is permitted to you"
(Talmud Tractate Yevamot 20a)
In general, Jewish custom is considered "Torah" as Tosafos says in Talmud Menachos 20b ("Nifsal") "The custom of our fathers is considered to be Torah."