As Oliver stated the specific ruling is based on the Rema, but the reason the Rema (and others) allowed refusing burial to the family of a get-refuser is not mentioned so I'm adding it here.
The origin of the idea of refusing burial to the family of a get-refuser or other person on violation of a beit din is Paltoi Gaon (841-858). He includes several penalties, including not counting the person in a minyan, pulling their children from school, banning a person's spouse from synagogue, declaring their wine and bread to be like the wine and bread of non-Jews, removing a person's tzitzit and mezuzot, not circumcising sons, and not eating or drinking with him. (תשובות הגאונים - מוסאפיה סימן י)
Paltoi Gaon's position is supported by many other rishonim including the Rif, Nemukei Yosef, Rashba, Rb. Yerusham, Rivash, Binyamin Ze'ev, and the Beit Yosef.
According to the Binyamin Ze'ev, who is the one cited by the Rema, the reason the beit din can do this is as follows. The gemarah gives the courts permission to uproot a mitzvah if it is for the sake of the Torah. (Yevamot 90b). Additionally, while one cannot take a shevua that only prevents a mitzvah, one can take a shevua that prevents both mitzvah and non-mitzvah activities. For example, if one makes a shevuah not to eat matza for a full year, they are forbidden to eat matzah on the first night of Pesach even though these is a positive Torah commandment to do so. Similarly, a Jew can take a shevua not to help a person in any way because that will include non-mitzvah things even though it may include not performing a positive mitzvah (like burial or circumcision). (Binyamin Ze'ev 289 and Yabia Omer EH 8:25).