If someone has a sibling who is mechalal shabbos and they try to tell them what they're doing is wrong but the sibling refuses to listen does one have to go even further in trying to stop them?
The following summary would seem to imply that your particular case would not require you to continue to "give mussar" to your sibling. I bolded the sections that would seem to apply. This is based on the Kitzur Shulchan Aruch 29:16 which I copy in after the summary. As always CYLOR to see what the best thing to do would be.
Someone who sees his friend who sinned, or who is walking in a bad path—it is a Mitzvah to return him to good, and to know that he is sinning in his evil ways, as it is written: “You shall surely rebuke your friend” .
Primarily, the biblical mitzvah of rebuke still applies nowadays. .
One who rebukes a friend , must rebuke him privately and gently, and must tell him that he is only rebuking him for his own good, and to bring him to life in the World to Come. .
Anyone who is in a position to rebuke, and does not rebuke, is punished for that very sin .
One may only rebuke if he thinks that if his friend will listen. If he knows that he won’t listen, it is forbidden for him to rebuke him .
Just as one is commanded to speak when it will be heard, so too one is commanded (or it is an obligation even) to not speak when one will not be heard .
If one sees a person sin unintentionally and knows that he will not heed rebuke, if the sin isn’t explicit in the Torah, one shouldn’t rebuke that person. Some say one should rebuke a person only if he is familiar with him. 
The mitzvah of rebuke does not apply to a person who has rejected the yoke of Torah or violates Shabbat in public. 
Bei’ur Halacha 608 s.v. Aval writes that the mitzvah of Tochacha doesn’t apply to a person who has totally rejected the yoke of Torah such as someone who violates Shabbat in public. The reason for this is that he is not included in the Torah’s description of “one’s fellow.” Aruch HaShulchan 608:7, Tzitz Eliezer 17:36, and Shevet HaLevi 1:205:608 agree.
Just like any halachic inquiry should be brought to a qualified posek, questions regarding this mitzvah should certainly be brought to a qualified posek.
בְּמֶה דְּבָרִים אֲמוּרִים כְּשֶׁהוּא מְדַמֶּה שֶׁיִּשְׁמַע לוֹ. אֲבָל אִם יוֹדֵעַ בּוֹ שֶׁלֹּא יִשְׁמַע לוֹ, אָסוּר לְהוֹכִיחוֹ, דְּאָמַר רַבִּי אִילְעָא מִשּׁוּם רַבִּי אֶלְעָזָר בְּרַבִּי שִׁמְעוֹן, כְּשֵׁם שֶׁמִּצְוָה עַל אָדָם לוֹמַר דָּבָר הַנִּשְׁמָע כָּךְ מִצְוָה עַל אָדָם, שֶׁלֹּא לוֹמַר דָּבָר שֶׁאֵינוֹ נִשְׁמָע. רַבִּי אַבָּא אוֹמֵר חוֹבָה שֶׁנֶּאֱמַר אַל תּוֹכַח לֵץ פֶּן יִשְׂנָאֶךָ, הוֹכַח לֶחָכָם וְיֶאֶהָבֶךָ. (יבמות דף ס"ה ע"ב) (אוֹרַח חַיִּים סִימָן תר"ח
If your sibling will not listen, you have done all that you can.
just a thought,
you need to estimate whether what you do will be helpful or harmful.
no straight answer here. each case is different. consult with LOA
if a sibling has gone off it is likely due to ruach hatuma (such as looking at bad things or reading bad things). so the solution is to try to bring some light of kedusha to banish the darkness such as getting someone to study torah with him.
not much else you can do. once the yetzer hara has camped in, it is very hard to get him out.