Is it a violation of halakah to pick your nose (unrefined act, possibly a violation of being a holy nation)? Would there be a difference in halakah if it were the sabbath (possible violation of smoothing)?
The Gemara (Shabbos 12a) says:
ת"ר אין פולין ברה"ר מפני הכבוד כיוצא בו אמר (רב) יהודה ואמרי לה רבי נחמיה אין עושין אפיקטויזין ברה"ר מפני הכבוד
Our Rabbis taught: One must not pick [lice from his garments] in the street out of decency. In like way R. Judah-others state, R. Nehemiah-said: One must not cause himself to vomit in the street, out of decency.
The trend of this passage suggests to me that the halacha would say that if it isn't common decency to pick one's nose in public, one shouldn't do so.
Shulchan Aruch HaRav 92:7 says that one should not touch the Tzoas Ha'Af during Tefila only through a handkerchief (Beged). This would indicate to me that there is no prohibition against picking a nose, since if was prohibited outright then why would it be mentioned as prohibited by Tefila.
Regarding Shabbos - I have heard that HaRav Ovadia Yosef Shlita has said it is permitted, however I have been unable to source it.
Chacham Ovadia Yosef in Yabia Omer 5:30 writes that picking ones nose is a davar she'eino mitkavein for pulling out hairs and should be permissible, but we are stringent and forbid it when it's a p'sik reisha. He also brings a case of rubbing ones beard is also a p'sik reisha seemingly. He then brings a story of the Arizal who on shabbas touched his beard and left it there until the end of shabbas because of this issue of ripping out hairs. See there.
Perhaps this Medrash can help. The Medrash 67:9 says that whilst Rivka described her animosity for the Bnos Ches to Yitzchok she picked her nose and then flicked the away the booty in order to show her revulsion for them. The language of the Medrash sounds as if she did it by hand. It would seem therefore that this is an acceptable thing to do otherwise Rivka wouldnt have done it. One could say that she permitted herself to do it in order to persuade Yitzchok to send away Yaakov, but seeing that he was blind he wouldnt have been able to see.
The Gemara (Pesahim 112a), as Rashi explains, says that someone who regularly places his hands between his upper lip and his nostrils creates a ladder for Ruah Ra'ah to bring fear into him.