Chacham Ovadia Yosef in Yabia Omer 5:30 writes that picking ones nose is a davar she'eino mitkavein for pulling out hairs and should be permissible, but we are stringent and forbid it when it's a p'sik reisha. He also brings a case of rubbing ones beard is also a p'sik reisha seemingly. He then brings a story of the Arizal who on shabbas touched his beard and left ithis hand there until the end of shabbas because of this issue of ripping out hairs. See there.
fixed the syntax to facilitate a clearer understanding of the authors words