The Rambam says one must leave at least 40 (some versions say 4) hairs on each side:
The Sages did not determine the amount [of hair] which must be left in
the corners of our temples. We have, however, heard from our elders
that one must leave at least forty hairs.
However, it seems he would allow plucking:
One may remove the [hairs from] the corners [of our heads] with
scissors. The prohibition applies only to total removal with a razor.
Shulchan Aruch says one should not even use scissors, but it's unclear if that extends to plucking:
He is not liable unless he uses a razor. And there are those who
forbid scissors that shave like a razor; and their words should be
regarded.
The Chasam Sofer has a lengthy analysis of the question, in which he concludes plucking is not an issue, especially when done without intent, like while combing one's peyos.