As explained in naming convention for a Jew with a non-Jewish father? such a person can be called to the Torah using his grandfather's name. Similarly, he would follow the minhagim of his grandfather. This is like a ba'al teshuvah whose (Jewish) father does not have a specific minhag, but who can identify the minhagim of his family farther back. That is in this case, his closest relative is his mother's father and that is the line that he should follow.
Since his grandfather can be identified, the family minhag can usually be determined. If the family minhag cannot be determined, then it would be like any other ba'al teshuvah who would then tend to follow the minhag of his community or the rabbi who has taught him.
This can be seen at Baal Teshuva Minhag which points to a number of rabbonim who state that if the immediate family minhag is not identifiable, then the minhag of the community the ba'al teshuva has joined should be followed.