I know that being Jewish is passed down maternally; however being a Kohen is passed down paternally. I was wondering in a theoretical situation, if a Kohen father were to have a child with a non-Jewish mother, what is the status of the child? I assume he wouldn't be Jewish, but is he still a Kohen? If he were to convert to Judaism, would he be considered a Kohen Jew, or would he be treated the same way as a Jew who didn't have a Kohen father?
Kohanim need to be Jews first. The child of a non-Jewish woman is not a Jew, and thus cannot be a kohen. Even if a kohen marries a convert (which halacha forbids, but if he did), so his child is Jewish, that child is not a kohen. Marrying someone who isn't even Jewish could not produce a better result in terms of the child's status.
If the child converts he has no halachic relationship to his birth parents. Thus, his halachic father is not a kohen and so neither is he. Converts are "ben Avraham Avinu".
This article from Chabad discusses children of forbidden marriages with kohanim. It takes as a baseline that the women are Jewish (kohanim face special restrictions in marriage). If even those children have defective kohen status, how much the more so would non-Jewish children be blocked from that path?
He would be a Ger. Not a Kohein. Not a Levi. Not a Yisrael. A Ger. Gerim (converts) are not halachicly related to their biological "relatives". It's a clean slate.
It's worth noting that in general the offspring of a Kohein with a woman he can't marry are not Kohanim anyway, but Challalim.