The mitzvah of blowing shofar is m'Doraita, meaning from the written Torah. (VaYikra 23:24 and VaYikra 25:9) This has the implication that in questions regarding it's fulfillment, one should be stringent. In other words, one should be stringent, for example, that what you are blowing is in fact a kosher shofar.
There are requirements on the horn itself and the animal which it comes from. See the following sources for details: Rosh Hashanah 26a and in Shulchan Aruch (and Shulchan Aruch Harav) Orach Chaim chapter 586. The following is a link providing a brief discussion of this subject:
In regard to the animal, it must be from a kosher animal, either behemah or chayah. But that presents an interesting question in terms of contemporary practice. In regard to the written Torah, it defines a kosher animal (mammal) as any animal with fully split hooves and that chews its cud (Is a ruminant.). (See VaYikra 11:1-3)
The practical side of this in regard to your question is that although you may be able to determine that the hooves are clearly split. It would not be possible to determine from an extinct animal with certainty that it was a ruminant. The Torah gives examples of animals that look like they are ruminants, but that are not. Since in the case of shofar you would be required to be stringent in your decision, on this alone, the extinct animal horn would likely be disqualified. For interesting details about the potential problems with these simanim, see the following link:
But for some, there is even another consideration.
Behemot are that class of animal that may be used as an offering in the Temple. Generally these are the herded animals mentioned in Torah, cows, sheep and goats. The horns of cows are specifically not allowed for use as a shofar.
Chayot on the other hand are definitely kosher animals. But according to some minhagim, like those who follow the Chazon Ish and possibly the Siftei Kohen, the simanim cannot be relied upon in this generation. Only chayot about which we have a definite mesorah are considered kosher for actual use and consumption. For people required to follow those minhagim, only the horns of chayot that have a definite mesorah as being kosher would be permissible for use.
Again, in the case of the extinct animal, this would exclude their usage.
For a very good discussion of the details surrounding this subject, see the following link: