Does טבע ever appear in Talmudic literature, such that it could mean "nature" in this context?
Yes. The Gemara Megillah 14b states:
אמרה לו: עדיין שאול קיים, ולא יצא טבעך בעולם
"She said to him, your teva has not yet issued forth from the world".
This could easily be interpreted to mean nature. (And indeed, it is).[1] The alternative is that it means your currency (being related to the word mishnaic Hebrew word מטבע - coin); the sign of a new monarch.
One very powerful piece of evidence for this explanation is the girsa of this text as presented in the Midrash Shemuel (Buber; 23):
אמרה ליה אף על פי שמשחך שמואל עדיין מטבעו של שאול קיים, ולא יצא מטבעך לעולם ,
"She said to him, even though Samuel anointed you, the currency of Saul is still extant, and your currency has still not [been] issued to the world"
Note also Rambam's formulation in Hil. G'zelah Va'avedah (5:18) which pegs the monarch's authority on the circulation of currency. This seems to be based on our passage in Megillah (as noted by R. Chaim Kanievsky):
במה דברים אמורים, במלך שמטבעו יוצא באותן הארצות, שהרי הסכימו עליו בני אותה הארץ, וסמכה דעתן שהוא אדוניהם והם לו עבדים. אבל אם אין מטבעו יוצא
[1] I assume virtually all acharonim read it this way, as were they to understand it to be a reference to currency, which is merely indicative of monarchy, they would not gloss over this point. One Acharon from whom this seems evident is R. Shelomo al-Fasi (18th cent.) who paraphrases this passage in his Kruv M'moshach to Hilcht Sanhedrin (10:5) as:
ועדיין לא נודע טבעך בעולם
If it meant currency then this statement is quite strange as the problem would not be that the currency was not known, but rather that it had not been circulated.
This is also implied by the common paraphrasing of the passage in Megillah in a variety of later sources. One such source is the Tanchuma:
מדרש תנחומא (ורשא) פרשת לך לך
א"ל הקדוש ברוך הוא לך לך מארצך ואני אודיע טבעך בעולם
Considering that I cant find a source in Chazal for the expression טבעך בעולם except in Megillah it seems probable that that is the source for the expression, and the Tanchuma does not seem t o be referring to currency. If, one thinks that the wording of the Tanchuma is not dependent on the passage in Megillah, then one would have an early source that implies teva means nature (although does not prove it).
This is also the implication of another Tanchuma:
מדרש תנחומא (ורשא) פרשת שלח
אמר אברהם לפני הקדוש ברוך הוא רבש"ע לחנם אמרת לי קח נא את בנך, א"ל לאו אלא להודיע טבעך בעולם שנאמר (שם /בראשית/ יח) כי ידעתיו למען וגו'
The same inference could be made form the Rashbash:
שו"ת הרשב"ש סימן רנ
וכך אמרו במסכת מנחות בפרק כל המנחות, אמרה כנסת ישראל לפני הקדוש ברוך הוא רבונו של עולם החזיק לי טובה שהודעתי טבעך בעולם
Another authority who evidently ascribed to a "coinage" reading of the Gemara (like Rambam's reading) is R. David Nietto (17th cent.) who writes that "teva" is a medieval word to describe nature. (De La Davina Providencia pp. 19-20 cited in Golden Doves With Silver Dots p. 139).