This answer is partially speculation; however, I feel that it is well-supported.
When you refer to a person's "secular birthday," I assume that you refer to his or her birthday on the Gregorian calendar since that is the calendar which most of the world uses. There are, however, plenty of other civil calendars used around the world (e.g. the Ethiopian calendar and the Iranian calendar). In addition, the Gregorian calendar has only been in use since the late 16th century (already into the period of the Acharonim!). I see no reason why the Gregorian calendar should have any particular significance in Judaism. The fact that something happened on some date in some random calendar system is irrelevant. This applies to all calendars--not just the Gregorian calendar. I can't just make up a calendar that has a three-day year and say that I want a birthday party every three days!
Furthermore, there is no reason why the Hebrew calendar cannot be used for "secular" purposes. In fact, the State of Israel requires that the Hebrew date be included in all official government documents.
On the other hand, the Talmud does recognize the concept of a 365-day solar year. The Gregorian calendar is the most accurate representation of the solar year around today. It is possible that there could be some mystical associations with particular solar dates although that is outside my realm of knowledge. Still, even if there is some significance to solar dates, that would still leave the Gregorian calendar as the sole exception to the rule that non-Jewish calendars are irrelevant.