If a family pet is dying, is there a specific written prayer to say for the pet?
Not that I'm aware of. But Jewish law traditionally has a high value on concern for animals' pain. So perhaps something like:
Almighty God, Whose mercy is on all His creatures; You commanded us not to stand idly by when a donkey is suffering from a crushing burden, and thus it's heartbreaking for us to see our pet suffer like this. We beseech You to ease its pain; to give us the strength to endure this difficulty; and to grant us compassion for all Your creations.
(I must add that in many cases the prayer may not be for a miraculous recovery, merely an end of the pet's suffering.)
I don't think so. I was studying in Israel when I found myself in that situation and asked one of the rabbis (don't remember which) then present at the Shalom Hartman Institute, who advised me to pray my own words from my heart. (I did, and made it home in time without adding to my pet's suffering.)
I modeled what I said loosely on the various conclusions to the t'filah given in Tractate B'rachot 16-17. (It was kind of a mash-up and I didn't write it down.)