Will Rogers commented on Mother's Day:
It's a nice idea and all, but whoever came up with it must have had a hurting conscience.
I can't see Father's Day as having blatantly pagan origins. If having one day a year to focus on recognizing your father is meaningful for your father (especially if he's expecting some recognition), then by all means do it. If neither he nor you really cares about it, then there's no need.
Look, today we like to turn values into simple actions that can be ceremonialized and commercialized. You bought dad a plasma TV on Father's Day? Yotzei. The true meaning of honoring your parents is complex; the medrish says that you could feed your elderly father the finest of aged steaks while dishonoring him ("it's food dad, just eat it, old man. I've got take this phone call now."); or give him heavy labor at the millstone while honoring him ("between the two of us, one needed to work the mill, and the other needed to be drafted into the army; I'm so sorry dad, I think this is the least-difficult thing I could do to you.") It's all about the attitude.
And from the father's perspective, well, Judaism expects men to civilize themselves and shoulder the responsibility of being an involved parent. If it makes you feel better that there's one day a year when everyone publicly gives you a salute for it, then fine. Hey the Talmud says you don't always have to have perfect motivations for doing the right thing.
I'll tell you how I was taught to observe Father's Day: my father would remind me to call my grandfather (his father) and wish him a Happy Father's Day.