As DanF already answered, the Torah only commands us to go for the 3 festivals.
However, if you look in the Mishna in Yoma, 7:2, and its commentators, you will learn that there were people who attended the Yom Kippur services in the Bet HaMikdash - and that it was a Mitzva to do so.
הָרוֹאֶה כֹהֵן גָּדוֹל כְּשֶׁהוּא קוֹרֵא, אֵינוֹ רוֹאֶה פַר וְשָׂעִיר הַנִּשְׂרָפִים. וְהָרוֹאֶה פַר וְשָׂעִיר הַנִּשְׂרָפִים, אֵינוֹ רוֹאֶה כֹהֵן גָּדוֹל כְּשֶׁהוּא קוֹרֵא. וְלֹא מִפְּנֵי שֶׁאֵינוֹ רַשַּׁאי, אֶלָּא שֶׁהָיְתָה דֶרֶךְ רְחוֹקָה, וּמְלֶאכֶת שְׁנֵיהֶן שָׁוָה כְאֶחָת
One who saw the High Priest reading the Torah couldn't see the goat and bull being burnt. And vice-versa. Not because it was forbidden, but because they happened simultaneously at 2 locations far away from each other.
The Bartenura there explains that watching the Yom Kippur service being performed is a Mitzva.
וּרְאִיַּת עֲבוֹדָה מִצְוָה הִיא מִשּׁוּם בְּרֹב עַם הַדְרַת מֶלֶךְ
From a practical aspect, keep in mind that between Yom Kippur and Sukkoth there are 4 days (and often one of them is Shabbat). So those that lived further away would probably plan to be in Jerusalem by Yom Kippur so as not to be "camping out" during the fast day.