The form 'כְּתוּבָה' certainly exists, as you state; it is the passive participle of the root כתב, and means "written", as in: "נבואתו כתובה על הקיר" = "his prophecy is written on the wall".
However, this is not the same as the noun which designates a "marriage contract". Although there are exceptions, for the most part nouns with specific meanings are not derived via the appropriation of participles, but rather they are derived by setting a root to one of the nominal mishqalim. In this case, the relevant mishqal used is the קְטֻלָּה mishqal; hence כְּתֻבָּה.
The original poster suggests that this mishqal is rather rare; however, that is not the case. Many nouns in Biblical, Mishnaic, and Modern Hebrew are formed from this mishqal. Here is a sampling:
אֲגֻדָּה, אֲחֻזָּה, אֲלֻמָּה, אֲסֻפָּה, אֲפֻדָּה, אֲצֻלָּה, אֲרֻבָּה,
גְּאֻלָּה, גְּדֻלָּה, הֲמֻלָּה, חֲלֻקָּה, חֲנֻכָּה, חֲנֻפָּה,
חֲתֻנָּה, יְרֻשָּׁה, כְּבֻדָּה, כְּהֻנָּה, כְּתֻבָּה, נְקֻדָּה,
סְגֻלָּה, עֲבֻדָּה, עֲרֻבָּה, פְּלֻגָּה, פְּעֻלָּה, פְּקֻדָּה,
קְדֻשָּׁה, קְוֻצָּה, קְצֻבָּה, קְצֻנָּה, שְׁדֻלָּה
Finally, in order to demonstrate that our earliest sources do indeed reflect a pronunciation of the marriage contract as 'כְּתֻבָּה', it is sufficient to look at Ms. Kaufmann 50, the authoritative vocalized manuscript of the mishnah, to see to that the word is consistently written with a dagesh in the bet (the word appears there in the left column, third line).
So, to summarize: just as it does with so many other roots, so too the Hebrew language conveniently distinguishes within the root כתב between the past participle and the specific noun by the use of two separate forms, כְּתוּבָה and כְּתֻבָּה: the first is the past participle, while the second is the noun meaning "marriage contract".