It's a very good question and is primarily aimed at the change in the text from בצלמו to בצלם.
But you additionally ask in the close of your question what is to be understood in the distinction between צלמו and דמות.
First of all, nothing went wrong. Rather it is following the paradigm that was already established in creation in the preceding days.
Like is discussed in many kabbalistic books, like Sefer Yetzirah, Otiot d'Rabbi Akiva and elsewhere, the language of the Torah, particularly in the first parsha of the first book is written with multiple layers of meaning and conveys many concepts that form the basis and foundation of our faith. The study of the various methods and rules that pertain to this type of understanding is sometimes referred to as Dikduk, the rules of Grammar.
This deceptively simple expression implies that without proper application of all these various methods, which includes Crowns, Trop, Nekudot, Letters, Verb form and structure and much more, it is impossible to truly understanding the plain meaning of the text.
With this brief introduction, the first thing to point out is that the two expressions being asked about, צלמו and דמות are actually referring to two names of G-d used in the Torah. The first, צלמו, is explicit and is actually the name intended when referring to אל עליון in the beginning of the Amidah prayer. That like is taught in Sefer HaShemot from Rabbi Moshe Zacuto, עליון is actually the numerical (gematria) value of צלמו.
It is important to note that gematria (numerical quantification, like the English derivation of Cipher) is another method contained within the rules of grammar mentioned previously like is stated explicitly in Sefer Yetzirah 1:1. The other two general categories mentioned pertain to letter formation and structure (as in Safrut) and pronunciation, both with and without sound, like in Sipur.
The second expression, דמות, requires a more complex permutation to understand to which name of G-d it is referring.
Another principle within the field of grammar is what is called in Hebrew, the Miluy (מלוי). It is the pronunciation of the individual letter name.
In trying to conceive of the concept of Miluy in relation to the individual letter, it is best to view this from the first letter used in all of creation, namely the letter Yud (י). Yud in the Miluy is (יו״ד).
It illustrates through the form of the letters of the Miluy another level of understanding of what is taught in the beginning of the Torah.
That the beginning of creation starts with Nekudah (Point, as in singularity and also what is referred to as unmeasured time, זמן בלתי משוער), proceeds to Kav (Line which includes the idea of linear time זמן משוער, past, present and future) and matures into Shetach (Field which is the beginning of space or place, מקום).
In other Torah nomenclature, this is called World עולם, Year שנה (or Change) and Soul נפש, like is found in Bereshit 15:17 and Shemot 19:18 and 20:14 (עש״ן) and in the writings of the Chida (Midbar Kedmot) as the Fangs of the Serpent (עטיו של נחש) because it was the beginning of the concealment of G-d through the creation of the world or universe.
The last two letters of the word דמות form one of the variants of the Miluy of the last letter of the Aleph-Beit, namely Tav (ת). Its Miluy in this variant is תו. The reversal of direction in this case relates to the letter transformation taught by Yaacov Avinu, which is At-Bash (א׳ת-ב׳ש).
And so the expression, דמות, is actually referring to the word אדם. And this word refers to G-d's name (יהוה) like is taught by Rabbi Menachem Azariya of Fano in his Sefer Kanfei Yonah which explains the source for the name of the first human being (Adam) meaning דומה לעליון (Similar to Above Please note the specific reference to Elyon previously) just as the text of the Torah states: בְּצַלְמֵנוּ כִּדְמוּתֵנוּ. The Kanfei Yonah points out that this is based upon the Gematria value of the Trop which appears over the letter Dalet in the word ולאדם (a value of 26, like יהוה) in Bereshit 2:20.
So what is being referenced is actually two of G-d's names (צלמו and יהוה). And these two names are the source for the division within Humans between male and female, husband and wife.
That based upon the form and structure of the letters of these two names, is found the form and structure of the human male and female that pertain to procreation. This is in keeping with the teaching of both Rabbi Moshe Kordevero in Sefer Pardes Rimonim and also Rabbi Nehunia ben HaKaneh in Sefer Temunah.
And the final element of your question relates to the change of the name relating to the Human female from a name possessing four letters to that of three (from צלמו to צלם).
This change is in keeping with the paradigm already established on day four of creation and the reduction of one of the two great(and equal) sources of light (the sun and the moon). In keeping with that paradigm, the letter Vav from צלמו was concealed leaving it with only three letters. So while the name of G-d pertaining to the wife possessed only three revealed letters, the name pertaining to the husband possessed four.
It should also be noted that these letters correspond to the number of Patriarchs and Matriarchs and that the union of these two names means "He commands Milah." (צוה מילה)
So to recap, nothing went wrong with the Divine plan and in fact, with careful reading it is understood to have executed perfectly.
