I think that we need to clarify what is "expected to be virgin". A divorced woman, or a widow, or a woman which is not virgin because of a reason x, has no problem to marry. To marry for a non virgin woman is not prohibited. For most men, to marry a non virgin woman is not prohibited.
An exception is the Kohen Gadol (Great priest) who is prohibited to marry a divorced woman (as last kohanim) or a widow (Leviticus 21, 14):
אַלְמָנָ֤ה וּגְרוּשָׁה֙ וַחֲלָלָ֣ה זֹנָ֔ה אֶת־אֵ֖לֶּה לֹ֣א יִקָּ֑ח כִּ֛י אִם־בְּתוּלָ֥ה מֵעַמָּ֖יו יִקַּ֥ח אִשָּֽׁה׃
A widow, or a divorced woman, or defiled, or a harlot, these shall he not take; but he shall take a virgin of his own people t wife.
No woman is in duty to marry a "virgin man".
Aside from this issue, there are difference in marriage settlement between a virgin and a non virgin woman. The virginity is linked to the hymen (virginal membrane).
The verse says in Exodus 22, 16:
אִם-מָאֵ֧ן יְמָאֵ֛ן אָבִ֖יהָ לְתִתָּ֣הּ ל֑וֹ כֶּ֣סֶף יִשְׁקֹ֔ל כְּמֹ֖הַר הַבְּתוּלֹֽת:
If her father refuses absolutely to give her to him, he shall pay money according to the dowry of virgins.
Rashi explains that the amount is written in Deuteronomy 22, 29:
וְ֠נָתַן הָאִ֨ישׁ הַשֹּׁכֵ֥ב עִמָּ֛הּ לַֽאֲבִ֥י הַֽנַּעֲרָ֖ חֲמִשִּׁ֣ים כָּ֑סֶף...
From those verses the Talmud (Ketubot 11a) learns the minimal value of a Ketuba for a virgin. For a non virgin, the minimal value is lower. (this last paragraph is oversimplified, to be accurate a length paragraph would be necessary).
Regarding extramarital (rules are different for men and women, because a woman becomes prohibited to her husband and not the contrary, a man can spouse several women and a woman cannot have several husbands), or non marital sexual relationship, indeed it is prohibited for men and women. But the past is not linked to the ability to marry.
A problem is when there is an accusation of lying (I do not explain the cases), when the bride was supposed to be virgin and is accused is accused of lying by the groom. If the bride discovered that the groom who was supposed to be "virgin" is not, this is not a case of taking a legal action. I think that for a man the fact to be "not virgin" is not different from every other sins, because this is not a special, different from other problems for a man, I have no proof from halachic litterature.
Is there any evidence that it is more desirable for a man to be a virgin at the time of his marriage?
Are you assuming that there is evidence that it is more desirable for a woman to be a virgin? | Do you mean desirable to God? Desirable according to the Torah? Desirable according to some social convention today? According to some social convention from thousands of years ago? Consider clarifying.