My brother in law suggested that the original test in verse 4 was only whether or not the Jews will follow the rules of the Mann (הילך בתורתי אם לא). However, after they failed the test, it was found that they had not only failed to keep the rules of the Mann (תורתי), but they had also disrespected Shabbos (מצותי).
This answer feels right, especially after reading closely the context in which both verses are found.
However, this answer is not compatible with Rashi's explanation on the earlier verse, which includes the prohibition to collect Mann on Shabbos as part of the "תורה" of the Mann.
(What follows is my original answer to the question, which does not answer both questions.)
I found an answer for your first question ("What were the מצות and what were the תורות that G-d is referring to?"), but not the second ("If the test was only for the תורות , why was G-d complaining that they didn't observe the מצות as well?").
Rav Hirsch writes (on verse 28) that the essence of the sin here was pursuing livelihood on Shabbos, against God's will, demonstrating the lack of belief and trust that God will provide. Shabbos is a special phenomenon, something that is both a מצוה (something that requires physical action/refraining from action) and a תורה (something that requires one to impress something in one's mind and feelings).
He who seeks his livelihood on Sabbath against God's Will denies that
his food is sent from God at all. [...] The Jew who seeks his
livelihood on Sabbath, completely turns his back to God, and to
following His Will, places himself on his own, and tears up the bond
with all His teachings. Hence the words עד אנה מאנתם לשמר מצותי ותורתי,
for with חלול שבת is expressed the denial to pay any attention at all
to the commands and teachings of God. Sabbath is especially both a
מצוה and a תורה as are all symbolic commands. It demands a concrete
doing something or refraining from doing something (a מצוה) to
demonstrate a truth and to impress it on the mind and feelings (a
Excerpted from the Judaica Press translation, second edition (Gateshead 1999)
The use of both of these words reflect the special nature of Shabbos, and the seriousness of its violation especially when it betrays a lack of trust and reliance in one's relationship with God.
(As far as I could tell, Rav Hirsch did not address the use of different words in verses 28 and 4)