Rashi on that same verse (Numbers 15:31) explains that the first two commandments were heard directly from God, and the rest from Moshe. That's why it's worded כי דבר ה' בזה - he transgresses the part of the commandments which was experienced as 'the word of God' (as opposed to the word of Moshe). Similarly perhaps we can explain the wording of the Sifrei (which is explaining that same verse) דיבור ראשון to mean the first part of the commandments, namely the part that was heard directly from God.