When Rosh Hodesh falls on Shabbat, we recite a special Amida for the Musaf prayer, which features the section of "Ata Yasarta." This section incorporates the themes of both Shabbat and Rosh Hodesh, and is thus recited in place of the standard Shabbat Musaf and the standard Rosh Hodesh Musaf.
If one mistakenly recited the Musaf service of a regular Shabbat, or the Musaf service of a regular Rosh Hodesh, he has not fulfilled his obligation. Since the Amida he recited spoke only of Shabbat or only of Rosh Hodesh, he did not fulfill his requirement, and he must therefore pray the correct Musaf prayer for Shabbat Rosh Hodesh. This is the ruling of the Hacham Ovadia Yosef, in his work Hazon Ovadia (end of Hilchot Hanukah, p. 313; listen to audio recording for precise citation), based on the position of the Hida (Rav Haim Yosef David Azulai, 1724-1806) in his Birkeh Yosef (490:8).
The section of "Ata Yasarta" on Shabbat Rosh Hodesh concludes with a Beracha that mentions both Shabbat and Rosh Hodesh: "Baruch Ata Hashem Mekadesh Ha’Shabbat Yisrael Ve’Rosheh Hodashim." If a person recited the proper Musaf for Shabbat Rosh Hodesh, but he forgot to mention Shabbat or Rosh Hodesh in the concluding Beracha, he has nevertheless fulfilled his obligation. Since he mentioned both Shabbat and Rosh Hodesh earlier in the section of "Ata Yasarta," he fulfills the requirement even if he omits one or the other in the concluding Beracha.
This applies also to somebody who mistakenly recited the word "Musaf" instead of "Musfeh" in the "Ata Yasarta" section. In this section we make reference to two Musaf offerings that were brought in the Mikdash – the offering for Shabbat, and the offering for Rosh Hodesh – and we therefore employ the plural form of "Musfeh." Nevertheless, if one mistakenly recited the singular form, "Musaf," he has fulfilled his obligation.
An interesting question arose concerning a person who on Shabbat Rosh Hodesh recited the standard Musaf for Shabbat, and then, realizing his mistake, recited the standard Musaf for Rosh Hodesh. Although he did not recite the Musaf for Shabbat Rosh Hodesh, he did recite the Musaf for Shabbat and the Musaf for Rosh Hodesh. One may have thought that such a person has fulfilled his obligation, since in the end he prayed the Musaf prayers for both occasions.
However, Hacham Ovadia Yosef (ibid. p. 317; listen to audio recording for precise citation) maintains that a person cannot fulfill his Musaf obligation on Shabbat Rosh Hodesh by reciting two separate Amida prayers. Once the Sages legislated reciting the Amida of "Ata Yasarta" on Shabbat Rosh Hodesh, one fulfills his obligation only by reciting this Amida. Therefore, even if a person recited both the Amida for Shabbat and the Amida for Rosh Hodesh, he must pray a third time, with the proper text of "Ata Yasarta."
Summary: On Shabbat Rosh Hodesh, one must recite the special Amida of "Ata Yasarta" for the Musaf service. If one recites either the standard Musaf for Shabbat or the standard Musaf for Rosh Hodesh – and even if he recites both these Musaf prayers – he does not fulfill his obligation until he recites the Amida for Shabbat Rosh Hodesh.
