Halacha requires reciting the paragraph of Ya’ale Ve’yabo whenever one recites Birkat Hamazon on Rosh Hodesh. If, however, a person forgot to recite Ya’ale Ve’yabo in Birkat Hamazon on Rosh Hodesh, he does not repeat Birkat Hamazon. Once he began reciting the fourth Beracha of Birkat Hamazon – the Beracha of "Ha’tob Ve’hametib" – he has lost the opportunity to recite Ya’ale Ve’yabo. Thus, even if a person realized his mistake immediately upon beginning this Beracha, he simply continues and does not recite Ya’ale Ve’yabo.
However, if one realized his mistake before he recited the Name of Hashem in the Beracha of Boneh Yerushalayim, then he should recite Ya’ale Ve’yabo at that point. Since he had not yet recited God’s Name in the Beracha, he still has the opportunity to recite Ya’ale Ve’yabo. He then continues as usual, reciting "Ve’tibneh Yerushalyim…"
A more complex question arises in a case where one realized his mistake only after he completed the Beracha of "Boneh Yerushalayim," but before he began the Beracha of "Ha’tob Ve’hametib." Many Siddurim present the text of a special Beracha to be recited in such a case – "Baruch Ata Hashem…Asher Natan Rosheh Hodashim Le’amo Yisrael…" However, while it is indeed proper to recite this text, it is unclear whether it should be recited with or without "Shem U’malchut," meaning, the phrase, "Hashem Elokenu Melech Ha’olam." This point is subject to a debate among the authorities. In light of this question, the Hida (Rabbi Haim Yosef David Azulai, 1724-1806) applies to this case the principle of "Safek Berachot Le’hakel" – that one should never recite a Beracha that is subject to uncertainty. As such, if a person on Rosh Hodesh realizes immediately after reciting "Boneh Yerushalayim" that he forgot Ya’ale Ve’yabo, he should recite the Beracha of "Asher Natan Rosheh Hodashim…" without the phrase, "Hashem Elokenu Melech Ha’olam." This is also the view of several other authorities, including the Ben Ish Hai, Hesed La’alafim, and Hacham Ovadia Yosef.
If one realized his mistake after he recited the words, "Baruch Ata Hashem," but before concluding "Boneh Yerushalayim," then he should immediately recite the words, "Lamedeni Hukecha," so that he will have recited a complete verse ("Baruch Ata Hashem Lamedeni Hukecha" – Tehillim 119:12), rather than saying God’s Name in vain. He should then recite Ya’ale Ve’yabo at that point and continue with "Ve’tibneh Yerushalayim…" Generally speaking, we do not employ this solution of "Lamedeni Hukecha" in the case of a recitation for which one is not required to repeat the given prayer (in this case, Birkat Hamazon). Accordingly, we would have expected that one should not use this solution in the case described above, since one who forgets Ya’ale Ve’yabo in Birkat Hamazon on Rosh Hodesh does not repeat Birkat Hamazon. However, as we saw, if one continues and completes the Beracha of "Boneh Yerushalayim," he puts himself in a situation of uncertainty regarding the Beracha of "Asher Natan Rosheh Hodashim," whether or not he should recite "Shem U’malchut." It is therefore preferable to recite "Lamedeni Hukecha" and then go back to recite Ya’ale Ve’yabo, rather than simply continuing and subjecting oneself to the controversy surrounding the text of "Asher Natan Rosheh Hodashim…"
Summary: If one forgot to recite Ya’ale Ve’yabo in Birkat Hamazon on Rosh Hodesh, then the following guidelines apply:
1) If he realized his mistake only after he began the next Beracha, he simply continues Birkat Ha’mazon.
2) If he realized his mistake after he completed the Beracha of "Boneh Yerushalayim," but before he began the next Beracha, then he inserts at that point the Beracha of "Asher Natan Rosheh Hodashim Le’amo Yisrael" (printed in some Siddurim), but without the phrase, "Hashem Elokenu Melech Ha’olam."
3) If he realized his mistake after he recited the words "Baruch Ata Hashem," but before reciting the words "Boneh Yerushalayim," then he should recite the words, "Lamedeni Hukecha" and then go back to recite Ya’ale Ve’yabo.
4) If he realized his mistake before reciting the words "Baruch Ata Hashem," then he simply recites Ya’ale Ve’yabo at that point and continues as usual with "Ve’tibneh Yerushalayim…"
