On Shabbat, the "Reseh" paragraph is added to Birkat Ha’mazon. Specifically, it is recited in the section of "Rahem." The reason is that in "Reseh" we make requests to G-d, and "Rahem" is the section in Birkat Ha’mazon in which we make requests. (The Mishna Berura clarifies that although we generally do not make requests on Shabbat, this is true only with regard to personal requests. Requests in standardized texts formulated by our Sages, however, may be made on Shabbat.)
On Rosh Hodesh and Yom Tob, we add "Ya’aleh Ve’yabo" in Birkat Ha’mazon. When Rosh Hodesh or Yom Tob falls on Shabbat, we first recite "Reseh" before "Ya’aleh Ve’yabo," following the standard rule of "Tadir Ve’she’eno Tadir, Tadir Kodem" – the more frequent Misva is performed before the less frequent one. Since Shabbat is observed more frequently than Rosh Hodesh or Yom Tob, the recitation of "Reseh" precedes the recitation of "Ya’aleh Ve’yabo." Moreover, the sanctity of Shabbat exceeds that of Rosh Hodesh and Yom Tob, and for this reason, too, "Reseh" should be recited first.
However, if a person mistakenly began reciting "Ya’aleh Ve’yabo" before "Reseh" on Shabbat Rosh Hodesh or Shabbat Yom Tob, and he realizes his mistake in the middle of "Ya’aleh Ve’yabo," he should complete "Ya’aleh Ve’yabo" and then recite "Reseh." This is the ruling of Hacham Ovadia Yosef, in Hazon Ovadia – Shabbat (p. 98; listen to audio recording for precise citation), where he explains that it would be disrespectful to "Ya’aleh Ve’yabo" to stop in the middle of the recitation to recite "Reseh." In such a case, the rule of "Tadir" is overridden by the concern not to disrespect "Ya’aleh Ve’yabo," and so one should complete "Ya’aleh Ve’yabo" and then recite "Reseh." He then does not have to repeat "Ya’aleh Ve’yabo."
Summary: When Shabbat coincides with Rosh Hodesh or Yom Tob, such that both "Reseh" and "Ya’aleh Ve’yabo" are added to Birkat Ha’mazon, "Reseh" is recited before "Ya’aleh Ve’yabo." If one mistakenly began reciting "Ya’aleh Ve’yabo" before "Reseh" and realized his mistake in the middle of "Ya’aleh Ve’yabo," should complete "Ya’aleh Ve’yabo" and then recite "Reseh." He does not then repeat "Ya’aleh Ve’yabo."
