If three people ate together, and two of them forgot that they needed to conduct a Zimun, and recited the first three words of Birkat Ha’mazon – "Baruch Ata Hashem" – before realizing their mistake, it is questionable whether or not they should continue reciting Birkat Ha’mazon. Seemingly, they should be able to correct their mistake by reciting the words "Lamedeni Hukecha," such that they would have recited the complete verse in Tehillim, "Baruch Ata Hashem Lamedeni Hukecha." This strategy is used when a person begins reciting a Beracha which he is not supposed to recite, and realizes his mistake after reciting the words "Baruch Ata Hashem." In order to avoid reciting a "Beracha Le’batala" (Beracha in vain), he should recite the words "Lamedeni Hukecha" so that he will have recited a verse from Tehillim, and not an unwarranted Beracha. It would appear that this strategy can also be used by people who mistakenly began reciting Birkat Ha’mazon before a Zimun. They can, seemingly, correct their mistake by reciting "Lamedeni Hukecha" so that they will not be considered as having begun Birkat Ha’mazon, and thus a Zimun can still be recited.
However, the validity of this course of action depends on a question raised – and left unresolved – by the Ben Ish Hai (Rav Yosef Haim of Baghdad, 1833-1909), in Parashat Vayakhel. He addresses the case of one who began reciting the Amida for Minha, and after reciting the first three words – "Baruch Ata Hashem" – he heard Kaddish being recited. The question becomes whether he should recite "Lamedeni Hukecha" so that he will be considered as having not yet begun the Amida and thus can respond to Kaddish. While at first there seems to be no reason not to allow this, the Ben Ish Hai is uncertain, as it is possible that the recitation of "Lamedeni Hukecha" was instituted as a solution only to avoid a "Beracha Le’batala." In a case where there is no "Beracha Le’batala" at stake, and one wants to discontinue the Beracha he had begun so that he can perform a different Misva – such as answering Kaddish – it is questionable whether it is proper to discontinue one’s Beracha by reciting "Lamedeni Hukecha." (The Ben Ish Hai does say, however, that if one heard Kaddish after reciting "Adon-nai Sefatai Tiftah…" before beginning "Baruch Ata Hashem," then he can answer Kaddish and then begin the Amida anew afterward.)
This question of the Ben Ish Hai is relevant also in the case of people who began reciting Birkat Ha’mazon and then realized that they need to recite a Zimun. It is uncertain whether they should recite "Lamedeni Hukecha" so a Zimun can be recited, or simply continue with Birkat Ha’mazon without a Zimun.
Summary: If people began reciting Birkat Ha’mazon without conducting a Zimun, and they realized their mistake after reciting the words "Baruch Ata Hashem," it is uncertain whether they should discontinue Birkat Ha’mazon by reciting the words "Lamedeni Hukecha," or simply continue with Birkat Ha’mazon without a Zimun.