When Yom Tob falls on Mosa’eh Shabbat, Habdala is combined with Kiddush, in a sequence known by the acrostic "Yaknehaz," which stands for "Yayin" (the Beracha over wine), "Kiddush" (the Beracha for the Yom Tob), "Ner" (the Beracha over a candle), "Habdala," and "Zeman" (the Beracha of "She’hehiyanu").
In order to be able to recite a Beracha over a candle on this night, some preparation is needed before Shabbat. On Yom Tob, one may light a candle from an existing flame, but one may not create a new flame. Therefore, one must prepare before Shabbat a candle that will still be burning on Mosa’eh Shabbat so he can light the Yom Tob candles and light a candle for Habdala. It is recommended to purchase a 48-hour yahrtzeit candle, as a 24-hour candle will not still be lit when the time comes for Habdala on Mosa’eh Shabbat. Alternatively, one can add some oil before Shabbat to a 24-hour candle so it will remain lit for several additional hours.
Another problem that must be addressed is the prohibition against extinguishing a flame on Yom Tob. If one lights his regular large Habdala candle, he will have to allow it to continue burning until it goes out on its own. To solve this problem, some stores sell a very small Habdala candle especially for this situation, which one lights for Habdala and then puts down, and it goes out soon thereafter. Or, one can cut two or three inches off his regular Habdala candle and connect it to a plate with its wax, and use this "candle" as the Habdala candle on Mosa’eh Shabbat. These two options – purchasing a small Habdala candle, or making one from one’s normal Habdala candle – are the preferred ways of fulfilling the Misva of the Habdala candle when Mosa’eh Shabbat falls on Yom Tob.
Summary: When Yom Tob falls on Mosa’eh Shabbat, one must light a yahrtzeit candle before Shabbat that will still be burning on Mosa’eh Shabbat, since creating a new flame is forbidden on Yom Tob. As extinguishing a flame is likewise forbidden on Yom Tob, and thus one cannot extinguish the Habdala candle when Mosa’eh Shabbat is Yom Tob, it is preferable to either purchase a very small Habdala candle for that night, or to cut a small piece of one’s regular Habdala candle and stick it to a plate, such that the candle will go out on its own very soon after Habdala.
