The Shulhan Aruch (Orah Haim 180:1) writes that it is not proper to recite Birkat Ha’mazon at an empty table, and one should therefore leave the leftover bread, and crumbs, on the table for the recitation of Birkat Ha’mazon. If all the bread was finished, one should not bring another full loaf to the table, as this was the practice of the pagans. If one has a partial loaf of bread, it may be brought to the table for Birkat Ha’mazon. A complete loaf that had been on the table during the meal may be left on the table for Birkat Ha’mazon.
The Ben Ish Hai (Rav Yosef Haim of Baghdad, 1833-1909) writes (in Parashat Shelah) that it is customary also to leave the bones, shells and peels on the table for Birkat Ha’mazon. These oftentimes have "Nisosot Kedusha" (sparks of holiness) within them, and reciting Birkat Ha’mazon with these on the table can have the effect of extracting the sparks. If one finds it uncomely to leave these things on the table, he may collect them onto a plate and move them to the side, but they should be left on the table.
Summary: It is proper to leave leftover bread on the table for Birkat Ha’mazon. If all the bread was eaten, one should not bring a new, complete loaf for Birkat Ha’mazon, but one may bring a new piece of bread. It is also proper to leave on the table bones, shells and peels for Birkat Ha’mazon.
