The Shulhan Aruch (Orah Haim 196) writes that if somebody eats non-kosher food, no Beracha may be recited over that food, neither before it is eaten nor after it is eaten. The Gemara comments that reciting a Beracha over forbidden food constitutes a form of blasphemy, as one mentions G-dís Name over food which G-d declared prohibited.
By the same token, if three people eat together, and one of them eats non-kosher food, they do not form a Zimun. A Zimun requires "Kebiíut" Ė that the group sat down together and formed a single entity. If one personís food is forbidden for consumption, the others cannot be said to join with him to form a single entity, since they cannot partake of his food, and thus a Zimun cannot be recited.
The Poskim discuss numerous examples where one member of a group eats kosher food which is forbidden for other members of the group. One classic example, addressed by the Kaf Haíhaim (Rav Yaakob Haim Sofer, 1870-1939), is when three people eat together during the Nine Days, and one of them eats meat. Some communities (such as ours) follow the custom to refrain from meat during the Nine Days, whereas other communities permit eating meat during this period. The Kaf Haíhaim thus writes that if two people who observe this custom are eating dairy foods, and the third Ė who does not observe this custom Ė is eating meat, they cannot form a Zimun, since they cannot eat from each otherís food. The two who observe the custom to avoid meat cannot eat the third personís food, because he is eating meat, and that third person cannot eat the other two peopleís food, because he is eating meat and they are eating dairy foods. Therefore, they cannot form a Zimun.
However, as Hacham David Yosef noted, there is a very important exception to this rule. If they have Parve bread, which is permissible for all of them, then they can form a Zimun. Since there is bread which can, Halachically speaking, be shared by everyone in the group, this suffices for them to combine into a single group for the purposes of Zimun.
Another example, discussed by Rav Yosef Shalom Elyashiv (1910-2012), is the case of people who eat Masa together on Pesach, and they follow conflicting stringencies. One follows the practice not to eat hand-made Masa, and the others follow the practice not to eat machine-made Masa. Since they cannot each otherís food, they do not form a Zimun.
It must be noted that this applies only if the members of the group cannot eat each otherís food, but not if the restriction applies in only one direction. An example would be if two people eat dairy foods that are not Halab Yisrael, relying on Rav Moshe Feinsteinís ruling permitting the consumption of such products in the United States, whereas the other does not follow this ruling and consumes only Halab Yisrael dairy products. Although the third person cannot eat the other two peopleís food, they can eat his food, and this suffices for them to combine to form a Zimun.
Summary: If three people eat together and one eats non-kosher food, they do not combine to form a Zimun. And even if they all eat kosher food, but they eat food which is Halachically forbidden for one another, they cannot form a Zimun. An example is during Nine Days, when two people who refrain from meat during the Nine Days eat dairy foods together with somebody who is eating meat, following his custom which allows meat during the Nine Days. Since they cannot eat each otherís food, they do not form a Zimun, unless there is also Parve bread from which they can all partake. If two people are eating dairy products which are not Halab Yisrael together with somebody who is careful not to consume such products, and so he eats Halab Yisrael products, they form a Zimun, since the first two are allowed to partake of the third personís food.