The Gemara in Masechet Shabbat (14) cites a Halacha in the name of Rabbi Parnach forbidding touching the parchment of a Torah scroll barehanded. One who must touch the scroll must do so indirectly, while holding some material in his hand, such as a cloth or Tallit. The Gemara warns that a person who touches the scroll barehanded forfeits the Misva in which he is involved, Heaven forbid. Thus, for example, if a Sofer (scribe) is writing a Sefer Torah, or if somebody studies from the scroll, lifts it to show it to the congregation, or rolls it, he forfeits the merit of the Misva if he directly touches the scroll while performing that Misva.

The Shulhan Aruch (Orah Haim 147:1) codifies this Halacha (listen to audio recording for precise citation), and adds that it applies even if one washes his hands just before handling the Sefer Torah. One might have thought that this issue depends solely on the Tum’a (impurity) of one’s hands, such that it would be permissible to directly touch the scroll after performing Netilat Yadayim. According to the accepted Halacha, however, even immediately after performing Netilat Yadayim it is forbidden to directly touch the scroll.

The exception to this rule is the scroll of Megilat Ester. Although the prohibition against directly touching a Torah scroll applies also to the Megila, Hacham Ovadia Yosef rules (based on the Kaf Ha’haim) that the one who reads Megilat Ester may touch the scroll after washing his hands, as the Megila may be treated more leniently than a Sefer Torah.

Summary: It is forbidden to directly touch the parchment of a Sefer Torah; one who must touch the scroll should make sure to do so indirectly, while holding a cloth or his Tallit. This applies even if one washed Netilat Yadayim. However, one may directly touch a Megilat Ester after performing Netilat Yadayim without a Beracha.