The Shulhan Aruch (Orah Haim 584) writes that Hallel is omitted from the service on Rosh Hashanah and Yom Kippur, despite the fact that we recite Hallel on every other Yom Tob (listen to audio recording for precise citation). The Mishna Berura (commentary by Rav Yisrael Meir Kagan of Radin, 1839-1933) explains (listen to audio recording for precise citation) that we omit Hallel because the festive singing of Hallel is inappropriate on these days of judgment, when the books of life and death are opened before God. The festive nature of Hallel is incongruent with the fear and dread we experience on Rosh Hashanah and Yom Kippur as we stand in judgment before God. The Mishna Berura adds that although we are confident on Rosh Hashanah that we will earn a favorable judgment, as God is a compassionate and loving judge, nevertheless, we must experience a sense of fear and dread. Interestingly enough, the Mishna Berura writes that the feelings of fear and anxiety experienced on Rosh Hashanah are themselves a source of merit for us as we stand in judgment, and help ensure a favorable outcome. Therefore, despite our confidence, we do not recite Hallel, as this festive reading in inconsonant with the emotions we are to feel on these days.
The Mishna Berura adds, however, that one who reads Tehillim on Rosh Hashanah may include the chapters of Hallel in his reading, since he reads them as prayer and not as a festive Hallel service.
In this Siman, the Shulhan Aruch also discusses the proper procedure for the Torah reading on Rosh Hashanah. Two Sifreh Torah are removed from the ark, and, on the first day of Rosh Hashanah, we read from the first Torah the section in Bereshit that tells of Yishak’s birth, until (and not including) the narrative of the Akeda. We make five Aliyot in this section, except when Rosh Hashanah falls on Shabbat, in which case we make seven Aliyot. For the Maftir reading, we read the verses from the Book of Bamidbar that describe the Musaf offering on Rosh Hashanah. The Haftara reading for the first day of Rosh Hashanah comes from the beginning of the Book of Shemuel, and tells the story of the birth of Shemuel and the song sung by his mother, Hanna, after his birth. The Mishna Berura explains that we read these stories because, according to tradition, it was on Rosh Hashanah that Sara and Hanna (as well as Rahel) conceived after many years of infertility.
It is customary on Rosh Hashanah to recite many Piyutim (liturgical hymns). These prayers were written by renowned and righteous Sadikim, and nobody should make the mistake of questioning or underestimating their importance. These are sacred prayers that must be carefully recited with seriousness and concentration. One certainly should not involve himself with other activities – even Torah learning – when the congregation recites Piyutim. Care should also be taken to recite the text in precise accordance with the community’s custom. The Maharshal (Rabbi Shlomo Luria of Lublin, Poland, 16th century) records an incident where a Rabbi of a certain community changed the text of a Piyut on one occasion, in deference to a Rabbi who was visiting from another country, where a different text was used. Tragically, that Rabbi lost a child during the following year. The Rabbi proclaimed that this tragedy was a punishment for the grave sin of tampering with his community’s customs by changing the prayer text.
The customs regarding the Piyutim, like all our customs, were established by outstanding Sadikim, and each community must carefully and strictly follow its time-honored traditions. It has been noted that the word "Minhag" ("custom") has the same letters as the word "Gehinam," indicating that by following our traditions, we are protected from the fires Gehinom and are deemed worthy of a favorable judgment.
Summary: Hallel is not recited as part of the prayer service on Rosh Hashanah or Yom Kippur, but one who recites Tehillim on these days may include the chapters of Tehillim. On the first day of Rosh Hashanah, we read (as the Torah and Haftara readings, respectively) the stories of Yishak’s birth and Shemuel’s birth, because it was on Rosh Hashanah when their mothers conceived after many years of infertility. It is critically important to recite the traditional Piyutim (hymns) on Rosh Hashanah, and to use the precise text, each community according to its custom.