Weekend plans? The Badawi moulid, baby.

At a jazz club on a recent Sunday evening in Cairo, serendipity placed me next to US Ambassador Francis Ricciardione.

Seeing as the Arabic-speaking ambassador knows a lot of things and gets to meet a lot of people that I don't, I sought to exploit the situation.

"So,'' I said, leaning over casually. "Doing anything interesting this week?"

The ambassador said as a matter of fact he was. Possible reporting scoops danced in my head. Then his answer brought me back to earth. "I'm going to Tanta."

Tanta? Certainly nothing scoop-worthy could draw the ambassador to Egypt's gritty, third-largest city in the middle of the Nile Delta, famously inundated with acrid smoke this time of year as farmers burn rice stalks in the surrounding countryside.

"It's the Badawi moulid,'' he explained. "If I'm in Egypt, I never miss it. You shouldn't either."

The ambassador was right about that. The three-day annual meeting of Egypt's largest Sufi orders celebrates Ahmad al-Badawi, a 13th-century mystic revered to this day by millions of Egyptians. Equal parts medieval carnival, music festival, and religious occasion, it blends piety with spectacle in ways that challenge conventional ideas of what it means to be Muslim.

Unlike other major religious events, which are tied to the lunar Muslim calendar, the Badawi moulid coincides with the end of the harvest. For most, it begins with a visit to the mosque and tomb of Badawi, which lies amidst a sprawling market in the center of town.

Men and women freely mingle inside the mosque – something that is generally frowned upon during more sober religious occasions – as they jostle forward to press up against the saint's tomb, garishly lit with green fluorescent lights and draped with plastic flowers and tinsel.

They lean their foreheads against the metal cage that surrounds the tomb, and murmur prayers for health, better financial fortune, or a child's success in school. The practice – similar to Catholic prayers to the Virgin Mary seeking intercession with God or Shiite prayers to Imam Ali – is strictly at odds with Sunni Islam, which is generally thought to prevail here.

Indeed, the leaders of Al Azhar University, the arbiters of Sunni orthodoxy in Egypt, have long assailed this and other popular moulids, or saint's festivals, like the ones to mark the Prophet Muhammad's birthday or the death of Zeinab, his granddaughter, whom the faithful believe is buried in Cairo. To these leading Sunni imams, praying to saints or even celebrating Muhammad's birthday is akin to idolatry.

But their long-standing efforts and those of groups like the Muslim Brotherhood to discourage expressions of popular Egyptian Islam have gained very little traction. A senior Brotherhood official rolls his eyes when asked about the moulids. "We're against it, it's a relic of jahaliya," he says, using the Arabic term for the age of ignorance before Muhammad's time. "We would really like this to stop."

For many of the thousands of Egyptians who descend on the festival, it's an opportunity listen to music, or play backgammon, or ride the Ferris wheels erected throughout town. But the serious celebrants are here to feel God in a deeper way, through music, dancing, and chanting.

Fateh al-Mussalami, a Sufi poet, retired government official, and – according to a genealogy card he carries in his wallet – a proud descendant of Muhammad, says religious leaders opposed to the festival fail to understand this popular reverence for saints like Badawi.

"Badawi was one of the figures that helped raise the veil of knowledge from people's eyes,'' he says, sitting in a coffee shop across from the shrine. "God worked many miracles through him to show that he approves of those who recognize and feel God."

He also nods agreeably when I tentatively mention how much the shrine area resembles Shiite shrines I have visited in Iraq. "Oh, absolutely,'' he says. "They're like us. We all love and respect the holy family." Indeed, scholars trace many of the moulids' traditions and styles to the Fatimid era of Shiite rule in Egypt, which ended 100 years before Badawi's birth.

On the second evening of the festival – the Leila Kebir, or "Big Night" – 50 tents set up by various Sufi orders on the city's fairgrounds are filled with chanting and swaying Sufis. Strung with blinking Christmas lights and kitted out with deafening sound systems, some are filled with men in strict ranks holding hands; in others, where the dancing and music are wilder, men and women mingle.

"This is all about bringing joy and celebrating God,'' says Abdul Bakr Habibie, a mercurial Habibiya Sufi in his mid-60s, who says that God is working miracles every day, all around us. "If people would open their ears and listen, we could end all the evil in the world."

Unfortunately, listening becomes increasingly tough as the night goes on, with each tent cranking up their sound systems in competition with each other, straining ear drums and drowning out much of their music with feedback and buzz.

Defeated by the amplifiers, my friends and I retreat at about midnight, the Sufi music echoing in our ears as we drive home.