The colt was in front coming down the stretch, and for a few moments the Hall of Fame trainer thought he had his fourth Derby victory.
"It just took the air out of us," Baffert recalled Thursday.
As was the case with most of the 150,000 people at Churchill Downs that day. Calvin Borel and 50-1 shot Mine That Bird blew by Pioneerof the Nile to win by 6¾ lengths in the second-biggest upset in Derby history.
Now Baffert and his colt are back for a rematch in Saturday's Preakness.
Not only will they have to contend with the Derby winner, but Borel has switched horses and will ride stellar filly Rachel Alexandra, the 8-5 morning line favorite who brings a five-race winning streak into the 1 3-16-mile race at Pimlico.
"I would've taken a shot at the Derby with her. She's just a tremendous athlete," Baffert said. "She's a good filly and these classics are huge. There's not a lot of money to run for fillies. She fits with these boys, so I don't blame them for taking a shot."
Baffert did the same thing with Excellent Meeting in 1999, but she was pulled up as a precaution and didn't finish the race.
He expects a better result for Rachel Alexandra, who will break from the No. 13 post on the far outside under Borel.
"He'll have her right in contention immediately," Baffert said. "She'll probably be sitting second or third and just cruising."
Pioneerof the Nile is the 5-1 second choice in the 13-horse field and drew the No. 9 post. Garrett Gomez and the colt figure to be stalking the pace from an outside position.
"The questions that weren't answered in the Derby we're going to find out in the Preakness," said Baffert, back in the race for the first time in six years.
Hall of Fame trainer D. Wayne Lukas is picking the filly to win, but he gives Baffert a solid chance at winning his fifth Preakness, which would tie him with Lukas and T.J. Healey for second on the career list.
Rachel Alexandra isn't the only unknown factor in the Preakness. The weather figures to play a part, too, with a 50 percent chance of afternoon thunderstorms.
That could turn Pimlico's dirt into mud, the same kind of slop that bogged down most of the 20 horses in the Derby, except Mine That Bird, who went flying through it.
"I still want to see what my horse does on dirt. He's seen sticky mud," Baffert said. "My horse has never run on a dry track. He's trained well on a dry track. I'm hoping it moves him up."
Pioneerof the Nile ran on dirt for the first time in the Derby after spending the winter on synthetic surfaces in Southern California, where he had a four-race winning streak.
"He really bounced out of it well," Baffert said, referring to the colt's post-Derby condition. "He's going to run a big race."
Owner Ahmed Zayat thought that in the Derby. Two days before, he was standing outside Baffert's barn when bird droppings landed on his glasses. He took it as a sign of good luck.
"He was so happy," Baffert said. "Little did he know two birds were going to crap on him."
In the cozy conditions of Pimlico's stakes barn, where the Preakness runners are stabled, Baffert has gotten a close look at the horse that vanquished Pioneerof the Nile in the Derby.
"I remember coming here and looking at Smarty Jones," he said, referring to the 2004 Derby and Preakness winner. "He was a nice looking little horse. He wasn't that big, Mind That Bird same thing. It's just amazing how these runners come in all sizes and packages."
Baffert took a call last fall from someone wondering if he'd be interested in buying Mine That Bird, who was a champion 2-year-old in Canada. He looked at the horse on paper and passed.
No regrets, though.
On Derby Day, Jill Baffert told her husband she thought Borel might jump up and beat them. "Honey, he's riding the longest shot in the field, don't worry about it," Baffert responded.
In the din that accompanies the Derby runners down the stretch, Baffert heard a tiny voice say, "Who is that?"
Looking closer, he confirmed his wife's worst fears. It was Borel on the rail.
He just hopes Borel doesn't burn him again - this time with Rachel Alexandra.