The Braves were trailing 2-1 after the Astros scored a pair of runs in the eighth inning off Atlanta errors before Conrad's shot to right field off Houston closer Matt Lindstrom (2-3).
Alex Gonzalez singled to set up Conrad's go-ahead shot. Conrad entered the game in the bottom of the sixth inning to replace Chipper Jones, who sprained his left knee.
Jones was injured fielding a ball in the sixth inning. He grabbed a grounder hit by Hunter Pence and jumped in the air as he made the throw to first base to get the out. He then fell to the ground where he remained for several minutes while trainers and coaches crowded around him. He was helped to his feet and walked gingerly off on his own power.
Troy Glaus added a homer in the ninth and also drove in Atlanta's first run in the seventh inning.
Jonny Venters (4-0) got the win after striking out Jason Michaels to end the eighth inning and Billy Wagner pitched a perfect ninth for his 28th save.
Atlanta's big ninth inning allowed the team to overcome its three errors.
Michael Bourn tripled in the eighth inning and tied the game on Gonzalez's bad throw home to Brian McCann on a ball hit by Jeff Keppinger. Keppinger reached first on a fielder's choice. Braves starter Jair Jurrjens allowed six hits and two runs in 7Q innings.
J.A. Happ yielded just two hits and a run in 6 2/3 innings in his third start for the Astros since being traded from Philadelphia for Roy Oswalt.
The win increased the Braves' NL East lead to 2 1/2 over the Phillies, who lost 15-9 to the Los Angeles Dodgers.