The Scottish-born Holden struck a loose ball at the top of the penalty area that Japanese goalkeeper Shusaku Nishikawa was able to slow, but couldn't keep from trickling over the line in the 47th minute.
A victory was critical for both teams with the more heavily favored Dutch and Nigerians yet to play in Group B.
Marvelle Wynne was critical in creating the U.S. goal, using sheer speed down the right side to get a half step on Yuto Nagatomo and send his low drive into the goal box. Japanese captain Hiroki Mizumoto deflected the ball out to the top of the penalty area, where Holden ran onto it unchallenged.
The Americans entered the game not having scored a goal since a 3-0 victory over Canada in the semifinals of CONCACAF Olympic qualifying. It lost the final to Honduras. The Americans also failed to score against Cameroon and Ivory Coast in exhibitions last week at the ING Cup in Hong Kong.
Maurice Edu was a force in the center of the American backline. Normally a midfielder for Toronto FC, he was paired with Michael Parkhurst in the middle of the U.S. defense and repeatedly stymied Japanese threats.
Japan is one of only three teams at the Olympics with a strictly under-23 team and no overage players. Its squad is nearly entirely composed of domestic based players with only Keisuke Honda of Dutch club VVV Venlo and Takayuki Morimoto of Italy's Catania not playing in Japan.
The United States had more possession in the first half but Japan created the more dangerous opportunities. While the Americans earned reactions from the crowd for their passing and dribbling, the Japanese exploited a corner kick and counterattacks for chances.
A short corner in the 21st resulted in Atsuto Achida's cross through the goalbox that Masato Morishige heeled wide of the far post. The Japanese found space on the American left flank that twice resulted in crosses by Hiroyuki Taniguchi, one in the 40th minute that Wynne had to race in from his right back spot to knock over for a corner.
The Japanese mounted some late pressure, with their best chance coming in the 80th minute when Keisuke Honda was able to shoot from near the top of the goal box. But with Edu marking Hirouyuki Tanaguchi, the shot was blocked, and the ball rolled harmlessly until cleared.