The first naval clash between the two sides in seven years broke out just a week before President Barack Obama is due to visit Seoul, raising suspicions the North's communist regime is trying to rachet up tensions to gain a negotiating advantage.
There were no South Korean casualties, the country's Joint Chiefs of Staff said in a statement. South Korea's YTN television reported that one North Korean officer was killed and three other sailors were wounded, citing an unidentified government source. The JCS said it could not confirm the YTN report.
Each side blamed the other for violating the sea border.
The exchange of fire occurred as U.S. officials said Obama has decided to send a special envoy to Pyongyang for rare direct talks on the communist country's nuclear weapons program. No date has been set, but the talks would be the first one-on-one negotiations since Obama took office in January.
"It was an intentional provocation by North Korea to draw attention ahead of Obama's trip," said Shin Yul, a political science professor at Seoul's Myongji University.
He also said the North was sending a message to Obama that it wants to replace the armistice agreement that ended the Korean War in 1953 with a permanent peace treaty while keeping its nuclear weapons.
Washington has consistently said that Pyongyang must abandon its nuclear arsenal for any peace treaty to be concluded. North Korea has conducted two underground nuclear tests since 2006 and is believed to have enough weaponized plutonium for half a dozen atomic weapons.
"We are sternly protesting to North Korea and urging it to prevent the recurrence of similar incidents," South Korean Rear Adm. Lee Ki-sik told reporters in Seoul.
North Korea's military issued a statement blaming South Korea for the "grave armed provocation," saying its ships had crossed into North Korean territory.
The North claimed that a group of South Korean warships opened fire but fled after the North Korean patrol boat dealt "a prompt retaliatory blow." The statement, carried on the official Korean Central News Agency, said the South should apologize.
South Korean President Lee Myung-bak, who convened an emergency security meeting, ordered the South's defense minister to strengthen military readiness.
South Korea's Joint Chiefs of Staff said in a statement that a North Korean patrol boat crossed the disputed western sea border about 11:27 a.m. (0227 GMT), drawing warning shots from a South Korean navy vessel. The North Korean boat then opened fire and the South's ship returned fire before the North's vessel sailed back toward its waters, the statement said.
The clash occurred near the South Korean-held island of Daecheong, about 120 nautical miles (220 kilometers) off the port city of Incheon, west of Seoul, the statement said.
The North Korean ship was seriously damaged in the skirmish, a Joint Chiefs of Staff officer said on condition of anonymity, citing department policy. Prime Minister Chung Un-chan told lawmakers the ship was on fire when it fled north.
Lee, the rear admiral, said the shooting lasted for about two minutes, during which the North Korean ship fired about 50 rounds at the South Korean vessel, about two miles (3.2 kilometers) away. He said the South Korean ship was lightly damaged.
He said several Chinese fishing boats were operating in the area at the time of clash, but they were undamaged. Chung, the prime minister, described the clash as "accidental," telling lawmakers that two North Korean ships had crossed into South Korean waters in an attempt to clamp down on Chinese fishing.
Lee, however, said the South Korean military was investigating if the North's alleged violation was deliberate.
The Koreas regularly accuse each other of straying into their respective territories. South Korea's military said that North Korean ships have already violated the sea border 22 times this year.
The two sides fought deadly skirmishes along the western sea border in 1999 and 2002.
No South Koreans were killed in 1999, but six South Korean sailors died in 2002, according to the South Korean navy. It said exact North Korean causalities remain unclear.
Baek Seung-joo, a North Korea expert at Seoul's state-run Korea Institute for Defense Analyses, said Tuesday's clash would not have a big impact on inter-Korean relations.
He said the Koreas held a landmark summit in 2000 and the North sent a cheering squad to the South for the Asian Games in 2002. Both events took place after the separate clashes in 1999 and 2002.
Baek, like fellow analyst Shin, said that North Korea caused the incident but that Pyongyang appears to want to create tensions and use them for domestic political consumption.
The two Koreas have yet to agree on their sea border more than 50 years after the end of their 1950-53 Korean War, which ended in an armistice and not a peace treaty. Instead, they rely on a line that the then-commander of U.N. forces, which fought for the South, drew unilaterally at the end of the conflict.
North Korea last month accused South Korean warships of broaching its territory in waters off the west coast and warned of a clash in the zone, which is a rich crab fishing area.
The latest conflict comes after North Korea has reached out to Seoul and Washington following months of tension over its nuclear and missile programs.
North Korea launched a long-range rocket in April and carried out its second underground nuclear test in May. But it subsequently released South Korean and U.S. detainees, agreed to resume joint projects with South Korea and offered direct talks with Washington.
Two administration officials said Monday in Washington that Obama has decided, after months of deliberation, to send a special envoy to Pyongyang for direct talks on nuclear issues.
Obama will send envoy Stephen Bosworth, although no date for his trip has been set, the officials said. The officials discussed the matter on condition of anonymity because the decision has not been publicly announced.
Hundreds of thousands of combat-ready troops on both sides face across the 155-mile-long (248-kilometers-long) land border that is also strewn with land mines and tank traps and laced with barbed wire. About 28,500 U.S. troops are stationed in South Korea to deter a potential North Korean aggression.