The incident took place Monday off the Malaysian part of Borneo island over the fiercely contested waters, where China and Malaysia have overlapping territorial claims.

The Chinese air force transport planes approached Malaysian airspace in "tactical formation" and flew to within about 60 nautical miles of the coast, Malaysia's air force said in a statement.

They were spotted by radar and several attempts were made to contact the planes.

As they approached, the air force sent planes to intercept and identify them, and they did not enter the country's airspace before flying off.

The Malaysian air force described the appearance of the planes as "suspicious".

"This incident is a serious matter in regards to national sovereignty and aviation safety," as the area is a busy place for aircraft, the statement said.

The foreign ministry did not respond to requests for comment.

China has laid claim to nearly all of the South China Sea and has built numerous military outposts on small islands and atolls, angering other countries with competing claims to the waters.

Despite their overlapping claims, relations between Malaysia and China are usually cordial, and Monday's incident was unusual.

There have been tensions in the area in the past, however.

Last year, a Chinese survey ship had a long standoff with a Malaysian oil exploration vessel off Borneo.

Other claimants in the sea include Vietnam, the Philippines, Brunei and Taiwan.

The US has also sent warships through the waters to assert international rights to freedom of navigation, angering China.