Thousands of protesters took to the streets of Hong Kong Friday to mark the anniversary of the city's handover from Britain to China, with pro-independence groups rallying for the first time amid fears Beijing is tightening its grip.

But a second demonstration planned for early evening at China's representative office in the city --- expected to be a flashpoint -- was halted due to a heavy police presence.

It comes as a Hong Kong bookseller who was detained and interrogated for months in mainland China, and was due to lead the annual march, also pulled out of protests at the last minute citing a "serious threat" to his security.

Tensions are high in the southern Chinese city after the explosive revelations by bookseller Lam Wing-kee, whose account of his detention on the mainland after he went missing last year sent shockwaves through a city where residents are worried of eroding freedoms.

The 61-year-old was one of five employees of a Hong Kong firm that published salacious titles about leading Chinese politicians and who disappeared only to resurface over the border.

His story fanned growing concerns that Beijing is increasing its influence in semi-autonomous Hong Kong, which is ruled under a "one country, two systems" deal enshrined in the July 1997 handover agreement, guaranteeing its freedoms and way of life for 50 years.

"Lam Wing Kee is facing a serious threat and is forced to not attend the rally," said protest leader Lau Shan-Ching, who spent 10 years in a Chinese jail as a prisoner of conscience.

The march organisers said Lam had pulled out because he had been tailed in recent days.

"He had observed that he had been followed by strangers in the last two days. He got greatly annoyed about his personal safety so he decided not to come today," pro-democracy lawmaker Albert Ho told reporters.

"Mr Lam has reason to be scared that this may be people from the mainland," Ho added.

The march got underway nonetheless at around 3:30 pm (0730 GMT) from the city's Victoria Park, with the front protesters holding a huge banner that read "Stand in Unity. Protect Hong Kong".

- Anger, disappointment -

"We want to choose our own government," shouted protesters, some of whom carried yellow umbrellas -- a symbol of the pro-democracy movement that brought parts of the city to a standstill for months in 2014.

"As a Hongkonger, I feel I need to speak out for the city. To pursue democracy is the issue this generation is most concerned about," rally participant Eva Li, 20, told AFP.

Rally organisers said some 110,000 took part in the protest, more than double last year's turnout, while police put the figure at up to 19,300.

Scuffles broke out when a small group of protesters who broke away from the main rally to march to Hong Kong leader Leung Chun-ying's residence were stopped by officers.

A government spokesman said in a statement: "It (the government) takes note of different views on policies and initiatives... it sincerely hopes that various sectors will, in the interest of Hong Kong, strive to seek common ground."

Leaders of several pro-independence groups who had planned a second rally outside China's liaison office in Western District said the demonstration could not go ahead due to a police cordon blocking protesters from entering the area.

"It is a violation of people's freedom of expression," Baggio Leung, of Youngspiration, one of the groups that organised the protest, told reporters.

The groups are calling for more autonomy for the city, including independence from China, following the failure of the 2014 protests to bring political reforms.

Some are now demanding a return to British rule as a stepping stone towards independence, while others say violence may be necessary to bring change.