UNITED NATIONS (CNN) -- "Sudanese forces" have removed several thousand people from a refugee camp in the Darfur region, despite assurances they would not, U.N. officials said Tuesday.

"As far as I'm concerned it has to stop ... and people will also have to be brought back, said Jan Pronk, the U.N. envoy to Sudan, who said the situation there is deteriorating. "Stop it and reverse what has happened."

Pronk said "a couple of thousand" refugees had been moved from El-Geer on the outskirts of Nyala in South Darfur to Sherif, just north of Nyala.

He said the government had misrepresented the facts in the case.

"The government has told these IDPS (internally displaced persons) that this was happening in close consultation with the United Nations and in consultation with nongovernmental organizations, which is not the case," he said.

U.N. spokesman Fred Eckhard said preliminary reports indicated that between 3 a.m. and 9 a.m. (midnight to 6 a.m. GMT), 15 trucks were used to relocate "a proportion of the population" north of the Nyala camp.