S&A Property Ltd harvests strawberries at Brierley Court Farm, near Leominster, in North Herefordshire.
The company had erected a utility building for temporary farm workers from eastern Europe.
But Herefordshire Council's Northern Area Planning Committee refused to grant S&A Property Ltd retrospective permission for the building and site works.
Members turned down the application at its meeting on May 12, 2004.
S&A Property appealed against the decision but the Office of the Deputy Prime Minister dismissed the appeal on Friday, November 4.
S&A Property Ltd has 93 days to comply with the order, although the company could challenge it in the High Court.
There is no indication at this stage if there would be a legal challenge. Herefordshire Council expects S&A Property Ltd to comply with the findings of the Office of the Deputy Prime Minister.
The Leader of Herefordshire Council, Cllr Roger Phillips said: "The findings of the ODPM fully justify Herefordshire Council's position.
“This is a landmark decision which sends out a clear message that blatant breaches of planning regulations will not be tolerated."
Herefordshire Council secured a court injunction at Hereford County Court, on May 14, 2004, preventing S&A Property Ltd from developing the site further.
The only works permitted under the order were health and safety works.
Mr Justice Burton, sitting at the High Court, London, in June this year, ruled that an order banning future works at Brierley Court Farm was breached and fined Stanley John Davies, the company's sole director, £65,000.
He also ordered he pay the Council's legal costs of £11,489. The breach centred around works to an amenity building.
The breach centred around works to an amenity building. Large openings previously boarded up with plywood had been fitted with new PVC windows.