The removal of horses from historic cliffs at a coastal beauty spot would be a loss for the community, a farmer has said.