The TV sho is set to be affected by a major ITV shake-up, with some of the older panellists reportedly vowing to “fight” against the younger stars