Basically, neither the Isle of Man nor the Channel Islands are members of the European Union and therefore the spouse of an EU national who was working there could not benefit from EU free movement law. If only Rees-Mogg et al just moved there, perhaps we could just cancel the whole Brexit thing?
The official headnote:
(1 ) The Isle of Man and the Channel Islands are not part of the United Kingdom and have only a very limited legal relationship with the European Union.
(2) An EU national who works on the Isle of Man is not thereby exercising EU rights of free movement for the purposes of the Immigration (EEA) Regulations 2006.
Source: Amsar (Isle of Man: free movement)  UKUT 12 (IAC).