Harold Wilson's resignation/coup still murky.

I’ve worked for decades to reveal the truth about the ‘Wilson plot’. But the cover-up continues
Richard Norton-Taylor

Files about alleged Soviet infiltration in MI5 and a plot centred on the former Labour PM are still missing. Why the secrecy?

mong the scores of files declassified today at the National Archives, several relate to persistent allegations of MI5 wrongdoing, including plots to undermine Harold Wilson’s Labour government, and the possibility of Soviet penetration at the top of the agency.

These claims appeared in the memoir of the former MI5 officer Peter Wright, whose book Spycatcher the government was determined to ban. As the Guardian’s spook-watcher of the time, I had been served an injunction in 1987 preventing me from revealing Wright’s claims, along with David Leigh of the Observer, beyond the one article I had already written for the newspaper.

I have been looking forward, then, after more than 35 years, to reading what ministers and their most senior officials in Whitehall recorded at the time. I was far too optimistic. The selected files released today, most marked Secret, reflect what was abundantly clear at the time – namely, that Margaret Thatcher fought tooth and nail in a desperate attempt to ban Spycatcher, first in Australia, where Wright lived after he retired, and then in the rest of the world.

The files contain no surprises. They confirm what was quite evident at the time, namely that the Spycatcher debacle of the mid-1980s triggered panic in the upper reaches of Whitehall, and that Thatcher was determined to block any attempt to subject MI5 to independent scrutiny. They reflect the painfully obvious concern in Whitehall’s security establishment about how the Spycatcher affair would resurrect evidence that MI5, worried about the loyalty of some of Wilson’s friends, had kept a file on the Labour prime minister (code-named Henry Worthington), and that there was a group of MI5 officers beyond Wright who shared paranoid delusions about Soviet spies in their midst.

The documents confirm that for years officials and ministers discussed how to block demands – with the Guardian leading the charge – for proper oversight of MI5. One of Thatcher’s closest advisers is quoted in the documents as claiming the pressure for reform came only from a “few MPs and unscrupulous journalists”.

Regardless, the pressure on MI5 grew amid evidence that it had targeted the National Council for Civil Liberties (now Liberty), trade union leaders and the Campaign for Nuclear Disarmament in the 1970s and 1980s. Meanwhile, senior MI5 officers were berated in a 1985 Security Commission report on how no action was taken to safeguard Michael Bettaney, an official on the agency’s Soviet desk, who had suffered a breakdown, drank a bottle of whisky a day, and was discovered inebriated in a gutter. He was arrested and sent to prison for 23 years only after he was caught trying to pass documents to the Russian embassy in London.

One Whitehall official referred in one document released today to the “somewhat secretive and cloistered existence which members of the security service inevitably lived”. After years of procrastination, Thatcher was persuaded to bow to the inevitable. Something had to be done to satisfy the campaign of “investigative journalists” – the view of cabinet secretary Sir Robert Armstrong – threatening to undermine public trust in MI5. But failing trust was beginning to become a concern for MI5’s own officers, especially with the European court of human rights expected to demand the agency was placed on a legal footing for the first time.

The government finally agreed to take the most cautious of steps. In 1989, a Security Service Act set up a parliamentary intelligence and security committee (ISC) to meet in private. (Only this year, the committee complained about the present government eroding its limited powers.)

The files declassified today are littered with yellow sheets signed by Whitehall weeders, signalling where page after page has been withheld under the catch-all exemption of section 3(4) of the misnamed Public Records Act. This states that files may be closed indefinitely for “administrative purposes” or “any other special reason”.

It is clear from the placing of the missing files that many relate to what became known as the “Wilson plot”. The security establishment has always insisted it was a figment of the conspirators’ imaginations. Yet even Wilson’s successor, James Callaghan, expressed concern about a group trying to undermine the Labour government, the declassified papers confirm.

In the preface to his official history of MI5, published in 2009, the historian Christopher Andrew referred to redactions he had been asked to make. “The most difficult part of the clearance process has concerned the requirements of other government departments,” he wrote. He added: “One significant excision as a result of these requirements in Chapter E4 is, I believe, hard to justify. This and other issues relating to the level of secrecy about past intelligence operations … would, in my view, merit consideration by the intelligence and security committee.”

Chapter E4 is titled The ‘Wilson Plot’. I asked Sir Malcolm Rifkind, former defence and foreign secretary and then chair of the ISC, whether he would take up Andrew’s invitation. He declined. I made a Freedom of Information Act request to the Cabinet Office asking them to tell me what Andrew was referring to. It refused, saying “intelligence” material was exempted under the act.

The Cabinet Office could do all of us a favour by publishing all the documents relating to this still mysterious episode in recent British political history.