Starting today and for the next two weeks, The Interpreter will host an online discussion on the theme of government secrecy. This is a perennial topic of dispute in democratic societies — I think Max Weber had a bit to say on the subject, and I have in front of me a 1998 book by Daniel Patrick Moynihan which traces the American culture of secrecy back to World War I — but it is particularly urgent and fast-changing in our information age.

We're proud to have the support of Unisys for this forum, which will include a live event in Canberra on 16 March for invited guests from various parts of the Australian foreign policy and national security community.

This introductory post is intended just to lay down some suggested themes, but hopefully the discussion will take its own course. I have already lined up a few participants, but as always, we welcome your thoughts via blogeditor@lowyinstitute.org . I know it is difficult for our many readers in Canberra's national security bureaucracy to participate, but requests for anonymity will be respected. Yes, I acknowledge the irony of that statement, given the topic up for discussion, but it's the lesser of two evils.

With those preliminaries out of the way, let me suggest some themes and questions:

• What is the cost of government secrecy, and what do citizens get in return?
• Does secrecy help nations avert crises and strategic surprises, or does it play a role in bringing them about?
• Is there a 'culture of secrecy' in government, and if so, what are the benefits and risks for national and international security?
• Is it possible that the challenges and possibilities of 'big data' are so huge that debates about secrecy are becoming archaic?
• Are current models of government secrecy viable in the information age and what does this mean for the Government 2.0 agenda? 
• Should government take a new approach to secrecy in the age of WikiLeaks?

We're not going to cover all of these topics, and I certainly don't want WikiLeaks to dominate the discussion; we debated that topic extensively in 2010. But this week and next week, our contributors will have at it, and I hope you will join in too.

Paul Monk is a founder, Director and Principal Consultant of Austhink Consulting.

Secrecy is a kind of dead weight on sound and accountable government, in much the same way that excessive and irresponsibly incurred public debt is a dead weight on the effective functioning of our market economies. Secrecy should be cut back to a minimum. The question is how to bring this about when the actors best placed to change the rules have strong perverse incentives for keeping them as they are.

Over-classification of documents occurs not only in the dramatic arenas of strategic intelligence and diplomacy, but in those of budgetary and fiscal affairs, of public policy deliberations more generally and of administrative and legal affairs.

We tell ourselves we have open societies and 'responsible' governments and even tend to think that relentless media scrutiny holds government to account. The truth is that secrecy and classification have been increasing relentlessly for decades and, like borrowing and spending, seem to increase regardless of which side of politics is in office. Both these trends are a dry rot within democratic capitalist polities; none of which seem able to rein them in. We should lead the way in heading in the opposite direction.

The case for keeping secrecy to a minimum is not new. Like free trade theory, it has been around for some considerable time. Lord Acton, famous for remarking 'All power tends to corrupt and absolute power corrupts absolutely' also declared, 'Everything secret degenerates, even the administration of justice; nothing is safe that does not show how it can bear discussion and publicity'.

Paul Monk is Director and Principal Consultant of Austhink Consulting. Part one of this post here.

That secrecy has been increasing decade by decade despite repeated, even official calls for the trend to be reversed is very telling. Two brief examples from the US help to underscore this point.

In 1994-97, the Clinton Administration flirted with the idea of substantially reducing secrecy, but failed. Its 1994 Commission on Protecting and Reducing Government Secrecy took three years to issue a report. That report declared 'It is time for a new way of thinking about secrecy.' It called for a new era of openness to replace 'the culture of secrecy...that we associate with Dulles and Hoover'. Fifteen years on, secrecy in the US, like the national debt, is considerably worse than it was in the Clinton years.

But the failure didn't start with Clinton. My own favourite case of a fruitless call for less secrecy is the Defence Science Board's Special Task Force on Secrecy in 1970. It argued that secrecy ran directly counter to the nature of the scientific research whose technological work it sought to conceal. All things considered, its authors urged, the US would be better off if it adopted, unilaterally if necessary, a policy of complete openness in all areas of information.

I've italicized those words for special emphasis. This task force was not composed of naïve or fellow-traveling 'liberals'. It included such notable cold warriors and weapons scientists as Edward Teller and Jack Ruina of MIT. Nonetheless, the report was classified 'For Official Use Only' and its recommendations disregarded.

There is a long history to all this and ample scope for prevaricators to tangle the debate up in the weeds. We should not indulge such prevarication, however. Rather, we should be working from first principles and seeking to push back against secrecy on the basis that its increase is inimical to what we stand for as citizens of a liberal democratic polity. But this push must be based on principles and not on Assange-like anarchism. The goal is better government, not the fracturing of government.

Paul Monk refers early on in his first post on excessive government secrecy to the fact that 'the actors best placed to change the rules have strong perverse incentives for keeping them as they are.' It's worth drawing out the details of this point a little, and there's an excellent summary of the problem in a recent report on security classification by the Brennan Center for Justice at New York University's School of Law:

Numerous incentives push powerfully in the direction of classification, including the culture of secrecy that pervades some government agencies; the desire to conceal information that would reveal governmental misconduct or incompetence; the relative ease with which executive officials can implement policy when involvement by other officials, members of Congress, and the public is limited; the pressure to err on the side of classification rather than risk official sanctions or public condemnation for revealing sensitive information; and the simple press of business, which discourages giving thoughtful consideration to classification decisions. By contrast, there are essentially no incentives to refrain from or challenge improper classification. After all, classification is an easy exercise that can be accomplished with little effort or reflection; those who classify documents improperly are rarely if ever held accountable—indeed, there is no reliable mechanism in place to identify them; classifiers receive insufficient training in the limits of their authority; and those who have access to classified information are neither encouraged to challenge improper classification decisions nor rewarded for doing so.

The report recommends a number of punitive measures against officials who routinely over-classify and cash rewards for those who challenge over-classification. Interesting.

Below, a comment from David Gizzi, but first, Alex Burns writes:

Austhink's Paul Monk raises several important points in his posts to Lowy's debate on excessive government secrecy. I discovered Monk's perspective on the openness of information through rather unusual circumstances. Whilst preparing for PhD research at Monash on strategic culture, I found several of Monk's second-hand books. Reading Monk's hand-written margin notes to Claire Sterling's controversial study The Terror Network provided a rare insight into the late Cold War and how one of Australia's most influential intelligence analysts thinks through ambiguities.

Australia's intelligence community is less open to constructive debate than its United States or Great Britain counterparts. Agencies like the Australian Security Intelligence Organisation ran highly visible media campaigns after the September 11 attacks to hire new intelligence analysts. But this didn’t necessarily translate into a more sophisticated understanding in Australia of how intelligence works. Recently, the Society for Historians of American Relations (SHAFR) has run several articles on freedom of information and digital archives. However, Australia has more restrictive laws on freedom of information disclosures — which the intelligence community can be exempt from — compared with the United States or Great Britain. These restrictive laws prevent SHAFR's research methods from being used here.

Our Unisys Forum on the Future of Secrecy, which kicked off on Monday, is gathering pace, and I'll have further substantive contributions on The Interpreter soon.

But first, I would like to get expressions of interest from you about attending the 'live' portion of this forum, which will take place in Canberra at the National Gallery next Friday, 16 March, from 9am to 2pm, including a lunch. Our speakers will include:

• Malcolm Turnbull, who will talk about his Spycatcher experience.
• Nicholas Gruen, economist, columnist, blogger and Chair of the Rudd Government's Government 2.0 Task Force.
• Maurie McNarn, former Director of the Defence Intelligence Organisation.
• John McMillan, the Australian Information Commissioner.

We have a small number of seats available for Interpreter readers, particularly those with a professional or academic interest in the topic, so please let me know via email if you'd be interested in attending, at no charge: blogeditor@lowyinstitute.org .

Daniel Baldino, a Senior Lecturer at Notre Dame University and editor of Democratic Oversight of Intelligence Services, writes:

The world of intelligence is closely intertwined in official secrecy. Traditionally, spy agencies have been inclined to operate behind closed doors. Their instinct has been to provide as little information as necessary – a rudimentary 'need-to-know' stance. Arguably, we have witnessed some degree of growing recognition that the requirements of an open and democratic society do require a greater dose of public examination, accountability and transparency in regards to the operations of the so-called 'secret state'. 

Certainly the burden of 'winning' the war against terrorism, and the protection of vital national interests, will continue to place a serious pressure on the performance of secret agencies. In 1929, Herbert Hoover's Secretary of State, Henry L Stimson discovered the existence of US interception operations that involved the code-breaking of sensitive diplomatic telegrams. With the mindset of eliminating funding to the cryptology office (the 'Black Chamber'), he broadcast the famous line, 'Gentlemen do not read each other's mail'. Today it is clear that policy prescriptions cannot be held hostage to Stimson's well-intentioned but naive logic.

Peter Timmins writes:

Numerous reports and inquiries have confirmed that we have a culture of secrecy in many areas of government. While there have been some positive reforms since 2009, we have a long way to go to move the culture along. As the Australian Law Reform Commission put it in its 2010 report on Secrecy Laws and Open Government in Australia (yet to receive any government response), '(o)fficial secrecy has a necessary and proper province in our system of government. A surfeit of secrecy does not.'

That report identified over 500 secrecy provisions in Commonwealth law — they add a chilling effect to a closed culture built around the premise that international affairs is the preserve of executive government. The blanket exemptions for the intelligence agencies from FOI is also telling, and not replicated in the US or even NZ.

The attacks of 9/11 brought about an almost instant reordering of newsroom hierarchies.

Old South Asia hands who knew their way around Afghanistan and Pakistan were suddenly in high demand. Former Kabul correspondents, neglected for more than a decade after the withdrawal of Soviet Union, found themselves speedily rehabilitated and sent back to the Hindu Kush. Arabists who could not only pronounce the names of terrorist suspects but also read the Jihadist websites that fired terrorist imaginations also raced up the rankings.

But in the new journalistic caste system, security specialists with good contacts in the intelligence community quickly rose to the top. The new Brahmins were the harvesters of secrets.

I was posted in Washington at the time, a BBC bureau that had traditionally favoured correspondents fluent both in American political history and the cold grammar of Beltway power. But our ranks came to be bolstered by reporters who arrived brandishing Rolodexes brimful with intelligence sources.

My friend and colleague Gordon Corera was a case in point. He was working on an expose of the Pakistan nuclear scientist AQ Khan, and would later write an excellent history of Britain's Secret Intelligence Service. Now he is the BBC's highly-respected Security Correspondent, a post that did not even exist at the time of 9/11.

In the disorientating aftermath of those attacks, Washington-based journalists who lacked intelligence expertise came to rely heavily on those who did. Here, Judith Miller of the New York Times was the Brahmin among Brahmins.

Anna Madeleine Solar-Bassett writes:

Social media is going to be a major issue of the secrecy debate moving forward. As Kony 2012 and Wikileaks show, governments that support open and relatively free internet systems (save for IP protections such as copyright in favour of economic incentivisation of creativity) are going to have to deal with the increasing openness of high level government, commerce and finance moving forward (unless you want to institute China/Iran/Kyrgystan/et al's  ideas about creating national 'intranets': completely filtered and blocked internets). 

Indeed, moving forward, a key distinguisher between states will be, as many commentators have noted, not about left and right, but open or closed. Australian, American, European, etc governments that purport to believe in liberal democratic ideas such as free speech, expression and internet are going to have to accept that, once information is out there on the net, it is very hard to redact, whether or not you can criminally prosecute post-facto. 

Nicholas Gruen is CEO of Lateral Economics and was Chair of the Government 2.0 Task Force.

I recently took my son to the stage play (pictured) of the TV show Yes, Prime Minister. One could predict the kind of plot that would ensue and ensue it did. The Prime Minister and his minions manoeuvre for advantage, navigate a range of dilemmas and eventually a resolution is arrived at. Life goes on happily enough for the characters and we appreciate the bon mot that Wikiquote tells us is wrongly attributed to Bismarck: 'Laws are like sausages — it is best not to see them being made'.

But the decades have made a huge difference in the sensibility of the new production, which is written by the original authors of the TV series.

The series ran through most of the 1980s, a period that contained its share of tumult, from the destruction of union militancy to the Falkland's War. The series reflected bastardry enough. But somehow the dramas were genteel, reflecting battles between those privileged enough to be in the system. Waste in government continued, powerful people and time-servers were protected when they should have been exposed and dealt with. But one could be forgiven for thinking, at the end of an episode, 'it was ever thus'.

Twenty years on, as the moral dilemmas piled up on stage, the governors conspired against the governed. The PM and his advisers address their common problems as insiders in the showbiz that is government by framing an innocent person at the bottom of the social pile (an immigrant). The fall guy was to be taken into custody as a suspected terrorist and kept there, incommunicado, for up to a year, after which he might be deported or his incarceration repeated.

This change in 'Yes, Prime Minister' points to a revolution our society has undergone in the way we 'do' government.

Secrecy was always the default presumption before the days of freedom of information and the craze for 'transparency'. Today we are into our second generation of freedom of information (FoI) legislation, which seeks to move the default more fully to a state where information is presumptively open (subject to exceptions where the case can be made) rather than the converse.

Yet while these changes have been taking place – through the first and now the second wave of FoI reforms – government is increasingly 'performed'. This has huge implications for the incentives operating within the system.

Steve Vinsik is vice president and partner in Global Security Solutions for Unisys Corporation, sponsor of the Lowy Institute's 'The Future of Secrecy' forum.

Open, but secure. That is the new approach governments need to take to secrecy and national security in the age of WikiLeaks. Social media is changing the way people and organisations work and communicate, and its impact on public sector agencies and national security strategies is a subject of both excitement and grave concern.

This new digital universe enhances governments' interactions with their global counterparts for matters of national security and diplomacy; it enables better information-sharing and collaboration among agencies, governments, private industry and the public; it can provide a human face to a government body; it can enhance a government's and nation's reputation locally and globally. But why do we seem to be less secure, more terror prone? Why, in a time when governments can engage directly with citizens, is our age characterised by fear and loss of trust?

In the virtual world, a vast amount of diverse information is available at the click of a button, but there can be adverse impacts on operations, assets and individuals when confidentiality, integrity, or availability of that information is compromised. Given the viral nature of social media and the wide geography and audience it spans, it is much easier for agencies to gather and disseminate information quickly and easily. However, it also makes sensitive information leakage much harder to contain.

As government agencies look to leverage new technologies to communicate with the public, move more citizen services online, share services amongst agencies, share intelligence for national security purposes and collaborate with other nations and private industry, they will need to take a more open stance to secrecy and information sharing.

Major Gen (Retd) Jim Molan is author of Running the War in Iraq.

There is a need for secrecy and there is abuse of secrecy. There is a lot to be protected and some good reasons for protecting it. One of the greatest forces for getting the balance wrong is government convenience.

Rather than protecting us from a real or potential adversary, a competitor or even a neighbour, governments use national security to protect themselves from domestic scrutiny of poor policies and incompetent implementation. Comment in this series reflects the failure to get the balance right. I think this applies particularly in the military arena.

I found WiliLeaks fascinating but the only thing it revealed to me was that, in relation to the important function of speaking to various foreign embassies, our professional diplomats (such as Ric Smith) came out streets ahead of our politicians (such as Kevin Rudd, once a professional diplomat). I also find it fascinating that most of the 'important' issues 'exposed' in WikiLeaks would have been known to foreign diplomats and intelligence agencies that were at all interested and half smart.

Therefore, you have to assume that a good deal of our secrecy is aimed at the domestic media and the Australian people.

In relation to the detritus of day-to-day staff work in Iraq as revealed through Wikileaks, I cannot think of much that would have any implications as long as it was revealed after the event. I certainly cannot think of any critical tactical techniques revealed by the routine communications published. They were interesting only to the uninitiated. This is the kind of stuff that is eventually released to historians to write the official history.

The piece by Nicholas Gruen that we published yesterday was cross-posted on Club Troppo, Nick's regular blog haunt, and it's worth pointing to the subsequent discussion in the comments thread.

In response to a reader, Nick mentions a section of his draft post that we agreed to omit from the final, in which he juxtaposes Churchill's willingness to show mercy during World War II with Western behaviour in the fight against terrorism: 

When the West fought its last war of survival – WWII – Churchill memorably offered these words. “be the ordeal sharp or long, or both, we shall seek no terms, we shall tolerate no parley; we may show mercy-we shall ask for none.” And yet, a decade after Western liberal capitalism became the unchallengeable default setting for modernity, at the first whiff of grapeshot countries which had honoured the Geneva conventions either engineered or became complicit in widespread abuse of prisoners involving ‘extraordinary rendition’ and ‘enhanced interrogation techniques’ such as waterboarding.

I'm attracted to Nick's broader argument that government has become more 'performative' in recent times, and that this manifests itself in what he calls 'soft secrecy'. But the Churchill example Nick cites does not illustrate this point very well. After all, although the breaches of the Geneva Conventions during the war on terror have been shocking, compared to the incendiary bombing of German cities carried out on Churchill's orders during World War II, they represent a minor lapse in just-war standards.

In fact, for a number of reasons, the kind of carpet bombing used routinely in World War II and as late as the Vietnam War is unthinkable today. Partly that's because technology makes it easier and more affordable to be discriminating in the use of aerial weapons. But the capacity of the mass media (and now the new media) to transmit to the world the evidence of mass suffering that would result from such tactics surely also plays a part. As a result, Western norms about proper wartime behaviour have changed dramatically for the better.

This returns us to Nick's point about modern government as performance. For modern warfare is increasingly a media event, where the battle for headlines and public opinion is considered almost as important as what happens on the battlefield. Modern war is in large part a public performance, and civilian casualties get bad reviews from the punters.

In the twin realms of politics and government, secrets are a tradable commodity. 

In this market, knowledge really is power – or a function and a facet of power. To use an economic framework: secrecy, knowledge and power are all 'coin of the realm', the legal currency of a political system.

The word 'market' is used with intent, because applying an economic model shows the reality of what politicians, minders and senior bureaucrats actually do with secrecy and secrets. Secrecy can confer monopoly power on a pollie. And the market model also leads quickly to that key economic question – who profits? Various participants in this market will price secrets in different ways. Demand, supply and sales get complicated when you stir in government ministers, journalists and the military.

An important distinction must be made between the values attached to secrecy by the political class (ministers, pollies, minders) and the military. The military believe secrets have an absolute value, while politicians view secrets as having relative value, according to the needs of the market and the size of the secret. Public servants are supposed to view secrets in the same way as the military (an imperative imposed by their customs, training and the law) but constant contact with pollies means senior bureaucrats can come to understand the benefits of trade, even if it is seen as black market activity.

The military must believe that secrecy is an absolute value; if secrets leak, operations can fail and people can die. This is core value stuff, well illustrated by Jim Molan's contribution to this debate: 'There is a need for secrecy and there is abuse of secrecy. There is a lot to be protected and some good reasons for protecting it. One of the greatest forces for getting the balance wrong is government convenience.'

If secrets are government's attempt at monopoly, then FoI is about liberalisation and opening the market. The FoI experience in Australia illustrates why free trade so often wins in theory but has a hard time in reality. As the previous column argued, politicians and bureaucrats draw both power and control from maintaining their monopoly.

But surely, you say, it is politicians that have passed the FoI laws. Why would they fight against their own handiwork? The answer is that advances in FoI usually come with new administrations; the commitment to making government more open fades as the experience of government grows. To illustrate, here is a rant from Tony Blair's memoir:

Freedom of Information. Three harmless words. I look at those words as I write them, and feel like shaking my head till it drops off my shoulders. You idiot. You naive, foolish, irresponsible nincompoop. There is really no description of stupidity, no matter how vivid, that is adequate. I quake at the imbecility of it. Once I appreciated the full enormity of the blunder, I used to say — more than a little unfairly — to any civil servant who would listen: Where was Sir Humphrey when I needed him? We had legislated in the first throes of power. How could you, knowing what you know, have allowed us to do such a thing so utterly undermining of sensible government? Some people might find this shocking. Oh, he wants secret government; he wants to hide the foul misdeed of the politicians and keep from ‘the people’ their right to know what is being done in their name. The truth is that the FOI Act isn't used, for the most part, by 'the people'. It's used by journalists. For political leaders it's like saying to someone who is hitting you over the head with a stick, 'Hey, try this instead', and handing them a mallet. The information is neither sought because the journalist is curious to know, nor given to bestow knowledge on 'the people'. It's used as a weapon.

But another and much more important reason why it is a dangerous Act is that governments, like other organisations, need to be able to debate, discuss and decide issues with a reasonable level of confidence. This is not mildly important. It is of the essence. Without that confidentiality, people are inhibited and the consideration of opinions is limited in a way that isn't conducive to good decision-making. In every system that goes down this path, what happens is that people watch what they put in writing and talk without committing to paper. It’s a thoroughly bad way of analysing complex issues.

The end bit is an almost reasoned discussion of the need for some secrecy in the discussion phase, if not the end phase, of a government process. But the stuff at the top is just a marvellous, almost glorious outpouring from a great media spinner who, at the end, could no longer either charm or cajole. Blair gives an unusually frank glimpse into the fears and insecurities that make even top politicians fret and froth.

This jeremiad against FoI comes from a leader who goes on to finish his book by arguing that the new divide in politics is not between left and right but 'open versus closed'. And progressives, Blair announces, must be champions of the open position. 

Last Friday The Interpreter's Future of Secrecy discussion culminated in a live event in Canberra. I hope some of the presentations made at the event will be released shortly, but meantime, Malcolm Turnbull agreed to a short interview after his speech.

Photo by Mark Graham.