Monday, May 25, 2015

What are we measuring?

@Silvanavf

Openness and transparency are overused buzzwords these days. At the risk of sounding redundant, these two qualities are related to the asymmetry of the informational resources between governments and citizens in an information age (in words of Castells). In this context, not only freedom of information (FOI) has amounted to more than a hundred laws worldwide but also open government data (OGD) initiatives have become a trend for governments everywhere (these numbers have increased after the launch of the Open Government Partnership- OGP[1]).

References to openness and transparency can be found in blogs, academic papers and now, more than ever, in political speeches (specially when close to election’s times). Not only these concepts are overused, there is also an over-demand for measurements, rankings and assessments. This last statement relates to the demand side of information as advocacy groups and practitioners find it difficult to promote something based on certain impact when it is difficult to prove. On the supply side of information, public administration’s champions need to “sell” the benefits that these types of polices bring (and to show them that the pros surpass the cons) to the staff implementing these policies as well as the heads of the agencies and/or governments taking the decisions. 

In this context, during the first semester of 2015 we have witnessed the launch of several initiatives that try to measure the impact, results and other variables related to the disclosure of government-held information and data. Some of those exercises are related to open government data, others to transparency in a broader sense. All these global exercises should be added to the long list of available measurements, rankings and index in FOI and OGD fields. Thus, are these exercises enough? Do we need to think a new way to assess them both?

- Rankings, ratings and other measurements

Global rankings, ratings, indexes, barometers, they all refer to the idea of placing the elements of a given universe in a certain order. However, this order is not arbitrary. Thus, rankings, ratings, among others, refer to the placement of a certain list of objects according to a set of criteria/variables. In this sense, it does not necessarily mean that the objects in the first positions of that ranking are more valuable/useful than the ones at the bottom. It just means that those first objects are closer to preferable conditions set by the author of that scale. Thus, it is important to highlight that in all cases (FOI and OGD) these measurements reflect the availability of information and not the impact of this initiatives (policies and/or legislations)

There are numerous examples to illustrate these ideas. Some of them from the Open Data field are, for example the indexes, such as Global Open Data Index[2] by OKFN, which presents information on the current state of open data release around the world, according to a list of criteria. In terms of ratings, a clear example is the RTI (Right to Information) rating developed by Access Info Europe and the Centre for Law and Democracy. This exercise is based on placing a large list of RTI legislation according to the correlation between the text of the law and a set of standards (most of them originated in Mendel’s work “Principles on Freedom of Information Legislation” back in the 90s when he was a member of Article 19)

The Open Data Barometer is another example of this list of initiatives. This exercise is more rigorous in the correlation between the information they present and the name of their initiative. A barometer reflects changes in circumstances or opinions and, thus, the gathered information aims to reflect changes in the disclosure of data disclosed in open reusable formats.

Despite the lack of literature, in particular in terms of measurement and impact, in the FOI and even more in the OGD field, there is some relevant work on the area. In the case of FOI, there is some analysis on the different types of rankings and index. In these exercises, they mostly refer to the link between FOI and transparency. This close relationship between FOI and transparency (as a necessary previous step towards accountability[3], in most cases) has been portrayed in most of the assessments. As explained by Sheila Coronel a couple of years ago, “the existing ratings differ only in the countries they cover and some of the indicators they use. There is already much overlap in this field” [4]. A similar situation can be pictured in the OGD field. 

Even though both fields are intrinsically related there are some important points that differ FOI and OGD (I won’t go into the details as I wrote about this in previous posts). Despite all the differences in the assessments there are also some other variables in both within and between both fields. This is my attempt at classification:

Scope:
       One Country (government and civil society) Even though this is not the focus of this post but it is important to include them here.
       Global (international). Mostly from INGOs and/or IGOs
Object:
       Information
       Open Data
       Proactive disclosure
       Reactive disclosure
       Public information
       Personal information
Types of Assessment:
       Performance/implementation
       Legal assessment


Thus, if we take these variables we can classify some of the assessments in both fields, as follow:

Access to Government-held Information
(RTI/FOI)



Open Government Data- Right to Data







-       Recommendation for future assessments combining both fields:
Summing up some of the above-mentioned ideas and looking into the future of both fields, there are some points I would like to share (hoping to get feedback to keep thinking about these issues).

Firstly, if we are talking about different initiatives (even though they share similar resources), different goals and different scopes, we also need to think about the need to design different assessments (or joint assessments but focusing on different elements).



Thus, we need to have a baseline in both areas (in global initiatives) to which the different assessments can add their own particularities. There are basic elements in both fields such as the presence of FOI legislation according to international standards and the actual possibility of reusing (licensing and copyright) the disclosed data in reusable formats. This means that in each of the accessed districts, users have a guaranteed right to access government-held information (legislation) and also a guaranteed access to preferable format (to request OGD and not only obtain the data via governments’ proactive disclosure).

In terms of implementation (as sometimes the letter of the legislation has little to say about the actual right to access government-held information and data in a given country), it is necessary to count with statistics on compliance in FOI and OGD. This type of information will allow advocates and researchers to produce an accurate picture of the access to information and data in a given district.

Another important point is to have statistics on the use of that government-held information and data (even though the proactive disclosure of OGD makes this type of figures really difficult to gather).

These are all basic elements that scholars and advocates in both fields already know. However, there are another sort of recommendations, closely related to the common sense, but it is important to remember when designing a global/international measurement in FOI and/or OGD. They are:

-       Clear goals (what do we want to measure and what for?)
-       Clear methodology according to the goals (don’t follow established practices just because everybody is doing the same thing but don’t try to reinvent the wheel if it is not necessary either)
-       If you are not adding value, don’t waste resources. In some cases, “the existing ratings differ only in the countries they cover and some of the indicators they use. There is already much overlap in this field”
-       Acknowledgment of the particularities of each initiative (different types of disclosure, scope, etc.)
-       Acknowledgment of the particularities of each reality (context matters)


I started this post saying that, following Sheila (Coronel Columbia Journalism School) we are in “the midst of an explosion in the measurement of government openness and the accessibility of information to the public”. In this context, it is extremely important to learn from past mistakes and also to avoid “reinventing the wheel”. And why this is important? Just because, as mentioned by Alan Hudson (Global Integrity), “assessments, done right, can be an entry-point for learning and reflection”, and both fields really need to learn from each other and to reflect about how to improve not only the availability of information and data but also the actual use to promote not only transparency but also accountability (among many other important goals).



Notes:  


[1] The Open Government Partnership is a multilateral initiative that aims to secure concrete commitments from governments to promote transparency, empower citizens, fight corruption, and harness new technologies to strengthen governance. In the spirit of multi-stakeholder collaboration, OGP is overseen by a Steering Committee including representatives of governments and civil society organizations.”
“The Open Government Partnership formally launched on September 20, 2011, when the 8 founding governments (Brazil, Indonesia, Mexico, Norway, the Philippines, South Africa, the United Kingdom and the United States) endorsed the Open Government Declaration, and announced their country action plans. In just two years, OGP has welcomed the commitment of 57 additional governments to join the Partnership.”
[2] It is used to be called Open Data Census, which it was clearly a methodological mistake as only some countries are reviewed in the exercises (not all the elements of the universe, clearly)
[3] Despite the many differences- emphasis on the formats in the case of the OGD assessments, for example- in both cases the measurements and assessments relate to the availability of different types of information. In both cases, the measurements are just indicators of opaque (mostly) and clear transparency (in words of Jonathan Fox), depending on the case.
[4] There is a broad agreement among transparency researchers that the presence of FOI laws is an insufficient measure of transparency (Coronel, 2012)