Government slams CIPFA library data as 'flawed' and 'unusable'

The government-appointed Libraries Taskforce has criticised the body responsible for publishing library statistics, saying its "flawed" data suggests the sector is in a worse state than it really is. The true picture shows there is grounds for optimism, the government has argued.

Library campaigners have welcomed the probe into accountancy firm CIPFA's data, but argued the responsibility for rectifying the problem lies with the government.

Although CIPFA's annual statistics are heavily used as a primary source of information for debate about the future of public libraries, the Department for Digital, Culture, Media and Sport's (DCMS) Libraries Taskforce has described the dataset as "problematic in a number of respects". Following an investigation, concerns highlighted relate to discrepancies in how the data is collected and reported, problems with the structure and accessibility of the data and its failure to "fully reflect the changing role of public libraries in the 21st century".

The dominant narrative surrounding English public libraries has been one of decline, with issues and visits down while libraries themselves cut hours and replace staff with volunteeers or else close. Last December CIPFA reported that 105 libraries had closed in 2016, £66m had been slashed from libraries' budgets and visitor numbers were down by 2.9%.

However, the Taskforce is arguing that were a more in-depth analysis of CIPFA's collected data possible, a more nuanced, less "rudimentary" picture could emerge - one in which there may even be reason for optimism.

For example, the Taskforce said the decline in book issues and visits to libraries was actually slowing. While between 2006/7 to 2016/17 the number of issues and visits had fallen 8% and 4% respectively, the decline last year of issues and visits had slowed to 4% and 3% respectively. Meanwhile, more than a quarter of all library services has actually improved their issue and visitor numbers, it said. The report emphasised: "The existence of trendbuckers, as well as the evidence of variation within library services, is evidence of a national library service with the capacity for resilience."

The Taskforce report also said there was "lack of good data" on how libraries were used, and the changes they've been undergoing, while describing what data it did provide as flawed and possibly incomplete or misleading.

"The data is not structured appropriately for analytics; The data reflects inconsistencies in reporting; The data's standardisation is opaque or misleading; The data does not measure the appropriate variables; The data lacks auditing and accountability," the report reads.

The excel sheet provided by CIPFA is branded "unusable" for analytics purposes and the "out of date structure ... hinders data analysis on the part of DCMS, the local authority, or interested citizens". Certain forms of analysis and comparison between library services in similar areas/ working with similar cost bases are not possible; high impact events such as renovations or temporary closures are not made obvious; and variables and measurement granularity is described as "not fit for purpose".

Another issue with the data is "inconsistency over time", with numbers of participating libraries dwindling from 100% of authorities in 2006-07 to 86% in 2016-17. Aside from how time-consuming it is to have to collect and record data manually, and the "intense public pressures" as a result of publishing, one reason suggested as to why fewer libraries are participating is that they themselves simply don't find the data returned to them useful.

"For the library services who do submit CIPFA data, it is increasingly difficult to justify the efforts to provide consistent and accurate data, especially given constrained resources," the report reads. "As a result variables are left blank, further decreasing the consistency of the dataset overall."

Variation in how data is recorded adds another layer of mistrust in the figures. For example, some libraries use electronic counters to count visitor numbers, CIPFA's preferred method, while others use sampling. The report describes a lack of incentive for libraries to check the accuracy of the data they provide and even notes instances where, instead of halving numbers from electronic counters to account for exits, some libraries have submitted the total tally (in effect double counting). The report laments the lack of an auditing system that it says means it's not possible to catch some of these errors.

The report's authors underline "the nature of CIPFA's statistics sheets goes against the very ethos of the library sector: proprietary data, expensive and inaccessible, it is the exact opposite of the free and open information libraries are proud to provide". It concludes: "The flaws in the data impair our ability to understand trends at the national level in a comprehensive way."

Library campaigners have welcomed the government's exposure of the issues with the CIPFA data, but criticised the Libraries Taskforce, saying the responsibility to rectify the problem lies with the DCMS.

Former Waterstones m.d. and library campaigner Tim Coates (pictured right) pointed to the 1964 Libraries Act, which he said made this plain. “The report rightly highlights many well known problems with CIPFA data. But it also points out that the figures are the only management data that exists and the only way of informing the public how their money on the public library service is spent," he said. "The DCMS has a legal obligation to obtain such figures as are needed effectively to superintend public libraries and therefore the onus falls on them, urgently, to address the problems this report identifies. The library task force should have done that four years ago and has failed."

Coates argued further that reporting on library statistics should be far more frequent than once a year. "Library systems operate in every single public library and they could easily be used to provide all the data - not just every year (as CIPFA do)  but actually every day to a public intranet at the close of business. That is the kind of reporting retailers use and the kind we should have for public libraries and is what local councillors need to fulfil their management role," he said.

Elizabeth Ash, a founding member of library campaign group Speak Up For Libraries, said she found it "frustrating" that CIPFA's figures weren't currently publicly available (at present to access the figures even participating libraries need to pay a fee), but, like Coates, she also criticised the Library Taskforce for "dragging its heels".

"It is frustrating that CIPFA report on public libraries but that the full report is not publically available. It is annoying that despite the Libraries Taskforce identifying the need for open access data on every library authority being made available in a more timely and complete manner, that all they have managed to produce is a list of libraries by authority with contact details. The public library service is being eroded and the Taskforce is dragging its heels producing any measure with which we can monitor this," said Ash.

Nick Poole, chief executive of CILIP, the library and information association, added his voice to calls for "open" and "more robust" data. 

"We support the report’s call for the production of high-quality open data about the public library network. This greater transparency would provide a range of benefits including greater support for effective service delivery and decision-making at a local and regional level, and national oversight and strategy," said Poole. "The identification of the importance of forward-thinking leadership, political support and the flexibility to innovate as key factors to the success of a library service are very useful, but these kind of insights would have much greater value if based on more robust open data."

A CIPFA spokesperson commented: "CIPFA is committed to providing robust and useful data on libraries, and, to do this, we believe we must not shy away from making improvements to data relevance, quality and access where they are needed. Therefore, we welcome this helpful report and look forward to working with the Department for Culture, Media and Sport (DCMS) to improve our public libraries data. It will be particularly useful to explore how best we can encourage all library authorities to respond to our survey and to respond accurately."