“You will all have experienced procurements that seemed to go on forever, cost millions of pounds and took countless hours of your employees’ time and energy. I know how frustrating this can be and I can promise you here today that we will do things differently…But there will also be things we expect from you. Government will no longer offer the easy margins of the past. We will open up the market to smaller suppliers and mutuals and we will expect you to partner with them as equals, not as sub-ordinates. The days of the mega IT contracts are over, we will need you to rethink the way you approach projects, making them smaller, off the shelf and open source where possible. We will expect you to be transparent in all your dealings with us and for the terms of the contracts we sign with you to be published online.”

Sounds promising…anything we can do to help?

The non-ideal 2012 [correction: 2011] Census will see Lockheed Martin paid £500m-odd of money we can ill afford to undertake a clunky process of data gathering which will take 2-3 years to complete and feed back.

But if everyone had a personal data store such as Mydex….

….one could simply add to the personal data store the fields needed to complete the Census questionnaire. ONS could invite people to volunteer this information, or could see how far it got compelling it by law with threats of dire consequences. It could poll the information once every ten years if that were good enough for statistical purposes and for planning public services. Or it could poll people’s personal data stores ever 10 months, 10 weeks, 10 hours, 10 minutes, or 10 seconds. Lockheed Martin could go back to making rockets and bombs. We’d save a pile of money. And we’d start to be able to plan public services based on real needs and preferences instead of an out-of-date decennial view.

The immediate question (to anticipate any ONS trolling [amendment: pushback]) is universality. How can you possibly make PDSs universal in the way the census needs to be? Perhaps the answer is: given the huge benefits both to the individual and the state of working with PDSs, how much incentive can we plan for the individual to help this to spread far, wide and fast? Remember the core principle has to be gaining the individual’s trust, so intrusive data gathering and playing fast & loose with the data is ruled out.

But if you want a Big Society fuelled on accurate, up to date data on personal needs, circumstances and preferences this has to be the way to go. The 2012 2011 Census is going to feel about as far from Ideal as procuring a Stealth bomber to run the country’s Neighbourhood Watch Schemes.

Mydex: A Manifesto for UK Public Services

One area government IT has made progress is with public data, with the “power of information” policy and the data.gov.uk portal, which recognises the value of ‘unlocking’ data held by the Government for reuse by added value service providers.

Next we need a comparably radical rethink on personal data. This starts with a return to the role of personal identifiers and intermediaries set out by UK officials a decade ago, and as recently adopted by the Obama Administration. This means:

• assume that access to on-line public services will be through a market or ecosystem of accredited third-party identifiers (issued for example by a range of existing online services, credit bureaux, or banks
• drop the false notion that it’s generally essential to know who people are
• challenge the assumption that personal data is “owned” by service-providing departments to be shared at their convenience
• instead, recognise that the individual is not only the rightful owner, but also the only technically feasible point of integration of exponentially growing volumes of personal data, and therefore the only possible place where “personalisation” can happen
• recognise furthermore that structured, scalable personal data managed by individuals is set to become the source of immense new economic value, and that the individual is a rightful
beneficiary.

This change in mindset includes a specific challenge to secret parts of government entrusted with keeping Britain safe. A safe society isn’t the outcome of dysfunctional public services designed to aid surveillance.

Britain has a far better chance of being secure with public services designed to work for individuals and front-line public servants, which respect human rights and dignity. When the data are cleaner, the relatively small number of exceptions stand out more clearly.

On-line identifiers need to work under the user’s control, with minimal disclosure and revealing information only to justified parties. They need to be consistent and convenient (see Kim Cameron’s “Laws of Identity”).

In the short term the UK can copy the US administration: announce that future access to online services will be via third-party identifiers, and then provide for the emergence of a “trust framework” so a range of identifiers are accredited for suitable purposes. Many services can be accessed anonymously, and for many more all that is needed is a consistent user experience. It’s not always necessary to identify people to check their entitlement.

But sometimes individuals will need to invoke stronger identification credentials online: for “Know Your Customer” processes or to meet the most stringent visa requirements for example.

Government IT therefore needs to anticipate a world where individuals are equipped with

• highly evolved personal data stores
• the ability online to invoke strong authentication or verification
(e.g. proof of qualifications, licences, credit, nationality or identity)
• selective disclosure, i.e. the ability to share the minimum necessary in a particular circumstance.

This doesn’t require major new procurement. It means:

• review each main service function to take into account the role
of user-driven records for health, education, welfare, transport, or
other areas such as the Census or the London Olympics
• quickly participate in at least two live prototypes of user-driven
services across multiple organisations supported by independent
online verification services
• where there is benefit, re-engineer the public services (health,
education etc) users can drive new services.

Just as the existing “Power of Information” has created new APIs to allow structured public data out of government systems to create new value, so this “empowered citizen/customers” agenda will see new APIs that allow structured personal data in. This means public services can be driven and personalised by users, and new service packages created for them by third parties.

This “empowered citizen/customers” agenda might even reveal a revised role for the National ID Register as a voluntary service offering online verification as part of a trust framework, for the most demanding cases.

Happily, the urge to save money will usher in the right identity policy and in turn protect our privacy.

The area to focus on is data logistics. When Alan Mitchell and I browsed through hundreds of complaints about public services recently we observed that very few are about privacy and none at all about problems with identity. But the vast majority point to poor information logistics. The key person – official, professional, or the unhappy individual – just didn’t have the right information at the right time.

This causes irritation, frustration, offence, and vast expense. It’s extremely annoying for individuals not to be able to get hold of information they need, to have the wrong information, or to have to give the same information over and over again. It’s unjust, time-consuming and possibly worse to get the wrong treatment or service because the service cant get the information or has the wrong information.

It’s unnecessarily expensive for public services to attempt to maintain hundreds of different records about the same person (but neither feasible nor desirable to amalgamate them into panoptical mega-records). If you provide services on the back of incomplete and inaccurate data there’s every chance the service will be poor and unnecessarily expensive.

And it’s hard to plan and prioritise if you’re not in touch with your customers and people try as far as possible to withhold data from you. If we built churches using the last census there would be a few Jedi cathedrals lying empty.

If we can fix this (and we think it can be done) then people can get better, more responsive service, restored individual responsibility with a path to empowered self-service. HM Treasury also gets a triple dose of cost savings.

It means restoring control over personal data to the individual and building trust on the side of the individual.

User-controlled digital identifiers within an identity assurance framework are prerequisite, and that is just what Cabinet Office is now quietly proposing. Better privacy is a by-product (and a legal requirement, let’s not forget). But the compelling reason to pursue better data logistics with user-driven services is saving money.

That’s not to say the underlying problems ContactPoint was meant to help with – caused by poorly co-ordinated and overstretched childrens’ services – have gone away; they haven’t.

The question of how technology best supports front line professionals, without disproportionate and unwarranted intrusion remains unanswered. It’s part of the scope of the Munro review, which provides first feeback in September, and a final report in April 2011. I suspect we’re in good hands here. I’d hazard a guess that Dr Munro will focus relentlessly on the crucial matter of protection of the relatively small number of children at real risk, and not attempt to boil the ocean of the welfare, diet, propensity to obesity and general wellbeing and conformance to social norms of every child. And I also bet that the role she recommends for ICT in helping child-protection professionals will be conformant to data protection and human rights law in a way that ContactPoint was not.

The Databankendämmerung must spread, just as we must escape the limitations of the Accentureweltanschauung. There are other ill-advised and intrusive central databases on which we should call time: eCaf; NHS SCR; the NHS Detailed Care Record; NHS Secondary Uses Service; long term comms data retention generally and the Intercept Modernisation Programme in particular. Kind friends won’t let me forget that I’ve promised to do a special celebration to mark the end of the Benighted ID Scheme and its lavish quantities of nugatory PA consulting.

The LibDems always opposed the “Database State”. The Tories were quick to spot that the last administration had taken a wrong turn and were politically vulnerable. But when Labour Ministers stopped listening exclusively to Cheltenham and Whitehall and resumed listening to the outside world (about eight weeks before the last election) they too quickly came to their senses as well.

It’s best not to see this in political terms, because really it’s a question of information logistics. Remember Troubleshooter? If John Harvey-Jones could revisit us and contemplate the dozens, hundreds of databases which public and private organisations run each trying to scrape, grab and update their versions of us, and then looked at the average householder spending a week and a half updating the different customer service systems of every entity we ever have to deal with (through episodes from moving house to losing a wallet) recording and sharing the same data over and over again, filling out endless forms with different callcentres and web sites and usernames and passwords, ….he would just laugh his vast laugh, wouldn’t he? And as he laughed he’d start to calculate the waste and loss of value, and huge tears would roll down his generous cheeks.

The Database State is an issue of civil liberties, justice and equality, of course. But there more than that: it’s been clear for a good year that the country heading for bankruptcy. It has been clear for a decade we need radical reform of public services. It has been clear ever since people started chipping in their ideas to IdealGov that the role of technology in this radical reform is about user participation, about quick wins and creating a foundation of trust.

The radical money-saving reforms have to be based on accurate personal data. They have to be built with tech systems that work. They have to draw on people’s supportive, active participation.

Some databases are valid and unobjectionable of course: DVLA, TV licensing, the electoral roll. Many public-sector databases can be fixed. The point about the Databankendämmerung isn’t that all databases are evil. It’s that the state can’t fix society’s complex human problems with giant databases.

Weirdly enough, however, the opposite will turn out to be true. Even the worthwhile databases are still plagued with errors, omissions and duplications, They need our help. Databases can’t fix society. But, given the tools, society can start to fix the databases. That’s a much more promising way forward.

We will take steps to open up government procurement and reduce costs; and we will publish government ICT contracts online.
We will create a level playing field for opensource software and will enable large ICT projects to be split into smaller components.
We will require full, online disclosure of all central government spending and contracts over £25,000.
We will create a new ‘right to data’ so that government-held datasets can be requested and used by the public, and then published on a regular basis.
We will require all councils to publish meeting minutes and local service and performance data.
We will require all councils to publish items of spending above £500, and to publish contracts and tender documents in full.
We will ensure that all data published by public bodies is published in an open and standardised format, so that it can be used easily and with minimal cost by third parties.

It’s an important read with some enlightened ideas in a realistic tone which acknowledges the real diffrences on important topics such as Trident.

It closes with the sobering reminder:

The deficit reduction programme takesprecedence over any of the other measures in this agreement, and the speed of implementation of any measures that have a cost to the public finances will depend on decisions to be made in the Comprehensive Spending Review.

Cheers Edgar.

The 2011 Census will adopt a similar processing strategy to that used in the 2001 Census.

But this is a £500m deal. And it’s 10 years on. Shouldn’t things have changed more than this?

The Times reported

The ONS has already formed a working group to investigate alternatives to the census. It will present it recommendations to the government after the 2011 survey.

“We’ve started thinking, ‘Okay, what comes beyond 2011?’” said Peter Benton, deputy director of the census at the ONS. “We’ve set up a programme to look at our need for information and the different ways we could meet that need in future.”

What do we do here? Does a bankrupt country want to spend £500m a time on a C19th process? How many Jedi will get recorded, as public trust in government data handling plummets?

But we need the data as much as ever; we need to build and improve the consistent data set, and we probably need more data now and in future rather than less. We need a census for the #PBAge.

ONS says it’s going to look at the sort of stuff that Finland has been doing for a decade. But have they looked at where the future is headed with VRM and volunteered personal information? What chance their supplier Lockheed Martin has been offering “thought leadership” on techniques that would slash the cost of the process (do they care if it were way more effedctive, and privacy-friendly?)

Perhaps when ONS feel sufficiently on top of the 2011 census, they’ll take a moment to look at this sort of stuff.

This almost comes onto the radar of big IT suppliers. It’s massive for smart little NGOs; it would have funded about a decade of early MySociety work.

I got it in an email (extract below). There’s probably a URL but I dont have it yet. This was announced by my local MP Jeremy Hunt. They’d take the cash from the Cabinet Office budget.

This is going to be fun!

Hi there – hope you’ve all had a merry and relaxing Christmas.

I just wanted to flag up the £1 million competition that we have
announced today for anyone who can develop an online platform that
enables us to tap into the wisdom of crowds to resolve difficult
policy challenges. In government, we will use this platform to publish
all Green Papers, and open up the entire policy making process to the
public. See briefing note below for more details.

This really is the most radical crowdsourcing announcement ever made
by a UK political party – not only in terms of our commitment to
opening up the policy making process, but also because of our use of a
Longitude/Netflix style prize.

We’d be really grateful if you were able to flag up this announcement,
and the press release below, to your contacts in the IT media. After
all, we want lots of people to enter this competition and develop
online collaborative platforms – so publicity is obviously crucial!

All the best

Hunt: Solving problems together – harnessing the Wisdom of Crowds

The Conservatives are today announcing a competition, with a £1million
prize, for the best new technology platform that helps people come
together to solve the problems that matter to them – whether that’s
tackling government waste, designing a local planning strategy,
finding the best school or avoiding roadworks.

This online platform will then be used by a future Conservative
government to throw open the policy making process to the public, and
harness the wisdom of the crowd so that the public can collaborate to
improve government policy. For example, a Conservative government
would publish all government Green Papers on this platform, so that
everyone can have their say on government policies, and feed in their
ideas to make them better.

This is in addition to our existing radical commitment to introduce a
Public Reading Stage for legislation so that the public can comment on
draft bills, and highlight drafting errors or potential improvements.

Launching the competition, Shadow Culture Secretary Jeremy Hunt said:

“Conservatives believe that the collective wisdom of the British
people is much greater than that of a bunch of politicians or
so-called experts. And new technology now allows us to harness that
wisdom like never before. So at this time of year, when families and
friends are getting together, we’re announcing a new idea to help the
British people get together to help solve the problems that matter to
them.

“There are currently no technological platforms that enable in-depth
online collaboration on the scale required by Government – this prize
is a good and cost-effective way of getting one.

“Too often policy has been ill thought through with disastrous
consequences. When formulating and implementing policy why should we
not listen to the hundreds of thousands of experts out there?”

ENDS

For further information please call Ramesh Chhabra on 07738 935 187

Notes to Editors.

In the bureaucratic age, decisions in government, business and other
organisations were typically made by a small, closed group of experts.
In the post-bureaucratic age, new technologies enable us to reject
this top-down approach to decision-making. These new technologies
allow us to harness the wisdom of the crowd, take advantage of the
power of mass collaboration and make use of the information and ideas
dispersed amongst large groups of people. Evidence from around the
world has shown that this post-bureaucratic approach can result in
more efficient and effective decision-making and problem solving than
relying on small groups of experts.

Harnessing the wisdom of the crowd in this way is a fundamentally
Conservative approach, based on the insight that using dispersed
information, such as that contained within a market, often leads to
better outcomes than centralised and closed systems. The Conservative
Party has already used crowd sourcing to develop new policies, for
example through our ‘Stand Up Speak Up’ initiative. To make sure that
we make best use of this approach, a Conservative government will
offer an unprecedented £1 million prize for any individual or team
that develops a platform that enables large groups of people to come
together online to solve common problems and develop new policies.

Harnessing the wisdom of crowds – case studies

Innocentive

Innocentive is a website used by leading companies such as Proctor and
Gamble and charities such as the Rockefeller Foundation, to tap into
the wisdom of the crowd and get answers to otherwise intractable
research problems. There are over 160,000 scientists and other experts
in the Innocentive network, and they are incentivised to take part
through cash prizes for solving problems.

Improvng the Netflix algorithm

Netflix, a US-based DVD rental company, wanted to improve the
algorithm it uses to recommend films to users. Instead of hiring a
research team itself, it threw open its dataset, and offered a $1m
prize for anyone who could improve its algorithm by 10% or more. This
approach yielded a solution far more cheaply and quickly than relying
on an internal team of researchers.

Peer-to-patent

Peer-to-Patent uses the wisdom of the crowd to improve the patent
process, and has been trialled by the US Patent Office. Under this
approach, patent applications are posted online, so that instead of
relying on a small group of bureaucrats, anyone in the world can check
whether the application is valid. This approach seems to be much
faster and more efficient than the traditional closed approach to
appraising patent applications.

Solving maths problems

In January 2009, Timothy Gowers, professor of mathematics at Cambridge
University and a holder of the Fields Medal, posted a hitherto
intractable maths challenge on his blog, and invited readers from
across the world to collaborate and solve the problem. The resulting
comment thread spanned hundreds of thousands of words and drew in
dozens of contributors. Six weeks later, the theorem was proved.

Harnessing the wisdom of crowds – 10 potential applications

Here are ten ideas to get the ball rolling: ten problems (ranging from
the serious to the somewhat seasonal) that we think could better be
solved by the collective wisdom of the British people than by a bunch
of experts sitting round a table. But the whole point of our
competition is to stimulate discussion about the different problems
that we can solve together if we had an easy to use online platform
for collaboration…so here are some of the possibilities:

1.　　　 Identifying and rooting out wasteful government spending.

2.　　　 Designing credit card bills that anyone can understand.

3.　　　 Finding a safe place to park your bike.

4.　　　 Rating the quality of schools and hospitals, to help other
people make informed choices.

5.　　 Making　government information – for example on how to fill in
your tax return or set up a new business – clear, simple and useful.

6.　　 Creating new technology that blocks all spam emails.

7.　　　 Locating current and planned road works, and working out a route
that avoids them.

8. Deciding how National Lottery good causes money should be spent.

9.　　 Picking the England squad for the 2010 World Cup.

10.　　　Designing a strategic plan for your community or city.

Harnessing the wisdom of crowds in policy making

In the post-bureaucratic age, opening up the policy making process can
help us to design better policy and transfer more control to
individuals and communities. The Conservative Party is committed to
harnessing the wisdom of crowds in a number of ways:

-　　　　　　　　 We will introduce a Public Reading Stage for legislation, so
that the public can help to spot errors in legislation, and feed in
their comments during the legislative

process.

-　　　　　　　　 We will set government data free, enabling the public to
collaborate and develop new social and commercial applications.

-　　　　　　　　 We are publishing online, and in real time, the expense
claims of our Shadow Cabinet, enabling full and instant scrutiny.

-　　　　　　　　 We have published online a leaked version of the
Government’s IT strategy, so that people can post their suggestions on
how to develop a better set of policies.

A Conservative government would seek to make extensive use of this
approach. However, there are currently no technological platforms that
enable in-depth online collaboration on the scale required by
government.

We are today announcing that a Conservative government will offer a £1
million prize for any individual or team that develops an online
platform that enables large scale collaboration and meets the
specifications that we will be publishing alongside the official
opening of the competition following the election. This platform will
then be used by a future Conservative government to throw open the
policy making process to the public, and harness the wisdom of the
crowd. For example, a Conservative government will publish all
government Green Papers on this innovative and open platform. The
source code of the platform will be made openly available, so that it
can be used by local councils, social enterprises and other
organisations free of charge.

While leading institutions such as the Gates Foundation, Google and
Netflix have successfully made use of procurement prizes, this £1
million prize will be the largest prize ever offered by a British
government in the modern era. The prize will be funded from within the
Cabinet Office budget.

Let’s do this together. Let’s say “wouldn’t it be better if” about how tech affects transparency, costs and the quality of public services and how they affect our lives.

@ntouk and I have long since been fed up with what one senior Whitehall official yesterday called “this £trillion attempt to drag us into 1983″. Many of us have had a go at the draft government IT strategy on the Opposition’s makeITbetter site. Officials across Whitehall are now furiously revising it, so let’s hope the final published version is better.

Meanwhile we can speak freely. We can look to the realities of the wider world, and we don’t have to pretend that everything to date has been fine. Now it’s time to find our voice and say what we want.

The Centre for Technology Policy Research and IdealGov are launching a six-week competition, which everyone wins. Everyone who contributes is invited to a party. And everyone can, like, bring stuff (as we did to mypublicservices).

Practicalities. Please add any comments of suggestions about the process to this post. The final crowd-sourced “White paper of Wibbi” will be created on an open wiki here. Please feel free to register and edit, or to add comments at the end.

Party: IdealGov and CTPR are chipping in £1000 to the launch party to which everyone who has contributed is invited. There will be prizes including signed photos of our very own tech mandarin Sir Bonar Neville-Kingdom.

Political engagement:An Opposition front bench team has already agreed to listen attentively to our results. We have also extended invitations to Labour and LibDem leaderships and to officials to attend the party or have the results presented. [UPDATE: Big news: still on day one and we've now also heard back that this will get presented to a LibDem front bench team and to the people drafting the Labour manifesto. This is subject to the project attracting enough substantial input of quality. So this is now definitely an opportunity to put good ideas in front of all three main UK parties. We're also up for inviting SNP, Plaid & Greens to launch party. Everyone needs a good government IT policy.]]

This initiative is formally adopting the principles of #CMRD, the “courteous and mutually respectful dialogue” called for last week by Michael Wills and first practised at an Intellect/Identity and Passport Service event this week.

Above all, we need to state in plain lay terms the role of contempory technology in future public services should be. Specifically, our work will need to cover off the main headings:

- governance of public-sector IT
- technical architecture which supports the real-world intention
- procurement of technology and tech-based services
- design that works for front line staff and users
- basis for participative public services
- public data
- personal data
- trust, dignity & legality under human rights & DP law
- political engagement, openness and trust in the political process
- and above all saving vast, vast amounts of money.

This is not a time to splash out. The country’s broke. So first we need to spend less on IT, existing contracts notwithstanding. But then it’s two orders of magnitude more important that our IT plans support far more efficient public services.

Suggesting we deploy hundreds of PA consultants (or Deloitte or whoever) to mooch around filling out timesheets and expense claims for absurd day rates is not going to get you invited to the party. But any suggestion that draws the best expertise available into the gift economy (and by no means are all consultants nitwits) is most welcome.

This project is not a platform for venting anger at wrong headedness or past mediocrity (whoops! did I just do it? Old habits…) Take that frustration but use it to say what you want in the spirit of the #CMRD. Please bring your beliefs, principles, and passion, but the IdealGov and CTPR moderators will give short shrift to anything actionable or which reeks of partisan preconceptions. Scepticism is justified, but cynicism not.

We may need a “babies and bathwater” section to set out for controversial systems such as CfH or the ID Scheme what must go but what also should be retained. We should give praise where due, eg for Power of Information work. And our suggestions must be practical enough to keep the lights on, ie to keep essential services running uninterrupted while new and better plans emerge.

Contributions from all stakeholders are welcome: officials, industry, front line staff, anyone who speaks from personal experience of public services. Pertinent Art is always welcome, because it can speak to our condition so powerfully.

We last did this in 2004, remember. Now its time to do it again.

IdealGovernment has wanted this for years. But – as he himself proves – it won’t be easy. Let me get some things off my chest. Then let the new era of civilised and mutually respectful dialogue commence.

I don’t agree with everything Mr Wills said, so I’ve taken his speech and commented in line. Attention conservation notice: this is quite a long post

Databases lie at the heart of this revolution. They offer the opportunity to improve dramatically the efficiency and responsiveness of public services. Take the Tell Us Once project, for example.

But the TellUsOnce problem won’t be solved by databases. If it was, you could have solved it by 2005 (as you promised you would in 2000).

OTOH a VRM platform would show this problem solved for so little money that the invoice wouldn’t even cross your desk – a fraction of the cost of yesterday’s “One Place” web site (which in turn you could have had for free a decade ago if you had practised respectful and courteous dialogue and listened to Stef).

The increasing sophistication of data management has sparked serious public concern about privacy and civil liberties

We’d be fine if the approach were sophisticated. It’s the crudeness, combined with raw power, that concerns us.

Go onto Google UK and search for ‘UK government big brother state’ and you get one and a half million entries.

WTF? Is this how we gauge if it’s of public concern? As a joke maybe, eight years ago. Oh look: go onto Google UK and search for: ‘Wills eejit or ****?’ and you get 1.21m entries. Does that make it the question everyone is asking?

He talks of ‘striking a balance’ between security and liberty. He’s no Benjamin Franklin. It’s not about striking a balance. It’s about getting it right because we need both privacy and security in the systems which underpin essential public services. Mr Wills claims that kids can’t get free school meals, that voters can’t vote without data sharing by public bodies. This is absurd. He says hardly any constituents complain about surveillance but they demand CCTV in their hundreds.

To reconcile all this we need, he says

“democratic discourse, rational and mutually respectful discourse, wary of anyone, on any side of the debate, who claims a monopoly of wisdom. These issues are complex and difficult and resolving them will require intellectual rigour, a willingness to learn from experience and to engage continually with alternative points of view. Only through such a democratic iterative process can we hope as a society to resolve this issue satisfactorily.”

Hurrah. This could be straight out of the Ideal Government guide to, well, ideal government. We like that.

Sadly, such a rational, respectful discourse, so essential to the creation of public policy on this crucial issue, has been largely absent in recent years, replaced all too often by reciprocal caricaturing and stereotyping, with understanding and respect all too seldom present. And this matters.

This is true too. I must say I welcome his recognition of it, and his acceptance that

Government must take its share of the blame for this failure of discourse.

Mr Wills’ spin doctors had leaked his speech to a class-obsessed tabloid journalist David Aa. I can see now that when I saw the next bit in that hostile context I misread it:

Too often, we have been overly defensive and dismissive of criticism. Government believes it is acting benignly and legally and has not adequately recognised the fears of those who believe this is not the case.

Mr Aa is paid to be annoying, and it works. In my irritation I took the extract to mean that government was defensive and should be more assertive, but Mr Wills’ point here (and the context makes it clear) is a more subtle one which I welcome.

He then says companies make mistakes too. True, but we withdraw our business from them when they are mediocre. They’re not entrusted with passing laws, taxing us, and trying to run public services. He goes on

Where government gets it wrong, we are learning to hold our hands up and take immediate steps to put matters right. The loss of Her Majesty’s Revenue and Customs disks triggered radical reforms of data security in government.

Her Majesty’s data was not on that disk. Mine was. Nevertheless, horrified, many of us welcomed that loss, because it finally started to show how bad things had really got.

Did it trigger radical reform? Well, it certainly triggered half a dozen reports from tame senior officials and management consultants. It triggered episodic bans on Blackberries and memory sticks, and the glueing up of USB ports. But we have yet to see the radical reform we need which is to restore control of personal data to the individual.

When we recognised that data sharing provisions in the Coroners and Justice Bill had been too widely drawn we immediately withdrew them.

It’s true. Eventually they did. This would be a good moment for Mr Wills to thank the selfless NGOs who campaigned so hard to rectify this awful mistake he was trying to make. But instead he speaks of

opponents…too quick to assume the worst of government without any evidence to support their assumptions, replacing argument with rhetoric.

But…but…but how then did the unpaid NGO volunteers successfully win the argument against your proposed policy and show it was so ill-conceived? Do you recall how hard you and your officials resisted?

Michael: you and your colleagues will find this cultural change you call for harder than you yet realise. But as long as you try, and appear sincere in your endeavour, we will support you every step of the way. Have we all taken on board the rules you call for? No rhetoric. OK. No rhetoric.

To reject all the benefits that databases offer the public, simply because a mistake might be made, is to strike the balance in the wrong place.

1. Name the benefits. 2. This isn’t about striking a balance, it’s about doing it the right way.

Should we really avoid trying to do all we can to prevent another Soham tragedy?

Oh, please leave the memory of those poor girls in peace! Why blame your poor personal data practices on them, and on the equally unfortunate Victoria Climbie? Michael: this is rhetoric.

Or stop doctors accessing vital medical records?

Doctors can have it with my permission. I choose not to share it with bureaucrats and the forces of law and order. That preference will not endanger society.

Or fetter the provision of welfare entitlements, such as free school meals, for the most vulnerable?

Just get your act together. Deliver the services to which people are entitled, without hoovering up their data when they are at their most vulnerable and brewing it up into HMG’s patent toxic soup.

Basic principles for protecting the use of data are that it should be proportionate and necessary

Indeed. Furthermore, that in the absence of a specific and lawful purpose there should be informed consent. Assuming that 12 year olds understand this and are cool about it is not sufficient. Allowing hundreds of thousands of officials access to childrens’ records or to identifiable medical data is unlawful, just as taking and retaining the DNA of innocents has proven to be.

But you acted all surprised when you lost the Marper case. And you’re behaving as if we had a proper implementation of the EU data protection directive, and are acting all surprised that Europe is taking enforcement action against the UK.

We don’t live in a database state as much as a database society.

Yes, but we’ll sort ourselves out with the Twitterverse. Government runs the state part of it. That’s what you’re screwing up, because of the history of groupthink and the poor dialogue you rightly point to. That’s what we want whoever wins the next election to sort out. Hey! It could be you! Maybe.

They deliver real benefits for the public and it skews debate about the challenge they pose to all of us if anyone ignores this or pretends otherwise.

Let’s see if we can find real evidence for these benefits, because there’s a strong case to be made that the database state is designed to serve an unholy alliance of administrative convenience and security fears.

Meanwhile it is becoming everywhere apparent that a wholly organisation-centric “CRM” approach to life is nothing like as advantageous to the public, or indeed to large organisations, as it was sold as being by the management consultants which Alistair Darling has just banned.

What’s it costing you per data subject to keep records up to date in WUYJ, MIAP, BusinessLink, ContactPoint, eCAF, CfH? How accurate are the data? How complete? And how much duplicated? How much is explicitly permissioned by the data subject? Is it proven to a legal standard that in every case government is holding only data which is necessary and proportionate in a democratic society? Can you demonstrate clear auditable and informed consent?

But, like all technologies, databases can do damage if misused. The issue is not whether to have them but how they can be deployed without damaging privacy.

Ah! Phew! Hooray!

It’s a question of balance and the challenge is how to strike it.

No it’s not! It’s not about balance, any more than climbing to the moon on a ladder is a question of balance. I don’t care how good your balance is: it’s the wrong way to go to the moon.

You get accurate data at lower cost and personalised services without privacy intrusion by putting people back in charge of their own data. The Internet works at both ends, you know. You just can’t have every part of public services grab every piece of data they can about everyone, take away the barriers to data sharing, then hope to create an accurate picture of everything such that you can eliminate fraud, keep the public safe, and provide personalised services. Try doing the maths.

You know all that great “Power of Information” work the government has done on opening up APIs and letting public data out? Sometime soon a Secretary of State or a PM will announce, in a second such major policy shift, that the really big prize is in how we work with personal data. Government will relinquish the desire to own and control it. You will open up government APIs and let people’s structured, scalable private data in, under thier control. You will leave people in charge of their own lives, which is how reality is because we have to put all the pieces together anyway.

I repeat: this is not a question of striking the right balance. It’s about creating a secure platform for personalised services and new value. Getting it wrong, which we described in our report Database State, wastes vast amounts of money and of people’s time; it fails to deliver good customer services and breaks the law. We the taxpayer are thus intruded upon, failed by public services, caused to waste endless time sorting it out. We’ll have to pay to get it wrong and then pay again to see it all put right.

But where we should have a constructive dialogue, we have all too often an impoverished discourse where slogans substitute for evidence.

Too true. I’ve seen so much of that from government politicians and officials. I’m so weary of it. So many of us are so weary of it. But it cheers me up when NGO-world gets creative and cheeky. I love the pertinent art, the campaigning. Not just because it’s often witty, but also because it’s based on a deeper truth, coupled with a better sense of human nature, than your officials have served you up with.

The Rowntree Report on what the authors called the Database State is a good example of how the public discourse is flawed.

It shows how bad things have got. It’s pretty shocking that the fact that much of what you’re constructing falls outside what it is legal to do in Europe should come as such a surprise. It has taken you a full NINE MONTHS to reply to it. Did you really not consider these questions before you embarked on Transformational Government, and mixed up the security agenda with the public-service agenda? Was that wise? Don’t shoot the messenger here.

This could have made an important contribution towards meeting the challenges of new technology. The subject matter was important and its academic authors have a distinguished provenance.

I think you’re building up to shooting the messenger.

However, a detailed reading of the report reveals it was riddled with factual errors and misunderstandings and reached conclusions without setting out the evidential base for doing so.

Argh! You just shot the messenger!

So opaque was its methodology that it has taken months to work through it to respond in detail.

Stop shooting! Remember, this is the new era of respectful dialogue!

The methodology is clear as day. We wrote up 46 large databases (with precious little help from government). We discussed them against European criteria for legality. During the course of the work the Marper judgement confirmed that the DNA database was indeed illegal. We gave all our references.

I’ll get on to the government’s response in due course. It has taken their however many dozens of staff nine months to reply, so working at the same pace I should be ready to reply by around 2030-2050.

I hope that all those who read the original report and provided publicity for it will do similarly for today’s response to it.

I’m sure they will if it’s interesting, credible and passes the Mandy Rice-Davies test.

It is important that we now move beyond rhetoric

I think I may find myself quoting this again, perhaps frequently We may need a new acronym to go alongside Wibbi: IMBR.

new and detailed dialogue between all concerned to ensure that we seize the opportunities of this new information age while protecting ourselves against its risks. So when government is considering how data might be used for the public good, the voices of users and practitioners can be heard. That requires an open, constructive approach on both sides.

Yup, we all sign up to that. High bloody time.

To that end, I am announcing today that the Ministry of Justice will host an event early in the New Year to consider how we approach the data sharing aspects of reforms to the electoral register.

Alright then. But not in Feb; I’m in Iran.

He goes on to talk about cross-referencing the electoral roll with the NI number. Well, you need to do something to make this process more secure. Not so sure about the NI number: it might suddenly look as if we have 80m registered voters in the UK.

Those identifiers will not be available to the public, for obvious reasons – they are solely for the electoral registration officer’s use.

Not obvious to me at all Michael – I think you have organisation-centric assumptions engrained so deeply you’re not even aware of them.

I also intend to jointly host an event with Delyth Morgan from the Department for Children, Schools and Families which will focus on ContactPoint.

Clearly it would have been helpful to have this dialogue before you specified the system and started spending hundreds of millions of pounds of taxpayers’ money on it with Cap Gemini. So when we say “better late than never” it is with quite a heavy sigh.

We can never be complacent about databases – the challenge in getting the balance right

No it’s not about balance! It’s about getting it right!

But we can only do this on the basis of a rational and mutually respectful dialogue between all concerned. I hope the measures I have announced today can be the start of such a dialogue

Indeed. IMBR. I think you, your speechwriters and spin doctors will find this a major cultural adjustment. You’re clearly not there yet. But we will support and uphold you in your attempt to do just that, and for my part (I cant speak for anyone else) I shall try to do the same.

Was this whole long rambling post worth doing? Is Mr Wills going to get his act together and sort this out? I checked out his currency with one of Westminster’s most effective and reliable lobbysists. The verdict: “I wouldn’t expend too much effort on trying to enlighten him.” I find that harsh. He’s still a Minister after all.

Let us give Mr Wills credit for officially ushering in – if not yet himself exemplifying – the era of mutually respectful and courteous dialogue about the right way for government to work with personal data in public services. From the moment I click on “Publish” on this post I undertake to exemplify it myself. Hurrah!