ABS 2017: Haunted by the ghost of ABS 2009?

My former employer, the Australian Bureau of Statistics (ABS), is currently embarking on an ambitious program of re-engineering its statistical systems and processes under the umbrella of the ABS 2017 program.  But the ABS is a damaged organisation. Morale is low and distrust abounds. The primary cause is the appalling actions of its management in 2009. Can the ABS overcome its self-inflicted wounds and succeed?

The ABS 2017 program is a very wide ranging program that affects most of the organisation and which requires a lot of work by a great many staff. Naming the program ABS 2017 proclaims the intent to substantially complete the work by 2017. It is certainly brave to embed the project deadline in the project's name.

The project's principal aim is to "industrialise" the process of producing statistics using standardized business processes and generalized  metadata driven enterprise software, which in turn should reduce costs, improve timeliness and better enable the organisation to deliver new products and services and to take advantage of new sources of data.

It may seem surprising that the ABS does not already have such infrastructure. There have been attempts in the past to build generalized  metadata driven systems, but they have not been completely successful. Past attempts have only tackled parts of the statistical production process, such as aggregate data management or seasonal adjustment of time series. What the ABS has ended up with is an ad-hoc mixture general systems and custom, survey specific, programs built in different eras which require a lot of effort to set up and run. There is a lot of duplication of metadata for different systems, incompatibilities between systems and missing functionality. The systems are mostly very hard to use and maintain.

The ABS 2017 project aims to build infrastructure based on the international standard metadata models DDI and SDMX. Using common metadata across all stages of the statistical cycle, eliminating functional gaps and incompatibilities with a greater focus on integration and ease of use should allow cut costs and allow greater flexibility in handling new statistical problems. All national statistical agencies face similar problems and there is momentum for some level of international cooperation in the development of statistical infrastructure. But international cooperation is never easy. With the current fashion for austerity by national governments, it remains to be seen whether any government, including Australia's, will fund the necessary (modest) investments.

Some links to background material:

• ABS 2017 Overview (2012) (Powerpoint - 1.7MB)
• The Generic Statistical Business Process Mode (GSBPM)
• Operationalizing Metadata Frameworks – An ABS perspective (2012) (PDF)
• UNECE Conference papers (2013)

In order to achieve the aims of the ABS 2017 project, the ABS needs an engaged, skilled and highly motivated workforce. There needs to be a spirit of cooperation and trust across the organisation. Innovative thinking and solutions need to be embraced and risks need to be taken.

But all is not well in the ABS. The ABS is a damaged organisation. Staff morale is low and the senior management is distrusted by the rest of the staff. The primary reason for this is the abrasive and confrontational style of the Statistician (CEO) Brian Pink and his fellow senior managers. Their true character was made very public in 2009 when they tried to sack about 90 staff in the middle and junior management ranks [1].

In 2008 the ABS senior management embarked on a program of trying to change the roles and duties of top level staff ("re-aligning" was the term used), starting at the top and gradually working down, with the intention of going down to at least EL1 level. The overall thrust was to "step-up" the roles, so that lower levels took on more responsibility. (I never worked out what that really meant in practice. Explanations were always vague and incoherent.) In late 2008/early 2009  the ABS faced budgetary problems. It was claimed that the ABS had too many staff at the AS, EL2 and EL1 levels, so it was decided to use the "re-alignment of roles" program as a way to get rid of staff at these levels.

Things all came to a head on Wednesday 15 April 2009. The details of exactly what happened and when are unclear, especially in the lead up to that day. During the course of that day a number of AS and EL2 staff, with decades of high level experience were told they had been assessed against the new "broader roles" and found wanting. They were given little or no notice of what those "broader roles" were and no chance to defend themselves against their assessments. The "assessments" were highly negative, focusing only on claimed personal failings. They were then suspended from duty (on pay) and given a number of "options" to consider. They had to immediately leave the building. There were unconfirmed rumors at the time that at least some were escorted out by security guards. Details of what the "options" were have never been disclosed, but continuing to work at the ABS at their level was not one of them. Apparently the "options" were different for different people. Some were offered "voluntary" redundancies, some were not. Taking a demotion was an option offered to some. Formal redeployment at level within the Australian Public Service (APS) was not an option. In all, 6 AS and 22 EL2 staff were removed in this way.

 Like all ABS staff, they had formal performance agreements, but they were not judged against them. Clearly, from the ABS's point of view, performance agreements mean nothing and can be disregarded on a whim.

ABS middle manager receives his assessment from senior management, 2009

The immediate reaction in the ABS was total shock and disbelief. Not much work was done that day nor on the days immediately after. There was chaos caused by the sudden departure of so many people in key roles - people suddenly found themselves without any line management, people were waiting for decisions or approvals from the suddenly departed. There was very little official information - people found out more or less by accident that people they were supposed to be working with were suddenly absent. People had no choice but to rely on rumors.  Key gaps were filled on a temporary basis with immediate transfers, creating gaps elsewhere. As it became known who had gone, it was clear that office politics played a major role in deciding who was to go. Also, senior technical roles proved to be especially vulnerable.

The union (CPSU) immediately got involved and commenced action in the industrial courts.

In the next few days, management  got stuck into the second round of the process - targeting the EL1s, such as myself. We were, eventually, given the new "role statements" against which we were to be assessed. The "role statements" were little more than the generic role statement for EL1's in the APSC's (Australian Public Service Commission) ILS. This "role statement" is very generic, full of vague managerialist language and completely lacking in anything about relevant technical skills, such as, for example, IT or statistical skills.  (Google APSC ILS if you are looking for an insomnia cure.) The value of such generic language in role statements was obvious from management's point of view. It gives a very wide scope for interpretation, so that with the right emphasis you can find great fault with the people you don't like and correspondingly find much to recommend people you do like.

Management got as far drawing up a "hit list" of about 60 EL1s (out of about 500), but a court judgement handed down just before the sackings were due to start put an end to that. The court ordered the ABS to immediately stop their actions to sack the 60 EL1s and invalidated the earlier actions with the AS and EL2 staff, and ordered the ABS to allow all the dismissed staff to return to work.

The CPSU website has details:

• ABS job cuts - ABSolutely appalling (21 April 2009)
• Union win puts ABS employees back to work (5 May 2009)

Read the comments in pages linked above. This one (from the 5 May article) in particular illustrates the callous intent of the process:

I heard a first hand account of the manner in which a staff member was treated by one of the senior management gang - I had difficulty in sleeping that night. I shudder to think how the person must have felt. However, concerned about his future working life, this person is hesitant about taking this matter further. To me, it appears to be a systematic and pre-determined bullying tactic, with the intent to cause maximum damage to people's self esteem...

The ABS appealed the judgement, but lost and eventually conceded, grudgingly, that the process they had tried to use was unlawful - something they would have known from the beginning if they had bothered to consult their own internal experts. Unsurprisingly, very few of the AS & EL2 staff who had been unlawfully sacked chose to return.  So, in effect, the ABS got away with it - a fact ABS staff haven't forgotten.

The ABS called for voluntary redundancies for EL1s. They were massively oversubscribed. Lots of people wanted to get out - who could have guessed?

During the later half of 2009, ABS management continued to pursue the program of "re-alignment" of roles for EL1s. The explanations they gave us about what it was supposed to mean were as incoherent as ever - mostly just general whinging - they didn't think any us were any good, the only important people were the senior managers and the only important work was what they did. Whatever responsibilities we were supposed to "step up" to didn't come with any "handed down" authority, nor did it mean any greater access to information or less low level interference from above. The senior managers appeared to be genuinely surprised when they encountered considerable passive resistance. Since roughly half the EL1s assumed that they were on that secret "hit list" they were in no mood to trust or co-operate with the managers who had just tried to sack them.

In this period, morale in the ABS hit rock bottom. The organisation became quite dysfunctional. The senior management was despised and distrusted and working relationships broke down. Many people left. Other employers in the market for statisticians and related staff did very well. The ABS had turned itself from an employer of choice to an employer of last resort. Some people found themselves in untenable situations - extremely overworked due to other staff leaving without being replaced and being ignored by an absent or distracted line management. People reacted in different ways. Some kept their heads down and withdrew - trying to ignore the chaos above and around them, some tried very hard to pretend that nothing had happened. Many showed increasing symptoms of stress. Canberra's GPs, psychologists and neurologists started to notice something was badly wrong in the ABS. Worse, some people reacted by becoming more aggressive and bullying - deliberately or unconsciously copying the style of the top leadership. ABS's output suffered - the loss of critical people resulted in more errors in published statistics. Eventually even senior management noticed that something was wrong.

The 2008/09 ABS Annual Report makes almost no mention of the major trauma in the organisation, saying only:

The Australian Industrial Relations Commission issued orders in May 2009 which put on hold the ABS process to address its excess staffing situation...

In the 2009/10 ABS Annual Report, referring to the fallout from 2009, Brian Pink says just:

The ABS went through a particularly difficult period.

He also bemoans the difficulty of recruiting skilled staff especially mathematics graduates. Perhaps he should stop trying so hard to get rid of the skilled staff he already has.

By early 2010 there was a bit of a change from the senior management. An all staff survey was announced. A change in leadership in TSD (Technology Services, where I worked) saw a more positive, inclusive and less hostile attitude from above. Some measures were put in place to improve two-way communication between the senior levels and staff and general improvements in engagement. There was at least some hope that things could change a bit for the better. The program of "re-aligning" roles was quietly forgotten. The staff survey results showed that morale was indeed very low, and that leadership of the ABS was rated as extremely poor, and that EL1s as a group were much more unhappy than the ranks above or below.

Welcome as these small steps were, for any real rebuilding of trust there had to be some kind of reckoning for the senior managers who had done so much damage to the organisation, especially the Statistician Brian Pink. They should have apologized and resigned. But there has been no apology, no concession that what they did was a mistake, no acknowledgement of the pain and damage inflicted on people and the organisation. To the contrary, the leadership continues to deny that they did anything wrong and maintain an attitude of sullen resentment that they were out-foxed by the wily unions and courts.

This "pretend nothing happened" attitude has consequences. When dealing with people distressed by the events of 2009, the policy of HR is now is to deny the reality of what happened and to refuse and obstruct requests for help.

Senior management had claimed that drastic steps had to be taken in 2009 to reduce the numbers at management levels and that natural attrition and voluntary redundancies could not work. This claim was soon shown to be, at best, dubious. Within less than a year about 8 AS positions were advertised and a new Deputy position was created. So much for reducing management numbers.

It was around 2009/10 that the program of work to transform the statistical production process that eventually became the ABS 2017 program was being geared up, expanding on work that had been going on for some time. The ABS leadership should be congratulated for having the courage to commit to such a far reaching program of change. Whether there is any relationship in the minds of the ABS leadership with the 2017 program and the reign of terror of 2009 I don't know. Did they think that middle management had to be "shaken up" or "renewed" to enable change? (But the top levels of  management did not need such "renewal"?)

In the short term, especially in 2010/11 with major staff morale and turnover problems, ABS management became more engaged with their staff, and were more open to changes initiated from below (in word if not in deed). There were some initiatives that gave staff some scope to influence the wider organisation. However by the end of 2011, at least in the part of the organisation I worked in (TSD), these initiatives had come to an end and some channels of communication between staff and management had fallen into disuse. A high level review of the delivery of technology services was conducted, but without any input from anyone who actually delivered any technology services. There were also another couple of changes in the leadership of TSD which I don't think were an improvement.

Major changes were afoot in the ABS at this time, especially in my part of TSD - Technology Applications (all the applications programmers). The financial year starting July 2012 would see a major increase in work on the ABS 2017 program, and consequentially major changes for Technology Applications, the branch with the programmers who would build and integrate the new statistical infrastructure. The branch had been mostly organised like a cottage industry, building custom, survey specific systems and working with just one client. But it needed to change to be able build generalized, integrated infrastructure serving the needs of multiple clients. Such organisational change ought to have involved sophisticated organisational change management, with a high degree of engagement across the levels of the organisation. Instead, an old-fashioned, top down, "we know best" approach was taken. Change management was entirely about a few managers imposing change on staff. Communication was practically non-existent - almost everyone was kept in the dark. There was, for example, no interest from management in hearing the perspective from anyone already working on current generalized infrastructure, nor from anyone knowledgeable of successes and failures of similar projects in the past. It was bitterly ironic that back 2009 EL1s had been exhorted to "step up" their responsibilities to take a more strategic focus under pain of instant dismissal, yet in 2012 they were locked out of have any role at all in planning the biggest change in way they worked in a generation.

By this time it was clear that there was no future for me in the ABS so I decided to take the offer of a VR and retire early.

Why did management take such a top-down, controlling approach? Partly, they were just reverting to type ("engagement"? been there, done that). But I also think that fear was a factor. Managers were fully aware of what had happened to their colleagues in 2009, so when budgets became tighter and big changes were happening, they do what any frightened manager does - try not to be noticed. Make "safe", timid decisions, don't take risks and exercise firm control of those under you to prevent them bringing you the wrong kind of attention. They may not acknowledge or be consciously aware of the fear, but you can't "unknow" the past and what it revealed of your boss's true character.

Of course, management being "safe", risk averse and excessively controlling is precisely the opposite of what is needed if ABS 2017 is to succeed. The ABS, especially TSD, needs to foster a culture of innovation and cooperation. At the time I left they were heading in the wrong direction. Managers just don't "get" innovation. Innovation is disruptive and threatens old certainties. On a number occasions over the the years I have listened while a senior manager congratulates someone for an innovative achievement while thinking "You hypocrite - we all know that if you had been asked for permission in advance you would have refused it".

An example of the risk averse and controlling attitude of the management of Technology Services Division was their restatement of their commitment to using the SAS language for programming statistical processing components [2]. Although SAS may continue to have some role in ad-hoc analysis, the fact that it had long been the default choice for programming statistical processing systems in the ABS was the primary technical reason why the ABS was still stuck in the "cottage industry" mode of building statistical systems. There was no reason for management to state a policy at all (and against expert advice) - it was really a technical software engineering issue that should have been left to those with software engineering expertise. But a number of managers have a strong attachment to SAS, so for them it is a "safe" choice and anyone going against it is "dangerous". Why they are so attached, I don't understand. My guess is that because SAS is extremely expensive, spending decisions and negotiations with vendors are made at a high level. Managers like vendor reps in suits telling them how clever they are making "strategic partnerships" with industry. It is very likely that the commitment to stay with SAS will be simply ignored - TSD managers have never shown any real interest in statistical computing so it is very unlikely that they will exercise any oversight over any future developments.

So how have the events of 2009 affected the prospects for ABS 2017? Its always hard to guess what might have been. Was such a traumatic shakeup really necessary to enable change? Surely not. The ABS lost many staff with valuable skills and knowledge (many more than just the ones targeted), and many working relationships were badly damaged. Mostly the same senior managers who were responsible for the 2009 events are still running the place. The rest of the staff have little trust or respect for them (according to staff surveys). ABS used to have a high degree of commitment by its staff to its mission but that is now much less, the ABS having shown how little commitment it has its staff. All this must have a serious negative effect of the likely success of ABS 2017. But on the other hand, people are people and want to do worthwhile work regardless of the chaos around them. Projects can succeed despite the best efforts of management.

TSD is especially important for ABS 2017. The project is a major software engineering undertaking, and it is particularly important that software engineers work in an environment that allows them to solve the right problems effectively. At the time I left, things were not looking good in that regard. Software engineering expertise is certainly not concentrated it the upper levels of TSD management. Managers must see their role as role as enabling and facilitating technical staff to work effectively, not micro-managing and controlling them. Often the most useful thing they can do is get out of the way. 

Its now quite a while since I left the ABS. There have been more personnel changes in management, especially in TSD where I used to work. I don't know how the planning for the July 2012 changes in my work area eventuated - all I know is that they were going about it the wrong way. Work on the ABS 2017 program is well underway. There are a lot of clever people working on it. Good luck to them. They'll need it.

fn1: A quick explanation of the Australian Public Service job classifications in the ABS:
The head of the ABS is a statutory position called the Statistician. Next down are Deputy Statisticians (currently 4), then First Assistant Statistician (FAS) (currently 8, who head Divisions), then Assistant Statistician (AS) (currently about 32, who head Branches), then Executive Level 2 (EL2 who mostly head Sections, but a few are specialists), then Executive Level 1 (EL1, who are mostly junior managers and/or team leaders, but quite a few are technical specialists of some kind). In the rest of the Australian Public Service the term Statistician is replaced with Secretary. AS and above belong, in public service terms, to the Senior Executive Service (SES).

fn2. SAS is a 1970s era statistical analysis package entirely unsuited to the task of programming robust, generalized, testable, efficient, modular software. The language lacks just about all the features serious programmers expect. It lacks functions or procedures or any equivalent. It lacks exceptions, there is no extensible type system and certainly nothing like a class. Its only means of organizing program logic is the text substitution macro.