A popular refrain in the private sector is every company is a technology company. It refers to the critical role technology plays in a company’s success.  It recognizes that technology investment is a top-priority for business leaders predicated on the best return on investment.  Whether the business is a financial services entity, a healthcare system, or a services company, technology lies at the heart of gaining a competitive edge and improving the customer’s experience.  Similarly, every Federal Agency should consider itself a technology Agency.

Federal Agency missions range from protecting our homeland, operating our financial system, caring for our Veterans, and so much more. As these missions evolve and grow in complexity, the demands on technology supporting these missions intensify, certainly over the past 10-20 years.  And from a digitalization perspective, our private and public sector organizations are largely only at the “end of the beginning” of this transformation.

Yet Federal Business and IT leaders are challenged to stay within their IT tight operating budgets as they also struggle to modernize their legacy systems. A starting point to meet this challenge is for the Agency business leaders to acknowledge their awareness of technology’s vital role in delivering outcomes. The leaders must also recognize that IT must continually keep pace with the relentless advances in technology. IT can no longer be viewed as a back-office operation, but rather as a front office, critical component in the efficient, secure, and reliable delivery of citizen services.

Tech Debt: The Federal Agency Imperative

Over the past decade, many IT portfolios increased some three to four times, as more and more Agency functions became digitized or automated.  But budgets and staffing to support these have remained flat.  All of this has created a “perfect storm” of increased demand and a proportional decrease in investment.  Federal Agencies are not spending on IT in proportion to the vital role of technology in delivering on Agency outcomes and citizen services.

Over the same decade, the concept of “technical debt” (aka “tech debt”) has become a stark reality and headwind for IT leaders. It refers to the suboptimal efficiency of writing code for a particular application or workflow, accompanied by the need to maintain that code… sometimes for many years. The expense of maintaining this code is a debt that lingers and robs precious resources from development and modernization.

The amount of precious capital servicing “software debt” may be fed by the best of intentions. Too often, in an effort to save money, IT departments – in both public and private sectors – choose to develop a solution in-house only to discover the cost of servicing that proprietary solution over time far exceeds any savings of writing it initially in-house. Tech Debt is pervasive and impacts the Agency’s business in terms of availability of systems, system performance, security, and other system interoperability. (For example, at Veterans Affairs alone, current plans for software modernization – to “buy-down” tech debt – are in excess of $2.5 Billion).

Tech Debt: Where Specific Burdens Emerge

As with the lack of any maintenance or modernization, this lack of investment creates business impacts, which directly affect Agency outcomes/services.  These impacts manifest in several specific categories, including:

• Otherwise known as “uptime,” availability is the coin of the realm for most businesses and end-users. It is often measured in terms of “9s” of reliability. Lack of maintenance with code can result in increased system downtime, and quantifiable impacts in terms of business functions not completed during an outage.
• No matter how good code is, when it is put into production, over time every application faces the tough test of the actual business user base. “Defects,” or improperly performing code, will occur. Also, even code operating as designed, will need to be adjusted and updated. Finally, business rules and user preferences will change. All of these create problems for Agency users, in terms of systems that provide bad results (“integrity”) and delay business processes and corresponding outcomes.
• Keeping pace with the growing sophistication of cybercriminals – many of whom are state-sponsored – is an ongoing challenge. Despite increased efforts to incorporate “security by design” (aka DevSecOps), every system will produce vulnerabilities or weaknesses that can be exploited. In the worst scenario, Federal Agencies can be impacted by “breaches” where valuable and/or sensitive information is exposed. This can result in systems being taken offline, significantly impacting business operations.
• In the past, systems were stand-alone often in some massive Enterprise Resource Programs (ERPs). Over the past decade, enterprise architecture and data standardization have driven Agencies to authoritative data sources and more system interdependency. While this has been a boon for Agencies and business value, it also creates complex dependencies among systems. Change management and code updates in one system will impact many others, creating cascading impacts on Federal Agency business users.

Tech Debt: Agency/Business Impacts

Three examples at Veterans Affairs provide insights to any Federal business leader or CIO on how to responsibly address tech debt while delivering value to Agency business partners:

Electronic Health Record. The core business system for Veterans Health Administration (VHA) is the Veterans Health Information Systems and Technology Architecture (VistA) application.  In 2018, the VA/OIT estimated that they expended about 700 FTEs per year worth of labor maintaining and upgrading this “home-brewed” application, across the over 130 versions of VistA code in the VA enterprise. Consolidating the versions or “instances” has started to reduce tech debt and aligned to VHA’s goal of truly presenting every day as an Integrated Health Care System.  Migrating to the Cloud has improved performance and availability.  And responsible configuration management has been scheduled and conducted in refined ways to reduce downtime (availability), while not deferring important maintenance to add functionality and keep code functioning (performance).

Identity & Access Management (IAM).  As more applications move to a Single Sign On (SSO) methodology, the value and speed of employee access created more debt and work for the IT staff. In the on-premise environment, SSO availability was far less than demanded by end-users, and so the system was migrated to a Cloud environment with better performance and reliability (including geographic fail-over), which increased to greater than 99%, for both employees and Veterans (availability). Think of the value and avoided access issues for approximately 425,000 VA employees every day that this IAM method delivered (interoperability), and the corresponding faster delivery of services and benefits to Veterans.

Benefits Management System.   The core system or VA Benefits is a decade-old, comprehensive home-grown application, which requires constant maintenance and defect management.  In discussion with business owners, goals were set to increase the velocity of code and improve user application performance, and to put a prioritization methodology around code, to address defects that were truly of the highest impact to claims employees.   As a result, application performance monitoring was put in place to reduce downtime by 200% (outages every 6 weeks versus 2 weeks) and mean time to restore (MTTR) from 38 hours to less than 4 hours (availability).  Think of the value of faster claims both in terms of employee processing time and Veterans receiving faster decisions and benefits.

Resolving Tech Debt

In subsequent articles, I share ideas on how Federal Agencies can tackle the growing tech debt crisis. Those ideas include (1) metrics and methods to ensure appropriate investment and “buy-down” of technical debt, (2) ensuring development programs address true software development, modernization, and enhancement (DME) costs, (3) per OMB guidance, implement an Investment Review Board (IRB) to prioritize limited and constrained IT budgets, and (4) drive Agencies to fully embrace an “operating expense” (aka “OpEx”) strategy, where software risk is shifted to software/platform as a service (aka “SaaS/PaaS”) to industry vendors.

In the Decade of the 2000s, a significant percentage of all productivity gains were attributed to information technology, and in the past decade, that trend has continued. It is time for Federal Agencies, their business leaders, Chief Financial Officers, and Chief Information Officers and others to quantify the full cost of software and align the costs to Agency outcomes and potential business impacts. Technical debt is alive and well in the Federal space, and needs to be resourced and addressed so that we are not running 2022 Federal systems on 2012 plans, methodologies, and resources.

Jim Gfrerer is CEO of Federal Business, LLC, the parent company of Business of VA media, and formerly the U.S. Dept of Veterans Affairs Assistant Secretary for the Office of Information and Technology (OIT), and Chief Information Officer.