Posts Tagged ‘data migration’

NewImageData migration is dead, long live the enterprise architecture! The days of planning and executing a large scale data migration from one enterprise system to another, while maintaining on operational business, should be gone. There are too many failure modes in a modern enterprise extraction, transformation, and load (eETL) operation. Risk management initiatives are ill-equipped to provide the aversion models necessary to ensure the availability of critical business process in the event of the inevitable migration failure. But even amidst such information technology despair there is a hope, but at what cost?

Before we get into the discussion why enterprise data migration fail, lets first spend a bit of time of the basics. Data migration is often defined as process of transferring data between storage types, formats, and/or computer systems.   In essence, it is the one off selection, preparation and movement of relevant data, of the right quality, to the right place, at the right time. It is usually performed manually and/or programmatically when organizations change (extend, upgrade, adopt) information systems, where the new data storage format is not the same as the old. But not all migrations of data are data migrations.

NewImageData migration is the irregular movement of data (an important concept that will come back to later). Regular movements of data, such as those found in data warehouse refreshes, are fully automated, highly repeatable, and take advantage of pre-mapped data schemas, are not part of this data migration space. With data migration, a single pass movement of data is often accompanied by low or inconsistent initial quality of data between disparate databases and/or applications. It is this large-scaled multi-factor characteristic (semantics, people, process, technical, workflows, data quality, and product types) of enterprise data migration that make is so complex, lending itself to numerous failure modes.

IDC has estimated that over 80% of data migration projects fail or have significant cost overruns, which is itself a type of financial failure. Within this cohort, 50% exceed their projected schedule by 75%,  their budgets are exceeded by 66%,  and 33% of these fail entirely. While these failure statistics are based on a wide range of projects, over have used some form of formal migration methodology. These are not good results if your looking for migrate a enterprise system while maintaining critical business function and financial predictability. Moreover, give the complexity of today’s enterprise systems, you can only reduce the risk of migration failure, but you can not eliminate it completely.


While there are no industry standards for migrating data within the enterprise that guarantee success, there are many migration methodologies designed around risk reduction. The most common (e.g., Practical Data Migration) are based on variants of proof of technologies/concepts, landscape analyses, data quality/cleansing, engaging the business early and often, testing, and automation through ETL tools. While these are good practices, or in some cases even necessary, for any project, they have limited demonstrated ability to ensure that the data is correct/error free after being transformed through complex data quality and business transformation rules.

So, with its high complexity, enterprise data migration should be dead, not because it is bad or evil, but because it the risk of failure is just too high. But, if not the enterprise migration, then what? Well, the answer lays in moving away from data migration as an unplanned irregular movement of data, something today’s enterprise architecture (EA) is all about. But what is EA in the context of migration?

NewImageEnterprise architecture is the continuous practice of organizing logic for business processes and IT infrastructure that reflect the integration and standardization requirements of the company’s business operating model (TOGAF 2009). EA is more than just structure, it is an dynamic means to realize architectural vision, business architecture, information systems architecture, technology architecture, governance, and even migration architecture. EA treats the business as an organic growing entity that evolves through a continuum of changes. Which should not be a surprise to most, given that people (organic by nature) constitute most of a business’s enterprise to begin with.

The key difference between traditional data migrations and migrations developed through enterprise architecture is that EA-driven migrations tend to be less complex and therefore more successful. EA-driven migration succeed because, by their nature, they are designed to succeed. Migrations are not an afterthought, a single one time event of irregular movements of data. Enterprise data migrations are designed in the context, not in leu of, an evolving business and its infrastructure and is the means to the businesses ends, not the ends itself.

Traditional data migration died a long time ago, we just never noticed it because we were too busy cleaning up all the failure debris. It is time for us now to start thinking differently so we don’t repeat this same migration mistakes. Enterprise Architecture-driven migration presents the best hope of success for those looking to continuously evolve their business model.

For more articles like this, please see LiquidHub.


Read Full Post »

Field notes1

The risk of human failure is possible in any endeavor, not with standing the work that I am doing to help a client plan for the migration of an enterprise claims management system. As part of developing an operational readiness plan, which spans two and a half years, we are developing a wide variety of governance characteristics which range from migration requirements to staging infrastructure to migration approach to business/risk assessments. It is a fairly comprehensive model for how, when, and where their business will be migrated.


As part of the process, we are now looking at some of the elements surrounding the human condition; specifically, the impact on productivity on the business during migration. Several studies have clearly demonstrated that there is a significant chance (30-50%) of decline in performance during periods of transition, during which new characteristics are being adopted. The question being address is how to deal with decline in a way that does not impact their clients. One can better prepare the employees which takes time or add additional temporary capacity (people) which takes money.


Our current thinking on how to address the impact on productivity is to create a Personal Operational Readiness program tailored to meet the individual capability maturity needs of each employee.  We are not only looking at different kinds of training (beyond the 3-5 day training programs), but training on operational data as well. When employees see their work in the new ontology, their productivity increases significantly, which is key for making the transition without adding large numbers of temporary employees.

More on this to follow…



Read Full Post »