Data Migration

icon of skill

Essence

Effective management of data migration projects demands a nuanced skill set encompassing technical expertise, project management proficiency, and strong communication skills. Skilled practitioners exhibit a deep understanding of data structures, employing analytical capabilities to assess data quality and identify potential issues. They orchestrate the migration process with meticulous attention to detail, ensuring accuracy, integrity, and security of data throughout the transition. Strong project management skills come into play, enabling managers to define clear project scopes, set timelines, and allocate resources judiciously. Communication is paramount, involving effective interaction with stakeholders, transparent reporting of progress, and adept handling of change management. These managers also excel in risk management, proactively identifying and addressing potential challenges, and possess adaptability to navigate unforeseen circumstances during the migration. Additionally, their commitment to continuous learning and staying abreast of technological advancements contributes to ongoing project success and optimisation.

Collaboration and vendor management play pivotal roles in the data migration landscape. Skilled managers foster collaboration among diverse teams, including IT professionals, business users, and data owners, ensuring a cohesive approach to the migration. Furthermore, they adeptly manage external vendors or tools, coordinating third-party services effectively. Documentation is a key aspect, with managers creating comprehensive records of migration processes, mapping rules, and transformation logic for future reference and auditing purposes. Successful data migration managers embrace a holistic approach that balances technical proficiency with soft skills, resulting in seamless migrations that meet project objectives while adhering to data integrity and security standards.

Experience

From my experience, business requirements are key to a successful data migration and there's usually a high demand for business decision making as you translate from one system to another. It's handy to have technical Business Analysts perform the field level mapping as the business can find the detailed work overwhelming. At IBA I brought in a Data Migration specialist who had the toolset proficiency to extract, transform and load the data. The complexity of the transformations however led to a daily workshop with the product owner to work through the issues. Often there are complixities in the mapping exercise, with mandated fields in one system but not the other, or a many-to-many data mapping that needs to be normalised.

At Rabobank I struggled at first to find an agile approach to data migration. Historically it had always been a very waterfall approach; requirements, mapping, build, test, resolve, dry run, deploy. However, I researched some agile approaches and they proved far more effective. Shifting mapping issues left by running migration cycles early, with stubbed out mappings in some areas, created better flow of issues for the product owner.

Another aspect of Data Migration is data governance. The movement of data from one platform to another is a business concern and should not be left to IT to perform itself. Clarity is needed on who in the business owns the data and who is a responsible steward of it. In a large mature organisation these roles are often defined, but at Greencross and IBA this needed more definition.

There should always be a user acceptance phase on a data migration, unless it's through a vendor approved and tested process (e.g. platform upgrades). Although it is important to always include some checksum process and sanity tests. For critcal applications, such as banking, there can be regulated independent audits of the migration. At Rabobank I engaged Ernst & Young in this process. Again communication is key, explaining through visuals the approach, in this case via staging table etc, and then determining the right way to audit the process end-to-end.

Credentials