….For all references and definition, see the page called “References & Glossary for the 5 pitfalls...”
Summary
This white paper provides an overview and a detailed view of pitfalls making painful a data migration related to Dynamics 365 or Dynamics CRM.
...
Those items should be categorized into 5 groups: data context, platform context, functional context, code context, architecture context.
Keywords
Data Migration, D365, Dynamics 365 On-Premise, Dynamics CRM On-Premise, Pitfalls, Data Context, Platform, Code, Customizations, CRM services, Data security Model, CRM Functionalities, CRM Database.
Confidentiality
The white paper does not provide any confidential client data related to data migration. All the code in this white paper is only related to the CRM SDK365 code.
Introduction: What are the main pitfalls during migration?
A data migration focuses on the movement of data between source (legacy data system and business) and destination (target system). Hence, in our white paper, a data migration will focus on the movement of data between Dynamics CRM / 365 on-premises (source) and Dynamics CRM / 365 on-premise or online (destination) [1].
...
OPTION - TYPE A (DB/DB) | DATA AND METADATA STRUCTURE | TECHNOLOGY/PLATEFORM | VERSION/PLATFORM BUILD NUMBER |
TYPE A – 1 | SIMILAR STRUCTURE | SIMILAR PLATEFORM | SIMILAR VERSION |
TYPE A – 2 | DIFFERENT STRUCTURE | SIMILAR PLATEFORM | DIFFERENT VERSION |
TYPE A – 3 | DIFFERENT STRUCTURE | DIFFERENT PLATEFORM | DIFFERENT VERSION |
1. Generic Framework
Based on a generic framework, we asked ourselves questions related to the following concept: pitfalls …what are the pitfalls we could face during a data migration? To answer this question, I decided to regroup them in 5 categories: data context, platform context, functional context, code context and finally, architecture context.
...
2. Pitfalls Overview
During the data migration process and precisely during the milestones of data migration, pitfalls should occur. Each pitfall mentioned above (data context, platform context, functional context, code context and architecture context) should occur in all milestones of your data migration and should delay the process. A data migration is composed of important milestones in transferring business and system data from one Dynamics CRM platform to a new Dynamics 365 platform. Hence, a milestone is not a key factor in Dynamics CRM data migration.
...
3. Pitfalls Details
3.1 Pitfalls from data context
Data context could be categorized into five categories: data security model, data sharing, data model denormalization, data logs and audit and finally, data volume.
Data CRM Security Model
Complexity of data security model in the source:
...
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
The complexity of the data security model could be a pitfall, specially with the Mailbox entity (New feature in D365 security model) ! If we start the data migration from an old version of Dynamics CRM (i.e. CRM 2011), we need consider that the native CRM security model has changed because the model includes the concept of “Mailbox”. Regarding the new versions (from CRM2013 to D365), the entity “User” is linked to the following entities “Mailbox” and “Queue”, creating a new CRM security model. |
Data Sharing
What can be shared? Is there a limit to migrate shared data?
...
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Complexity is increased because POA is linked to native and custom business entities (i.e. “account”). |
Data Model Denormalization
Dynamics CRM allows to create a lot of relationships between entities and so, between tables. To do so, we can create as much as possible, those relationships in several ways:
...
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Directly create relationships from the following CRM features: customization through one-to-many relationships, customization through many-to-one relationships and customization through many-to-many relationships. |
Data Logs and Audit
Different types of logs should be considered as pitfalls because the data could be relevant. But do we need to keep logs related to internal CRM process or custom process we developed?
...
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Regarding business needs or organizational constraints, we need to enable auditing in CRM. “Audit” table could have become bigger and bigger and the challenge is how to maintain audit history creating an unmanageable database size over time and specially, in data migration process. Then, what business rules could help us to clean the data or archive data we will not want to migrate? Audit can be a pitfall because “Audit” table does not have any constraints that would be violated by moving data from the table to an audit archive database. |
Data Volume (Row Number and Denormalization)
Data volume could be a pitfall because:
...
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
PrincipalObjectAccess (POA) : a lot of data could be related to business data. Bigger is the size, more complex should be the strategy of data cleaning. |
3.2 Pitfalls from platform context
Platform Cumulative Updates and Hotfixes (Version 8.2 & 9.0)
Regarding each major version of Dynamics CRM/365 on-premise (i.e. version 8.2 or version 9.0), there are a lot of cumulative updates: each month, until December 2020, there is a new update rollup include all the hotfixes that were released for limited distribution. The rollups have 2 purposes: enhance unexpected behavior and repair functionality.
...
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
More we need to test and validate the impact on the data previously migrated ! |
Black Box Behavior (Front-End and Back-End CRM Services)
Dynamics CRM/365 on-premise is a black box and so, unexpected behavior could derive from two services: CRM frontend services (CRM Client-code) and CRM backend services (CRM Async Service & CRM Async Service Maintenance). All the behaviors and the communication between the frontend and backend services are not easily apprehended, although we can:
...
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Using external tools (decompiler, sql analyzer, …) could make it painfull. |
System and Custom Metadata (Mapping old to new D365 values)
All retrievable, system and custom metadata has a “MetadataId” primary key to make it unique and has values that can be retrieved by primary key or by name.
The values extracted from the database source could be different from the database or platform destination. Then, we should map the old values (source values) to the new values (destination values).
3.3 Pitfalls from functional context
Non-Supported Functionalities (system functionalities)
Regarding Dynamics CRM, there are two groups of functionalities : one group related to inner CRM workings (i.e. auto-numbering management, relationship roles management, duplicate detection rules management, and many more) and another group related to integration CRM workings (i.e. SharePoint integration and document management, Exchange integration and mailbox management, and many more). If custom functionalities are replacing native functionalities, they could become pitfalls in a data migration process because:
migration code must adapt to new data and metadata model (dynamics on-premise, on-line);
or replace custom functionalities with native functionalities and then, adapt migrated data to the new data model.
Overridden Functionalities (unsupported JavaScript)
Web Resources in Dynamics CRM could be the place to override native CRM functionalities. Exactly, it is possible to override native functionalities with JavaScript (client-code), functionalities related to:
...
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Validation process could not pass. So, Custom and overriding code (JavaScript) is becoming a part of the data migration process; And Data migration process could be delayed because we would have to adapt our code or redesign our client-code to make sure it will work in our new platform. |
3.4 Pitfalls from code and customization context
Customization Dependencies
In dynamics CRM, there is a way to determine the level of customization and so, the level of customization dependencies: DependencyBase and DependencyNodeBase tables are the starting point of our analysis and help to establish 2 points:
...
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Bigger is the number of dependency nodes, riskier is the data migration process and bigger could be the impact of CRM Deployment (CRM solutions). |
Code and Customization Cohabitation and Dependencies (C# and JavaScript)
In Dynamics CRM, there are mainly 2 forms of development: no-code development (customization with workflows) and code development (plug-ins and code activity in C#, client-code in JavaScript). However, when real-time workflows are combined with plug-ins, 2 problems can occur:
...
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
A design issue has created useless dependencies and a potential negative impact on the migration process. |
3.5 Pitfalls from Architecture context
Embedded CRM Systems (Dynamics 365 App for Outlook)
Dynamics CRM offers the opportunity to use an embedded application: “Dynamics 365 App for Outlook”. The latter helps the users to track CRM entities (CRM data) from MS Outlook (Office 365, Microsoft 365). Dynamics 365 App for Outlook integration could be a pitfall for the following reasons :
...
Knowing this major issue, we must include the configuration of the “server-side sync” related to Exchange Online in the data migration process.
CRM System Maintenance (Log and AsyncOperation table)
Because “AsyncOperation” table is become too large, it could affect the performance and delay the data migration. The database is become bigger because asynchronous background operations (i.e. asynchronous workflows, bulk operations, system events, updates match codes…) save data in “AsyncOperation” table (i.e. which entity and with entry (id) is processed, at which time the operation finished, who is the owner of the task and many more).
...
However, process needs to be very accurate because all data in “PrincipalObjectAccess” and “WorkflowLogBase” tables should not necessarily be deleted and should be kept because of business needs and business audits.
CRM System Externalization
Queue-Based System
Custom CRM functionalities developed for custom business needs need to be executed in batch mode. But, they can be and should be externalized because it will help to:
...
→ In Dynamics CRM V. 9.0, Webhooks pattern has been introduced for connecting Web APIs and service based on the publish/subscribe model [4]. On the contrary of the previous versions of Dynamics CRM, V.9.0 has encapsulated all the objects in the CRM database we need to create notifications: endpoint URL, authentication options and execution mode. Then, we must register a webhook in the Plugin Architecture of Dynamics CRM (not a step) in synchronous or asynchronous mode.
...
Architecture notification-based could help to notify external applications. However, data between applications may be out of sync. And this could be a pitfall. Regarding notification-based architecture, question is the following: How do we include data out of sync in our data migration process? Regarding the pattern used to notify external applications, we might not include data in our process, but at what cost?
CRM Solutions Deployment (CRM Application Lifecycle Management)
Because we need to validate data which has been migrated, we need to deploy our CRM solutions (zip files) to evaluate the impact of the new data during and after the CRM deployment process. However, different ways of organizing CRM solutions in a CRM organization could be a pitfall because CRM solutions create dependencies into the database:
...
→ If in the previous CRM Organization, Third-Party has been deployed in PROD environment and not in DEV or TEST environment, how could we include this pattern into our data migration?
Conclusion
...