This white paper provides an overview and a detailed view of pitfalls making painful a data migration related to Dynamics 365 or Dynamics CRM. Regardless the type of data migration (migration to Dynamics online or migration to Dynamics On-Premises), native and custom items could delay the data migration process. Those items should be categorized into 5 groups: data context, platform context, functional context, code context, architecture context.
Keywords
Data Migration, D365, Dynamics 365 On-Premises, Dynamics CRM On-Premises, Orchestration, Guide, Ordering, Prioritization, Grouping, Reporting, Logging, Tables, Attributes, Columns, Entities, Data Model, Customer Data Model, Activity Data Model, Custom Entities, System Entities, Core Data Model, Security Data Model, Internal System Process, Sequencies, Validation.
Confidentiality
The white paper does not provide any confidential client data related to data migration. All the code in this white paper is only related to the CRM SDK365 code.
INTRODUCTION: MIGRATION CONTEXT
Data Migration options
Without emphasizing the technical aspect of an option, in this case, whether it is a "Cloud" technology (cloud technology) or an "on Premise" technology, there are two execution contexts:
OPTION
DATA SOURCE
DATA DESTINATION
DESIGN OF SCENARIO
TYPE A
DATABASE (DB)
DATABASE (DB)
DB/DB
TYPE B
DATABASE (DB)
SERVICE
DB/SERVICE
Data migration options applied to Dynamics CRM
Emphasizing the technical aspect of an option, what does it mean regarding the platform Dynamics CRM or Dynamics 365?
OPTION
DATA SOURCE
DATA DESTINATION
DESIGN SCENARIO
DYNAMICS 365
TYPE A
DATABASE
DATABASE
DB/DB
FROM DYNAMICS ON-PREM TO DYNAMICS ON-PREM
TYPE B
DATABASE
SERVICE
DB/SERVICE
FROM DYNAMICS ON-PREM TO DYNAMICS ON-LINE
Pitfalls of a data migration
A data migration focuses on the movement of data between source (legacy data system and business) and destination (target system). However, pitfalls related to the CRM data context are real and can delay the data migration: data related to security model, data related to shared data, data model related to denormalization, data logs and audit and finally, data volume.
Even with the most thoroughly tested tools and procedures, we need to ask ourselves how to orchestrate a data migration, mainly from database to database.
1. PROBLEM: HOW TO ORCHESTRATE THE DATA MODEL
Here is the opportunity to orchestrate a data migration code related to Dynamics 365. So, we need to have a guide helping us to develop the right, accurate and structured code regarding the data model.
The questions raised by this problem could be challenging: How to group the entity model? What is the sequence of execution? How to prioritize the entities? How to exclude the unnecessary entities to migrate? Our guide could help to organize and structure the data migration and in so doing, we could establish the right requirements to fulfill.
2. REQUIREMENTS RELATED TO THE ORCHESTRATION
2.1 Requirements related to Database SQL as migration tool
First, we need to create at least 3 databases: a « source » database (Dynamics 365); a « target » database (Dynamics 365) and finally, a « tool » database we could call « CRM Tool ».
2.2 Requirements related to organization of entities.
Here is the opportunity to define our strategy and actions related to grouping, prioritizing and excluding entities in the process.
3. DATA PREPARATION: BEFORE DATA ORCHESTRATION
Here is the first steps to fulfill before migrating our data and orchestrating our data model management.
3.1 START THE DATA PREPARATION
Initialization of the logging table in the database “CRM Tool”
Initialization of the backup repository: repository will contain all the backups
Cleaning and initialization of the reporting statement in the database “CRM Tool”
Deploy the new customization to update the native data model of the new CRM organization: it will add the custom entities to the system entities and potentially update the relationships.
3.2 VARIABLES INITIALIZATION
Initialization of the following variables: identifier of the “Organization” entity, identifier of the “Solution” entity, identifier of the “Business Unit” entity, identifier of the “System User” entity.
Complete the identifier “System User” with its properties and the “User Settings” entity.
3.3 PREPARING CRM SECURITY MODEL
Update the following entities in the new database (target): System User, Business Unit, Team.
Complete the update with the following entities: organization, solution if needed.
3.4 END THE DATA PREPARATION
Handle the indexes of the “target” database, executing a SQL command to disable the indexes.
Logging the preparation in the database “CRM tool”.
Create the backup of the preparation: “PREPARATION BEFORE DATA ORCHESTRATION”.
4. CORE MODEL DATA ORCHESTRATION: WAVE 0
SEQUENCE
ACTIONS
SEQUENCE
ACTIONS
SEQUENCE 1
Logging of the group 1: process has started.
Preparing initial values: Organization and solution identifiers of the databases “source” and “target”; Organization name of the database “target”.
Handling “Currency” entity if needed: Update “Organization” entity with the value of “currency”; Handle missing “Currency”.
Handling “Owner” entity if needed: Handle the native service accounts “Integration” and “System”
Generating report of the records.
Logging of the group 1: process has ended.
Creating the backing of the group 1.
SEQUENCE 2
Logging of the group 2: process has started.
Preparing initial values: Organization and solution identifiers of the database “target”.
Handling “System User” entity.
Handling “Business Unit” and “Business Unit Map” entities.
Handling “Client update” entity, if needed.
Handling “Connection” and “Connection Role”.
Handling custom entities if necessary, only because of the relationships.
Generating report of the records.
Logging of the group 2: process has ended.
Creating the backing of the group 2.
SEQUENCE 3
Logging of the group 3: process has started.
Preparing initial values: organization and solution identifiers of the database “target”.
Handling entities related to the security model: “Role”, “Role Privilege”, “Internal Address”, including “User Settings” and “Business Unit”, “System User Business Unit Entity Map”, “System User Principals”, “System User Role”, “System User Profiles”, “Queue”, “Queue Membership”.
Handling the other entities: “Mail Merge Template”, “Metric”. “Site” entity, if needed, “UoMSchedule”, “UoM”, “Territory”, “Calendar”, “Calendar Rule” if needed.
Handling custom entities if necessary, only because of the relationships.
Generating report of the records.
Logging of the group 3: process has ended.
Creating the backing of the group 3.
SEQUENCE 4
Logging of the group 4: process has started.
Preparing initial values: organization and solution identifiers of the database “target”.
Handling entities related to “Calendar Rule” if not needed in sequence 3
Handling custom entities if necessary, only because of the relationships.
Generating report of the records.
Logging of the group 4: process has ended.
Creating the backing of the group 4.
SEQUENCE 5
Logging of the group 5: process has started.
Preparing initial values: organization and solution identifiers of the database “target”.
Handling entities related to “User” entity: “User Settings”, “Team”, “Team Profiles”, “Team Membership”.
Handling custom entities if necessary, only because of the relationships.
Generating report of the records.
Logging of the group 5: process has ended.
Creating the backing of the group 5.
SEQUENCE 6
Logging of the group 6: process has started.
Preparing initial values: organization and solution identifiers of the database “target”.
Handling entities related to the “User” entity: “User Query”, “User Query Visualization” if needed, “User Entity UI Settings” if needed.
Handling custom entities if necessary, only because of the relationships.
Generating report of the records.
Logging of the group 6: process has ended.
Creating the backing of the group 6.
5. PRODUCT MODEL DATA ORCHESTRATION: WAVE 1
SEQUENCE
ACTIONS
SEQUENCE
ACTIONS
SEQUENCE 7
Logging of the group 7: process has started.
Preparing initial values: organization and solution identifiers of the database “target”.
Handling entities related to the “Product” entity: “Production Association”, “Product Price Level”, “Price Level”, “Equipment”, “Product Substitute”, “Service”, “Product Sales Literature”, if needed.
Handing the other “system” entities regarding the relationships created in the customization process.
Handling custom entities if necessary, only because of the relationships.
Generating report of the records.
Logging of the group 7: process has ended.
Creating the backing of the group 7.
6. CUSTOMER MODEL DATA ORCHESTRATION: WAVE 2
SEQUENCE
ACTIONS
SEQUENCE
ACTIONS
SEQUENCE 8
Logging of the group 8: process has started.
Preparing initial values: organization and solution identifiers of the database “target”.
Handling entities related to the “Contact” entity: “Contact”, “Contact Address”.
Handing the other “system” entities regarding the relationships created in the customization process.
Handling custom entities if necessary, only because of the relationships.
Generating report of the records.
Logging of the group 8: process has ended.
Creating the backing of the group 8.
SEQUENCE 9
Logging of the group 9: process has started.
Preparing initial values: organization and solution identifiers of the database “target”.
Handling entities related to the “Account” entity: “Account”, “Account Address”, “Competitor”, “Competitor Address”.
Handing the other “system” entities regarding the relationships created in the customization process.
Handling custom entities if necessary, only because of the relationships.
Generating report of the records.
Logging of the group 9: process has ended.
Creating the backing of the group 9.
7. CORE BUSINESS MODEL DATA ORCHESTRATION: WAVE 3
SEQUENCE
ACTIONS
SEQUENCE
ACTIONS
SEQUENCE 10
Logging of the group 10: process has started.
Preparing initial values: organization and solution identifiers of the database “target”.
Handling entities related to the “Opportunity” entity: “Opportunity”, “Opportunity Close”, “Opportunity Product”, “Opportunity Sales Process”, “Lead”, “Lead Address”, “Quote”, “Quote Close”, “Quote Detail”, “Incident”, “Incident Resolution”, “Queue”, “Sales Order” “Sales Order Details”.
Handing the other “system” entities regarding the relationships created in the customization process.
Handling custom entities if necessary, only because of the relationships.
Generating report of the records.
Logging of the group 10: process has ended.
Creating the backing of the group 10.
8. ACTIVITY MODEL DATA ORCHESTRATION: WAVE 4
SEQUENCE
ACTIONS
SEQUENCE
ACTIONS
SEQUENCE 11
Logging of the group 11: process has started.
Preparing initial values: organization and solution identifiers of the database “target”.
Handling entities related to the “Activity Pointer” entity – orchestrate with object type code: type opportunity, type incident, type appointment, type recurring appointment, type email, type fax, type letter, type phone call, type task, type campaign response, type bulk operation.
Handing the other “system” entities: “Campaign Activity”, “Campaign Item”, “Campaign Activity Item”, “Email, “Email Hash”, “Email Search”, “Activity Party”.
Handling the other system entities if needed: “User Entity Instance Data”, “User Entity UI Settings”.
Handling custom activity entities if necessary, created as activity pointer.
Generating report of the records.
Logging of the group 11: process has ended.
SEQUENCE 12
Logging of the group 12: process has started.
Preparing initial values: organization and solution identifiers of the database “target”.
Handling entities related to the binary data: “Attachment”, “Activity Mime Attachment”, “Annotation”.
Handling custom activity entities if necessary, because of the relationships.
Generating report of the records.
Logging of the group 12: process has ended.
Creating the backing of the group 12.
9. THE INTERNAL SYSTEM PROCESS DATA MODEL ORCHESTRATION: WAVE 5
SEQUENCE
ACTIONS
SEQUENCE
ACTIONS
SEQUENCE 13
Logging of the group 13: process has started.
Preparing initial values: organization and solution identifiers of the database “target”.
Handling entities related to a data repository: “Audit”, “Principal Object Access”, “Principal Object Access Read Snapshot”, “Principal Object Attribute Access”, “Bulk Operation Log”, “Bulk Operation”, “Bulk Delete Operation”, “Bulk Delete Failure”, “Trace Log” “Trace Association”, “User Form”, “User Query Visualization”.
Handling other system entities if necessary, because of the relationships.
Generating report of the records.
Logging of the group 13: process has ended.
Creating the backing of the group 13.
10. OTHER CUSTOM/SYSTEM BUSINESS DATA MODEL ORCHESTRATION: WAVE 6
SEQUENCE
ACTIONS
SEQUENCE
ACTIONS
SEQUENCE 14
Logging of the group 14: process has started.
Preparing initial values: organization and solution identifiers of the database “target”.
Handling entities related to a system or native entity: “Resource”, “Resource Group”, etc...
Handling entities related to a custom entity, created for business needs.
Generating report of the records.
Logging of the group 14: process has ended.
Creating the backing of the group 14.
SEQUENCE 15
Logging of the group 15: process has started.
Preparing initial values: organization and solution identifiers of the database “target”.
Handling entities representing a N-N relationship, entities not already handled: “Account Leads”, ...
Handling other custom entities if necessary, because of the relationships.
Generating report of the records.
Logging of the group 15: process has ended.
Creating the backing of the group 15.
11. AFTER DATA MODEL ORCHESTRATION: WAVE 7
SEQUENCE
ACTIONS
SEQUENCE
ACTIONS
SEQUENCE 16
Logging of the final process: process has started.
Handling the indexes: rebuild indexes.
Logging of the final process: process has ended.
Creating the backing of the whole database: whole data migration.
CONCLUSION
Between each wave, we could integrate a validation process in order to validate the migration of the data. This validation process could be composed of 2 successive sub-processes: one process related to the deployment (letter A) and a second process related to functional tests (letter B).
If the deployment or the functional tests do not succeed, the wave has to be fixed before moving forward. The reasons of the failure could be the followings:
Data or records that have been migrated are the source of the error.
Strategy related to the grouping, the ordering or the prioritizing has to be considered.
