Summary
This white paper provides an overview and a detailed view of pitfalls making painful a data migration related to Dynamics 365 or Dynamics CRM. Regardless the type of data migration (migration to Dynamics online or migration to Dynamics On-Premises), native and custom items could delay the data migration process. Those items should be categorized into 5 groups: data context, platform context, functional context, code context, architecture context.
...
Summary
This white paper provides an overview and a detailed view of pitfalls making painful a data migration related to Dynamics 365 or Dynamics CRM. Regardless the type of data migration (migration to Dynamics online or migration to Dynamics On-Premises), native and custom items could delay the data migration process. Those items should be categorized into 5 groups: data context, platform context, functional context, code context, architecture context.
Keywords
Data Migration, D365, Dynamics 365 On-Premises, Dynamics CRM On-Premises, Orchestration, Guide, Ordering, Prioritization, Grouping, Reporting, Logging, Tables, Attributes, Columns, Entities, Data Model, Customer Data Model, Activity Data Model, Custom Entities, System Entities, Core Data Model, Security Data Model, Internal System Process, Sequencies, Validation.
Confidentiality
The white paper does not provide any confidential client data related to data migration. All the code in this white paper is only related to the CRM SDK365 code.
INTRODUCTION: MIGRATION CONTEXT
Data Migration options
Without emphasizing the technical aspect of an option, in this case, whether it is a "Cloud" technology (cloud technology) or an "on Premise" technology, there are two execution contexts:
...
Even with the most thoroughly tested tools and procedures, we need to ask ourselves how to orchestrate a data migration, mainly from database to database.
1. PROBLEM: HOW TO ORCHESTRATE THE DATA MODEL
Here is the opportunity to orchestrate a data migration code related to Dynamics 365. So, we need to have a guide helping us to develop the right, accurate and structured code regarding the data model.
The questions raised by this problem could be challenging: How to group the entity model? What is the sequence of execution? How to prioritize the entities? How to exclude the unnecessary entities to migrate? Our guide could help to organize and structure the data migration and in so doing, we could establish the right requirements to fulfill.
2. REQUIREMENTS RELATED TO THE ORCHESTRATION
2.1 Requirements related to Database SQL as migration tool
First, we need to create at least 3 databases: a « source » database (Dynamics 365); a « target » database (Dynamics 365) and finally, a « tool » database we could call « CRM Tool »What is the sequence of execution? How to prioritize the entities? How to exclude the unnecessary entities to migrate? Our guide could help to organize and structure the data migration and in so doing, we could establish the right requirements to fulfill.
2. REQUIREMENTS RELATED TO THE ORCHESTRATION
2.1 Requirements related to Database SQL as migration tool
First, we need to create at least 3 databases: a « source » database (Dynamics 365); a « target » database (Dynamics 365) and finally, a « tool » database we could call « CRM Tool ».
Cfm card |
---|
layout | Simple |
---|
color | hsl(214, 100%, 85%) |
---|
id | izxsvbhgoi |
---|
title | CRM Tool Database as Tool |
---|
body | 1-Create tables of reference in “CRM Tool” database: mapping, logging and reporting. 2-Create SQL objects (SQL Synonyms and functions): to map, ta extract, to validate, to count…data. 3-Create SQL scripts as stored procedures. |
---|
|
Cfm card |
---|
layout | Simple |
---|
color | hsl(214, 100%, 85%) |
---|
id | izxsvbhgoi82pzx7n2wwc |
---|
title | CRM Tool Dynamics 365 - Database as Toolsource |
---|
body | 1- Create tables of reference in “CRM Tool” database: mapping, logging and reporting. 2-Create SQL objects (SQL Synonyms and functions): to map, ta extract, to validate, to count…data. 3-Create SQL scripts as stored proceduresImport the database source in the development environment. 2- Administrate the database as requested. 3- Backup the database source. |
---|
|
Cfm card |
---|
layout | Simple |
---|
color | hsl(214, 100%, 85%) |
---|
id | 82pzx7n2wwck6p4p6dcymn |
---|
title | Dynamics 365 - Database as source | body | 1- Import the database source in the development environment. 2target |
---|
body | 1- Create the new database with the Deployment Manager 365 (Deployment SDK). 2- Handle the entity model customizing the native model of CRM. 3- Administrate the database as requested. 3- Backup the database source. |
---|
|
Cfm card |
---|
layout | Simple |
---|
color | hsl(214, 100%, 85%) |
---|
id | k6p4p6dcymn |
---|
title | Dynamics 365 - Database as target |
---|
body | 1- Create the new database with the Deployment Manager 365 (Deployment SDK). 2- Handle the entity model customizing the native model of CRM. 3- Administrate the database as requested. 4- Backup the database target. |
---|
|
2.2 Requirements related to organization of entities.
Here is the opportunity to define our strategy and actions related to grouping, prioritizing and excluding entities in the process.
...
3. DATA PREPARATION: BEFORE DATA ORCHESTRATION
Here is the first steps to fulfill before migrating our data and orchestrating our data model management.
3.1 START THE DATA PREPARATION
Initialization of the logging table in the database “CRM Tool”
Initialization of the backup repository: repository will contain all the backups
Cleaning and initialization of the reporting statement in the database “CRM Tool”
Deploy the new customization to update the native data model of the new CRM organization: it will add the custom entities to the system entities and potentially update the relationships.
3.2 VARIABLES INITIALIZATION
Initialization of the following variables: identifier of the “Organization” entity, identifier of the “Solution” entity, identifier of the “Business Unit” entity, identifier of the “System User” entity.
Complete the identifier “System User” with its properties and the “User Settings” entity.
3.3 PREPARING CRM SECURITY MODEL
Update the following entities in the new database (target): System User, Business Unit, Team.
Complete the update with the following entities: organization, solution if needed.
3.4 END THE DATA PREPARATION
Handle the indexes of the “target” database, executing a SQL command to disable the indexes.
Logging the preparation in the database “CRM tool”.
Create the backup of the preparation: “PREPARATION BEFORE DATA ORCHESTRATION”.
4. CORE MODEL DATA ORCHESTRATION: WAVE 0
...
SEQUENCE
...
ACTIONS
...
SEQUENCE 1
...
4- Backup the database target. |
|
2.2 Requirements related to organization of entities.
Here is the opportunity to define our strategy and actions related to grouping, prioritizing and excluding entities in the process.
...
3. DATA PREPARATION: BEFORE DATA ORCHESTRATION
Here is the first steps to fulfill before migrating our data and orchestrating our data model management.
3.1 START THE DATA PREPARATION
Initialization of the logging table in the database “CRM Tool”
Initialization of the backup repository: repository will contain all the backups
Cleaning and initialization of the reporting statement in the database “CRM Tool”
Deploy the new customization to update the native data model of the new CRM organization: it will add the custom entities to the system entities and potentially update the relationships.
3.2 VARIABLES INITIALIZATION
Initialization of the following variables: identifier of the “Organization” entity, identifier of the “Solution” entity, identifier of the “Business Unit” entity, identifier of the “System User” entity.
Complete the identifier “System User” with its properties and the “User Settings” entity.
3.3 PREPARING CRM SECURITY MODEL
Update the following entities in the new database (target): System User, Business Unit, Team.
Complete the update with the following entities: organization, solution if needed.
3.4 END THE DATA PREPARATION
Handle the indexes of the “target” database, executing a SQL command to disable the indexes.
Logging the preparation in the database “CRM tool”.
Create the backup of the preparation: “PREPARATION BEFORE DATA ORCHESTRATION”.
4. CORE MODEL DATA ORCHESTRATION: WAVE 0
SEQUENCE | ACTIONS |
---|
SEQUENCE 1 | Logging of the group 1: process has started. Preparing initial values: Organization and solution identifiers of the databases “source” and “target”; Organization name of the database “target”. Handling “Currency” entity if needed: Update “Organization” entity with the value of “currency”; Handle missing “Currency”. Handling “Owner” entity if needed: Handle the native service accounts “Integration” and “System” Generating report of the records. Logging of the group 1: process has ended. Creating the backing of the group 1.
|
SEQUENCE 2 | Logging of the group 2: process has started. Preparing initial values: Organization and solution identifiers of the databases “source” and “target”; Organization name of the database “target”the database “target”. Handling “System User” entity. Handling “Business Unit” and “Business Unit Map” entities. Handling “CurrencyClient update” entity, if needed: Update “Organization” entity with the value of “currency”; Handle missing “Currency”. Handling “Owner” entity if needed: Handle the native service accounts “Integration” and “System”Connection” and “Connection Role”. Handling custom entities if necessary, only because of the relationships. Generating report of the records. Logging of the group 12: process has ended. Creating the backing of the group 12.
|
SEQUENCE 2 3 | Logging of the group 23: process has started. Preparing initial values: Organization organization and solution identifiers of the database “target”. Handling “System User” entity.Handling “Business Unitentities related to the security model: “Role”, “Role Privilege”, “Internal Address”, including “User Settings” and “Business Unit Map” entities.Handling “Client update” entity, if needed. Handling “Connection” and “Connection Role”. Handling custom entities if necessary, only because of the relationships. Generating report of the records. Logging of the group 2: process has ended. Creating the backing of the group 2.
| SEQUENCE 3 | Logging of the group 3: process has started. Preparing initial values: organization and solution identifiers of the database “target”. Handling entities related to the security model: “Role”, “Role Privilege”, “Internal Address”, including “User Settings” and “Business Unit”, “System User Business Unit Entity Map”, “System User Principals”, “System User Role”, “System User Profiles”, “Queue”, “Queue Membership”. Handling the other entities: “Mail Merge Template”, “Metric”. “Site” entity, if needed, “UoMSchedule”, “UoM”, “Territory”, “Calendar”, “Calendar Rule” if needed.“System User Business Unit Entity Map”, “System User Principals”, “System User Role”, “System User Profiles”, “Queue”, “Queue Membership”. Handling the other entities: “Mail Merge Template”, “Metric”. “Site” entity, if needed, “UoMSchedule”, “UoM”, “Territory”, “Calendar”, “Calendar Rule” if needed. Handling custom entities if necessary, only because of the relationships. Generating report of the records. Logging of the group 3: process has ended. Creating the backing of the group 3.
|
SEQUENCE 4 | Logging of the group 4: process has started. Preparing initial values: organization and solution identifiers of the database “target”. Handling entities related to “Calendar Rule” if not needed in sequence 3 Handling custom entities if necessary, only because of the relationships. Generating report of the records. Logging of the group 34: process has ended. Creating the backing of the group 34.
|
SEQUENCE 45 | Logging of the group 45: process has started. Preparing initial values: organization and solution identifiers of the database “target”. Handling entities related to “Calendar Rule” if not needed in sequence 3“User” entity: “User Settings”, “Team”, “Team Profiles”, “Team Membership”. Handling custom entities if necessary, only because of the relationships. Generating report of the records. Logging of the group 45: process has ended. Creating the backing of the group 45.
|
SEQUENCE 56 | Logging of the group 56: process has started. Preparing initial values: organization and solution identifiers of the database “target”. Handling entities related to the “User” entity: “User SettingsQuery”, “TeamUser Query Visualization” if needed, “Team Profiles”, “Team Membership”User Entity UI Settings” if needed. Handling custom entities if necessary, only because of the relationships. Generating report of the records. Logging of the group 56: process has ended. Creating the backing of the group 6.
|
5.
...
Creating the backing of the group 5.
...
PRODUCT MODEL DATA ORCHESTRATION: WAVE 1
SEQUENCE | ACTIONS |
---|
SEQUENCE 7 | Logging of the group
|
67: process has started. Preparing initial values: organization and solution identifiers of the database “target”. Handling entities related to the
|
“User” User QueryProduction Association”, “Product Price Level”, “Price Level”, “Equipment”, “Product Substitute”, “Service”, “
|
User Query Visualization” if needed, “User Entity UI Settings” if neededProduct Sales Literature”, if needed. Handing the other “system” entities regarding the relationships created in the customization process. Handling custom entities if necessary, only because of the relationships. Generating report of the records. Logging of the group
|
67: process has ended. Creating the backing of the group
|
6...
6.
...
CUSTOMER MODEL DATA ORCHESTRATION: WAVE
...
2
7 78: process has started. Preparing initial values: organization and solution identifiers of the database “target”. Handling entities related to the
|
“Product” Production AssociationProduct Price Level”, “Price Level”, “Equipment”, “Product Substitute”, “Service”, “Product Sales Literature”, if neededContact Address”. Handing the other “system” entities regarding the relationships created in the customization process. Handling custom entities if necessary, only because of the relationships. Generating report of the records. Logging of the group
|
78: process has ended. Creating the backing of the group
|
7 6. CUSTOMER MODEL DATA ORCHESTRATION: WAVE 2
SEQUENCE | ACTIONS |
---|
SEQUENCE 8SEQUENCE 9 | Logging of the group 89: process has started. Preparing initial values: organization and solution identifiers of the database “target”. Handling entities related to the “Contact” “Account” entity: “ContactAccount”, “Account Address”, “Competitor”, “Contact Competitor Address”. Handing the other “system” entities regarding the relationships created in the customization process. Handling custom entities if necessary, only because of the relationships. Generating report of the records. Logging of the group 89: process has ended. Creating the backing of the group 8. SEQUENCE 9. Creating the backing of the group 9.
|
7. CORE BUSINESS MODEL DATA ORCHESTRATION: WAVE 3
SEQUENCE | ACTIONS |
---|
SEQUENCE 10 | Logging of the group
|
910: process has started. Preparing initial values: organization and solution identifiers of the database “target”. Handling entities related to
|
the “Account” entity: “Account”, “Account Address”, “Competitor”, “Competitor Addressthe “Opportunity” entity: “Opportunity”, “Opportunity Close”, “Opportunity Product”, “Opportunity Sales Process”, “Lead”, “Lead Address”, “Quote”, “Quote Close”, “Quote Detail”, “Incident”, “Incident Resolution”, “Queue”, “Sales Order” “Sales Order Details”. Handing the other “system” entities regarding the relationships created in the customization process. Handling custom entities if necessary, only because of the relationships. Generating report of the records. Logging of the group
|
910: process has ended. Creating the backing of the group
|
9...
8.
...
ACTIVITY MODEL DATA ORCHESTRATION: WAVE
...
4
10 1011: process has started. Preparing initial values: organization and solution identifiers of the database “target”. Handling entities related
|
to the “Opportunity” entity: “Opportunity”, “Opportunity Close”, “Opportunity Product”, “Opportunity Sales Process”, “Lead”, “Lead Address”, “Quote”, “Quote Close”, “Quote Detail”, “Incident”, “Incident Resolution”, “Queue”, “Sales Order” “Sales Order Details”.Handing the other “system” entities regarding the relationships created in the customization process.
Handling custom entities if necessary, only because of the relationshipsto the “Activity Pointer” entity – orchestrate with object type code: type opportunity, type incident, type appointment, type recurring appointment, type email, type fax, type letter, type phone call, type task, type campaign response, type bulk operation. Handing the other “system” entities: “Campaign Activity”, “Campaign Item”, “Campaign Activity Item”, “Email, “Email Hash”, “Email Search”, “Activity Party”. Handling the other system entities if needed: “User Entity Instance Data”, “User Entity UI Settings”. Handling custom activity entities if necessary, created as activity pointer. Generating report of the records.
|
Logging of the group 10: process has ended.
Creating the backing of the group 10.
8. ACTIVITY MODEL DATA ORCHESTRATION: WAVE 4
SEQUENCE | ACTIONS |
---|
SEQUENCE 11 | Logging of the group 11: process has started. Preparing initial values: organization and solution identifiers of the database “target”. Handling entities related to the “Activity Pointer” entity – orchestrate with object type code: type opportunity, type incident, type appointment, type recurring appointment, type email, type fax, type letter, type phone call, type task, type campaign response, type bulk operation. Handing the other “system” entities: “Campaign Activity”, “Campaign Item”, “Campaign Activity Item”, “Email, “Email Hash”, “Email Search”, “Activity Party”. Handling the other system entities if needed: “User Entity Instance Data”, “User Entity UI SettingsLogging of the group 11: process has ended.
|
SEQUENCE 12 | Logging of the group 12: process has started. Preparing initial values: organization and solution identifiers of the database “target”. Handling entities related to the binary data: “Attachment”, “Activity Mime Attachment”, “Annotation”. Handling custom activity entities if necessary, created as activity pointerbecause of the relationships. Generating report of the records. Logging of the group 11: process has ended. SEQUENCE 1212: process has ended. Creating the backing of the group 12.
|
9. THE INTERNAL SYSTEM PROCESS DATA MODEL ORCHESTRATION: WAVE 5
SEQUENCE | ACTIONS |
---|
SEQUENCE 13 | Logging of the group
|
1213: process has started. Preparing initial values
|
: organization and solution identifiers of the database “target”.Handling entities related to the binary data: “Attachment”, “Activity Mime Attachment”, “Annotation”.
Handling custom activity : organization and solution identifiers of the database “target”. Handling entities related to a data repository: “Audit”, “Principal Object Access”, “Principal Object Access Read Snapshot”, “Principal Object Attribute Access”, “Bulk Operation Log”, “Bulk Operation”, “Bulk Delete Operation”, “Bulk Delete Failure”, “Trace Log” “Trace Association”, “User Form”, “User Query Visualization”. Handling other system entities if necessary, because of the relationships. Generating report of the records. Logging of the group
|
1213: process has ended. Creating the backing of the group
|
12...
10. OTHER CUSTOM/SYSTEM BUSINESS DATA MODEL ORCHESTRATION: WAVE
...
6
13 1314: process has started. Preparing initial
|
values: organization and solution identifiers of the database “target”.Handling entities related to a data repository: “Audit”, “Principal Object Access”, “Principal Object Access Read Snapshot”, “Principal Object Attribute Access”, “Bulk Operation Log”, “Bulk Operation”, “Bulk Delete Operation”, “Bulk Delete Failure”, “Trace Log” “Trace Association”, “User Form”, “User Query Visualization”.
Handling other system entities if necessary, because of the relationshipsvalues: organization and solution identifiers of the database “target”. Handling entities related to a system or native entity: “Resource”, “Resource Group”, etc... Handling entities related to a custom entity, created for business needs. Generating report of the records. Logging of the group
|
1314: process has ended. Creating the backing of the group
|
13 10. OTHER CUSTOM/SYSTEM BUSINESS DATA MODEL ORCHESTRATION: WAVE 6
SEQUENCE | ACTIONS |
---|
SEQUENCE 14 | SEQUENCE 15 | Logging of the group 1415: process has started. Preparing initial values: organization and solution identifiers of the database “target”. Handling entities related to a system or native entity: “Resource”, “Resource Group”, etcrepresenting a N-N relationship, entities not already handled: “Account Leads”, ... Handling entities related to a custom entity, created for business needsother custom entities if necessary, because of the relationships. Generating report of the records.records. Logging of the group 15: process has ended. Creating the backing of the group 15.
|
11. AFTER DATA MODEL ORCHESTRATION: WAVE 7
SEQUENCE | ACTIONS |
---|
SEQUENCE 16 | Logging of the
|
group 14final process: process has
|
endedCreating the backing of the group 14.
SEQUENCE 15Handling the indexes: rebuild indexes. Logging of the
|
group 15final process: process has
|
startedPreparing initial values: organization and solution identifiers Creating the backing of the
|
database “target”.Handling entities representing a N-N relationship, entities not already handled: “Account Leads”, ...
Handling other custom entities if necessary, because of the relationships.
Generating report of the records.
Logging of the group 15: process has ended.
Creating the backing of the group 15.
11. AFTER DATA MODEL ORCHESTRATION: WAVE 7
SEQUENCE | ACTIONS |
---|
SEQUENCE 16 | Logging of the final process: process has started. Handling the indexes: rebuild indexes. Logging of the final process: process has ended. Creating the backing of the whole database: whole data migration.whole database: whole data migration.
|
CONCLUSION
Between each wave, we could integrate a validation process in order to validate the migration of the data. This validation process could be composed of 2 successive sub-processes: one process related to the deployment (letter A) and a second process related to functional tests (letter B).
If the deployment or the functional tests do not succeed, the wave has to be fixed before moving forward. The reasons of the failure could be the followings:
Data or records that have been migrated are the source of the error.
Strategy related to the grouping, the ordering or the prioritizing has to be considered.
...