In my previous post I left off with a couple of questions that hinted at this post. What I want to talk about is the architectural decisions we made when developing our solution for migrating data between on-premises instances of Dynamics CRM/365 and the Microsoft Cloud hosted version.
Building a Dynamics 365 Solution
First, let me tell you my 5 most important design characteristic of any solution built for Dynamics 365/CRM.
- First and foremost it must rely on only supported customizations.
- It has to be able to run on all versions of Dynamics 2011 UR12 and up, on-prem and online and using the same code base whenever possible.
- There should be no browser or OS requirement associated with running the solution including mobile capability whenever possible.
- A pleasurable user experience is an absolute must. It should be intuitive and easy on the eyes.
- Robust error handling and reporting is essential.
Sharing the Load
The way Migration Dynamics works is to split the load of the migration process between 3 systems.
- The source Dynamics organization: The source system is responsible for queuing data to be migrated. This includes queuing data that gets created/modified/deleted after the initial queuing of data has already started. This is what we refer to as the ‘Live Migration’ function of Migration Dynamics.
- Azure Web Apps, Storage and Web Jobs: The Azure Web App component is responsible for storing the data to be migrated temporarily until it can be written to the destination organization using Azure Web Jobs. The Azure Web App is also responsible for error and status reporting. Any client, including the source organization can get error reports and status information via web services from the Azure Web App. The Azure web app is essentially a pass through between the source Dynamics system and the destination.
- The destination Dynamics organization. The destination system is responsible for the actual CRUD operations associated with updating any individual record being migrated to the target system. Since the CRUD operations happen in the context of a single SQL transaction using the built in plugin transaction capability the integrity of the destination data is almost entirely guaranteed.
Starting at the End
Let’s take a step back and consider a migration scenario. Say, you’re migrating a Invoices and the Invoice Details from a source Dynamics instance to Dynamics 365 online. The correct order to migrate those records would be to first migrate the invoice and then the invoice details to ensure the details are not orphaned. Now consider that any number of those invoices have been cancelled or completed. In Dynamics a cancelled or completed invoice is considered read-only. That means the invoice can’t be updated even through the APIs without first being re-opened. When we migrate the closed invoice it will be marked as read-only. Subsequently, when a child invoice detail we have to re-open the invoice in order to associate the invoice detail. If this process did not happen inside of a single transaction what would happen if the invoice detail failed to write would ultimately result in an open invoice with no details. Now imagine this happens a couple of hundred times on a bunch of different entities in the system. It would quickly become very difficult to say with any confidence that the data in the destination system was reflective of that in the source system without rerunning the entire migration. For this and a number of other reasons, utilizing transactions in the target system when migrating data is ideal.
Azure Pass Through
But what about the queuing of the data to Azure? Is it necessary to have the intermediate step of moving all the data from the source to a holding area in Azure and then on to the destination? For a couple of reasons it really is necessary. Here’s why.
- Persistent Queues: Due to the nature of migrating data and maintaining the integrity of relationships within the data the order in which records get written to Dynamics is important. I discussed this in my previous post. The ability to queue up records to be written in a specific order means less possibility of orphaned data during a migration as well as the ability to retry when errors occur using the built in Azure queue functionality. Queuing the data in an intermediate storage also makes Live Migrations possible by ensuring the operations that take place on a record in the source organization happen in the same order in the destination.
- Web Jobs: Azure provides a service known as Web Jobs. These jobs trigger based on a number of different things including data being written to a storage location in Azure. Just as queues in Azure are perfect for migrating data, Web Jobs provide a great way of processing data out of those queues. Web Jobs provide robust logging and real time monitoring. They also scale out based on load and are always on waiting for the next message to become available.
Back Where We Started
At this point we’ve worked our way from the destination back to the source system and the process of pushing all of the data out to Azure. So, what does that look like and how does that tie in to our 5 important design characteristics? Well this is the only piece that the user sees really and therefore is the most important when it comes to our design principles. The Migration Wizard that runs inside of Dynamics as an HTML/JS Web Resource consists of 5 steps.
- Choose whether to enable Live Sync for your migration
- Choose / enter your destination connection information
- Map users
- Validate your setup
- Select the entities / records you want to migrate.
The wizard includes step-by-step guidance and executes the majority of its functionality server side and through web services in Azure to connect to and pull information from the target system prior to migration. Since the application runs inside of Dynamics itself you can not only monitor it from anywhere (including from your phone or tablet) it’s accessible to any user of Dynamics that has permissions.
Of course, last but not least, it will run just fine on both Dynamics CRM 2011 UR12, 2013, 2015, 2016 on-premises and Dynamics 365 online in the browser of your choosing.