A recent spike in traffic on a CRM 2013 client’s portal site revealed a performance issue that we have long known existed in the MS Dynamics CRM Organization Service. The issue ultimately comes down to overhead introduced by Claims Authentication on the Organization Service in custom applications that use the service to read / write data in CRM. To date we have been able to work around it through multiple techniques including using the 2007 endpoints with Windows Authentication in CRM 2011 and better caching of data in 2013 / 2015. However, this particular spike in traffic brought the issue back to the forefront for us.
The test fixture to prove this is simple. It uses code from the CRM 2015 SDK Samples and executes 2 test scenarios.
1. Run 200 threads to query CRM for 1 column in 1 record 1000 times. (source/bin VS 2013 / .NET 4.5.2)
2. Run 1 thread to query CRM for 1 column in 1 record 1000 times. (source/bin VS 2013 / .NET 4.5.2)
The only variable introduced when running these 2 test scenarios on an instance of CRM 2013 On-Premise was enabling / disabling Claims Authentication in the Deployment Manager with each run of the tests. This means SSL remained enabled, forms authentication was disabled, the application code remained the same and the same dataset was utilized. The only difference was the method of authentication the organization service used. The results are extremely revealing. Note: These results are for an instance of CRM 2013 On-Premise. The same tests were run against CRM 2011 On-Premise with the same results. The only difference was slightly longer times to retrieve in CRM 2011 for all tests.
200 threads each making 1000 org service calls to retrieve 1 column from 1 CRM systemuser record.
1 thread making 1000 org service calls to retrieve 1 column from 1 CRM systemuser record.
Again, the only variable introduced between 1 and 2 above was disabling / enabling claims authentication in Deployment manager and then restarting IIS and the CRM Services before running the test application from the CRM server itself.
What these results reveal is that the overhead introduced by Claims authentication is huge, not only under load with many threads hitting the service at once, where it becomes nearly unusable, but also on individual calls to the tune of being more than 10 times slower.
After uncovering this our thought was that we must have misconfigured something in ADFS or we were missing some configuration trick to eliminate the per call overhead. Numerous searches online came up with nothing. We eventually decided to open a ticket with MS to get their feedback and were quickly contacted by a Support Engineer. After describing the issue and running the tests while they watched via webcast, I shared the test application with their tech and he set out to run it on the sandbox environments on their network. After a lot of back and forth and an acknowledgement that they saw the same results we had, a week later, I was contacted by a supervisor and told they were closing the ticket as the overhead was “expected” behavior. I begrudgingly agreed to let them close the ticket since we’d made no progress and they weren’t able to provide any assistance or insight into why the overhead was so significant, except that Claims Authentication “works different” than Kerberos.
At this point I can’t just let this go as expected behavior. I’d classify such a significant issue as a bug or at least a design flaw that should be addressed in a subsequent release. I’m hoping someone can prove me wrong or help me understand where all of that overhead comes from other than it “works differently”. As such, I’ve included the source code above for anyone that wants to run the tests for themselves and provide any feedback or insight they may have. The only configuration change you may need to make on your On-Premise CRM instance is to set the Web Addresses in the Deployment Manager to the FQDN, as seen in the example below, to ensure you can connect to the organization / discovery service with and without Claims enabled.
Feel free to post your results and / or thoughts in the comments below