Jukka Niiranen

Category: performance

  • To update or not to Update Rollup? That is the question

    Some time ago I wrote a parody about Dynamics CRM blogs that mainly seem to post news about the latest Update Rollups being released. Now I find myself writing an Update Rollup blog post myself. Is that a sign that I’ve run out of topics and slipped into the autopilot blogging mode? I hope not, but after reading quite a lot about the latest Update Rollup 10 for Dynamics CRM 2011 I decided to contribute a bit into the discussion.

    It’s always great to receive improvements to the software you spend all your days working with, isn’t it? Compared to a product like Java which Oracle promises to patch only four times a year, even if it’s about critical security updates (did you remember to uninstall your Java 7 yet?), Microsoft is doing quite an OK job with their update process and we see a steady stream of releases to make Dynamics CRM better one bit at a time. Even though Update Rollup 9 was cancelled after the beta program, MS is now back on track with their release schedule in the form of UR10 that came out two weeks ago.

    We have once again a great in-depth article from Dynamics CRM in the Field about all things Update Rollup 10. It tells all the details of the story from Microsoft’s perspective, but let’s evaluate the ups & downs of this latest release from the Dynamics CRM community perspective to get a complete picture, shall we? OK, here goes:

    The Ups

    There was quite a number of fixes promised for the CRM Outlook client in Update Rollup 10 that would potentially solve some of the stability issues many users have encountered when trying to use the CRM Outlook client for actual work in their environment. It’s now 2 years since the first beta of the completely rewritten Dynamics CRM Outlook client came out and many of us are certainly hoping that it would have reached a mature enough level where you don’t have to be afraid of if causing a flood of tickets to your helpdesk. Yes, some users may be quite happy with it as it is, but you’d be foolish not to prepare for potential issues when planning a large scale deployment of the Outlook client. Reaching that good ol’ Service Pack 2 type of confidence level would be just great.

    Another promising deliverable from UR10 is the improvements to query performance on large data sets. A new version of the performance optimization whitepaper has been released alongside the update, which describes the new options made available to tune the Quick Find performance. If you read the results of these tests by Chris Cognetta where the query time was decreased by over 80% in an environment with 1.5 million rows, it’s obvious this is an update worth paying attention to if you have or expect to have some “big data” in your CRM system.

    Finally, as many MSDN subscribers are now eagerly updating their PC’s to run Windows 8, Internet Explorer and even the Office 2013 Preview, the added support for all of these in CRM 2011 Update Rollup 10 is reason enough to grab the very latest CRM bits. Just don’t forget that UR10 alone is not enough, there’s some things you should know about Windows 8 RTM & CRM 2011.

    The Downs

    So, we now have a Quick Find that is working faster, which means it must be doing something differently, right? It is, and unfortunately in some cases this will result in a Generic SQL Error message being shown to the user. It appears that the new update is not quite comfortable dealing with a set of Quick Find columns that contains address fields. As some of you may remember, the address 1 & address 2 fields on the account & contact forms are actually not stored in the same database table as the rest of the default fields, rather they reside in the CustomerAddressBase table. The UR10 version of Quick Find may therefore not work properly if you try to search for records based on address fields like city or zip code. Yes, Microsoft says using these as search fields is against best practices, but the users will very often want to narrow down the customer data based on geographic variables, so taking it away from them can only act as a workaround until we have a new hotfix from MS.

    CRM developers might not be too concerned with performing actual queries on live customer data, but they will certainly be annoyed when encountering the following prompt after editing a JavaScript web resource: You have exceeded the maximum number of 200 characters in this field; it will be truncated. Doh! Making small changes to script files just became very tedious until Update Rollup 11 rolls along in 2 months time. Again, there’s a workaround to it (don’t use the editor, instead upload the file or use some helpful web resource tool from CodePlex), but it doesn’t really make anyone want to rush UR10 onto all their servers.

    Then we have stories of people applying UR10 and finding they can’t access any of their CRM organizations, due to an unhandled exception: Method not found: ‘Void Microsoft.Xrm.Sdk.Query.FilterExpression.set_IsQuickFindFilter(Boolean)’. Uninstalling UR10 appears to be the only known cure for this show stopper, which obviously is related to the aforementioned Quick Find changes on some level.

    Alternatively, if you’ve updated your CRM server from 4.0 to 2011, you may experience errors when trying to import a solution file: The element ‘savedquery’ has incomplete content. List of possible elements expected: ‘LocalizedNames’. Manually editing the XML to remove a view prior to importing the file is the available workaround, but again it can make the life of a system customizer quite difficult until there’s a fix available.

    OMG, what should we do with these UR’s?!

    The short answer is: always test first, then evaluate the ups & downs for your particular case. If you are struggling with the Outlook client, face performance issues with your huge CRM database or simply want to run CRM 2011 on Windows 8, going for Update Rollup 10 may be perfectly sensible for you. For someone who doesn’t have a compelling need to update, you may well decide to wait a bit longer and no one should blame you for it.

    That’s pretty much how it is with any Update Rollup. In reality there’s never going to be the perfect time to update. Knowing ahead what hotfixes will deliver the biggest gains for you or potentially break your CRM deployment would be a great skill to have. Instead of dreaming about it, here’s a few guiding principles I would recommend you to follow:

    • Always test the updates in your own test environment before planning live deployment. A virtual machine with your solutions & data is a good compromise, identical hardware is for people with enterprise level budgets (and problems).
    • Never be the first to update, always wait a couple of weeks from the release and read the CRM forums for experiences from others. Trust the wisdom of the crowds.
    • Coordinate the server and client update schedules to limit possible issues and maximize the benefits from the updates. While mixed environments tend to be supported, why risk it?
    • Do update your CRM. Staying too far behind in this “cloud first” era can soon turn out to be an expensive strategy, so reserve adequate time and resources for keeping your CRM up to date – on a regular schedule.

    Do you have any words of advise for people who are struggling with the “should I update to Update Rollup X” question? What’s your survival strategy?

    Edit 2012-09-09: Based on the discussions on the Dynamics CRM Forum, I’ve created a new wiki page for Microsoft Dynamics CRM 2011 Update Rollup 10 on the CRM Wiki on Technet. The purpose is to provide a single place where the Dynamics CRM community members could share their information on potential issues and known problems that they encounter when applying the latest Update Rollups, as well as naturally links to any solutions or workarounds that can help others who are struggling with the same problem. I’ve initially added links to 10 suspected issues with UR10, but you are very welcome to contribute by adding new links or additional information to the wiki page. In an ideal world, we could have similar pages for each new rollup that gets released and be able to more easily identify both the benefits and the dangers or applying the updates into our CRM environments. Let’s see if the wiki can serve us as a platform for such information sharing.

  • Dynamics CRM 2011 performance improvements: CSS sprites

    Today’s business applications are mostly delivered through a web client that lives inside your browser. While Dynamics CRM has a “fat client” in the form of the Outlook components that can serve the CRM functionality even in an offline environment, Dynamics CRM itself is inherently a web application. The benefits of browser based application delivery are quite plain and obvious, but one must remember that there are still downsides to this approach. Everything relies on the client PC having a fast and reliable connection to the web server, which is not always the case. This particularly affects the mobile workforce or global companies with multiple offices spread around the world. Downloading the UI components involved in rendering a page of the web client will often mean loading tens of different image, script and other files, which requires good bandwidth and low latency to provide a smooth user experience.

    The user interface of Microsoft Dynamics CRM 2011 is much more graphical than in the previous versions, thanks to the new ribbon menu. While this does in my opinion make navigation inside CRM much more user friendly than the old text based menu hierarchies, it does make you wonder if all this comes with a price you have to pay in terms of slower page load times. Well, the good news is that Microsoft has also paid attention to the performance aspect and is now utilizing CSS sprites to render the default icons of the CRM UI.

    The image above is not a strange collage made by me, rather it is a single file called ribbon.png that is loaded when you access the Dynamics CRM through a web client. The image is a whopping 580 kb, but the good news is that you only need to load it once and then it will be cached by your browser. More importantly, each and every icon is rendered from this single image file, by displaying different parts of the image where a specific icon is needed in the UI.

    This same technique was already used in SharePoint 2010 and I’m glad to see it make its way also to Microsoft Dynamics CRM 2011. It will significantly reduce the amount of HTTP requests needed when loading a CRM page. Of course, the technique is only applicable to the default icon site. I’m not aware of any tricks for displaying a set of custom icons through this way, so adding a huge number of new ribbon options for your XRM application may come with a cost on the page loading performance.

  • Dynamics CRM summer news round-up

    The number of blogs, forums and news sites revolving around Microsoft Dynamics CRM can feel somewhat overwhelming, especially when you’ve spent some time being “unplugged” from the constant feed of information that surrounds our everyday lives. During my four week summer vacation I did managed to keep my hands off CRM most of the time, but the news and posts accumulating on my Google Reader still kept me quite firmly in the loop. Here’s a summary of a few Dynamics CRM related topics that caught my attention this summer.

    Refreshed Virtual PC image with Portal Accelerators

    Microsoft released an updated version of the VPC image that comes with Dynamics CRM 4.0 preconfigured, available for download on CustomerSource or PartnerSource. There’s more on it than just CRM, check out the following list for all the goodies:

    • CRM 4.0 with Update Rollup 11
    • Windows Server 2003 R2 SP2
    • SQL Server 2008 SP1 – SSRS, SSAS
    • SQL Server 2005 Express Edition
    • Office SharePoint Server 2007
    • Office Communications Server 2007
    • Office PerformancePoint Server 2007
    • POP3 Server
    • Visual Studio 2008
    • IE8
    • Office 2010 SP1
    • Windows Mobile 6 SDK and Activesync

    The image has been tweaked to include two virtual hard drives (VHD), allowing the swap file to be physically located on a USB drive, which is promised to improve performance by ~30%. How that figure has been determined is beyond me, but the environment works quite well on my Core i3 2.26GHz, 4GB, Win7 x64 setup. It would of course have been super nice to get your hands on a SharePoint 2010 environment, but the hassle of x64 environment virtualization with Hyper-V instead of Virtual PC would have not been worth all the trouble (although I’ve heard VirtualBox should be able to run also 64-bit images without Windows 2008 hosts). Let’s just settle for MOSS 2007 with Office 2010 client components.

    What made this VPC image especially interesting to me was the latest versions of the Portal Accelerators. Now, I never really had hands-on experience with the first wave of Portal Accelerators released for CRM 4.0, but from what I’ve heard the results delivered didn’t quite live up to people’s expectations. This new breed of accelerators, on the other hand, is based on the technology found in the commercial products of Adxstudio, who have built a full blown content management system on top of Dynamics CRM. I simply had to test drive the Customer Portal and Partner Relationship Management Portal, after seeing them promoted in Convergence 2010 Atlanta. (more…)

  • Enterprise Scalability and Performance with Microsoft Dynamics CRM 4.0

    As promised in my earlier post, here are some notes I wrote down during Convergence 2008 EMEA. In my opinion, the best session was “Enterprise Scalability and Performance with Microsoft Dynamics CRM 4.0” held by Amir Jafri. Not so much because of the original agenda and slides, but mostly thanks to all the questions and comments from the audience. Here goes:

    Database

    • SQL Server is where performance tuning gives the most “bang for the buck”
    • Creating custom indexes is supported and encouraged
    • Custom search attributes for entity Quick Find view do not automatically get indexes
    • Database index should be moved to a different disk system
    • Old customization versions are stored in the database
      • Performing many customization changes will slow the system down over time
      • Data is not and cannot be utilized in the UI, so what’s the purpose?
      • There are workarounds for cleaning the old data
    • Performance comparison of out-of-the-box setups between v3.0 and v4.0 on identical hardware show 50% reduction in CPU utilization and disk queue length
    • SQL Server 2008 will provide performance improvements for CRM out of the box

    IIS

    • Disabling authentication from static content has been done in v4.0, heavily recommended for also v3.0
    • 401 error page size has been reduced, should be done manually for v3.0
    • v4.0 uses content compression to reduce the amount of data transferred
      • Compression can create an overhead on the client when rendering content
    • Image Strips(?) are used in v4.0 to reduce round trips
    • CRM default home page (my activities) round trips have been reduced by 50% between v3.0 and v4.0

    Workflow

    • Common client for Windows Workflow Foundation is being developed
      • May ship with CRM v.Next
    • Workflows are still a heavy operation even in v4.0
    • Workflow performance measurement is almost impossible, due to the unlimited number of variations how workflows can be built
    • Workflow queue database table is not automatically purged of old data or indexed, should be done manually
      • Table size can grow to become so big that maintenance cannot be run, hotfix is available

    Performance general notes

    • Tests done using Cisco WAAS application accelerator in WAN environment have proved to improve performance by up to 50%
    • Official toolkits for measuring the performance of an existing CRM implementation do not exist
      • Performance measurement toolkit can be used only with test data, prior to live environment deployment
    • New MOM (a.k.a. Systems Center) pack for CRM v4.0 will include CRM specific performance analysis tools
    • Outlook client improvements in v4.0:
      • Reduced memory footprint, but still quite resource intensive, also due to new requirements from Vista and Office 2007
      • CRM client start-up should no longer slow down the email data transfer upon Outlook start-up, client loads in the background

  • Optimizing IIS Peformance for Microsoft CRM, Part 1

    If you ever need to access the MS CRM server from a remote office with not such a speedy connection, you will notice that CRM is not a particularly light weight application. In a WAN environment where the server is located in a different country or even on a different continent as the users, this can cause some serious issues. Slow performance is a very real threat for CRM user adoption.

    In an attempt to respond to the complaints of branch office users about the system usability issues caused by network latency, we have started investigating what really goes on between the CRM server and the CRM client in some of the basic use cases, such as opening CRM default front page (my activities) and opening an account form.

    Loading the CRM front page without cahced data can include close to 300 different HTTP requests. To me, that sounds like an insane amount for any web page! Sure, there are various little icons on the screen and many controls to load, but still that is way over what I would expect. My personal installation of SugarCRM only takes 65 HTTP requests to give me a nice dashboard of accounts and activities. Where do all these requests originate from?

    Let’s look at the individual requests that take place during a normal CRM page loading:

    CRM page load HTTP requests

    First you see the request for the 16_print.gif, then a while later you see the same request again. What’s up with that? Going through the list, you will notice that in practice all the items are requested twice from the application server. The first one has a result code 401, the other 200.

    Welcome to the world of IIS, or more precisely version 6.0 of it. This is how content protected by integrated Windows Authentication is handled by the server. You can read some discussion on the subject from this thread: “Integrated Windows Authentication – Double Hits“. Brian Reid from C7 Solutions also covered this subject in his blog post “Improving the Performance of IIS 6.0 Applications”, where he also suggests quite an elegant solution of modifying the AuthPersistSingleRequest setting in IIS metabase. However, this being not just any in-house developed site but Microsoft CRM, a business critical application, I’m hesitant to try anything that has not been officially promoted by MS as a performance tweak. Therefore I decided to have a look at what this article had to say about IIS: Optimizing the Performance of Microsoft Dynamics CRM 3.0 White Paper. I’ll share my findings in part 2 of this post.