Jukka Niiranen

Tag: Relevance Search

  • Dataverse results inside Microsoft Search in Office, SharePoint, Bing

    Dataverse results inside Microsoft Search in Office, SharePoint, Bing

    Whether you are configuring Dynamics 365 Customer Engagement apps from Microsoft or building custom applications with Power Apps, often the solution design and implementation focuses too much on the data entry experience and not enough on data discovery. Yet it’s the process of searching for information rather than entering new records that consumes far more time in the lives of information workers.

    Throughout the history of my blog (originally “Surviving CRM”), topics such as the Advanced Find feature or configuration of views in XRM/CDS/Dataverse have been the most popular ones. To put it another way: a lot of people search for information on how to make search better in Microsoft business apps.

    While they are nothing like Google, Microsoft still has a wealth of R&D budget spent on search related services. Just because Bing isn’t what you’re likely using in your private life, that doesn’t mean you couldn’t benefit from the Microsoft Search infrastructure. Today we’ll explore how data from Dataverse environments can now be exposed in Microsoft Search.

    Enabling Dynamics 365 results in Microsoft Search

    The feature for showing Dataverse results in Microsoft Search was originally promised at Ignite 2021 in November, but on the release plans it has been postponed a few times. Now the ability for enabling search federation has appeared and the roadmap item for it says “GA in June 2022”. The feature appeared in a couple of tenants, so it’s time for a test drive.

    First you should ensure that these prerequisites for configuration are met. The modern Dataverse search needs to be enabled for the environment you want to target. Second, the search admin user account that you’re using to run the configuration must have both admin access and a valid Dynamics 365 license for the environment.

    Next you can proceed to Microsoft 365 Admin center. Go to “All admin centers”, choose “Search & Intelligence” and select the “Data sources” tab. You should see the section “Microsoft apps and services” that allows you to add a new app for search federation.

    What if that section does not appear? It may be that you don’t have a paid Dynamics 365 subscription in the tenant. The detailed requirements haven’t been documented by Microsoft yet, but based on my experiments, having just Dataverse with Power Apps premium licensing is not enough. Neither is a trial subscription for Dynamics 365.

    In the configuration section for Microsoft apps and services, it says that “connections to these data sources do not count toward your index quota limits.” If you’re not familiar with Microsoft Search, then the data sources accessible via Microsoft Graph connectors are indeed a paid service. With Dynamics 365 we’re seeing the search capability bundled into the product, but for things like Azure Data Lake, Azure DevOps or third party services like Salesforce or ServiceNow, you’ll need a Microsoft Search paid license.

    If we were to use the paid features, how much would it cost? The required product can be found under “purchase services” in the M365 admin center – although not with any generic term like “search”, of course. You’ll need to know that SKU name is “Extra Graph Connector Capacity”. The minimum purchase is 1 million items and that would be around $1.000 per month, or in euro prices these sums below:

    If you had Microsoft 365 E5 or Office 365 E5 licenses, those would also accrue some index quota per each licensed seat, as shown here.

    Luckily we don’t need to make purchases since our tenant has Dynamics 365 Customer Engagement licenses in place. We can proceed to adding a new app under the Search & Intelligence data sources, by selecting the only supported MS app at this time: Dynamics 365.

    Next we get to choose the Dataverse environment. Except not all your environments may be listed here. Why? I did not discover a valid explanation for it in my initial tests. Neither the environment type (sandbox/production) nor existence of Dynamics 365 apps was the reason why environments didn’t always show up. Oh well, let’s proceed with what we’ve got and pick “Jukka’s Business Cloud” that contains the core CRM data in this tenant.

    If you’re wondering whether you could configure data from multiple Dataverse environments from your tenant to show up in the Microsoft Search results, the answer currently is: no.

    Once you’ve added the data source, it can take a while before the results will appear in the search experience. In our production tenant this took less than 30 minutes, in my personal tenant it’s been ~2h and nothing yet. MS says it can take 24 hours before the indexing is complete. Patience is a virtue – if you’ve got some to spare. I’ll just hop over to the production tenant now and explore the end user experience.

    How Dataverse records show up in Microsoft Search

    The users will discover the results from the previously configured Dynamics 365 connection when they access search either in Office.com, SharePoint Online or Bing. The UI to get to those results varies slightly, but the listed results from the Microsoft Graph connector seem to be identical.

    Obviously in Bing you need to be signed in and access the Work tab that switches you from the public web search to the internal results from your work tenant. In addition to all the regular results, plus custom search bookmarks like the one you see below, there will be a new tab to represent the “search vertical”. In the case of our production tenant, I named it “Business Forward” as it is the primary app used in our production Dataverse environment.

    I personally am more likely to leverage these search results from under Office.com. Here’s an example of what data is pulled from the Dataverse environment for a basic query term “forward”:

    We immediately see that the results cover both standard tables from CDM like accounts and contacts, but also custom tables like our work orders that are managed in Dataverse. We see a preview of fields from the matching records, which varies based on what columns are non-empty for each row in the corresponding Quick Find View.

    Configuration of the searchable fields is done in the Dataverse search settings, which the Microsoft Graph connector for Dynamics 365 then respects for each table. This means there’s very little to be configured on the Microsoft Search experience specifically. You really only decide A) which 1 environment to point it at, and B) what name do you want to show in the search vertical in Office.com, SharePoint, Bing.

    Dataverse in-app search vs. Microsoft Search experience

    Now that we have enabled the discovery of Dataverse data from within Microsoft Search, should that become the recommended entry point for all your business data searching needs? The answer to this is: through Microsoft Search you’ll only get a subset of search features that are available in a full model-driven Power App. Whereas with Microsoft Search you may well get close enough results, the detailed search capabilities are better within Power Apps native Dataverse search.

    If I enter a search term like “governance” into Office.com search bar, I’ll get a long list of results that I can not filter nor sort in any way. When I do the same search inside our Business Forward app (a model-driven Power App on Dataverse), I get the results tabbed per specific tables that allow me to narrow the query by record type. Also the filter pane gives access to owner and date filters, which get even more detailed as I select one specific table from the results.

    Another feature that can be a welcome addition is that the Dataverse search within the app UI covers only the tables included in that app. This can lead to search challenges in a larger Dynamics 365 environment that covers multiple different processes.

    For example, our production environment covers not only our CRM info on customers, sales etc. but it also hosts our internal IT Asset Management solution and its data. Now, if I search for “Nokia”, the results could include both the account management data related to Nokia Corporation as well as Nokia smartphones (from HMD Global) that our team members have registered as their IT assets. Sure, all results are filtered based on security roles and won’t reveal any data to unauthorized users, but anyone with broader access to Dataverse will also get broader, nonfilterable results with Microsoft Search.

    Sometimes you may get more results from Microsoft Search than from within Power Apps, because of the way documents stored in notes and attachments get indexed. I even discovered that the note record (annotation) which traditionally hasn’t had any user accessible table form in the native UI can actually be opened independent of the parent record. The form of a note is very nice and readable, providing a link to the related document:

    Things aren’t quite as awesome when it comes to attachments of email records. While these also open a form within a model-driven app, the attachment table form doesn’t offer any link to either the document or the related email message from where the file could be opened.

    The many faces of Dataverse search experiences

    It’s really awesome that we’re now seeing the mainstream search capabilities in Microsoft cloud services reaching into the domain of business applications with these out-of-the-box capabilities. The investments that Microsoft has made into Dataverse Search as a service that isn’t just for doing freetext search inside your CRM system is starting to pay off. Just like makers can tap into Dataverse Search from within Power Automate actions, now information workers can do simple queries into enterprise systems from within Office experiences.

    In its current state, with support only for Dynamics 365 customers and only a single environment per tenant, the Microsoft Search experience isn’t yet as powerful as it could become. Covering the Power Platform use cases where business data is managed in various environments and via specific custom apps would be a logical direction to broaden Microsoft Search with a true low-code platform story.

    The way I see it, the search experiences for business data managed within Dataverse are being developed in three separate areas:

    1. Free text search: building search indexes that cover several sources/tables, with multiple entry points (in-app, Office, flow, APIs), evolving into support for natural language queries and supporting conversational UIs (Teams chatbots etc.). This is the area where Dataverse Search and Microsoft Search are operating.
    2. Structured queries: building complex query criteria to filter the rows in a specific Dataverse table. “Show accounts with orders in last X months and no activities from members of sales team Y”. The Advanced Find feature and the FetchXML query language have traditionally covered this front. The modern advanced find experience is making these filters easier for casual app users to approach, while FetchXML is still alive (now available for download in the UI) and can help flow makers design complex queries more efficiently, for example.
    3. References: the next generation of “set regarding” features are aiming to broaden the traditional scenario of associating emails in Outlook with records from CRM. If the MS vision for Context IQ comes to life, we should be able to at-mention basically any record from a Dataverse environment and collaborate interactively on it via Loop components. Similar to the Microsoft Search initial limitations, it will be interesting to see how the lookup field experience can be optimized with machine learning algorithms when all records in a large tenant are behind a single @ symbol…

    Returning back to what I mentioned at the start of this blog post, as a solution designer it’s very important to understand all these different means through which users can search for your business application’s data. There’s a lot you can do by configuring the settings in Dataverse views to make the experience enjoyable for the user.

    Studying the Dataverse search documentation is a good start. However, it’s all just theory until you have some actual data and real user interfaces to test the usability of your search configuration with. What this means is that you’re unlikely to ever get all the settings right in the V1 release of your app. In practice the user experience for business data search requires attention throughout the lifecycle of your app. Microsoft will continue to change and expand the ways in which search queries and results are handled, so you better keep up with these new features and explore what configuration options are being introduced in the future, too.

  • Make your Power Apps search experience more Relevant

    Make your Power Apps search experience more Relevant

    Microsoft recently launched a brand new search experience for Model-driven Power Apps and therefore also first-party Dynamics 365 apps operating on top of CDS. This is a significant step forward in the “non-structured” search capabilities in Power Apps. The structured queries one can construct via Advanced Find were always one of the most powerful features that the XRM platform could offer on top of relational data (and remain one of my most read posts on this blog). Now also the free text based search experience is catching up nicely:

    Being a low-code application platform, there are several configuration options related to how the search features in Power Apps behave. This can also result in people missing out on how they could ensure that the search really works the way that the users would expect. In this blog post I’ll go through some common challenges that the Dynamics 365 professionals may have already learned to overcome but which can be surprising to new Power Apps makers.

    Challenge 1: Relevance Search is not enabled

    If you’re not seeing anything even remotely like the MS blog posts when performing a search in a Model-driven app, then you may well be using the limited Categorized Search instead of the fancier Relevance Search. When attempting to change the search type, you might discover that the dropdown actually doesn’t give you any choices, i.e. it’s a dead end with Categorized Search as the only option:

    Why does this Categorized Search option even exist, and why do we need to specifically enable Relevance Search? It all comes down to the on-premises legacy of the platform, where the single CRM server had to be capable of offering a standalone search option. In the cloud era we can leverage Azure Search as the external search index database to where CDS data is syncrhonized to, thus offering a far richer search experience.

    Now, Azure Search being a separate service and possibly having different types of SLAs and other legal variables than the core Power Apps or Dynamics 365 services follow, it’s not on by default even in the MS cloud. An administrator will need to navigate to the Power Platform Admin Center, choose the environment, then go under Settings / Features / Search to flip on the swith that turns on Relevance Search.

    Once it’s enabled, only then will you see the option to also turn on the new search experience (which probably will become the default option in the future). It won’t show up there immediately, so give it a lil’ time.

    Challenge 2: Relevance Seach doesn’t search my entities

    Cool, we’ve now got the new UI up & running. When we do a search from the new prominent search bar always visible at the top, the results are returned for some of the entites. But why aren’t we getting any hits for Widgets, for example? In this scenario we know there are records in there that contain matches to our search string.

    Even though Azure Search provides a richer search index for your Model-driven Power Apps, it doesn’t directly copy each and every table from CDS into it’s database automatically. When working with a Dynamics 365 environment there are some default entites covered (like accounts, contacts, opportunities). For any custom entities and especially with custom apps, you’re gonna have to explicitly tell which entities should be enabled for Relevance Search.

    To access this configuration option you’ll need to leave behind the modern Power Platform Admin Center and venture into the land of the legacy web client and Dynamics CRM era customization UI. You can either add “/main.aspx?settingsonly=true” to your environment URL or find another path into the land of more advanced options. Once there, find Settings / Customizations / Customize The System and open up the Entities menu in the Default Solution window. Now you’ll see what entities are included in your Relevance Search Index and which are not, i.e. they are in the Available Entities list:

    Select the entities you want, click OK, then Publish All Customizations and now they are enabled for the modern Relevance Search experience.

    Challenge 3: Relevance Search doesn’t search the right fields

    Hmm, why aren’t we still getting the full search results for our Widgets entity, even though we added it into Relevance Search? Well, technically you didn’t add the whole entity into the search index. By default a custom entity seems to pick up only the primary field (usually “name”) into the search scope, but it’s likely that you’ll have plenty of other important field that should also get indexed. The same goes for default entities: only a subset of the standard fields are included, and none of the custom ones are, unless you configure them to be included in Relevance Search.

    If finding the menu from where the entities list for Relevance Search is maintained was a bit tricky, then figuring out the place for specifying the field level search settings ain’t that obvious either. You see, it’s not a property of the field in the entity itself. It’s the inclusion of that field in the entity’s specific view called Quick Find View that determines what gets pushed to Azure Search database and back to Power Apps UI.

    Even though the entity view editor can already be found in the modern Maker Portal for Power Apps, these settings aren’t there yet, so I hope you didn’t close that legacy customization UI just yet!

    From the legacy Solution Explorer, open the Quick Find View for your entity, select Add Find Columns from the right and check the boxes for the fields you consider to be potentially useful for Relevance Search. Save & publish your changes, then verify that the search results match with your expectations.

    Challenge 4: Relevance Search doesn’t cover inactive records

    Sometimes the information you’re looking for isn’t found on records that are actively edited anymore, as you may be digging further into the history of your business data to check up on past transactions. In the typical CRM scenarios where Model-driven apps are leveraged this could mean searching through historical sales opportunities that are either won or lost, to find examples of what offers have been made to which clients on what specific terms.

    With the out-of-the-box configuration of Dynamics 365 or Power Apps, you won’t find them via Relevance Seach. Yes, the vast majority of records that you’re storing in CDS may well be excluded from the search index by default. This is because of the same Quick Find View we visited earlier. Not only does it control the fields being indexed, as well as the column layout of the search results page. It also controls the static filters which are applied to the search results before they are shown.

    Using the OoB opportunity entity as an example, you’ll find a system view called “Quick Find Open Opportunities” under the entity. Click on Edit Filter Criteria to reveal the fact that it indeed does only include open opportunities. While we can’t create additional Quick Find Views for entities, luckily we have the freedom to modify or delete the existing filters. So, just remove that status field filter criteria, save & publish. Now your Relevance Search will also return results from the inactive records for this entity.

    Challenge 5: Relevance Search suggestions aren’t found in the search results page

    The new search experience has a nice UX for suggesting matching records immediately as you type the letters in your the search term. Let’s say that we’re interested in the search term “relevance”. By the time we’ve entered “relev” into the search bar there’s already a number of matches in the suggestions box:

    Looks great, but I think I want to see more details, so I’ll click on “show more results” at the end of the search suggestions. Hey, where did all those matching records go?!?

    Unfortunately this is a challenge where you can’t configure way around it (at least yet). For the time being, this actually is by design. Taken from the documentation, Microsoft lists the suggestion feature separately from the actual search requests:

    Suggestions provide a list of matches to the specified search parameter value, based on an entity record’s primary field. This is different from a regular search request because a suggestion search only searches through an entity record’s primary field, while search requests search through all relevance search-enabled entity fields.

    So, the search box and the search results page follow different logic, which is why it is expected that they will show different matches. This may be seem quite complex to explain to the app end users on a detailed technical level, so it’s better to just instruct them to always apply the wildcard character * at the end of their search term. The results are likely to be closer to what they expected to see:

    Hopefully these 5 tips will help you in setting up the basis for more relevant search results and a better user experience in your Model-driven apps. For more comprehensive guidance, check out these three different levels of documentation that Microsoft provides on the Power Apps seach features: