UPDATE 2020-11-17: Microsoft’s initial product name “Dataflex” has been replaced with “Dataverse” in the General Availability announcement. Otherwise everything announced originally in July 2020 remains the same.
On July 21st Microsoft announced a new service called Dataverse for Teams. In short, it is a data platform for low-code apps that can be built right within Microsoft Teams. This is a brand new commercial offering and it comes bundled with every subscription of Office 365 / Microsoft 365 that includes the right to use Teams.
At the same time, Dataverse became the new name for what used to be called Common Data Service (CDS). For those who are more familiar with Power Platform technology, you’ll know that CDS has served as the foundation for building your custom Power Apps as well as leveraging the many Dynamics 365 app products built by Microsoft.
So, what is actually new, what is just rebranding, and why is the arrival of Dataverse into Microsoft Teams such a big deal?
One platform for many app types
Let’s look at how Dataverse is positioned and licensed for different scenarios. Starting from the left, the new capability announced at Microsoft Inspire 2020 can be seen as the entry point into app creation on top of Dataverse. Its license is seeded within Microsoft Teams, just like the Office 365 / Microsoft 365 plans already come with seeded rights to use Power Apps & Power Automate to extend the various business productivity apps found within the realm of Office. It’s important to understand that the “free” edition of Dataverse only allows the creation and usage of apps from within the Teams clients. Technically these are not the same as the official Teams custom apps built by pro developers, but for the sake of this discussion let’s call them “Teams apps” that happen to be powered by Dataverse and Power Apps.
When the needs for building a low-code business app can no longer fit within the Teams app usage scenario, customers can upgrade the Dataverse for Teams environment to full Dataverse. Essentially this means moving into the existing licensing model of Power Apps, where you can either license a single application at a time ($10 Per App) or purchase the “platform SKU” that allows an unlimited number of apps ($40 Per User). This unlocks the full feature set of what the Common Data Service used to provide, meaning advanced data types, environment management, security & compliance tools, integration options and ALM tools.
Continuing further on this journey, customers could also choose to purchase a license to the readymade business apps from Microsoft, branded as Dynamics 365. Many of these (but not all) are built on top of the same Dataverse platform that powers the Power Apps solutions created in-house by citizen developers working within the business teams. Underneath the surface, a sophisticated enterprise system like Dynamics 365 Connected Field Service with signals coming in from IoT sensors and the information presented to field technicians wearing a Hololens device in front of their eyes is actually assembled from many of the same building blocks that any Teams user now has at their disposal via Dataverse.
Microsoft low-code aPaaS evolution
Looking back at how the business application platform technology (aPaaS) from Microsoft has evolved over the years, we can trace the origins of Dataverse back to the launch of MS CRM in 2003. The platform capabilities for creating customer-specific business apps with a custom data model were added in 2005 when the concept of XRM (eXtended Relationship Management) emerged. The application layer on top of this platform evolved from Dynamics CRM to Dynamics 365, until in 2018 the XRM platform merged with Power Apps and XRM became CDS. The great thing about this long journey has been that the fundamentals of customer specific data management have been carried over to new generations of apps, thus ensuring that my first XRM custom entity created in 2007 would still be supported in Dataverse today, had there been a need to keep the same application running through 13 years.
The citizen developer focused layer of app maker capabilities sold as Power Apps is now further expanding in 2020 when Microsoft Teams is becoming an even more lightweight app platform on top of the existing technology stack. Using Dataverse from within Teams does not offer app makers all the advanced functionality that the full Dataverse platform contains, thanks to its roots in enterprise grade applications like CRM. Instead it is a simplified experience that is removing much of the complexity in areas like data models, security roles and environments. Microsoft wants to optimize the app creation experience in Teams to be in line with what the original collaborative purpose of the teams and channels is – so that the citizen developers can use suitable tools for their application back-end right from the start.
The obvious reason for lowering the barrier to start using the ex-CDS features in new app creation is because of the fact that the earlier default option has been suboptimal for everyone. I’m of course talking about SharePoint Lists, which have been the Power Apps data storage option not requiring any additional licenses beyond the O365 / M365 plans all users have already had. Even though there are concurrent investments being made into the newly announced Microsoft Lists, too, the path for low-code custom app creation has now been pointed towards Dataverse. It is the platform designed specifically for this purpose, whereas Lists doesn’t offer the same “no cliffs” development story from citizen developer apps to pro developer & IT managed systems.
Microsoft Teams is becoming a true app platform
Thanks to in part the strange times we’re living in 2020 righ now and the WFH trend that’s made all teamwork remote first, Microsoft Teams has skyrocketed into a true mainstream service with its 75 million daily active users. As more and more work outside just meetings and chats happens within the Teams client, it’s a logical place to also introduce new digital tools that can support asynchronous teamwork and potentially reduce the volume of meetings and calls that distract information workers. Presenting structured business data via a modern UI in the context of Teams and automating processes around it is something that organizational teams surely will have many use cases for. With Dataverse now bundled into Teams and the simplified app creation tools if offers, it’s hard to see how we would not see a sharp rise in the number of Power Apps built by business users very soon.
The professional developer story of Teams apps has been a logical model for ISVs to build commercial app products that are to be deployed across many different customer organizations. The low-code tools offered by Dataverse and Power Apps will be the mechanism through which the organization specific digital tools are likely to be developed, then distributed across the various teams within the same organization. Now that both citizen developers as well as commercial app developer companies can soon offer their solutions within the Teams app store, this can present synergies that will drive user adoption of Teams based tools in various more sophisticated and high value business processes.
How customers will be able harness the new powers that the built-in Dataverse support in Microsoft Teams presents to them will be interesting to see. An immediate result from the coming GA launch of Dataverse will likely be the explosive growth of different app environments within the tenant. Just take a look at the total number of teams that your organization has already created, then imagine X % of them getting a dedicated Dataverse environment provisioned when the power users start exploring these new tools. The growth in potential app makers is surely going to put more pressure on building a Power Platform governance model that can scale to meet the rise in environments, apps and Flows where business data is processed.
Conclusions
Microsoft Dataverse is the logical next step in democratizing the app building and process automation tools in Power Platform. The version included with Microsoft Teams is aimed at new app makers who want to solve business problems for their own teams, for whom the organization level admin and development capabilities in the full Dataverse might be overkill. While primarily a subset of the platform capabilities behind current CDS based Power Apps and Dynamics 365 solutions, Dataverse can offer a streamlined experience for Teams users to configure data models and manage security. These new options will introduce additional solution architecture considerations for organizations who are already leveraging CDS and Power Apps today. For new customers, the Teams based entry point into the world of Dataverse will likely become a significant driver of Power Platform usage growth in many, many organizations.
Read our highlights from the General Availability announcement on November 16th from this blog post: Welcome to Microsoft Dataverse for Teams.
Interested in reading our latest Power Platform blog posts?
You’re welcome to subscribe to our email newsletter: Forward Forever News. Max 1 email per month, with a curated list of latest insights and articles on Power Apps, Power Automate, Power BI.
13 comments
Pranav
2020-07-22 at 18:32Great insights and explanation along with visuals Jukka – appreciate it as always!
Paul
2020-07-23 at 23:42Microsoft should just make Azure SQL a standard connector again (rather than premium) then we can have a real relational database with a mature feature set, great capabilities and a wealth of understanding and learning resources available. Also make it possible to use SQL language within PowerApps so we don’t have to rely on the vagaries of the translation process from PowerApps expressions to SQL and spend our time rewriting expressions in the hope of getting them to delegate properly.
Jukka Niiranen
2020-07-24 at 09:04Paul, the ability to use SQL for querying Dataflex is indeed coming into more and more scenarios in the future. The Tabular data Stream endpoint will bring T-SQL type of capabilities for Power Apps developers, which should definitely make it more accessible to a wider audience.
Yes, it will still be a premium feature, but I don’t think it’s too unfair to ask some money for these type of low-code app development features in the Dataflex Pro edition. The bundling of Dataflex non-Pro in the Teams subscription to me sounds like a very sensible compromise in the licensing model. It means that pretty much every Office 365 / Microsoft 365 customer will have the possibility to use a proper relational data service for their apps (which runs on Azure SQL behind the scenes, of course), instead of this being restricted to only Microsoft 365 E5 plans, for example. Now, it’s of course interesting to speculate that since the new Dataflex “free” and Pro editions are following the Power BI model of product packaging, will we eventually see a Dataflex Premium plan added as well. This is something I’ve written about in my personal blog.
Barney
2020-07-26 at 01:22Hi Jukka, excellent post and blog thank you for your insights. A question for you what do you mean by “…restricted to only Microsoft 365 E5 plans…”? Are you suggesting that m365 E5 plans have access to CDS/dataflex pro?…
I’m super excited by your comments that dataflex std will be included and this is indeed a step forward from SharePoint lists, however I’m with Paul on the cost..I have my users several hundred with M365 E5 (I don’t think we have cds access btw but hoping I’m wrong??) and we also use dynamics fin& ops this is already a significant cost per user to then ask for a powerapps license on top it’s becoming very pricy… really appreciate your thoughts on this thanks, Barney ?
Jukka Niiranen
2020-07-29 at 19:57Barney, I was using E5 as an example of a strategy that Microsoft has used earlier in a similar case, where the Power BI Pro was packaged into the M365/O365 offering. No such option exists at this point in time for Power Apps Pro / Dataflex Pro, but I wouldn’t see it as an impossible scenario for future expansion. My point was that I find it a much, MUCH better strategy to open up the basic features of Dataflex to all users of Teams, rather than making bundling the premium features with another premium license. This way people will become familiar with the value-add offered by a data platform designed specifically for the low-code scenarios, so they can also better understand how in practice it differs from things like SharePoint Lists or raw Azure SQL databases.
I believe this has ultimately been a bigger challenge with the Power Platform licensing discussion, rather than the license cost directly. Those Dynamics 365 professionals who have worked with this technology starting from the XRM era will have plently of experience from implementing systems for customers where the business value from the resulting application was higher than the Dynamics licensing prices (otherwise those projects wouldn’t have happened). Professionals coming from the Office 365 side haven’t necessarily gone through similar discussions with customers and when they’ve been faced now with an additional license cost for Power Apps it’s been difficult to present the equation in a way where the business value from the platform gets articulated clearly enough for the purchase decision to take place.
Even with the Azure folks there will be surely be challenges in approaching the low-code world where the licenes are per-user and not pay-as-you-go. Instead of paying for a software development project with multiple developers to achieve a very specific and tailored result, now the world is slowly but surely turning toward the aPaaS model where the cost of developing additional applications may reduce dramatically but the productized plaform and app maker tools will have a higher upfront cost. This is the shift which I’ve covered in a previous article here on the Forward Forever blog, “take ownership of your digital tools”.
Even if the low-code technology found in Power Platform today is just a product of evolution, and there’s been plenty of RAD etc. tools in the past to offer something similar, I do believe we’ve seen the start of a brand new era here. The path to business value will be different here (more user driven rather than MS partner driven) and the way how budgets for this technology will be allocated is also a whole new story. I bet we also haven’t seen the “final licensing model” for Power Platform services either, but this latest approach with Dataflex for Teams (the new name for the bundled offering) and the Dataflex Pro as a premium data platform seem to me like a pretty well though out strategy from MS for pricing their products for the coming world dominance in low-code apps 😉
Paul
2020-07-25 at 02:40Thanks for the reply Jukka. My issue is that Dataflex (formerly CDS) is *not* a relational database. It is not possible (AFIACT) to query across multiple entities. E.g. I have ‘Appointments’ for ‘Jobs’ and each job has a number of ‘Job Products’ referencing a ‘Product Entity’ and I want to run a single query to list ‘all upcoming appointments that involve product X’ because I am running low on stock and want to send out a notification to the relevant customers – I don’t believe this is possible, certainly not from Canvas Power Apps. In SQL I can quickly and easily write a single query/view to return the desired results.
If I cannot *extract* information in a relational fashion (across multiple entities) it is not a relational database in my view. Hopefully TDS will bring this ability (I guess this is like using the ‘SQL Mirror’ of a Dynamics instance for querying – I have done that before where a data retrieval process taking 15 minutes through the CDS connector was reduced to 6 seconds by using T SQL), but the post says it is preview and only for analytics and reporting.
As I understand it, the lack of relational querying is why data in CDS is not properly normalised (you see this even with the standard entities – out-of-the-box the Account entity has over 150 fields with many ‘duplicates’ for Address1, Address2, Address3, etc.). It is also why in actual deployments you often find the same data repeated across multiple entities with a reliance on work-flows to try to keep that data in sync which I have seen cause many issues.
Yes, CDS sits atop Azure SQL, but so do SharePoint lists and you wouldn’t recommend using SharePoint as the database for an enterprise App. It doesn’t matter what technology is underneath if we cannot directly reach it.
I happily pay for Azure SQL, I happily pay for PowerApps, but I resent having to pay again to *connect* PowerApps to SQL, especially when I can connect anything else I want (including competitor products) to Azure SQL at no extra charge.
Jukka Niiranen
2020-07-25 at 20:34Paul, Dataflex is grounded on relational data models. If you look at the timeline picture in this post, you’ll see that Dataflex originates from a system that used to be called “Any Relationship Management” (XRM). Queries like the one you described are exactly what the CRM systems built on this technology have been using for a couple of decades already. When in a Model-driven Power App, there is a GUI that allows you to construct very complex queries that have conditions across multiple entities. “Show me accounts that have open opportunities with value over €100k and where there are product line items from category X” can be created by any user – not just the app maker. No, you can’t do everything you could do in raw SQL of course, but the capabilities offered by the Advanced Find feature have been one of the key reasons why power users have loved the platform over the years.
Now, going over to Canvas apps, which of course have originated from a completely different evolutionary path originally and only merged with XRM 2 years ago, there hasn’t been such a great story on how to leverage relational data. However, you can already today build a view in Dataflex and then reference that as the data source in a Canvas gallery. That particular view can have query conditions across several related entities, many relationships deep in the hierarchy.
What about the pro developers, how would they build such queries? Today, the answer is FetchXML, which is a proprietary query language for XRM (and CDS, and now Dataflex). Sure there are excellent community tools like FetchXML Builder which lower the barrier for newcomers to this platform to take advantage of the relational data processing capabilities. But it isn’t quite as simple as using the universal language of SQL of course. This is exactly the reason why Microsoft has decided to invest in making the Dataflex data available via the TDS endpoint. It’s a brand new feature, and it’s only the start of this journey, but you can imagine that MS wants to make Datalfex easier to approach by new audiences – compared to what the plaform was in the Dynamics CRM days. So, there’s likely a lot more to come on this front.
That of course leads us to the business model of how will these investments into the platform be funded. You say you’ll happily pay for Azure SQL and for Power Apps. The latter will give you access to Premium connectors like Azure SQL, so that solves the problem. No, the versions of Power Apps that have been bundled with Office 365 & Microsoft 365 do not provide access to external services. They contain features that are meant for customizing the Office tools, but they are not ment for building full custom apps that aren’t associated with the context of the products you’re paying for. I bet that the other services which allow you to connect to Azure SQL aren’t completely free either, so it’s hard for me to see a big disconnect here in the pricing model of Power Apps and Dataflex Pro. With these two services you can build low-code apps that are as complex as those enterprise Dynamics 365 apps that MS charges $95/user/month, so I find it quite understandable that such capabilities aren’t included in an Office bundle for no additional cost.
Per
2020-07-27 at 08:10Great article and totally agree that this will “put more pressure on building a Power Platform governance model” – this will for sure be one of the most important tasks to handle for every organisations in the future.
Paul
2020-07-28 at 04:01Thanks again for engaging Jukka.
I’m coming at this from the perspective of a *canvas* app developer, so my criticisms of the shortcomings of the platform need to be taken in that context. At present, as you say, the story for canvas apps is not very good (with severe limits on relational querying), though I do appreciate the efforts to improve things.
I am curious about the claim of CDS being based on relational models when the platform uses such concepts as polymorphic relationships and MultiValueFields / OptionSets which don’t conform to the tenets of relational database design – https://www.techopedia.com/definition/27292/multivalued-field-mvf.
I’m no expert in CDS, as you can tell 😉 , but when I have asked those who are (or claim to be) experts in CDS why data is not well normalised I get the response that it is because it is so difficult to work relationally in CDS and this appears to be backed by the design of the standard entities (such as Account with its 150+ fields and multiple address fields).
Licensing is a sort point for those of us who were caught out by the license changes in 2019 when SQL (among many other connectors) where moved from ‘included’ to premium. A model where you could create an App for 100 existing office users against a SQL Azure DB for just the additional cost of the DB (say $50 per month) suddenly leapt up by $1000 a month (yes, *existing* Apps were given a 5 year exemption, but for those of us who invested time and energy learning the architecture have lost out as customers move to other technologies due to the costs).
I can connect desktop Access/Excel/Word to an Azure SQL DB without incurring any additional ‘premium’ charges (just the cost of Azure SQL DB and the Office licenses or one off cost for perpetual Office). I think the comparison with $95 for Dynamics is false as that is a complete product whereas the premium PowerApps charges are for the tools/platform on which to build, maintain and support a product of your own.
Jukka Niiranen
2020-07-29 at 20:29Paul, I come from a functional consultant background with zero academic education on computer science topics, so I tend to always first frame this technology within the context of the business problem that it is aiming to solve. In this case, “relational” refers to the V1 scenario that the ancestor of Dataflex Pro, meaning MS CRM 1.0 in 2003, was built around: Customer Relationship Management. It is from the demands of these real world business processes that the platform today inherits features like polymorphic relationships (“Customer”, “Regarding” etc.) or option sets. They have been built to address the need to manage things like sales processes with easily configurable multi-langue dropdown fields, tracking of emails against an unknown number of possible custom entities, keeping things in sync with the data model of Outlook/Exchange and a million of other requirements. Would things look different if they were built from scratch right as of today? Most probably. However, we need to keep in mind that Microsoft actually tried that option with CDS v1 and the old PowerApps initially, then decided to scrap it all and switch to XRM instead. So, while it might not be the perfect fit for everything that today’s low-code app developers are looking to accomplish, we must remember that the relational business data management capabilities in this platform haven’t exactly appeared overnight but rather they’ve been built to match requirements from thousands & thousands of projects dealing with customer relationship management processes. Things like the security model will undoubtedly make it seem complex for app makers who are only now getting acquainted with the platform, but when it comes to the flexibility of what sort of business requirements can actually be met by leveraging the platform, then the evidence is quite clear.
If you or any of the readers of this blog are interested in learning more about how & why such functionality was brought into this world (and now ended up being Dataflex that’s soon found in any tenant using MS Teams), then here’s a great podcast episode that sheds light on the topic: The Origin Story of Dynamics 365 CE. BizApps MVP Mark Smith interviews the original Lead Application Architect of the platform, Aaron Elder, who shares his experiences from 2001 on how a then cutting-edge web app and the foundation of the XRM platform came to be.
Peter Schmidt
2020-07-28 at 15:45Paul, customers that had apps that were using the SQL Connector prior to the licensing change were granted 5 years grandfathering rights, which I think is the most generous Microsoft have ever been. All other enterprise database connectors – Oracle, SAP, etc require premium connectors – it was a anomaly that SQL was not, but it had been done this way to encourage early adopters to build enterprise apps with Power Platform and their reward was being granted 5 years of additional free use.
Paul
2020-12-14 at 00:19Hi Peter (very late reply – just came across this article again when researching something else and caught your comment) – should we assume that as any enterprise database connector is premium and SQL was a temporary anomaly (although never advertised as such) that Dataverse in Teams is also an anomaly and that additional licensing costs will be introduced one the early adopters have built apps that rely upon it?
Deron
2021-02-13 at 02:59Jukka and Paul, just wanted to say thank you for the dialog you’ve engaged in here. Coming from much the same perspective as Paul, I resonate strongly to the points he raised. Yet, Jukka’s well-stated counter arguments were very useful and gave me a fresh take on the situation. With my lack of Dynamics dev experience, and coming in from the Canvas Power Apps “doorway,” it’s been all too easy for me to bitch and moan about the convoluted structures in CDS/Dataflex/Dataverse/Dyn365. I know realize this is probably a bit of “when your (i.e. my) only tool is a hammer, everything looks like a nail…..and all the screws behave like terrible nails.” Thanks Jukka and Paul. I appreciate your insights and time taken to share them.