Since 2019, Microsoft has had a long-standing partnership with OpenAI. The goal: to accelerate breakthroughs in AI and ensure the benefits are broadly shared with the world. A significant milestone in this collaboration occurred in January 2023 with the introduction of Microsoft’s Copilot offerings. Referred to as an “AI collaborator,” Copilot functions as a valuable assistant that works alongside you, enhancing your productivity and efficiency. Our thought leaders across the organization have shared many perspectives on its use spanning approaches to responsible AI to its use in low-code app development. In the last few months, we have also shared several copilot announcements regarding model improvements. AI brings the potential of innovation to every line of business, and in this post we focus on our journey with Copilot in Dynamics 365 Marketing.
The Dynamics 365 Marketing journey began when we launched our first previews of content ideas in May 2022 and query assist in October 2022. The real-world feedback we have received has been instrumental in our learning process. It has allowed us to gain valuable insights, but we acknowledge that there is still much more to learn and accomplish to truly revolutionize the marketing domain. With the technology changing so rapidly, it has also become very clear that we will—the use cases for Copilot in customer experience (CX) are endless. Let’s delve into how these copilot features came to be and highlight our generative AI roadmap moving forward.
Dynamics 365 Marketing
Transform your customer experience with generative AI
Query assist, using Copilot to democratize segmentation
Marketing segmentation enables targeting—choosing an audience based on their characteristics. It could be demographics, psychographics, behaviors, preferences, or a combination thereof. Effective segmentation has traditionally required a thorough understanding of complex data models, database management, and SQL (Structured Query Language). Consequently, marketing organizations frequently face the challenge of relying on a single individual to create segments for the entire team, leading to a bottleneck in the process. The Copilot feature query assist is built upon OpenAI’s GPT foundational model to empower marketers to easily target the right audience by using everyday language to describe and quickly build segments without having to learn data models and construct queries. Directly within their workflow, marketers provide the segment description, and query assist starts to create a new segment and automatically generates the segment rules that marketers can add to the segment definition before publishing the segment.
“Our goal was to develop powerful and user-friendly features to enable everyone to create segments with confidence, while still providing flexibility and power for advanced users. OpenAI was the foundation, but customer needs were the central focus.”
Gaju Gatera, Dynamics 365 Marketing Product Manager for query assist
Ultimately, marketers who use query assist spend less time on segmentation and more time on creating campaigns that truly connect with their customers.
Query assist was initially released with our own AI model. We then experimented with OpenAI GPT and found that query assist could understand different syntaxes from formal language to more conversational language. For instance, users could type “Show me our Seattle based contacts who attended the Contoso coffee tasting event last year” rather than the prior version’s requirement for more structured or SQL-like inputs such as “contacts who live in Seattle city and attended the Contoso coffee tasting event 2022.” This was very encouraging.
We collaborated with customers to improve the UX and embedded query assist seamlessly into marketers’ workflows. Originally marketers would input a description powered by query assist, land on Copilot, and get stuck without a way to transition back to manual segmentation. So we made sure to highlight query assist in the entry card, and we introduced a button that lets you switch from Copilot and manual segmentation in the canvas, allowing marketers with different levels of expertise to choose the experience that they like most.
We also changed the interaction design and streamlined the UX so that complex segments (requiring relationships traversals) are now created automatically without human intervention; what required three to four manual steps before is now one click as Copilot can create the relationships automatically.
Immediately following the launch of the enhanced version of query assist in March, we’ve seen substantial results where we now see 66 percent of marketers creating segments with query assist. We also saw the accuracy (as measured by thumbs up or down) improve from 15 percent to 86 percent within a month of the release.
We are particularly excited to witness how Copilot is driving positive change and contributing to the success of our customers. Chris Barnhart, Marketing Director at NC Fusion, a professional soccer team and sports organization serving the Triad region of North Carolina is an early adopter of Copilot. As a SQL expert, Chris knows how to design relationships among tables to create segments that engage a wide range of audiences from fans, to parents, to donors. “When I saw how you just type in common phrases and query assist will connect everything for you, I was blown away!” He was thrilled to discover how query assist could simplify his work and make his team more efficient, especially since many of his interns haven’t learned about data structures in school.
Sidney Fernandes, Chief Information Officer and Vice President for Technology at the University of South Florida, shared with us his excitement for the potential of using copilot to empower stakeholders across the organization.
“Once we share this with our teams, it has the potential to be quite transformational. Students and staff won’t need to be educated on the nuances of student data or copywriting to engage the right audiences impactfully.”
Sidney Fernandes, Chief Information Officer and Vice President for Technology at the University of South Florida
Tyler McClain, Senior Solution Consultant with Connected Experience partner Coffee + Dunn tried the feature and shared their experience, saying “Using query assist was quick and easy. I just typed ‘anniversary within the past five years,’ and it gave me exactly what I was looking for. This feature can further empower our clients to engage their customers with ease, and I’m excited to share it with them.”
Encouraged by these positive results we continue to iterate on query assist. We are excited to announce that Copilot now supports segments that include marketing interactions or specific customer behaviors such as “customers who opened an email recently” enabling them to optimize segment creation to drive more effective campaign engagement. We can’t wait to see how marketers use it to drive better results and achieve their business goals.
Content ideas, simplifying email creation using Copilot to generate compelling content
Email marketing can be a highly effective way to connect with customers and drive business growth, but it also comes with its challenges. Over time, content can begin to feel stale or repetitive. Producing engaging content for email campaigns is time-consuming and requires significant creative energy.
To assist marketers in their email creation, we’ve used OpenAI GPT to develop content ideas. Given a few short key points, Copilot generates engaging email content ideas within seconds that marketers can select from and tweak to perfectly match their needs.
“Content ideas was developed with one goal in mind–saving hours of copywriting, by helping marketers kick-start their email creation and create compelling content within minutes.”
Aga Miskowiec, Principal Product Manager for content ideas
With this innovative tool, writer’s block is becoming a thing of the past, allowing marketers to accelerate the delivery of engaging email campaigns that resonate with their audiences.
After releasing the first iteration of content ideas, we rapidly realized its potential and have been working to refine the approach for over a year. This includes tuning that we haven’t seen in any other products. We tune based on a library of public emails, not for any branding but rather for the type of tone and content a marketer would use. We also enable companies to use content they’ve created in the past to tune the generation of new content.
In March, we took content ideas to the next level by upgrading the language model to OpenAI GPT-3.5, and this had a significant impact on the quality of ideas generated.
Our customers started to see a positive impact on their business Chris Barnhart Marketing director at NC Fusion shared with us that “Normal engagement runs about 20 to 30 percent and there’s a lot of factors that go into the email itself, but with the email written with content ideas the engagement was about 70 percent.”
But that’s not all, we’ve also added a range of different tone of voice options to help marketers tailor the content to their brand and audience. Whether they’re looking for an engaging, adventurous, casual, luxury, or formal tone, content ideas has got marketers covered.
“Setting the tone allows Campari Group, a global company, to tailor the tone of voice for each brand and region’s communication style saving us hours of copywriting” according to Liam Barnes, Global IT Director – Marketing Technologies from Campari Group, a worldwide leader in the spirits industry.
To level up usability, we put content ideas right in the email creation flow where marketers are adding or editing text as our data showed that switching to the Copilot tab was a significant discoverability usage drop off.
We’ve also gamified our UX to better guide marketers and help them craft higher quality key points used to generate content. “It’s pretty impressive. I was very happy using the feature. Our team has been very happy using the feature–it’s making delivering great content faster and easier.” shared Keith Perfect, Director of Technology and Intelligence from Northrop & Johnson, a luxury yacht brokerage.
As a result, the number of daily users who opened the feature doubled, the percentage of users who generated ideas (out of those who opened the feature) increased by 75 percent. Furthermore, the number of organizations using content ideas daily increased more than 5 times.
Receiving early feedback from our customers and partners who tried the latest version of content ideas was a rewarding experience. Their input helped us prioritize our focus and further enhance customer satisfaction.
“It’s awesome, content ideas helps us tremendously to efficiently produce emails. In fact, I can tell you it was normally at least an hour, and I did it in 15 minutes, it takes me ~25 percent of the time it took before to create email content!”
Chris Barnhart, Marketing Director at NC Fusion
“The results are very creative now, and tone of voice capability is great,” said Steve Smith, Principal Solutions Architect – Data & Artificial Intelligence at TTEC Digital.
We also found that our users needed help with what to ask the copilot. We redesigned the suggestions UI with sample topics, such as an abandoned cart, referral, or event invitation, to receive instant inspiration for commonly sent marketing emails.
And we’re just getting started! We are working on adding personalization, journey creation, brand style creation, and conversation help capabilities, just to name a few.
Our goal is to empower marketers in delivering exceptional customer experiences that drive business success with less effort and time than ever before. With the limitless potential of AI, we eagerly anticipate the future of customer experience.
Get started with Dynamics 365 Marketing
Start using content ideas and query assist today, and see how it can help you boost your productivity while creating deeply personalized experiences.
Not a Dynamics 365 Marketing Customer? Take a guided tour and get a free trial.
We’re always looking for feedback and would like to hear from you. Please head to the Dynamics 365 Community to start a discussion, ask questions, and tell us what you think!
