Copilot Studio Kit

Copilot Studio Kit, published by the Microsoft Power CAT team, is a comprehensive set of capabilities designed to augment Microsoft Copilot Studio. The kit helps makers test custom agents, use a large language model to validate AI-generated content, and track aggregated key performance indicators.

Testing capabilities

Makers can configure and run test sets against the Copilot Studio APIs to verify that the agent responses are as expected. Test results can be enriched with additional data points from Azure Application Insights and from conversation transcripts in Dataverse. For AI-generated answers, AI Builder prompts are used to compare the generated answer with a sample answer or validation instructions.

Conversation KPIs

Conversation KPIs are designed to help makers track and analyze the performance of their custom agents. This feature complements the existing built-in analytics of the Copilot Studio and simplifies the process of understanding conversation outcomes by providing aggregated data in Dataverse rather than requiring you to analyze the complex conversation transcripts.

SharePoint synchronization

SharePoint synchronization allows makers to configure selective content synchronization from SharePoint locations to a custom agent knowledge base as files. Compared to using SharePoint site as a knowledge source, larger files and more file types are supported. Additionally, PDF file indexing is improved, and SharePoint pages (.aspx) are supported as well.