Microsoft Copilot has recently made headlines after researchers revealed a potential security vulnerability within its Recovery Augmented Generation (RAG) systems. Nicknamed “Confused pilot”, They claim that Copilot could be tricked into providing sensitive data from the company’s systems.
Beyond Microsoft, other co-pilot programs using GenAI are causing a serious increase in security breaches that many organizations appear woefully unprepared to prevent. In fact, by 2025, Gartner predicts The generation of AI will cause an increase in cybersecurity resources needed to protect it, resulting in an incremental spend of more than 15 percent on application and data security. This is consistent with what we’re hearing from security leaders, many of whom are concerned about the impact of using co-pilots on their security infrastructure.
To properly protect an organization, it is important to understand why these security problems persist.
Setting the context: the rise of GenAI and AI assistants
Generative AI continues to be a market disruptor in organizations of all sizes and industries. Standalone and integrated solutions promise to radically improve workflows, improve customer service, and save teams hours of manual work.
In the Asia-Pacific region, the market size GenAI is expected to show an annual growth rate (2024-2030) of 46.46 percent, resulting in a market volume of 86.77 billion US dollars by 2030. This growth is driven by the fact that GenAI is lends itself to personalized and efficient digital experiences. including virtual assistants and chatbots that understand and cater to individual preferences.
As highlighted GartnerGenAI is the number one type of AI solution deployed in organizations. Additionally, GenAI integrated into existing applications, such as Microsoft’s Copilot for 365 or Adobe Firefly, is the number one way to meet GenAI use cases. Overall, 34 percent of respondents said this is their primary method of using GenAI and that it was more common than other options, such as custom GenAI models.
Leinar Ramos, senior director analyst at Gartner, discussed how this rise of GenAI solutions is driving conversations about their appropriate use. He said: “GenAI has increased the level of AI adoption across the enterprise and made issues such as upskilling and AI governance much more important. “GenAI is forcing organizations to mature their AI capabilities.”
Microsoft Copilot is a clear example of a GenAI solution that can be integrated across an existing platform and requires additional attention to mitigate security issues. As several analysts and statisticians have attested, Microsoft Copilot is one of the early notable entrants into integrated GenAI. As shared by Microsoft itself, “Microsoft Copilot is an AI-powered digital assistant designed to help people with a variety of tasks and activities on their devices.”
Understanding the reality of the Copilot threat
While the promise of GenAI solutions like Copilot is certainly attractive, it is crucial to understand why they also present significant security issues, with prominent organizations already attempting to resolve security breaches. Basically, it’s all about data and who has access to what.
The main concern with Microsoft Copilot and similar tools is that they have access to the same sensitive data as users. Unfortunately, many users using Copilot don’t fully realize how overly permissive access to data can greatly increase the chances of cybercriminals infiltrating sensitive information and systems, leaving CISOs to mitigate the inevitable consequences.
Another key concern is that Copilot can rapidly generate large amounts of new sensitive data based on requests and may reference data that was legitimately accessed that it should not. For example, Copilot might formulate an answer to a question with sensitive information for which the questioner is not authorized, such as information about future product launches, company restructurings, or high-level operations.
There have been some attempts to address issues related to data misuse. For one, Microsoft does not use an organization’s data for the purpose of training Copilot. Data remains within its own Microsoft 365 tenant. Additionally, companies are also being more careful with user access. And yet, these measures only go so far: the documents and responses generated by Copilot remain primarily about knowledge sharing, rather than security policies or principles.
So what can be done?
Trust is everything, and for GenAI integrations to be successful and not represent a data breach in the pipeline, systems and processes must be reoriented toward the security of personnel and operations. In essence, there are three key things we need to remember: GenAI is a force multiplier for everyone, these attacks still require action on the target and can be detected and stopped.
As a basic and important first step, before adopting Copilot, organizations should conduct a very strict and thorough access control review to determine who has access to what data. As demonstrated with principles such as zero trust, best practices often advocate for reduced user access.
Once Copilot is integrated, organizations can apply sensitive labels. Microsoft itself recommends applying sensitivity labels through Microsoft Purview. Here, users configure labels to encrypt sensitive data and ensure that they do not receive the Copy and Extract (EXTRACT) permission. EXTRACT prevents users from copying sensitive documents and blocks Copilot from referencing the document.
It’s also a good idea for organizations to consider implementing additional security services and expert solutions that identify suspicious user behavior and flag priority alerts so SOC professionals can detect threats before they go too far. A SOC defender can use additional information to respond authoritatively, for example by locking a “user” (a hackerbot disguised as an employee) out of your account. Quality tools will also provide greater visibility so SOC teams can see who is using Copilot and what data is being leveraged, so they can stay informed and prepared.
Like Vectra AI 2024 Threat Detection and Response Status Report As an example, AI can also be used to help security teams reduce data vulnerabilities, not only in their GenAI integrations but also across workflows.
The report highlights that almost all SOC professionals (97 percent) have adopted AI tools, and 85 percent said their level of investment in and use of AI has increased in the last year, which has had a positive impact in their ability to identify and address threats. Additionally, 89 percent of SOC professionals will use more AI-powered tools over the next year to replace legacy threat detection and response, and 75 percent said AI has reduced their workload in recent years. 12 months.
Partnering with a security expert can go a long way in helping teams understand security across their organization and know how to successfully adopt GenAI tools while utilizing other AI-powered solutions for greater threat detection and response. .
When it comes to Copilot, security experts can help configure it in a way that reduces data leaks, monitors prompts and responses to ensure sensitive data is not misused, and detects abnormal behavior or misuse of the device. solution.
