For many organizations, AI is no longer only a future risk or a theoretical topic. On the front line, AI is already changing how attacks look and feel. Sachiko Hasumi, CISSP, CCSP, argues that, from her experience, we may be missing another serious problem. That problem is the gradual erosion of human trust.
Disclaimer: The views and opinions expressed in this article belong solely to the author and do not necessarily reflect those of ISC2.
Attackers can now create what I call “synthetic legitimacy”. Their messages, voices, identities and behavior can look normal, professional and trustworthy. This makes our traditional identity and security models much weaker than before.
To defend ourselves against this, we need to change how we think about security, accountability and the people who operate our systems every day.
The End of the Traditional Yardstick
For many years, our defense was based on one simple assumption: malicious activity usually leaves some kind of visible trace. Malware dropped a noisy payload, lateral movement triggered alerts. Strange login locations, poor grammar, unusual timing, or clumsy behavior gave security operations centers (SOC) something to investigate.
Today, this traditional yardstick is no longer reliable. We often see users authenticating from known devices, passing multifactor authentication (MFA) and operating from locations that look reasonable; the signals look normal and the activity appears legitimate. Then a wire transfer is approved, a privileged portal is opened, or sensitive data quietly leaves the environment.
The problem is not always that our controls have completely failed. The bigger problem is that malicious activity can now arrive wrapped in legitimacy.
By way of example: my team recently contained a coordinated AI-supported phishing campaign written in natural and contextually correct Japanese. In the past, Japan’s complex language, business etiquette and hierarchical communication style created some natural friction for foreign attackers; a poorly translated email, or the wrong level of politeness, was often easy to notice. However, this is no longer the case.
This was an important lesson for me. This shows “synthetic legitimacy” is no longer limited to English-speaking environments or global companies. AI can reduce the friction of an attack in almost any language or business culture. It gives attackers more speed, more scale and, most importantly, more credibility.
The Four-Dimensional Battlefield
To defend a modern enterprise, I believe we must look at AI across four dimensions at the same time:
- A defensive tool: we can use AI to summarize large numbers of alerts, identify patterns and reduce triage time. This allows human analysts to spend more time on investigation, strategy and judgment.
- An attack surface: organizations are deploying their own AI systems and large language models (LLMs). These systems may introduce new risks: prompt injections, data leakage, insecure integration or data poisoning.
- An offensive capability: attackers can use automation to perform reconnaissance, identify misconfigurations, generate phishing content and weaponize information much faster than before. What used to take days may now take minutes or seconds.
- An amplifier: AI can scale social engineering through deepfakes, synthetic voice, fake identities, realistic emails and highly personalized fraud. This is especially dangerous because it attacks our trust and judgement directly.
I realized that if we focus only on technology, we miss an important point: that many AI-enabled attacks are designed to target human judgment. When a technical defense strategy meets an attack designed to manipulate human behavior, the defense can break in unexpected ways.
Operationalizing a Human-Centric Doctrine
To survive in the age of synthetic legitimacy, we need to adapt our security practices to support cybersecurity judgement and human decision-making. I don’t believe we can solve this problem only by buying more tools, as attackers exploit humans as efficiently as they exploit unpatched systems through the lifecycles of systems and data.
The Red Chair and “Design By Hacking”
In my practice, we use a concept for new projects that I call “design by hacking”. This is not only a compliance activity, but a way to change the thinking of the project team. Like threat modeling, we ask two simple but difficult questions:
- “How can this workflow be hacked?”
- “What do attackers really want from this system?”
Addressing these questions helps the team move away from only thinking about functionality, deadlines and business convenience. They force us to think about abuse cases, manipulation and attacker motivation.
We also use a method I call the “Red Chair” exercise. During design meetings, we place an empty chair in the room. This chair represents the attacker. We tell the team to imagine that the attacker is sitting there, listening to our business case, design and architecture discussion.
Team members then take turns sitting in that chair; while someone sits there, their role is to think like an adversary. They ask how the design could be misused, which user could be pressured, where trust is assumed but not verified. They ask what would happen if an email, voice call, approval request, or identity signal was fake but looked legitimate. This exercise changes the atmosphere of the discussions. Engineers, organization owners and security members begin to see the system from another side. They stop asking “Does this work?” and start asking “How will this be attacked?”
Most importantly, the Red Chair helps us consider how attackers may manipulate the humans using the system. This is very important in the AI era because the attacker may not break the system directly. They may persuade a trusted person to use the system in a harmful way.
“Cybersecurity Judgement” Training
While the Red Chair helps us build more resilient systems, we also need to build more resilient teams. To truly strengthen the human element of cybersecurity, we have introduced cybersecurity judgment training.
Rather than focusing purely on rules and awareness, we help people build practical habits. It is about learning to recognize pressure tactics, pausing before reacting and verifying information with intention.
In an age of increasingly convincing digital deception from generative AI to deepfakes the core risk is not necessarily a lack of understanding. It’s fact that sound judgment can be quietly disrupted by urgency, fatigue or emotional pressure. We built this particular training resource to help organizations and individuals develop the reflex to stop, verify and make the right call before a small lapse turns into a massive incident. We call this a “digital evacuation drill”, applying Japanese disaster prevention culture to cyberspace.
Aligning the Framework
In my practice, I connect this human-centered approach with familiar frameworks, including NIST CSF 2.0. The framework gives structure, but the value comes from asking how each function supports human judgement in real situations:
- For Govern and Identify, I clarify decision ownership, especially where AI-generated outputs are used. I also pay attention to roles exposed to pressure, authority or urgency, such as finance, help desks, executive assistants and security analysts.
- For Protect, I look for workflows that are overdependent on perfect human judgement. Access controls and MFA remain important, but they don’t always protect someone who is tired, rushed or pressured. In high-risk actions I try to build in practical friction, such as a second channel, independent confirmation, or step-up verification that gives the person permission to pause.
- For Detect, I look beyond whether an action is technically authenticated. An action can appear legitimate while still being influenced by manipulation. I therefore pay attention to context: unusual timing, urgency, changes in communication style, unexpected approval patterns or behavior that does not match the relationship.
- For Respond and Recover, I treat recovery as more than technical containment. Blameless retrospectives help the team understand how one decision became dangerous and make it easier to discuss fatigue, escalation paths and unclear accountability.
For me, aligning these functions is about making human judgement visible inside the security operating model itself.
The True Shift
The defining challenge of the AI era is not only speed. It is that adversaries can now appear convincingly legitimate. This has changed what I pay attention to. I still protect networks, endpoints and applications, but I also look closely at decisions, trust relationships and the signals people use to decide what is real.
The Red Chair exercise brings this shift into the design process. By asking how a workflow could be misused, who could be manipulated and where trust is assumed but not verified, the team begins to see security as part of the operating environment, not something added later.
The digital evacuation drill supports the same shift from the human side. Instead of telling people only to “be careful,” it gives them a way to notice urgency, fatigue, authority pressure or emotional discomfort before acting. Cybersecurity judgement becomes something people can practice. In incident response, I apply the same lens. I look at fatigue, alert overload and incident stress as part of the security picture, not only as personal issues.
For me, this is the real shift: security becomes stronger when it supports the people who make decisions every day. Human cybersecurity judgement is not a soft topic. It is becoming one of the central capabilities to be protected in the age of synthetic legitimacy.
Sachiko Hasumi, CISSP, CCSP, has more than 20 years of experience in international organizations, global financial industries and retails. She has held leadership, management and technical roles, with responsibility for security strategy, incident response, governance, resilience and awareness. Her cybersecurity work spans global operations, human-centered security, AI-era risk, cybersecurity judgement and organizational resilience.
For You, By You: Join Us and Build ISC2's AI Security CertificationAs cybersecurity professionals evolve and adapt their skillset and job functions because of AI, ISC2 has announced the development of a new AI security certification to recognize and benchmark AI skills and competence within the cybersecurity workforce. The AI security certification development process presents an opportunity for cybersecurity professionals to input into the process and help define parameters for the certification. This is your moment to play a defining role at the foundation of this new certification:
For more information about the ISC2 AI security certification program and how to contribute to the various development activities taking place, go to https://www.isc2.org/new-ai-certification. |
