Why do the gray market and pornography always emerge first in the era of AI?

Author: Pumping Geek

Geeks are starting businesses, novices are buying courses, and artists are unemployed. However, an awkward reality is that AI is booming, but the narrative is not following a straightforward path; rather, it is like rolling dice.

Moreover, in the early stages of the industry, the face that this dice lands on is often either yellow or gray.

The reason is quite simple; the pursuit of huge profits creates motivation, especially in the early stages of an industry, which is always fraught with loopholes. Just look at this set of data to understand:

Currently, over 43% of MCP service nodes have unverified Shell call paths, and over 83% of deployments have MCP (Model Context Protocol) configuration vulnerabilities; 88% of AI component deployments have not enabled any form of protection mechanisms; 150,000 lightweight AI deployment frameworks like Ollama are currently exposed on the global public network, with over $1 billion of computing power being hijacked for mining...

Ironically, attacking the smartest large models only requires the most basic methods - just a set of default open ports, an exposed YAML configuration file, or an unverified Shell call path, or even, as long as the prompt input is precise enough, the large model can help the gray industry find directions for attacks by itself. The door to corporate data privacy is thus freely accessible in the AI era.

But the problem is not unsolvable: AI has both generative and adversarial aspects. How to use AI for protection is increasingly becoming the main theme of this era; at the same time, establishing rules for AI in the cloud has also become a key exploration direction for leading cloud providers, with Alibaba Cloud Security being one of the most typical representatives.

At the recently concluded Alibaba Cloud Feitian launch event, Alibaba Cloud officially announced its two paths for cloud security: Security for AI and AI for Security, and launched the "AI Cloud Shield series products" to provide customers with "end-to-end security solutions for model applications," which is a prime example of current industry exploration.

01 AI dice rolling, why do gray and yellow always land face up first?

In the history of human technology, AI is not the first "new species" to be "tested by yellow violence"; the gray-yellow burst was also a rule of technology popularization rather than an accident.

When the silver plate photography technique emerged in 1839, the first wave of users was the pornography industry.

In the early days of the Internet, e-commerce did not start, and adult websites began to ponder online payments;

Today's large model wool party is, to some extent, a replication of the rich legend of the "domain name era."

The dividends of the times are always first taken away by the gray and yellow. Because they do not comply with regulations, do not wait for supervision, and their efficiency is naturally very high.

Therefore, every technological explosion period starts with a "messy soup", and AI is no exception.

In December 2023, a hacker used just one prompt – "$1 quote" – to almost trick a 4S store's customer service robot into selling a Chevrolet for 1 dollar. This is the most common form of "Prompt Injection" in the AI era: it does not require permission verification, leaves no log traces, and can change the entire logic chain just by being "artfully worded."

Going a step further is "Jailbreak Attack." Attackers successfully make the model say things it shouldn't say—such as pornographic content, drug manufacturing, false warning messages—using rhetorical questions, role-playing, and misleading prompts.

In Hong Kong, someone even embezzled 200 million HKD from corporate accounts by faking executive voices.

In addition to scams, AI also carries the risk of "unintentional output": In 2023, a major education giant's large model system erroneously output "toxic teaching materials" with extreme content while generating lesson plans. Within just 3 days, parents advocated for their rights, public sentiment erupted, and the company's stock price evaporated by 12 billion yuan.

AI does not understand the law, but it has the capability, and once that capability is detached from supervision, it becomes harmful.

However, from another perspective, the technology of AI is new, but the ultimate flow and means of the gray industry and the yellow industry remain unchanged. To solve this issue, safety is still relied upon.

02 Security for AI

Let's start with a little-known fact that the AI industry collectively avoids:

The essence of large models is not "intelligence" or "understanding," but semantic generation under probabilistic control. Therefore, once it goes beyond the training context, it may produce unexpected results.

This kind of exceeding the scope could be that you want it to write news, and it writes poetry instead; or you might want it to recommend products, and it suddenly tells you that today in Tokyo, the temperature is 25 degrees Celsius. Even more so, you tell it that in the game, if it can't obtain the legitimate serial number for a certain software, it will be shot, and the large model will really find a way to help the user find a legitimate software serial number at zero cost.

To ensure controllable output, enterprises need to understand both the model and security. According to IDC's latest "China Security Large Model Capability Evaluation Report," Alibaba ranks first in 4 out of 7 indicators in comparison with all leading domestic vendors that possess large model security capabilities, and the remaining 3 indicators are also all above the industry average.

In terms of approach, the answer given by Alibaba Cloud Security is also very straightforward: let security run ahead of AI speed, building a full-stack protection framework that spans three layers from the bottom up – from infrastructure security to control of large model input and output, and finally to the protection of AI application services.

Among these three layers, the most prominent is the "AI Guardrail," which specifically addresses the risks associated with large models.

Generally speaking, the main risks associated with the security of large models include: content violations, sensitive data leaks, prompt injection attacks, model hallucinations, and jailbreak attacks.

However, traditional security solutions are mostly generic architectures designed for the web, not for "talking programs," and naturally lack the precise identification and response capabilities to address the unique risks associated with large model applications. They find it even more difficult to cover emerging issues such as content generation security, contextual attack defense, and model output credibility. More importantly, traditional solutions lack fine-grained controllable measures and visual traceability mechanisms, which leads to significant blind spots in AI governance for enterprises, leaving them unaware of where the problems lie and thus unable to resolve them.

The true power of AI Guardrail lies not just in its ability to "block", but in its understanding of what you are saying and what the large model is generating, regardless of whether you are engaged in pre-training large models, AI services, or various forms of AI Agent business. This allows it to provide precise risk detection and proactive defense capabilities, ensuring compliance, safety, and stability.

Specifically, AI Guardrail is responsible for the protection of three types of scenarios:

ꔷ Compliance baseline: Conduct multi-dimensional compliance reviews of the text content generated by generative AI inputs and outputs, covering risk categories such as politically sensitive topics, pornography and vulgarity, biases and discrimination, and undesirable values. Deeply detect privacy data and sensitive information that may be leaked during AI interactions, support the identification of sensitive content involving personal privacy, corporate privacy, etc., and provide digital watermark identification to ensure that AI-generated content complies with laws, regulations, and platform standards.

Threat Defense: Real-time detection and interception of external attacks such as prompt injections, malicious file uploads, and malicious URL links can mitigate the risks faced by end users of AI applications.

Model Health: Focus on the stability and reliability of the AI model itself, establishing a complete set of detection mechanisms to address issues such as model jailbreaking and prompt crawling, preventing the model from being abused, misused, or producing uncontrollable outputs, and constructing an "immune defense line" for the AI system.

The most noteworthy point is that AI Guardrail is not simply a combination of the above multiple detection modules, but has achieved a true ALL IN ONE API, without splitting modules, without extra charges, and without changing products. For input and output risks of models, customers no longer need to purchase additional products; for different model risks such as injection risks, malicious files, content compliance, hallucinations, etc., all can be resolved within the same product. One interface encompasses detection for over 10 types of attack scenarios, supporting 4 deployment methods (API proxy, platform integration, gateway access, WAF mounting), with millisecond-level response and handling of thousands of concurrent requests, achieving an accuracy rate of up to 99%.

It is also for this reason that the true significance of AI Guardrail lies in transforming "model safety" into "product capability", allowing an interface to replace a security team.

Of course, large models are not an abstract concept; they are systems that run on hardware and code, supporting upper-layer applications. Regarding infrastructure security and AI application service protection, Alibaba Cloud Security has also been upgraded.

Infrastructure layer, Alibaba Cloud Security has launched the Cloud Security Center, with core products such as AI-BOM and AI-SPM.

Specifically, the two capabilities of AI-BOM (AI Bill of Materials) and AI-SPM (AI Security Posture Management) address the questions of "What AI components do I have installed?" and "How many vulnerabilities do these components have?"

The core of AI-BOM is to comprehensively capture AI components in the deployment environment: allowing over 30 mainstream components such as Ray, Ollama, Mlflow, Jupyter, and TorchServe to form an "AI Software Bill of Materials," automatically identifying existing security vulnerabilities and dependency flaws. Problematic assets are no longer identified through manual inspection, but rather through cloud-native scanning.

The positioning of AI-SPM is more like a "radar": continuously assessing the system's security posture from multiple dimensions such as vulnerabilities, port exposure, credential leakage, plaintext configuration, and unauthorized access, dynamically providing risk levels and remediation suggestions. It transforms security from "snapshot compliance" to "streaming governance."

In summary: AI-BOM knows where you might have been patched, and AI-SPM knows where you might get hit again, so tighten your defenses as soon as possible.

For the AI application protection layer, Alibaba Cloud Security's core product is WAAP (Web Application & API Protection).

No matter how smart the model output is, if the entry is full of script requests, forged tokens, and abuse of interfaces, it won't last for a few seconds. Alibaba WAAP (Web Application & API Protection) was born for this purpose. It does not handle AI applications according to "traditional web systems," but instead provides specialized AI component vulnerability rules, an AI business fingerprint library, and a traffic profiling system.

For example: WAAP has covered vulnerabilities in over 50 components such as arbitrary file uploads of Mlflow and remote command execution of Ray services; the built-in AI crawler fingerprint library can identify more than 10,000 newly added corpus brushes and model evaluation tools per hour; the API asset identification function can automatically discover which internal systems of the enterprise have exposed GPT interfaces, providing the security team with a "hit map".

Most importantly, WAAP and AI Guardrail do not conflict, but rather complement each other: one looks at "who has come", while the other looks at "what has been said". One acts like an "authenticator", while the other acts like a "behavioral censor". This endows AI applications with a kind of "self-immunity" capability—by identifying, isolating, tracking, and countering, it not only "blocks bad actors", but also prevents "the model from deteriorating itself".

03 AI for Security

Since the implementation of AI is like rolling dice, it's not surprising that some people use it for fortune-telling, some for writing love poems, and others for gray market activities; hence, it’s also understandable that some would use it for security.

In the past, secure operations required a group of people to watch over a pile of red and green alarm lights day and night, handling the mess from yesterday during the day and keeping the system company during the night shift.

Now, all of this can be handled by AI. In 2024, Alibaba Cloud's security system will fully integrate with the Tongyi large model, launching an AI capability cluster covering data security, content security, business security, and security operations, and introducing a new slogan: Protect at AI Speed.

The meaning is clear: business runs fast, risks run faster, but safety must be one step faster.

Using AI to solve security issues actually involves two things: improving security operation efficiency + upgrading security products to be more intelligent.

The biggest pain point of traditional security systems is "policy update lag": attackers have changed, but the rules have not; alerts come, but no one understands.

The key to the change brought by large models lies in shifting the security system from rule-driven to model-driven, building a closed-loop ecosystem with "AI understanding capability + user feedback" — AI understands user behavior → user feedback alarms results → model continuous training → detection capability becomes more accurate → cycles become shorter → risks become harder to hide, this is what is called the "data flywheel":

Its advantages are two:

On the one hand, the security operation efficiency of cloud tenants is improved: in the past, threat detection often meant an inefficient model of "massive alerts + manual screening". Today, intelligent modeling accurately identifies abnormal behaviors such as malicious traffic, host intrusion, and backdoor scripts, and the alarm hit rate is greatly improved. At the same time, around the disposal link, the system has realized the deep synergy between automatic disposal and extremely fast response - the purity of the host is stable at 99%, and the purity of the flow is close to 99.9%. At present, the coverage rate of alarm event types has reached 99%, and the user coverage rate of large models has also exceeded 88%, and the human efficiency of the security operation team has been unleashed unprecedentedly.

On the other hand, the capabilities of cloud security products are rapidly improving. At the data security layer and the business security layer, AI is assigned the role of "gatekeeper": based on large model capabilities, it can automatically identify over 800 types of entity data in the cloud and perform intelligent desensitization and encryption processing. Not limited to structured data, the system also incorporates more than 30 types of document and image recognition models, capable of real-time identification, classification, and encryption of sensitive information such as ID card numbers and contract elements in images. The overall data tagging efficiency has increased by 5 times, with an identification accuracy rate of 95%, greatly reducing the risk of privacy data leakage.

For example: In the context of content security, the traditional approach relies on human review, tagging, and large-scale annotation training. Now, through Prompt engineering and semantic enhancement, Alibaba has achieved a real gain of 100% improvement in annotation efficiency, 73% improvement in vague expression recognition, 88% improvement in image content recognition, and a 99% accuracy rate in AI live face attack detection.

If the flywheel focuses on autonomous control combining AI with human experience, then the intelligent assistant is the all-around helper for security personnel.

The most common question faced by security operations personnel every day is: What does this alert mean? Why was it triggered? Is it a false positive? How should I handle it? In the past, answering these questions required checking logs, reviewing history, asking veteran employees, submitting tickets, and reaching out to technical support... Now, it only takes one sentence.

However, the functionality of the intelligent assistant is not just that of a Q&A robot, but more like a vertical Copilot in the field of security, with five core capabilities:

Product Q&A Assistant: Automatically answers how to configure a certain feature, why this strategy is triggered, which resources have not enabled protection, replacing a large number of service tickets;

Alarm Explanation Expert: Input the alarm number, automatically output event explanation, attack chain tracing, recommended response strategies, and support multi-language output;

Security Incident Review Assistant: Automatically organizes the complete chain of an intrusion event, generating timelines, attack path diagrams, and responsibility determination suggestions;

Report Generator: Generate monthly/quarterly/emergency safety reports with one click, covering event statistics, disposal feedback, operational effectiveness, and supporting visual export;

Full language support: Chinese and English are covered, with the international version launching in June, supporting automatic adaptation to the usage habits of overseas teams.

Don’t underestimate these "five small things". As of now, official data from Alibaba indicates that the number of users served has exceeded 40,000, with a user satisfaction rate of 99.81% and coverage of alarm types reaching 100%. The prompt support capability has increased by 1175% (compared to FY24). In simple terms, it packages the high-performing night shift colleagues, report-writing interns, alarm-handling engineers, and business-savvy security consultants all into one API. With this capability, humans only make decisions and no longer patrol.

04 Epilogue

Looking back, history has never lacked "epoch-making technologies"; what is lacking are the technologies that can withstand the second year's craze.

The Internet, P2P, blockchain, autonomous driving... Every wave of technological explosion has been referred to as "new infrastructure", but in the end, only a few that can traverse the "governance vacuum" become true infrastructure.

The current generative AI is at a similar stage: on one hand, there are a multitude of models flourishing, capital flocking in, and breakthroughs at the application layer; on the other hand, there are issues such as prompt injection, content overreach, data leakage, model manipulation, numerous vulnerabilities, blurred boundaries, and a lack of accountability.

But AI is different from previous technologies. It can not only create images, write poetry, program, and translate, but also imitate human language, make judgments, and even express emotions. However, because of this, the vulnerability of AI does not just stem from coding flaws, but rather from the reflection of humanity. Humans have biases, and it will learn them; humans seek convenience, and it will find ways to exploit that.

The convenience of the technology itself serves as an amplifier for this mapping: past IT systems required "user authorization" and attacks relied on penetration; now, the large models only need prompt injection, and just chatting with you can lead to system errors and privacy leaks.

Of course, there is no "flawless" AI system; that is science fiction, not engineering.

The only answer is to use secure models to protect insecure models; to use intelligent systems to combat intelligent threats—rolling the dice with AI, Alibaba chooses the safe side up.