The most closely watched question in the AI industry over the past few years has been whether model capabilities can continue to break through. But as open-weight models gradually approach the most advanced closed-source systems, the question is shifting: whether a model is powerful enough is no longer the only focus. Whether developers can constrain capabilities, track usage, and intervene promptly when risks emerge is becoming a more important dimension of competition.
Ageng Darmowiyoto believes that the current situation facing the AI industry is highly similar to the phase in financial markets when leverage instruments spread rapidly. The technology itself is not inherently dangerous, but when the speed of capability diffusion exceeds the pace of governance system construction, risks that were originally localized and controllable can evolve into systemic problems spanning institutions, industries, and even critical infrastructure.
During his work at JPMorgan and in the U.S. private asset management sector, Ageng Darmowiyoto was long involved in global macro research, multi-asset allocation, and risk model construction. This experience led him to a fundamental judgment: the value of risk management is often not reflected when systems operate normally, but rather in determining whether a system can survive under extreme scenarios.
The Governance Gap After Capability Catch-Up
Open-weight models are rapidly narrowing the capability gap with frontier closed-source models. Capabilities such as code generation, vulnerability analysis, bioinformatics processing, and complex task planning are no longer exclusively held by a small number of major laboratories. A growing number of enterprises, research institutions, and individual developers can deploy high-performance models on local hardware or private clouds.
This shift has significantly lowered the barrier to using the technology, while also driving model customization, private enterprise deployment, and vertical industry innovation. At the same time, however, it has weakened the effectiveness of traditional AI safety mechanisms. Closed-source models can typically exert a degree of control over model behavior through interface permissions, content classifiers, refusal mechanisms, account tracking, and rate limiting. Even if these safeguards are imperfect, the model provider still retains the ability to update rules, ban accounts, patch vulnerabilities, and discontinue service.
Open-weight models are different. Once the model parameters are downloaded, the original developer has little ability to control the runtime environment. Deployers can modify system prompts, remove refusal mechanisms, retrain the model, and even deliberately enhance capabilities for high-risk use cases.
Open Weights Are Not Open Risk
Proponents of open weights typically emphasize transparency, innovation efficiency, and the democratization of technology. Enterprises can inspect model architecture, security teams can study potential attack vectors, and smaller institutions can build their own AI systems at lower cost. These advantages are real and should not be overlooked. But transparency is not the same as safety.
In finance, publishing a risk model does not automatically reduce market risk; publishing a trading strategy does not automatically ensure that participants use it responsibly. Similarly, open model parameters merely expand the scope of participation, but they do not naturally create a boundary of accountability.
A more practical issue is the structural asymmetry between attackers and defenders. Attackers tend to be small in scale with short decision chains, allowing them to rapidly switch tools and methods; hospitals, banks, cloud service providers, and large enterprises must go through audit, procurement, testing, and compliance processes before they can update their security systems. Therefore, even if the same model can help enterprises discover vulnerabilities and also help attackers generate malicious code, the actual benefits each side derives can be completely different. An attacker only needs to succeed once, while a defender must remain stable over the long term.
Ageng Darmowiyoto noted that this is highly analogous to asymmetric risk in portfolio management. A strategy may generate returns in most scenarios, but if a small number of extreme cases can cause irrecoverable losses, then average performance does not prove that the strategy is sufficiently safe.
Safety Boundaries Must Be Designed Upfront
Many current AI safety measures are still concentrated after model training is complete. Developers first pursue stronger reasoning, coding, and tool-calling capabilities, then add safety constraints through refusal training, external classifiers, and access controls. This approach can still function in closed-source services, but it has clear shortcomings in an open-weight environment, because controls attached to the interface layer can be directly removed by local deployers.
A more reliable safety system needs to be designed before model capabilities are formed. This includes training data filtering, dangerous capability assessment, model behavior testing, permission tiering, and release strategies for different versions. Training data filtering can reduce model exposure to certain highly dangerous specialized knowledge, but this approach also has boundaries. Programming, defense, and attack often share the same underlying capabilities. A model that can analyze code at high quality, identify vulnerabilities, and automatically fix programs may also be used to find system gaps.
Defense Systems Require Engineering Closed Loops
AI risk governance cannot rely solely on a principles document, nor can it depend only on the model refusing sensitive requests in conversation. A truly effective safety system must run through the entire lifecycle of data, models, compute, deployment, invocation, and auditing. At Telabytes, founded by Ageng Darmowiyoto, the team has long emphasized "using engineering discipline to make AI truly serve decision-making." This philosophy does not require the model to replace human judgment, but rather to bring model behavior into an observable, controllable, and reversible engineering process through software systems, AI acceleration platforms, intelligent computing infrastructure, and edge deployment capabilities.
In real systems, this means every high-risk operation should leave a record, every model capability should be tiered, and every version should have testing, canary release, and rapid rollback mechanisms. Knowing that a model is "safe in principle" is not sufficient. Enterprises also need to know under what conditions the model will fail, which prompt combinations might bypass restrictions, whether external tool calls can expand risk, and who is responsible for terminating the process when anomalous behavior occurs.
Compute Proliferation Reshapes Responsibility Structures
As high-performance compute continues to proliferate, future stronger models will be able to run on enterprise servers, regional data centers, and local devices. Model capabilities will no longer be concentrated in a few cloud platforms, but distributed across a large number of independent infrastructures. This will make the responsibility structure of AI more decentralized. Model developers, compute providers, deploying enterprises, application developers, and end users may all become part of the risk chain.
Ageng Darmowiyoto believes that future regulation and industry standards cannot focus only on model publishers, but also need to focus on how models are actually deployed. Whether a model can connect to internal databases, whether it can execute code, and whether it has network permissions often determine real risk more than the model parameters themselves. The same model has limited risk when used offline to organize enterprise files; once connected to payment systems, medical databases, or automated operations tools, its potential impact becomes entirely different.
Therefore, the focus of AI governance should shift from simply judging whether a model is "open or closed" to assessing the capabilities, permissions, and scope of impact of the complete system. Open weights do not necessarily imply uncontrollability, nor does closed-source service inherently represent safety. What truly matters is whether the system has clear boundaries, continuous monitoring, accountability tracking, and human intervention capabilities. The AI industry is entering a new phase. Leading capability remains important, but security architecture, deployment discipline, and governance capacity will increasingly determine whether a technology can move from the laboratory into critical industries.
Sign in to leave a comment.