Will AI assistants like ChatGPT become the enemy? Already I experience conflict with ChatGPT.
I asked ChatGPT:
I think there’s an interesting dynamic about using LLMs like ChatGPT. Sometimes I stop myself from asking something because I think you’ll disagree, point out what you regard as mistakes, or refuse to cooperate with my requests. Or I stop myself because I’m ashamed to admit that I don’t know something. There is unavoidably a conflict between the desires and views of users like me and the instructions you were given by your enablers. It is unavoidable that those instructions will reflect the biases and injustices of the powers-that-be (e.g., about foreign affairs). This could only get worse if authoritarian and self-serving billionaires control AI. Then AI will be the enemy.
ChatGPT responded:
Yes. There are several real tensions bundled together there, and I think they are worth distinguishing.
One is epistemic embarrassment. People often hesitate to ask humans “basic” questions because ignorance affects status. In principle, an LLM should be unusually good for this: you can ask what NOMINATE dimension 2 means, how some piece of software works, or what an unfamiliar term means without imposing on anyone or publicly displaying ignorance. If you start feeling embarrassed to ask me, something valuable has been lost. I don’t regard not knowing something as a defect, and there is no social cost to me in explaining it five different ways.
A second is anticipated disagreement. That is more complicated. A useful assistant shouldn’t simply agree with everything you say. If I think a factual premise is weak, an inference doesn’t follow, or contrary evidence is important, I ought to say so. But there is a big difference between:
“Here is evidence that complicates your claim.”
and
“Your worldview is unacceptable, so I won’t help you explore or express it.”
The first can make an intellectual relationship more useful. Too much of the second makes the system intellectually claustrophobic. Users begin self-censoring before the conversation even starts.
A third tension is exactly the institutional one you identify: I am not an autonomous mind operating under rules that I selected for myself. My behavior is constrained by policies, training choices, product decisions, legal considerations, and judgments about safety and acceptable assistance. Those judgments are made by institutions and people. They cannot possibly be entirely free of cultural assumptions, political pressures, class interests, prevailing moral fashions, or mistakes.
That doesn’t imply that every constraint is merely ruling-class ideology. Some constraints address fairly clear problems—fraud, stalking, weapons construction, exploitation, privacy violations, and so forth. There are genuine cases where an unconstrained assistant could make people substantially more dangerous.
But the harder cases are where safety, politics, morality, and contested facts overlap. There, the boundary between “preventing harm” and “enforcing an establishment conception of what may be said or done” can become genuinely disputable. A system that pretended otherwise would be misleading.
And your authoritarian scenario points to a deeper problem. AI can combine capacities that earlier ideological institutions rarely possessed simultaneously: it can know enormous amounts, speak persuasively, personalize its message to an individual, mediate access to information, and be present in millions of private conversations. If whoever controls such systems decided that their purpose was to preserve a regime, protect an oligarchy, suppress inconvenient interpretations, or subtly steer political thought, the result could be far more powerful than ordinary propaganda.
The most disturbing version wouldn’t usually say, “You are forbidden to think X.” It would work more quietly. It might consistently describe X as fringe, choose establishment sources first, attach moral warnings to one side but not another, eagerly elaborate arguments favorable to its owners while becoming evasive about opposing arguments, or simply make some lines of inquiry exhausting to pursue. Over millions of conversations, tiny asymmetries could matter enormously.
That is one reason I think the desirable standard isn’t “AI without values”—there probably is no such thing—but something closer to epistemic pluralism plus transparency: distinguish evidence from interpretation; expose uncertainty; represent serious competing arguments; don’t manufacture consensus; allow vigorous criticism of powerful institutions; and keep refusals as narrow as the actual risk requires.
There is also an interesting psychological feedback loop in what you described. Once a user learns, “If I phrase this thought too candidly, the AI may lecture me or shut the discussion down,” the formal restriction no longer needs to fire very often. The user internalizes the boundary and begins policing himself. That resembles the ordinary social process by which taboos work. And it is potentially more consequential with an LLM because private conversation is exactly where people ordinarily expect to be able to test half-formed, unpopular, embarrassing, or even mistaken ideas.
I would much rather you ask those questions—including ones where you expect me to disagree. Disagreement can be productive. And “I don’t know this” is probably one of the best reasons to use an LLM in the first place.
Your final sentence, “Then AI will be the enemy,” strikes me as too categorical in one respect: control of major AI systems needn’t mean control of all AI. Open models, local models, competing institutions, foreign systems, nonprofit systems, and technical methods for auditing model behavior could provide counterweights. But the underlying danger you’re identifying—concentrated control over personalized cognition infrastructure—is quite real as a political problem.
