cognitive security
This is a follow-up from my previous post about AI as an economic weapon. If you haven't read it, I suggest that you do before proceeding.
Geoffrey Huntley
I closed that post with the following, and we're going to expand on it:
The real question, however, is trust. I'm not saying that the Chinese models are dodgy. It's more of a meta question. You see, the question of trust extends to the frontier labs as well. As we enter this weird new space where businesses are being automated with AI, it essentially hands over your business's operations to another entity.
Another question on my mind is what happens to a country that lacks AI capabilities? When their businesses depend upon AI, and thus the country's economy depends upon AI, what happens if the spigot ever gets turned off through sanctions or war?
If we zoom forward a couple of years. It's now the future, and the future is undefined because no one really knows what it will look like, because it is the future, but if we play with the notion of extrapolating trends that are just starting right now, where founders, including myself, are building businesses with a mindset of autonomous software and product factories. We will soon be in a place where businesses are highly dependent upon the capabilities of AI, whether that be in access or how the models function.
This concerns me greatly, and it extends far beyond business and into society itself. You see, right now, society is already going through a Harry Potter-style sorting hat event where people are picking and choosing which tribe they belong to.
If you use any of the Frontier Lab models enough, you develop an eye for their tendencies, how they write, how they think, and how they communicate. If people are picking a single AI producer and using that AI daily in their day-to-day life to make decisions, they are outsourcing their cognitive security to someone else.
What concerns me is that, almost three or four years ago, Anthropic conducted research that allowed Frontier Labs to perform laparoscopic keyhole surgery to change how the models perform after they were made. This experiment was called Golden Gate Claude.
In this experiment, Anthropic performed surgery on the model's weight dimensions for the Golden Gate Bridge and no matter what you did when you were having a conversation with this model, the Golden Gate Bridge was always top of mind for the model.
For example, let's say that you wanted to go get some Panadol. It would give you driving instructions to a pharmacy via the Golden Gate Bridge. If you wanted to write a poem, that poem would prominently feature the Golden Gate Bridge.
Through modification of the model weights, the Golden Gate Bridge became a black hole where you could not escape from the gravity of the Golden Gate Bridge.
Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet: https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html
I'm surprised most people don't even know that this research exists and haven't really talked about or even written about what it could mean for society in the future.
Here's a scenario for you to chew on:
What happens when, in a couple of years, if a famous web search company, which was also a frontier AI lab, retires ads in their search product and instead offers their advertisers the ability to bid on the ability to rank higher in model weights than the competitors?
Most people would never know because they've outsourced their cognitive capability to a model.
What happens is if a famous social media network, which is also a Frontier lab, starts allowing their advertisers, to similarly bid to rank higher in the model weights, of their open source models that they're releasing for free.
Now, what happens if you extrapolate this? I don't know if this will actually happen. It's just baseless speculation, but one thing is for sure: we're entering into this world where a select few companies, these frontier labs, will have significant power over the world and how society functions, thinks and operates.
If these topics concern you, then perhaps the only true solution is that you should raise your own model, because by doing so, you gain the ability to protect your own cognitive security, business operations and supply chain.
