Article reprint source: AI Trends
Original source: Quantum Bit
Image source: Generated by Unbounded AI
Legendary programmer John Carmack and the father of reinforcement learning Richard Sutton have joined forces to go all in on AGI.
The goal of demonstrating general artificial intelligence to the public by 2030 is feasible.
And unlike mainstream methods, it does not rely on large model paradigms, but rather pursues real-time online learning.
The two made the announcement at a special event at the University of Alberta’s Machine Intelligence Institute (Amii), where Sutton teaches.
Sutton will join Carmack’s AI startup, Keen Technologies, while maintaining his teaching position at Alberta.
Both acknowledged at the event that Keen Technologies’ team is small compared to larger companies with hundreds or thousands of employees.
It is still in its early stages, and the company's entire technical team is on site——
There are only four people standing.
Its financing scale is only 20 million US dollars, which cannot be compared with OpenAI and Anthropic, which have raised billions of dollars.
But they believe that ultimately the source code of AGI will be at a level that can be written by one person, and may only be tens of thousands of lines.
Moreover, the AI field is currently at a special moment with the greatest leverage effect, and small teams also have the opportunity to make great contributions.
Legendary programmer and father of reinforcement learning
Carmack's legendary experience, from developing the world's first 3D game, to transforming into rocket manufacturing, to joining Oculus and becoming a key figure in Meta VR, is already well known.
Later he became involved with AI and OpenAI.
He revealed in another interview that Sam Altman had invited him to join OpenAI, believing that he could play an important role in system optimization.
But Carmack thought he had no understanding of modern AI in the machine learning paradigm, so he did not agree.
This became an opportunity for him to begin to understand AI.
He asked OpenAI's chief scientist Ilya Sutskever for a list of must-read articles for beginners, taught himself from scratch, and first gained a basic understanding of traditional machine learning algorithms.
When he had some free time and planned to continue to dabble in deep learning, he took on a one-week programming challenge:
Print out a few of LeCun's classic papers, practice them without an Internet connection, and start by deriving the back-propagation formula.
After a week, he came out of the retreat with a hand-crafted convolutional neural network in C++, without the help of modern deep learning frameworks on Python.
I can only say that I admire the great god.
At this time, his main job was still researching VR at Oculus, a subsidiary of Facebook (later renamed Meta), and he led the team to launch products such as Oculus Go and Quest.
However, during this process, conflicts and disagreements gradually arose between him and the company's management. He believed that the company was inefficient and publicly expressed his dissatisfaction.
In 2019, he resigned from his position as Oculus CTO and became a "consultant CTO", and began to turn more of his attention to AI.
In August 2022, he announced that his new AI startup Keen Technologies had raised $20 million in funding from investors including Sequoia Capital, former GitHub CEO Nat Friedman, and others.
Later he also revealed that he could actually come up with a mere 20 million US dollars.
But taking money from others gives him a sense of crisis and urgency, and a stronger determination to get things done.
At the end of 2022, he officially left Meta and regarded VR as a past stage of life, turning completely to AI.
In addition to this obvious main line, Carmack also has some inexplicable connection with AI.
His 3D games stimulated the demand for graphics computing, and GPU also began to grow and develop in the gaming field.
It is the computing power of GPUs that supports the explosion of AI today, and he still feels proud of his contribution when talking about it.
……
Today's other protagonist, Sutton, is also a legendary figure.
He is known as the father of reinforcement learning, has made important contributions to methods such as reinforcement temporal difference learning and policy gradient, and is also the co-author of the standard textbook on reinforcement learning.
In 2017, he joined DeepMind as a distinguished scientist and participated in the AlphaGo series of research. His student David Silver was one of the main persons in charge of AlphaGo.
Sutton wrote a famous essay, The Bitter Lesson, in which he argued that trying to teach human experience to AI is not feasible. All breakthroughs so far have relied on improvements in computing power, and continuing to utilize the scale effect of computing power is the right path.
Before the two formally communicated, Carmack had expressed his concern and approval of the article.
But the only direct communication between the two was initiated by Sutton.
A few months ago, Carmack received an email from Sutton after announcing funding for his AGI startup.
Sutton wanted to ask him whether he should take the purely academic, commercial or non-profit route in his research.
However, in subsequent email exchanges, the two found that there was a surprising consistency in their AI research directions and concepts, and gradually established a cooperative relationship.
Specifically, the two reached four consensuses:
They all believe that the current development of AGI is limited to a few narrow directions, relying too much on big data and computing power while ignoring innovation.
Both believe that early commercialization will hinder the development of AGI
Everyone believes that AGI will not be too complicated in the end, and one person will be able to master all the principles and even write the main code.
Both believe that the emergence of an AGI prototype by 2030 is a feasible goal.
Don’t just rely on big models, small teams also have opportunities
It was a very bold goal, and the audience thought so too.
When asked "how can a small team accomplish such a grand goal?", Carmack believes that the amount of data and computing power required to achieve AGI may not be as large as imagined.
A video of what a human sees in a whole year at 30 frames per second can be put into a thumb-sized USB drive. A one-year-old child only has this much experience data, but has already shown obvious intelligence. If the algorithm is right, there is no need to use the entire Internet's data for AGI to learn.
He also uses this intuitive way of thinking to consider computing power requirements: the computing power of the human brain is also limited, far from the level of a large computing cluster.
It is larger than a server node and larger than a rack, but only an order of magnitude larger at most.
And as time goes by, the algorithms will become more efficient and the computing power required will continue to decrease.
If there's one thing Carmack's seemingly disparate work in, 3D gaming, rocketry, and VR, has in common, it's the optimization of large, real-time feedback systems.
This is also what Sam Altman saw when he invited him to join OpenAI.
The AGI architecture he envisions should be modular and distributed, rather than a huge centralized model.
Learning should also be continuous online learning, rather than the current pre-training where most parameters are no longer updated.
My bottom line is that if a system can't run at 30 Hz, which means updating every 33 milliseconds or so while training, I won't use it.
He further stated that as a low-level system programmer who can write original Cuda code and manage network communications by himself, he may be able to do some work that others would not even consider.
It is not even limited to the existing deep learning framework, but will try more efficient network architecture and computing methods.
The overall goal is to simulate a virtual agent with intrinsic motivation and continuous learning ability to continuously learn in a virtual environment.
No robots, because his experience building rockets had taught him that the fewer physical objects he had to deal with, the better.
Compared to Carmack, who has just started working on AGI, Sutton has spent decades on this issue and has a more specific research plan.
Although not much was said at the event, the main part has been written up in an arXiv paper in the form of “Project Alberta”.
Project Alberta proposes a unified agent framework that emphasizes general experience rather than specific training sets, focuses on temporal consistency, prioritizes methods that scale with computing power, and multi-agent interaction.
A 12-step roadmap was also proposed.
The first 6 steps focus on designing a model-free continuous learning method, and the last 6 steps introduce environment models and planning.
The final step is called Intelligence Amplification, where one agent can use what it has learned to amplify and enhance the actions, perceptions, and cognition of another agent based on some general principles.
Sutton sees this kind of enhancement as an essential component to realizing the full potential of AI.
In this process, determining indicators to evaluate AI progress is very important but also very difficult, and the team is exploring different developments.
In addition, Carmack has always been an advocate of open source, but on the issue of AGI, he said that he would maintain a certain degree of openness, but would not disclose all the details of the algorithm.
As a small team, Carmack believes that it is necessary to maintain a pioneering spirit and focus on long-term development rather than short-term benefits.
Commercialization will not be considered prematurely, and there is no intermediate form like ChatGPT that can be publicly released.
As for what can be achieved by 2030, Carmack believes that "AGI can be demonstrated to the public", while Sutton said that "AI prototypes can show signs of life."
2030 is a critical year
This is not the first time that 2030 and AGI have appeared at the same time.
The top AI teams all unanimously regard around 2030 as the key node for achieving AGI.
For example, OpenAI wrote in its announcement that it would set aside 20% of its total computing power to establish a superintelligence alignment department that it believes superintelligence will arrive in this decade.
Even the investment community has similar views. Masayoshi Son just presented such a PPT at the SoftBank World Enterprise Conference.
Aside from OpenAI and Keen Technologies, there aren’t many organizations working on developing AGI.
OpenAI's biggest competitor, Anthropic, which just received $4 billion in financing, its CEO Dario Amodei mentioned in a recent interview that within two to three years, AI will be able to behave like a well-educated human.
When Transformer authors Vaswani and Palmer left Google, they founded AdeptAI, whose goal was also to create general intelligence.
However, the two suddenly left the company earlier this year, leaving only David Luan (far right) among the co-founders.
The two Transformer authors founded another company, Essential AI. The vision of this company is not so "starry-eyed", but more pragmatic, focusing on the commercialization of large models.
There are also not many domestic companies that have explicitly stated their AGI goals, mainly MiniMax and Dark Side of the Moon, a company newly founded by Yang Zhilin.
Reference links:[1]https://www.amii.ca/latest-from-amii/john-carmack-and-rich-sutton-agi/[2]https://www.youtube.com/watch?v=uTMtGT1RjlY[3]https://arxiv.org/abs/2208.11173