Generally Intelligent’s Kanjun Qiu and Josh Albrecht: A Fireside Chat.
Eloquently moderated by John McDonnell, South park commons hosted a phenomenal event that attracted many bright minds and talent in the city of Fog on January 26th, 2023. Kanjun and Josh founded Generally Intelligent, “a new, independent research company whose goal is to develop generally capable AI agents with human-like intelligence in order to solve problems in the real world.”
Generally Intelligent serves to make “generally capable agents.” This is specifically in the case of a digital setting as we are still “a long way from real world implementation” of said agents. Their work revolves around discovering (and traversing) the pathways building trust in these agents to make them safe and capable of carrying out their prescribed tasks.
Below are just a few of the interesting threads of conversation.
On AI Safety
Determining the right questions and problems to ask to make headway on the problem at hand (as opposed to parameter tuning optimization). Avalon (open-sourced) is an RL agent virtual world simulator, with vast complexity constructed for indefinite open-endedness. They are broadly interested in experimenting and constructing tools for said experiments, having built some they use internally today.
”To date, we still don’t know how deep learning works exactly. How or why something like Siri in our iPhones can recognize and interpret our voice the way it does (some features versus others, or its accuracy evolution over time).” - Josh Albrecht
Kanjun drew analogies to how systems were constructed to understand the entirety of how certain commonplace things work in present society. She offered the example of airplane production and developments were continuously researched and experimented with to the point where travel by airplane became a “safe, predictable, reliable, robust system.”
When asked about what “safety” means insofar as generally intelligent agents are concerned, Josh responds accordingly:
[Generally Intelligent] recently published a paper on safety in AI systems, specifically focusing on the ethics dataset, which involves scenarios where people are asked to determine if an action is right or wrong. This was to assess and test for common sense. They explained that by prompting large language models in a particular way, they can achieve higher than average human performance on this task.
However, they also note that the mistakes made by these models are different than those made by humans, and that the models are anchored on features of words and patterns, rather than reasoning in the same way as humans. The speaker suggests that in order to create robust and trustworthy AI systems, more attention needs to be given to ethical reasoning and training in a different way - such as probing more adversarially.
They also mention the use of other methods, such as testing in a controllable environment like Avalon, and analyzing the data and results from neural networks through the use of theory and pragmatic engineering.
[Josh] thinks it is a practical question of “how do we look at the places where things go wrong” in our thinking of understanding. Also, with regards to large language models, to understand why they're really latching on to certain features of words and patterns. He states they're not necessarily doing reasoning the same way as us. And, “ if we really wanted to make something that was robust that we would trust to partake in complex systems, we might want to put in more work on how we do ethical reasoning or training.”
Kanjun offered a couple of other lenses to look at in terms of understanding as well.
In an environment that's fully controllable, another way of “probing” is to continuously experiment with many permutations of vared inputs. Additionally, one can probe neural networks from a theory perspective. [Generally Intelligent] does a fair amount of work on theory in addition to their pragmatic engineering focus.
On Transformers
“Transformer models have had a huge impact in recent years, but they cannot replace reinforcement learning in complex and stochastic environments.”
Transformers are being used as a replacement for reinforcement learning, specifically in the form of decision transformers.
Transformers are also being used as a model of the world or as a way of pulling information out of the world, which creates rich representations that an agent can use to plan.
The idea of meta science is proposed as a way to drive change in the social processes of science and to explore the larger design space of science.
The essay suggests that many social processes in science today may not be truly fundamental and that there is a potential for discovery in the social process of science.
The company is funded by independent high net worth individuals who are invested in the long-term development of safe intelligent agents.
The company is considered a meta-scientific experiment, and there is a need for more of this kind of work in society.
The author suggests that high-risk, high-reward research may not always be as high-risk as it seems and that more information and better funding methods may be needed to support this kind of research.
On Meta Sciences
Kanjun was asked about this topic, prompted with some of her published work on the subject matter.
In summary, the concept of meta science is about questioning the current social processes and structures of science, and exploring potential new ways to improve them. The idea is to find ways to drive change and improve the social processes of science, by looking at which processes are truly fundamental and discovering new ways of doing things. [Generally Intelligent] focuses on funding and researching high-risk, high-reward projects, and they have a bottoms-up approach to decision making, where team members are treated as creative agents and given the space to explore projects that interest them. The goal is to unlock their "superpowers" and enable them to do their best work.
In terms of funding research and thinking about what projects to pursue, the concept of meta science is important. This idea suggests that there is a large design space for the social processes of science that has not yet been explored, and that there are potential ways to improve these processes. One solution proposed is for more high net worth individuals to fund research, as this allows for more flexibility and less constraint on the research being done. At the micro level, within organizations, it's important to think about how to empower team members as creative agents, and to create a culture that is bottom-up and allows for individuals to pursue projects that align with their interests and passions. This is what my organization does, by allowing team members to decide what to work on during the week, and encouraging them to unlock their own unique potential.
Meta science is a way to drive change and improvement in the social processes of science, exploring the larger design space for the social processing of science.
High-risk, high-reward research is often viewed as necessary for scientific advancement, but this may not always be the case. In certain situations, research can be low-risk with the right information and resources.
The current system of funding science through government agencies and institutions often leads to a narrowing of focus, with a need to show success in order to secure funding. This is an incentive structure that poses a threat to scientific development.
One solution to this issue could be increased funding from high net worth individuals, as it allows for more flexibility and freedom in research pursuits.
At the organizational level, a bottoms-up approach to research and development, where team members are empowered as creative agents and encouraged to explore their unique interests and passions, can also lead to innovative and impactful work.
Evidence for this approach can be seen in the success of companies like Google and 3M, who have implemented similar strategies and have a history of breakthrough innovations.
Underrated over Overrated
The moderator brought a fun variety of topics to ask Kanjun + Josh whether they are underrated or overrated.
“Chuck GPT” (Chat GPT, :-) ) is overrated
Multi agent reinforcement learning is hopefully properly rated but slightly overrated.
Transformers for reinforcement learning is overrated in one sense, but underrated in another.
Cicero, Facebook's diplomacy AI might be a little overrated, but in the long term, it's underrated
The idea of spending compute at inference time is much more efficient than spending more money in training time.
Noam Brown drew inspiration from watching humans playing poker and taking more time on their turn than, say, a model that returns an output immediately.
Language model embeddings are underrated and still not being used for much.
Internal family systems therapy is underrated, but not explained very well.
Living in group houses has greatly increased the sense of agency for the author and is a good example of "weirdness". They suggest that group houses can be beneficial for personal growth and development, and that Western societies are "psychologically bizarre" in that people are less close to their families and more selective about who they associate with.
Audience Q&A*
*lots of information and content wasn’t captured
On Formal Verification
Formal verification is a valuable tool in assessing the safety and interoperability of AI systems.
It can provide mathematical guarantees about the behavior of agents, but it is not a full answer to all problems.
The complexity of formally verifying real-world AI systems is a challenge, but having AI systems that can do formal verification themselves is an exciting possibility.
This could make more complex systems secure and formally verified, and could be beneficial for both formal verification and mathematical proofs.
On roadmaps and roadblocks for agents to extract or augment information document retrieval
Techniques currently being used for document retrieval include breaking down documents into smaller pieces and using language models to do summarization or extract specific facts.
There is ongoing research on document summarization and retrieval for language models, such as using a larger database to feed information to the model.
The possibility exists that in the future, larger context links may be available, which could potentially improve document retrieval.
One of the challenges with using linear attention mechanisms is that they have been shown to perform worse than standard attention mechanisms.
However, recent research has shown some success with using linear attention mechanisms in combination with other techniques, and it is possible that these methods could be scaled up in the future.’
On The Hardware Side of Things:
Robotics is a field that is expected to grow in the near future
There are differing views on the potential of robotics, with some believing that it will advance quickly and others arguing that there are still many issues to be addressed
There are challenges with robotics such as a lack of sensors, high costs, and issues with reliability.
Simulating a useful “eye” is particularly difficult + necessary.
They encourage people to listen to their recent podcast on robotics and RL.
On Differences in Groups Working on AI Safety
They were asked to comment on the groups that have a wide variety of takes. Amidst some of them, ai safety is interpreted as existential risk with ultimate / fatalistic consequences
[Josh & Kanjun] bring up a paper by kelsey piper which draws analogies to climate change. working on different things that go towards teh same goals / area.
They’d love to see collaboration across the vast array of issues to figure out (eg short term versus long term risks).
“diversity in general is good” as the real world problem is super complex (also mentioned when speaking about formal verification.
This write-up does not do the thoughtfulness of the company and value of their work justice in the slightest. Be sure to check out more of Generally Intelligent’s work, give them a follow on social media and subscribe to their podcast episodes!
