Meta AI: Agents, more GPUs and a new infrastructure come before AGI

Naila Murray talked about AI at Metas "Open Innovation AI Symposium". Murray is a Director at AI Research at Meta in London. She workes in the field of AI for more than ten years. In her current role at Meta AI, Dr. Murray leads an interdisciplinary research team exploring innovative applications of natural language processing, neural networks, and other advanced AI methods. heise online talked to her at an AI Symposium in Berlin.

Meta AI is open source. Why did Meta decide to go this way?

We take an open science and open innovation approach because for META, we really believe it is the best way. We're a company that is open to the public, that really focuses on helping folks to communicate and be creative in their communication. Because we care a lot about having users being able to express themselves on our platform. It's fundamentally a creative activity. So we want to put the tools out there for folks to experiment with and find new use cases for new ways to express themselves. So that's something that fundamentally requires a more open approach.

Gen AI in its current form is two years old, more or less. So it's really new and it's changing all the time, and new capabilities are changing all the time. So rather than us kind of have to figure the options all on our own, we want to figure this out with our community of users together. AI is a community that always has been very open as a community, because that just makes progress so much faster, and it's a type of research area where openness is more possible than it is in maybe like hard sciences or life sciences, where it's not as easy to share research results, because here we're sharing code, which is very, very easy to do. So we have this opportunity to do it with AI, and then it just accelerates the field much faster to work with a global community of researchers beyond our walls, because we are a big well-resourced company, but we're still finite, and we still do need and can benefit from working in collaboration with folks to just make progress as a whole, as an entire industry, which takes everybody forward.

You just said generative AI is like two years old, but you're in AI for much longer. What did change for you since ChatGPT was released, and there's this big hype, and now everybody's talking about AI.

Let me be clear. Generative AI is very old. I don't want anybody to think that this is two years old, but for sure, there has been a paradigm shift in the last two years. ChatGPT was showing exactly how far a scale can take you. Because we've had even the very same technologies that were used for ChatGPT. They were available for quite a lot of time – most of it. There was research done to get ChatGPT out there, but kind of like the baseline technology was available for a while. Put a lot more compute at it, a lot more data at it, like make the model sizes much bigger, just showed you, wow, if you do that, which seems somewhat simple, at least in theory, but it can get you to what we call these emerging capabilities. Capabilities that you didn't see going in, that just kind of pop out of the fact that these models are much bigger and seen a lot more data.

I think what that changes for me and I think for researchers more generally is really taking scale really seriously and thinking about where else can we scale. So that's one thing. I think we've been really trying to push the limits of scale. I think we're starting to get to the point where we're starting to see those limits, but I think we had to do a lot of work and a lot of important work to actually figure out where those limits are. And I'm excited for that.

Where are those limits? Everybody thinks AI will be everywhere, I am not sure about that. What do you think where the limits are? Where is AI maybe even useless?

There are obvious limits in terms of what people call hallucination or what we like to call confabulation. The fact that these models are basically statistical in nature mean that for them to work properly, they kind of have to really represent the full distribution of data that's out there. And that really means the full distribution of use cases that are out there. That is fundamentally tough to scale.

And that is not how intelligent agents like humans work. I mean, not to say that there can't be other types of intelligence, but at least human intelligence is one where rather than try to basically almost memorize or get a sense of the full distribution of problems we want to handle as intelligent agents. We try to develop models of the world that allow us to just reason, right? Reason from first principles about, I need to solve this problem. I don't need to try to think of a similar problem I've seen. I can just think about how does the world work and try to figure out how to solve a problem from that basis. That is going to be important to get truly intelligent agents. Agents that can solve pretty new problems, or problems where they can't kind of revert to 'I've seen this before, and therefore I know how to solve it'. That is to me, what intelligence means.

And for us to do that, I think we need to have agents that can really fundamentally understand how the world works. I don't think models are fundamentally capable of that.

It was a surprise to me, the extent to which relying on data that already exists can do so many interesting things. But I think maybe we're coming up to the limits of that, but I wouldn't bet that we're there yet. It could be that we have a bit more runway on just consuming more data and being able to solve a lot of tasks that way. But for me, this is one of the key limitations. Once we get to that point where we want to solve a fundamentally new task, I don't think our models are capable of that currently.

And maybe one last thing I'll say is that even in the current paradigm, there are clear limitations on being able to really rely on these models in critical applications where getting something wrong is not an option. So for example, if you want to use these models for things like making decisions about loans, or making decisions about things having to do with the judicial system, where you really don't wanna get this wrong, then I think there have to be humans in the loop.

And the AI agents, can you tell me how far your research already is? Do we have them in like a year or two or 10 years?

I can not say too much about exactly where we are. What I would say is that it's a very active area of research for us. We think that having agents that can operate more autonomously but with a very strong sense of what is our users intent, that's very crucial to have agents that work the way we want. It's something we're very excited about. The timelines are not entirely set yet, but once again, I'd encourage you to kind of just keep your eyes peeled for that, because with everything that we do here, we're gonna continue to have an open approach. Then folks will be able to hear about it and use it as it comes out.

Can you tell me the difference between AI agents and AGI?

What the word agent brings is this notion of being a bit more proactive than the type of AI systems we typically interact with. For example, if you think of something like Llama or Meta AI, that we have in our products, this is very much sort of like waiting for you to kind of come in with a question, with something you want to interact with the AI agent on, and then the AI agent interacts with you in that format. Almost like a chatbot in a lot of ways. So what we like to do is to start having the agent being able to solve more complex tasks like using tools.

Let's say, for example, you ask Meta AI who was the chancellor of Germany last year? Then you would like the agent to be able to figure out, okay, what was last year? What is this year? This is 2024. Last year was 2023. And to use that, you might want to use a calendar. You might want to be able to use a search engine. It is about to figure out: for this complex task, what are the tools I need to use, and then go out there and use them. This notion of proactive problem solving and being able to leverage external tools, for me, is some of the core principles of agents.

And then to the difference to AGI. For me AGI is a very vague term. I try to think of it as agents that are able to solve very novel problems relatively autonomously. And so agents probably gonna be able to solve tasks that are pretty similar once again to tasks that they've kind of seen before. But having them being able leverage tools in a novel way would be what would take that to AGI for me.

Do you see any risks?

Sure, I see a couple. Agents will be more proactive about solving things, figuring out how to solve a problem. We're gonna need to make sure that they understand the intent of the user, so exactly what the user wants to accomplish. That's going to be key. Because you don't want agents having access to very powerful tools without being sure they're gonna use them in the way we would want them to be used. So that's like probably the key thing we need to get really, really right.

No robots taking over the world?

Very much not. If once we get to AGI, even then it's not something that's top of mind for me as a concern, just because, you know, it's very hard to think about risk in the abstract, and this is a very abstract concept right now. What infrastructure would they train on, AGI need chips to work on, there's a lot. To think about in which ways they're risky, requires us to think about exactly how they're implemented, and we have no idea how to implement them. It's just too early for us to really think about that. We should be thinking about it even as an abstract level, but I think we have to be realistic about how far we can go along that path.

You've talked about infrastructure. Can you tell me something about your plans for infrastructure?

We are working on a very large-scale GPU cluster. We've had large-scale clusters before, so I would call this maybe a hyperscale cluster, and we're very excited to work on that, so this is going to happen We'll have hundreds of thousands of GPUs, and hundreds of thousands of the latest generation GPUs. So this is going to be a major effort, but we're a full steam ahead on that effort. So we're excited about that, both for accelerating research, but also for accelerating our products and services. I will be leveraging a lot of this. Not just for generative AI, but GPUs are used for all of our AI services, and as I mentioned, we have a lot of things that are not Gen AI. So this is gonna be critical for just Meta as a company.

(emw)