Enterprise AI Agents with Jun Qian, VP of Generative AI Services at Oracle

Richie Cotton: Hi Jun. Welcome to the show. 

Jun Qian: Hi Richie. Nice to meet you. 

Richie Cotton: At the moment, I think AI agents are the hottest topic around. So first of all, can you just tell me what's the coolest AI agent you've seen so far? 

Jun Qian: Ah, that's a very tricky question. I think you know it. You yield it every single. Go on. What's that?

Yeah, which is chat. CBT. 

Richie Cotton: Okay. We're counting chat GBT as an agent now then. 

Jun Qian: Yes. I think you think about it, right? And everyone have the model access for open end models, but why? And the chat CBT experience is quite different from a lot of agent you saw, right? And then think about right, and go back to Taiwan once and even more than 12 months, right when Chatt P first started.

Missing a lot of features. For example, web search features and the memory features, code generation features, right? So now actually it to generate code, the wrong code, right? So think about it, right? And from the foundation model perspective, right? And we have a lot of good models, right? And so big models and Gemini models, right?

Open-end models or what make the tr d difference. It's agent features. Think about it. And I'm pretty sure you use chat B all the time, but what makes JT p different? I think that's a secret sauce. And also my prediction is like TBD five is not a model, it's a agent. 

Richie Cotton: Interesting. Yeah. That would be very cool.

And I certainly right there with you, the just having these simple sort of tool use features to being able to search the web, being able to just interact with python or whatever, code tools, it just makes such a difference to like the quality and the range of things that you can do well with drone tube ai.

Jun Qian: Yeah, and also key thing is memory. Memory is also very important. Because, and you probably noticed, right? And open launch called memory features. And not long ago, right? Three, six months ago, however, they actually using these memory features, but more than six months ago, right? If you go to your she PT settings, you can actually see what they memorized, all the combinations you had with Che g, bt.

And also there are another secret command you can type in Chei called it BIO. We just had a BIO chat, GPT. You can see all your informations they saved with the for. So I think that's one of the secrets, right? And why it makes Chad g BT so relevant. A lot of questions you have, but I think that's agent, right?

So memory to calling that's the key component of the agents. Yeah. 

Richie Cotton: Yeah. The chat gd memory feature is really interesting because I went into my settings, had a look at what he was remembering about me. It thought I was really into ultra running, which is very odd. I'm very sure. Like I will go jogging once a year.

I dunno where it's got the idea from me, from so yeah. You just need to be careful that it is remembering the right things about you. I know at Oracle you've just launched a RAG agent. Talk me through what is that and what's it for? 

Jun Qian: Correct. Correct. I think everybody knows rag right?

And we don't need to extent what is a RAG and how, however, I think why it makes re so popular if we ask these questions. For example, rich, I think, the re concept. From your perspective, what do you think? Why is re suddenly too popular? And remember about it eight years back, right? We had chatbot, right?

And everyone's building chatbot, but now everyone is building rag based chatbot. But what is the difference between the chatbot we built eight years ago now, the rack, but I'll give you my answer, right? Because eight years ago I was at Microsoft, right? We're building this chat bot. For enterprise customers, right At that time, we built chat bot for Macy, for hp.

And what is the key challenge at that moment when we build a chat bot in the high level, there's two technologies, right when you build a chat bot. First one is we call internet understanding. When you ask the questions. And then the model need, understand what kind of questions you're asking. We call the in implant classifier.

It is a very generic technology, right? Just like Alexa Asur, they all have similar technology, right? You ask the questions, they try to identify what is your gene type in the high you can think about as a classification problems. So you can have 100 inten, 200 inten, right? So that's the first step, right?

When you ask the questions, the model, try to understand what's your in intent. The second part is once you understand the intent and will try to give you answer. So essentially you will hardcode all these answers related to this intent. Again, I'm trying to simplify the product and now you have this intent classification and the knowledge retrieval two part view, this chat bot.

But what's the problem? The problem is most of the chat bot cannot follow up your questions. Because they don't understand the context as that one challenge. And the second challenge is because your knowledge base has to keep updated all the time, right? Because once you have a new product launch, then you have updated your answers.

So this ongoing knowledge base maintenance become a challenge. And also when you ask the questions for example, you ask, okay, what's what's your open hours in Christmas time for me? And then you have to understand all this details, right? Daytime location, right? We call it slow failures, right?

And you need to understand all the details, right? We can use a classic NRP technology, right? Processing all the data. This is a classic NRP technology, right? Eight years ago. Internet understanding, right? And knowledge building, right? And that's one of the parts. However, I think the challenge is.

This kind of chat bot or you can think about Sri and Alex as well. It's silly, right? You can only answer specific question you have. You cannot really have a dialogue, so that's why the chat bot, I get a popular, solve certain problem, but didn't really go too far. But now. In this large language model world, you see how this large language model will help to solve the problem.

You don't need internal classification anymore, essentially, right? Because this the language model understand your language really well. And the secondly, all this large language model can very easily understand your follow-up questions. You don't need another, like a guess, right? So basically the building, natural language understanding of the large language model is so advanced.

So we just solved some of the critical problem. So you don't have any language understanding challenge anymore. And however, the second part, you should still exist because the re system, right? The re system, right? And so we still need to keep updating. All the knowledge base or the database you have, keep all the knowledge.

I think that part is still existing today, no matter what kind of mplementation you use. Or do you keep updating your knowledge base or your database to reflect the question you're asking? I think there's still one of the challenge we're dealing with today. That's from the high level.

Richie Cotton: Absolutely. And it sounds very simple, but just the idea of having chat bots that your customers don't hate, that's just absolutely huge for so many organizations, I think. 

Jun Qian: Exactly. Exactly. Yeah. I think one of the success use case we have because once we have the right system and even we only have the generation model you can sing, A lot of teams started building their own racks.

Because on high level, right? And very simple knowledge, right? And if you have already have a database, for example, if you already have your data, you open search, right? You can simply do the, you the query, right? To the search to the open search database, right? And then you got the search without back.

You do rerun. You do generation or everybody understand, right? Even very simple few steps you can already generate. Very useful. System. Of course, right? We need to do other issues, right? And knowledge updates, hallucination, however, just introduce a very simple right concept and already solving a lot of issues, right?

For example, right? And inside Oracle, right? We have this Slack channel, right? Essentially can ask for strengths when you run into issue, oh, I cannot log in my laptop. I have the. That string issues right before we have the human support team right to open ticket for each question you ask.

Now it's just like the. Automatic chat bot, right? Rag bot. Just answer the question. Do actually give you the answers and the instructions. But I think it's just like a very simple implementation rag in the support system can see a lot of the effort, right? And support effort any enterprise, right? I think that's a low hanging food, right?

To build a rank system for support. I think right. That is a very basic starting point, and you can achieve the math results immediately. 

Richie Cotton: Yeah. I like the with these sort of large language models, okay. You solve the sort of natural language understanding. You've solved a lot of the issues around context with rag.

It's like you've got accuracy from your knowledge base. But you mentioned the big challenge before that, keeping knowledge up to date and making sure you're retrieving exactly the right information. In a scalable way, that seems to be the big problem. So it sounds like knowledge management, data management, these are the big challenges.

Do you have any advice for organizations on how they can manage all this data for their agents, for their chat bots? 

Jun Qian: This actually go back to the traditional enterprise search problem. So let's say forget rack, right? And you just wanna build the enterprise search. Because search the same thing, right?

You want the latest information. So essentially you can think about, okay, yes we're not born the ocean here. If you already have very good enterprise search system, right? For example, like a lot of teams using open search, if you already have your knowledge base and data in the open search, which you already keep refreshing regularly.

Essentially, you already have a good knowledge base to start with, and then you can build the rag layer on top of it, right? So basically you have the query rewriting and the re rank and the generation. So you already build a very decent rack system to start with. But again, that's not the end. But, so essentially you already have a real time rag system with your existing knowledge base management.

But I think that's actually one of the interesting challenge in the enterprises because when you saw a lot of the open source POC implementation site, it sounds very easy, right? You can easily build a POC. And you can easily show the result, however, right? Once you get enterprise customers, you will find the challenge because a lot of enter enterprise system, you, they already have a knowledge base, right?

Clearly it's a lot of effort. If you wanna rebuild the knowledge base, for example, using back the dbs, that will be huge effort. It's a lot of challenge. However, you can leverage your existing knowledge page systems, right? You don't have to rebuild everything from scratch. For example, in this case, if a customer already have, let's say they already have an open search base knowledge base, you can easily add another layer of the rag on top of it, right?

Build a decently enough re system. And however, your data, refreshing data management system can be as the same as they have today. You don't have to redo everything. So this is one possible way, right to. Moves rag system forward quickly to have the team enterprise systems right to adopt the rag concept to start with.

Richie Cotton: Okay. So it sounds if you've got really good enterprise like knowledge management, then creating the, that rag layer, like it's just embed everything thrown into effective database, that's probably gonna be give you a decent start. There was a big if though. That's if you've got the good enterprise management system.

Jun Qian: Exactly. So that's actually get the second part. If you're just nine ly and oh, okay, plug into this open search, right? I send a query, get a result back. I do a rerun generation. I'm done. Clearly it's not, I give a example, right? We started, I said yeah, that sounds very straightforward.

And then we run the issues. Why? Because the traditional search results, right? And if you don't do specifically chunking, it could return the whole documents. So you run a query and the search system get a return, the whole documents, which is not useful. For the further processing and which, because once we get the results back and we want do the rewrite for example, to marry, which result is back, however, you just return the whole documents back, it will helpful.

So that way then you have to think about, oh, okay, our data ingestion pipeline actually needs a change. In this case, when you ingest data into the open search, then you have to change it, right? Instead of send the whole document as a single chart, then you wanna jump into smaller pieces. And then for each smaller pieces, right?

And for example, open search, they already have the building in embed max, right? So basically now you get into the second part, right? And how can you modify your existing knowledge management system? How do you. Modify your existing data ingestion process to make it. Better for the new re system. So I think that's exactly getting to the second phase of the re implementation, right?

So the nine new phase is, oh, I just poke into the existing safe cell and do re ranking generation reflection. That's probably, it's okay start. And then you get, oh, that doesn't work. But some use cases, why? And then you get, oh, actually we need a more smarter trunking algorithms, right? So you get a second part of the problem.

And then eventually you have get into the more concrete problem. For example, when you est data, if you have some graphs image, the existing data ingesting pipeline may not work because you don't really process in all the image data. Then you get the next base of the RE systems. Oh, actually we need to do this multimodality ingestion and the generation, right?

So I think that's why I like your podcast, right? With that guy for the rec 2.0, he is very practical. I don't remember his name. Yeah. Because once you start building the system layer by layer problem, then you get the deep and the deep. All these issues. However, if you just read a paper, read some blog, you don't build it, you won't be able to fully understand what are the challenges there.

Clearly, opportunity is huge, but if you don't. Build it in real. You won't understand all these challenging features we're facing in the real enterprise environment. Yeah. 

Richie Cotton: Okay. Yeah. That was Dawa. Kila won the advantages of Raku, who was on the show recently talking about the place innovations in retrieval augmented generation.

But yeah, so that's interesting. It seems like then you've got a very iterative process, so you're gonna start off, you're gonna have some kind of chat bot, and then there's gonna be a lot of. A lot of tuning goes on, just like figuring out all the agro algorithm stuff to make it work properly.

So you covered the technical side. I'm curious as to who's involved like on the talent side, like which teams or roles tend to be involved in this? 

Jun Qian: Oh, yeah, that's a great question. Yeah, I think Jen is speaking right and you need. Data scientists, right? Engineering. Yeah. For example, the rack system, right?

And it's, of course you need the engineering team to do all the integration work. However, I think the data scientist is super critical, right? Because once you build a re system how good it is, right? You need a measure, right? So essentially you need to marry not the single system. You want to marry the end-to-end accuracies and recourse.

This is typical data scientists work, right? We propel the sample questions, right? We know the ground truth, right? You want to build the end eye evaluation dataset to start with, to systematically measure the improvement of your rec system. So I think that's, is, that's the one key part is right.

Not only build direct system, don't matter of course, right? You have a human right and testing to marry yourself. However, you should have, you should build your evaluation data set. You should have a data scientist and measure this right system magically and continue to improve your evaluation data set as well.

You should add more and more questions, right? More and more. Answers are continuously to, to measure and they improve your csap. 

Richie Cotton: Okay. Measuring just incredibly important. I agree that's very much data scientists work, but what sort of metrics will you use to see? Does the system work on that? Especially if it's non-deterministic, so you're gonna get different site, different results each time.

Talk me through it. 

Jun Qian: Yeah, definitely. I think that Linda, in the support. Scenario that we have one key metric called deflection rate. So this is very standard across industry, even eight years ago, right when I was in Microsoft, right? And this deflection rate is the the question, the bulk can answer and result, need a human involvement.

For example, right when we started the factory rate, the 20% is pretty good, right? Essentially 20% of question you can answer by bot accurately without talking to cma. That's great. 20% is pretty higher. Years ago, right when we started this, so wow. You can solve 20% of the question. That's awesome. But nowadays, and for example, the, and the rack system support system we built, we couldn't even reach 80% infection rate.

So that's a dramatically improved, right? And for example, the experiment had eight years ago. Right now we can from like 2015, 20% to 70, 80%. That's the dramatic difference. Again, that's just one of the metric you could measure like a deflection rate. And we can use as one, as a proxy of the. And accuracy of the system.

Richie Cotton: Okay. That seems like a pretty useful measure. And I guess, do you need is it gonna be lodging around like end user measures? This person found the answer to the question they wanted, or like they clicked on the first results? Or if it feels like there's gonna be, you need like a whole suite of different measures or different stages of Oh yeah.

Jun Qian: This is one simple, for example, for any chat bot system, right? And once you once you provide answers. And if you don't have further questions and they just create a session. However, if they continue to have questions and there always a button there, right? You can kick talk to human, for example, right?

I think every single third bot system has that functions, right? I think that's a good proxy. So if the customer don't click button right and talk to the humans, I think that's one very strong signal. And also because when we change our answers that you also have the links. And if a customer click links, right?

That also very strong signal that means, oh, okay, you provided the useful information. Once they click link. And can find the answers they need. So yeah. But I think all the chat bot system will build like Microsoft and Oracle, right? And also and on this market. And this signal is very clear.

Talk to fma. As obviously though, 

Richie Cotton: oh yeah, that's true. It is oh, this jab was annoying. I wanna speak to you, and that's a sign that your product is not working. That's yeah very clear, very straightforward, but very useful. Okay. Actually that's often my default behavior, so I, because straight to you. So I'd like to talk about some of the issues about adopting, chatbots and agents within the enterprise. I guess it seems data privacy, data security are top of mind in terms of the challenges. Can you talk me through what people should be worried about with these things?

Jun Qian: Yeah, I think this is one of actually the advantage for the existing enterprise systems, right? Including Oracle, Microsoft as traditionally, they're very strong on data privacy and the securities, right? Yeah. Same thing. I remember you you talked with the entitlement, right? In the previous podcast.

I think that's actually, it's very true, right? Because when you build a new rec system and I think one of the key challenge is how to integrate with existing enterprise system, including entitlement and authorizations, right? Because certain information is only available for certain customers.

You cannot ly index all your knowledge base, right? And then just throw out, right? I think you probably saw some of the use, right? And some, the, some company build some rec system and now every single employee you get the same information as a CEO. I think the key thing is right and security and the data privacy compliance is super, super important.

Okay? So when you design your system, you had have this in your mind. How do you in integrate with your existing security and authentication? Also sensing system, which is actually is very tricky. Because if you don't design your system carefully, including if you just stumbling index everything, I think it'll be very tricky.

But the good news is a lot of the enterprise providers in including Oracle, like Microsoft, they already have very strong security and compliance policies. I think for for us, right? If we start building the new. Agents or the Genta Air Solutions, we work very closely with all the security, legal and compliance teams right within the automation and leverage their existing experience.

I think that is the key things that collaborated with your existing security compliance and the data privacy group. Every single solution we build, we always go through the security compliance reviews. With the internal team, so make sure the solution we build is fully compliant and secure with your existing policies.

You don't have to invent new things, but make sure the thing you build is right, secure, and compliant with your existing. Policy. I think that's the case thing. 

Richie Cotton: Okay. Yeah in theory the existing sort of enterprise software using, hopefully those policies that you've already developed over the years, they're just gonna naturally apply in the AI use cases.

Alright. One thing that's I was having a conversation with One Data Comes customers that they were struggling with. Okay. The technology works, but in terms of the organizational side of things, there's teams wanting to like. Get started with AI and build things as quickly as they can. And then you've got a compliance team that is like, Nope nope, nope.

Because they move a lot slower. Yes. Do you have any sense of like how you can resolve this sort of tension between governance and speed? Yes, 

Jun Qian: exactly. I think that's is our ongoing challenge. However, I think the challenge is also opportunities. I think the first question is why compliance? No. Then you had to go back to, I get the, I get a no every day.

You have to doubt back. So I think the question is you need to educate. For example, two years ago, right when we started, a lot of our compliance legal don't really understand how large the model works, I think. And we spend a lot of time working together with our legal team, compliance team.

It's more like the education as well as learning together. So I think the build a partnership is very important of course. And from the compliance legal process, it's so much risk. From their perspective, they seem, oh, everything is so risk, right? If you model general anything, how do we control this behavior, right?

I think it is more about education, right? And you, and so service for example, right? And we need and say, oh, okay, larger model is itself. It's also software, right? It's a piece of software. It's not like magic things, right? And it is a piece of software. And then how do we control this behavior? How do we monitor the behaviors?

And once they understand the concept and the ones understand how do you control the behaviors of the large language model? And then they will have bad incentives. Actually, sometimes we will provide even more insight, suggestions. For you to solve it as a problem. And now our legal team is now get very knowledgeable, right?

When we start discussing about compliance data of privacy, sometimes they're actually on our side. Try for other people and, okay, this is another legal issue. And as you do the zero data retention you don't keep any data. Yeah. It should be okay. Because we start some discussion with the partners, right?

And then you understand the concept, then they can help you sometimes, right? And to even accelerate some of the process. I think the key thing is really build a partnership, right? And educate. And your team, your legal compliance team, and also work together. It's not easy, right? I spend a lot of times working very closely with it, but now once they have the knowledge once they have the understanding, and then it's become much, much easier.

Of course, certain compliance you still have to do, for example, right? Yeah. We, like lama, when we launch LAMA stream model, so it's multimodality capability we cannot launch in you, right? Because that's just a law. We have to comply with the new law. You cannot use the mortality models honestly, model then the certain law you have to comply.

They just had no way around, but some of other things more about I think the understanding. Collaboration and make sure and we all on the same page. 

Richie Cotton: Okay. Yeah. So I'm definitely in agreement that just educating different teams on what's possible, what should be allowed, what's a good idea or what's a bad idea?

That's really important. And yeah, it sounds that. The trick then is that you've just gotta have some sort of agreement with these compliance teams on general principles for what should be allowed in general, what shouldn't be allowed, and that should resolve most of the sort of use cases just to enable you to get on building stuff.

Jun Qian: Yeah, exactly. I think I, I will put this in the beginning. You probably want put extra effort because from legal complaining, they also want to learn the new technology. Think for example, I build a few POCs for demo to them, right? And have some sessions, work session together with them to tell them what exactly is the knowledge behind the scenes.

I think that could help right them to understand the technology and in the long term, to benefit you to move the project forward more quickly. You don't have to repeat the sum of the discussion again and again. Once we truly understand. And how the large language model get trained, and how the large language model get called in the inference in time once, and they understand all this concept and it'll be much easier. 

Richie Cotton: I just wanna return to something you said a moment ago about not being able to use the LAMA for the large language model from meta in the eu. This is something I, I've not heard about.

Do you wanna talk me through what are the issues there? 

Jun Qian: I think they have policy, right? If you process y image. And there are certain restrictions in certain region, right? You cannot using the image processing features in the EU regions. I remember right when the LAMA three start, right? I think Lama three on three right?

Starts poly image and sending features. However, I think that certain EU rules, right? You cannot process image. I think that's why we had to turn off the image processing features right? In the EU regions, right? We only allow that in the US regions, for example. Yeah. That's one of the very specific rules, so it had to complex.

Richie Cotton: Okay. Yeah, that definitely seems worth taking note of at the point where you're accusing your models and also deciding where things are being deployed. And 

Jun Qian: yeah, I think that, again, that's actually go back to the data privacy and the sovereignty seems right each region and states right, may have their own policies.

For example, I did privacy policies right in Singapore and Australia. And it could be different from the us. You had to really compliance and with your local policies. 

Richie Cotton: Absolutely. Yeah. Always a good idea to, to check all the different local policies around the world. Okay. For organizations who are wanting to get started with these things. We talked about having to put governance in place beforehand. Do you have any advice on what are good first use cases? I dunno whether you can jump like straight to where we're creating some fancy agent or whether like you wanna start with something simpler?

Jun Qian: Oh yeah. I think I mentioned right. I think red actually is really good use case because I think it is a solve a lot of problems right to service. And if you already have the knowledge base built today, right? For example, you already have some in internal search systems and build a rack system on top of it, I think that is very tangible, right?

You can generate very tangible results. And for example, the the use case I gave, right? And support, support the chat bot, click the system, right? I think that's a very tangible things, right? You could could build and show immediate results. Of course. And a lot of other use cases you can see.

But I think so start from support channel. I think that should be a very decent starting point and you can see some tangible results. If you already have some knowledge base system right? And inside your company, I think that should be a very straightforward starting one. 

Richie Cotton: Okay. So yeah start with chatbots and then work forward from there.

I guess at the other extreme of things. So there are a few startups that are trying to have complete. AI employee, so you've got cognition Labs has this Devvin software engineer. This Julius AI has an AI data scientist. EMA is trying to create like a universal AI employee.

Like some really ambitious projects that I'm not quite sure what. E in entirely. Yeah. Is there like a sweet spot for like where you should be aiming with with agents? Yeah, I think that's 

Jun Qian: actually if you think back right, in the past two years, what are the two prominent use case about large model?

I think one probably we already talked about rag and then the second one is really code agent. I don't know. I'm pretty sure that you already try some of the code tools, right? Like a, Windsor right cursor. Yeah. I use Cursor as well. Yeah. In my spell time. So I think that if you using in cursor for long enough right?

Or any other code agent then evolved quickly, right? And so start with is just complete your code and then you can generate the whole code for you. And then they can read your repository, generate more company code, and now they can generate the multi steps. And they say you give a client, they can execute.

So you can you just from, for example, for any of these code generation tool you can see very involving from a very simple task completion to now a fully semi auto coding. Task, right? You can run, for example 20 minutes, right? All the tasks, right? And you can plan the task and run the tasks I think we call a async, right?

And code, right? And the latest, I think the codex and cloudy code, they already went to the, into that direction. So clearly I think a code demolition automation is one trend very clearly and is very productive already. I'm pretty sure a lot of the, developers, if you're not using code generation tools you're probably out.

Yeah. But again, go back to your question, I think. And this is I think there is still just the beginning, right? And we saw the adoptions, internally, we also have the code generation tools. We developed and you know our team, right? We called Oracle Code Assist. And we built our launch, this service and last year, right?

And internally, and we have a lot of teams are using it as well. So I think a code clearly is another trend. You probably, we all know, right? The wind serve news, right? And all this news, right code. I think the code is clearly one winner writing this China AI world right rag cold, clearest winner.

And we'll discover more use case. I think this just started right. And we clearly see some patterns, but I would assume and once we get into deep into the applications, yeah. For example I'm always thinking because I've been doing software more than 30 years, so we're always thinking, right?

Is there any fundamental, like a computer system operating system, right? And even compilers, right? And should we rethink some of the fundamental building block we're doing today, like operating systems, right? And for example, all the common line today we use still a little bit of done, right? You need to find the grab, whatever, right?

And come out some of the information you need from your laptop, for example. However, I think about, right? If you have a natural language interface, you should just say, oh, okay, can you find me the resume I saved in my dis a month ago and with the name, blah, blah, right? And then this smart command line should find data from you.

I don't need to roll crazy, uni command, right? Python information, right? And this could be. Build in very soon. I would assume register should be available very soon as a part of a building your opportunity system, right? For example, when you might get a max. You should already have this very smart command line, right?

You don't need to remember all this unique command anymore, right? It's just there. 

Richie Cotton: That's funny. Yeah. On the coding side, I'm right there with you. If you don't know how to use these sort of AI assisted coding tools, then you're not gonna make it as a software developer these days. I think the point is though, that they're quite easy to use, so it's something that everyone should look into if you're writing code.

But yeah I do like that idea of just. Better software search. There's so many times where yeah I cannot find a file that I'm pretty sure I downloaded or I created like a while ago. Even like we use the Google office suite and yeah, somehow, even though Google is a search company, like Google drives search is absolutely terrible.

Like they're supposed to be, they're supposed to be good with this stuff. 

Jun Qian: Yeah, exactly. Same thing. Dropbox, all these things so think about the software you're using every day. Clearly there's a lot of things they can do better. So that's why if you think about and always get very excited, and basically I'm thinking the software we built in the past 30 years can be rewritten in know, in a dramatic way.

I'll give you another example, right? So we all the web scrapper five years ago when we built this web scrapper, right? Oh, you get a HMR page, right? Then you try to do regular expressions. I do all the things. I try to match a certain. Data view, for example. However, every website is different and then the web that change all time and then you have to change your, web spreading tools.

For example. However this, it's very simple. You just send this to H tool, that language model. You don't h ml send to the area, send, oh, I want this too. Then they will give you one version back. 

Richie Cotton: Oh yeah, I'm with you. The on that, like web scraping is. Such a tedious task to do as a human. It's so annoying.

And yeah, I do like that. I actually, yeah, 

Jun Qian: So actually just I, I like to do all the experimental things like the, can I do two things, right? One thing is, oh, I just don't download the whole, right? I sent to the area. They gimme all the answers back really decent. Another way is actually you take a screenshot center, give you also back too.

So either way it works. Perfect. Nice. 

Richie Cotton: So yeah lots of opportunities then I think for just making software better. So you mentioned the idea of enterprise search and also just like operating system search. Since you've been in this game 30 years, do you have any more predictions on like things that.

Can be improved using AI over the next few years. 

Jun Qian: Yeah. I think, like for example, and if we, this industry long enough, right? For example, 30 years back, right? Yeah. I still remember when I was high school, right? I heard I was AI the expert system. I was so excited. Wow. Expert. See, that sounds great, right?

And then I studying, I remember it's a ProLock, right? That's one of the language, at that time, ProLock right's called this ai language. So I started using ProLock. I build my own. Expert system and then I find out, oh, actually the expert system, just else I got quite disappointed. I said, wow.

Then you had to build all this knowledge based knowledge graph where handcraft all these things, it's just like another practical, right? And then 30 years later we had this life language model. Essentially any of the life language today is a super, super smart expert ci. We cannot even imagine to build 30 years ago.

Whereas if you're 30 years ago, you can build a expert system like any of these today. It's not like a impossible task. That's why, I was so excited when this thing happens. I said, wow, finally after 30 years, we finally built some decent expert system. And then I think thinking forward, I see, I think again, I think just like a plenty of things even go back to 30.

50 year scope when we start thinking about the ai and I think a lot of the vision and the Pioneer had 50 years ago, now we can finally implement that. Like the third tooling test, right? All we finally and concrete and now the expert system. All we already building, I think I probably could just go back to read all these old app books.

Say what kind of dream they have. 50 years maybe now it's become a reality. 

Richie Cotton: Absolutely. Yeah, definitely there's just been so much progress over the last few decades. Hopefully it, it's gonna continue in, in this area. 'cause like stuff that we've been trying to get working for decades, it now actually does work.

Yeah, 

Jun Qian: robotics, I think robotics probably Next line. I was just joking with my kids. I think mostly maybe, five years later everyone gonna have this R 2D two. And we all like R 2D two I think. I think that's. It's, I think that's a clearly trend, right? And we all have our digital companion physically, right?

And now today we already have this digital companion like Chad, GPG conduct. However, I think after five years, I would assume, right? We'll all have our. Little cute robotics robot with us, and we will all have the autonomous driving, I would assume. I don't particularly drive myself, I think.

And autonomous driving should be here. 10 years. Why we need to drive. And we have all this way more. And the Tesla building, autonomous driving, I think this reality is already here. And just on the hidden side. A lot of things already happening, we just didn't realize.

Yeah. 

Richie Cotton: Jensen swung earlier this year. The CEO of Nvidia Iwi is talking about physical AI being the next big wave, which I think is just a rebranding of robotics. But yeah it seems like this is gonna be an important big thing. 

Jun Qian: That's exactly the thing, for example, right? The robotic scene where we have been envision for more than 50 years now, just become a reality.

So I think we are just living in our own fantasy, right? We imagine all these things 50 years ago. Now it's happening one by one. 

Richie Cotton: What's the impact then of of physical ai? I presume it's gonna hit manufacturing and industrial use cases first. Is it? 

Jun Qian: I think industrial only happening. I am not talking about that, how this impact our daily life actually and act.

This is the only one area. Actually, I do have concerns. If you have a physical. Robots in your home or control it if it's misbehavior, right? I think this actually, once you get into your daily life, I think there's a lot of things we have to thinking carefully. If you build a small, cute robot dog, probably it's okay.

However, if you have a really human bot right, can lifting scenes right and can do all the heavy, labor work. But what happened to the out of control. And actually, I don't have concerns. I put such you huge human body in my house. Not only privacy, it's just you know what happened to the malfunction?

Richie Cotton: Yeah. Isaac Asimov, he wrote hundreds of books on this topic. It's like robots going wrong all over the place. Yeah. 

Jun Qian: It's the only place. Yeah. I think that's yes. I'm always optimistic about technology, however, I think and once we get into the physical space I think right.

We definitely need a more careful, how do we design this robots, kind coli with us peacefully. I think that's gonna be one of the things we have the thinking right now. Yeah. 

Richie Cotton: Yeah. Definitely some, potentially great things or potentially very scary things going on. Terminator was not necessarily a happy movie.

Yeah. Before we wrap up, I'd love to talk a bit about Stargate. 'cause Oracle's obviously involved in this is a ridiculously large infrastructure project. Can you just tell us a bit about it? 

Jun Qian: Yeah. I think you already saw all the news and I won't repeat right. And, but however, I think I would like, like more think about from computing perspective.

And for example, right when we start building all the GP cluster, and to start with, 1000 GPOs, right? Back to three years ago, for example, when the GPD two GBDs three open when we started, right? I think about like probably thousands of the GPOs, right? To build a large models. You probably heard from Eli, right?

He built like a 100 k, h 100 in his emphasis, right? Data center. Now the star gate is already close to equivalent, right? About 500 KH 100 skills, right? So beside think about this a computing infrastructure, right? It's getting more and more powerfuls, right? And very, it's very strong, right?

We're gonna have a 1 million TPOs, right in a single TPO clusters. I think that's the target, right? And Sge and also other provider too, right? I think Microsoft and Amazon, right? Meta, they're all trying to build a large scale GP clusters. I think the what's unique about the Oracle Open AI collaboration is right.

This is huge CPU clusters in many regions. The textile one is just the first one, right? You probably heard right. We are gonna build by the Middle East. And we're also working with a lot of the, state and government. We're building the large CP cluster for them. I think again, this, I think this just another starting point and we just dramatically change our GPU infrastructures to start with.

And more generic is model computing infrastructures. Back to again, back to, that was 10 years ago, right? And when I lead a team launched the first Amazon returning service, right? We're only talking about a CP, so hundreds of CPUs, but now we're talking about thousand even. So the GPOs. It's only 10 years.

And we dramatically already advanced the whole computing industry, so given another five years. And that's why I think, I'm super excited, right? And because from the computing infrastructure, right? Computer science perspective, it's just like a so traumatic exchange in the past few years.

This gonna lead us to many things, right? We, it's very hard for data put this way, 

Richie Cotton: having a million GPUs in a single cluster. That's just a ridiculous I mean it's a crazy ambitious engineering challenge. Can you talk us through what are the benefits gonna be? Like, what do you expect to come out having such a massive compute infrastructure?

Jun Qian: Yeah. I think the word perspective is just intuition. For example, right? And let's just use the TB industry as December, right? Before it could again, it could take your 12 months, right? Do one iterations. But now if you a hundred x tower. Then you could do eTrition super quickly, right?

Everything else you could probably had one tion, right? So the way you do the, I think JD have very good blog post about this, right? Think couple years back. So essentially you increase your eTrition speed. That makes a huge difference, right? Before, for example, if you take your three months to experimental one algorithm, now probably gonna take you three weeks.

Once you have more convenient powers you could take you to three days. So that means your innovation speed is accelerating. That's the difference. It's not necessarily your model get a larger and larger. It's more about the intuition, speed, and I think about it. Then you could have a more teams.

Doing all the experimentation in parallel once you have the computing powers and then I think the integration could accelerate. I think that makes a huge difference. It's another, not only about the sex, it's really about the speed you ate. 

Richie Cotton: Okay, that's interesting. 'cause I had assumed it was just gonna be like, okay, we're gonna create some like giant new model, but actually just being able to create similar sized models.

Faster and being able to keep tweaking them, not just make a Exactly. 

Jun Qian: Yeah. Before probably gonna, take, one team three months to one ations. Now you could have a hundred team doing same thing. So basically you do a hundred x your iteration. I think that's gonna dramatically accelerate.

And no matter what you build agent or foundation models, I think that can accelerate. Our AI revolutions. 

Richie Cotton: Okay. Yeah, certainly a bright future for being able to develop stuff once all the infrastructure's in place. Okay. Cool. Just dcia, what are you most excited about in the world of ai?

Jun Qian: Yeah. I think for me, I you, because I've been doing the computer science AI for a really long time. My, my thinking is still I think there's still a lot of fundamental computing problem, right? One, one, I think one thing really excite me is I am thinking is right. The way we build a computing system, right?

And how the way we build a computing software could dramatically be changed in next few years. I think that's, one dimensions, right? What you've been thinking. Should we just, do we still need a compiler, for example, for building other things, right? Is our current MacBook or window system. Do we rebuild all this opportunity system as well?

And again, I think that clearly a lot of the lot of things we could do, right? We could make our software computer system much smarter and much better. However, I think the second thing is I still believe. They's scientific wise, right? We haven't make many breakthroughs, of course is great.

We still have a lot of potential like build super powerful models, agents. But again, I think some of the fun fundamental technology like reinforced learning is still 50 years ago, right? It is not many traumatic I would say right, scientific advancement yet, and I think right, still have a lot of opportunities.

So I develop a new algorithm. And have a new thing. I, another, I think another thing is maybe as a current algorithm, large language model, for example, right? Make it p right. In the next few years, then we still need to keep thinking, you put this way, I didn't think we done right. We just started. And as a humanity, right?

I think right. Lifetime is perfect. Great, right? However, it can solve all the problems at the very beginning, I think, right? From a research science process, a lot of things, right? And I think we should continue to evolve, right? And continue to do more. Not stop here, right? No, lifetime not is great, right?

We can build a lot of applications, but I still see there's a lot of potentials for science community, right? Continue to. Involved where continue to invent, right? Not stop here. Yeah, 

Richie Cotton: absolutely. I'm right there with you. Like a lot of the kind basic scientific research questions. These are like very cool things to work on.

There's a lot of ways that both data and AI can help out. Hopefully if we can speed up the scientific research, that's that's definitely gonna be a big win for humanity. Wonderful. And just finally I always want recommendations for people to follow. Whose work are you most excited about?

I think

Jun Qian: not that particular, of course, tradit this like Angeli or angio in jacking and all this euro follow them. And I started to follow, for example, like open pic. And because you're the top tiers, right? Frontier and Deep Mind and all this. And enterprise, we follow them very closely and yeah, I think be open-minded.

I put this way and podcast including your podcast, things Absolutely. Gotta follow data frame. Exactly. If you gotta follow data frame right. The one you had in this Rag 2.0, I think that's fantastic. There's a lot of practical, I think. My my recommendation is just listen and read as much as you can.

Don't be best by any one of them, right? The more you learn and the longer you learn, then you can build your own judgment, right? And but don't best by single person, for example, oh, I will only follow this province blog or suggestion. And I think I, I be very open-minded. And try to learn, right?

And from different companies or different personnel, right? Broadcast, podcast, YouTube, right? And anything right? You could learn and, that's a nice suggestion. Yeah, 

Richie Cotton: Definitely. I like the idea of just speaking to different people, getting different points of view. Yeah, I speak to different people on the podcast every week.

I do like it when guests disagree with whatsoever. That was last week's brilliant. Yeah. But good advice just to get all those different points of view. Super. Thank you so much for your time. 

Jun Qian: Yeah, so Rich, having me here, I'm super excited. You can see I'm super, super excited. I think we just started, yeah, I think a long, again, it's very exciting work ahead of us, but still, I think a long way to go, right?

The fundamentally change how we do computing and I think right, we still have a lot of opportunities and we can make 

Richie Cotton: a difference. Absolutely. It's all about making a difference. Alright, thanks. 

Jun Qian: Okay, thanks so much for having me here and very nice talk to you.