Reviewing Our Data Trends & Predictions of 2024 with DataCamp's CEO & COO, Jonathan Cornelissen & Martijn Theuwissen

Richie Cotton: Hey there, Jo and Martijn. Welcome back. Glad to chat with you again for another round of data trends and predictions.

Cool. So we're going to talk about your new predictions, but before we get to that, I want to talk about your predictions from last year and see how many you got right. Your first prediction was that generative AI is going to hit the mainstream. So well do you think you did on that? Joe, do you want to talk us through it?

Jonathan Cornelissen: I think this one was a little bit of a layup. It's not. the most risky prediction that generative AI was going to get mainstream a year ago. But I think we can confidently say that that has happened in the last I think some research indicated at the beginning of the year, 20 to 30 percent of people were using chat GPT in the U.

S. The latest research I've seen, it puts it at about half of Americans using chat GPT or one of the other models. And that's in line with what we're hearing from clients that monitor. AI, Jenny, I usage on their networks. So, yeah, I think, I think we can confidently say that we, we, we got this one, right?

The one thing I would notice is that if you look at daily usage, though, like we're still at the beginning of this, I think estimates range from anywhere between 5 10 percent of Americans using Jenny, I daily. And it very much skews towards students, data folks, people who are part of the AI space, of course, people in marketing, customer support.

So there's a whole group of people that doesn't use this at all or doesn't use this on a daily basis. So we still have a long way to go for kind of mass adoption, but it's definitely gone mainstream. So I would give us a point on this one.

Richie Cotton: Yeah, that's an interesting point about there's a difference between you having heard of it and used it like once and being a daily user. Martin, was there anything you wanted to add to that?

Martijn Theuwissen: Yeah, maybe like, yeah, I think we got the point here. Maybe it was a bit too easy. I think one thing I would add is that there's a lot of like explicit adoption a lot of the stats are about explicit adoption as in like, you're actually interacting with chat GPT. Now there's also like this.

Implicit adoption happening which is growing very quickly. Like the best example to me is like, if you do a Google search today, it's more and more serving these like AI generated responses, which obviously help as well, but making generative AI more and more mainstream. And I think we're going to see much more of that as well in the future, like that keeps going

Richie Cotton: Yeah, that's true. So even if you didn't intend to use generative AI in some way, just like you have to search for things on the internet sometimes. So you're going to be at least passively consuming those AI generate results.

Martijn Theuwissen: these.

Richie Cotton: cool. All right. Yeah. So I think, I think you got one, but will you be the point for prediction? next prediction was that AI literacy becomes a universal skill. So, talk me through how well you think you've done on this. Martin, do you want to go first this time?

Martijn Theuwissen: I think if you look at it from like data cams perspective, like it's definitely the part of the curriculum where we've seen the most growth. I think we've been growing over like 60 percent year over year there. So that's really like, literally like hundreds of thousands of learners starting courses.

We also see like quite a lot of like, users within like large enterprise uh, clients. So like more and more folks are taking up AI. Literacy building it as a skill. Now at the same time I think there is like still a long road to go. I also think that there's definitely in the U and I'll get to that, like for my future predictions, like there's still like some hurdles to take there.

 So I would say like, we're pretty good on this one as well. But it's definitely not the other universal skill. Yeah,

Jonathan Cornelissen: agree with that. I'd give us maybe half a point or 60 percent of a point or something, because on the one hand, AI literacy is the fastest growing skills area. On the other hand, is it a universal skill as in most people are AI literate? I think the answer is clearly no. so we're on the way towards that being true, but I think it's not yet quite as true as we expected.

Maybe.

Richie Cotton: Yeah, absolutely. I suppose it depends on how you word it a bit. Like there's millions, maybe tens of millions, hundreds of millions of people interested in this. It's just there's billions of people in the world. And yeah, so the majority is still not illiterate yet. Alright, so lots of work to do there.

 I guess we got our work cut out in DataCamp try to teach the rest of the people. Okay so, prediction number three was that the data space and the software space are going to overlap more. do you want to talk us through what this means and how you're going to judge Uh, Yeah, Joe, do you want to

Jonathan Cornelissen: I think this definitely happens. It's hard to quantify. It's hard for me to prove that, but I think there's definitely been a convergence of the data and the software engineering space in a way that hasn't happened really before. There's a lot of software engineers building AI products. There's the role of the AI engineer.

It's one of the new roles the marketplace, one of the fastest growing roles. and we've seen this at DataCamp as well. So historically, DataCamp has been very popular with data analysts, data scientists and especially for more technical learners. They aspire to become active members of the data community.

More recently, we've seen a lot of software engineers coming to the platform. Looking to build skills, for example, build on top of Lama, build tweak their own LLMs, or just understand what it's all about. And so I think we've clearly entered a new area where software engineering will be much less deterministic going forward.

And the skill set for software engineers is shifting towards Understanding AI and logical reasoning skills will still be important, but I think creativity and AI skills will have a huge premium for software engineers in the next few years to come. So I think. It's hard to quantify this, but it's definitely happening.

So it gives us definitely a full point for this one. 

Richie Cotton: a lot of software engineers into this space. And so, yeah, there's a lot more overlap in that way. Martin, was there anything you wanted to add to that?

Martijn Theuwissen: maybe like, another perspective here is that I think AI is causing that the data space and the software space is like converging more and more towards like a common language like having more of this universal tool that both groups know how to interact with, which naturally brings them closer together get to get, for example, like, Python, and like Python was doing like, Python is most heavily used like in software engineering, as well as in data science, but they still have their own libraries, their own frameworks and so on.

But now it's something like prompting, both groups use prompting, like you have something that's much more universal. It's often based on the same like natural language that people use. So I think that's also like playing a role there and like, putting these groups closer together.

Richie Cotton: Nice. Yeah. I like the idea of having a common language just to bring different groups closer together. Nice. Okay. I, I give you the point

Jonathan Cornelissen: Richie, question for you, because I totally agree with Martijn, Python is kind of the universal language that brings software engineers data folks together. JavaScript is still very popular amongst, software engineers and is it going to become more popular amongst the data community then, you think?

Richie Cotton: Yeah. So I think people have been trying to make JavaScript a language for data science for years, just because it is a popular language and you can sort of do some things with data there. I think it's going to become more popular. I don't think it's going to uh, Python anytime soon. Like a lot of the stuff isn't very natural working with data. You've got no objects that are a bit like a data frame. So yeah, it, it just makes things harder. So, I think we're two and a half out of three so far. And the next prediction that you made was that generative AI is going to move from prototype to production.

So I guess in 2023, we saw a lot of companies just Experiment with generative AI. And the idea was that this year things are going to move into production. So do you think that's happened?

Jonathan Cornelissen: yeah, I think so. I think if you look at the last year, every self respecting company has launched AI features at a minimum. They've added a chat bot. So that also points to how much potential there really is for truly integrating AI into the product. And then at the same time, if you look at, like, startup companies in YC and in Techstars the vast majority of them are AI first and they're building AI first alternatives to some of the existing software companies.

So I think we're, we're kind of in classic. Can the startups get distribution before the incumbents get their act together and innovate type of question? So it very much feels like that definitely shifted in the last 12 months where 18 months ago you saw a lot of demos and prototypes and now everybody's actually using it and implementing it.

So I would give us a thumbs up for this one.

Richie Cotton: Voting for your own points. Uh, Yeah. Uh, Yeah, so I agree. There's definitely been a lot of AI features, AI products been released in the last year. So, it just seemed like, yeah there's a lot going on in the space with like stuff actually hitting production Martin, you look like you wanted to say something.

Martijn Theuwissen: Well, I was just going to say, like, I think my mailbox would agree. If I look at my mailbox in the last 12 months, I've probably got like every week, like, some company announcing like some type of AI implementation into their product. So, yeah, I definitely think we went from pilot phase to like, Hey, it's now in production.

Richie Cotton: Yeah, certainly I get a lot of emails too, but yeah, companies trying to advertise that all these AI features I see, I guess my big query on this is bigger companies tend to move slower. So are these big companies enterprises, are they managing to get things into production yet? Or do you think it's just smaller companies that are a bit more nimble?

Jonathan Cornelissen: I actually think the surprising thing about this kind of innovation wave is how fast the incumbents are innovating in almost every sector. Google used to be the exception to that rule, but it seems like They've gotten their act together. But if you look at some of the most innovative implementations in production, they actually come from the largest companies out there. 

certainly some startups that are really taking off. I think compared to other waves, it's shocking to me how incumbents are moving. And there's, obvious reasons for that, but I think it's, worth pointing out. It's a bit different than other kind of platform shifts.

Martijn Theuwissen: Yeah, and it even says, like, it's outside of the typical technology players. Like I'm in the market for a car and like the amount of AI advertisement I get when I walk into like a car shop, like it's unbelievable. Like every car seems to be equipped with AI. Let's not go into the details what it means, but it's actually pretty fascinating.

That like from a, for non tech, not traditionally tech company.

Richie Cotton: Yeah, certainly like a car sales room. You don't think of it as being like sort of tech hub necessarily. But yeah, the fact that they're pushing AI as well means yeah, it's gone everywhere. Okay, nice. All right. So, we'll give you that point. And the next one was that it was two parts of this.

So you said that image generation AI is going to mature, go mainstream, but video generation AI won't hit the mainstream until 2025. So I guess yeah, Martin, do you want to take the image part of this then talk us through that?

Martijn Theuwissen: probably if he would have scored this like two months ago, we would have done a little bit worse, but in the last two months, like you had both open AI with, with Zora and Google with with VO, which is which both are actually pretty impressive. But at the same time, like, okay, I haven't seen a lot of video generation being like productized.

So like the, the tools are nice, like they're not fully accessible yet to everyone. What they can do and like if you look around the corner, like it looks all very exciting. It looks very promising. People can directly see the applications. If you think on social media, if you think in the marketing realm, but at the same time, like I haven't yet encountered it actually being used, like marketing teams having their hands on it.

So I think the way that we, that we made the prediction of okay, well, it mature, it looks like it's like. What you can do with lightning, what you can do with. The angle points, like it's, very cool today at the same time. it's not yet in production.

it's not there for the masters yet. The same way, like, like the chat, you can see us

Richie Cotton: Yeah, interesting. So, with video is actually, I think back in January when we did the 2024 predictions it was like a week or two later that open AI announced Sora and it was like, Ooh, maybe this is coming soon. We thought, and it sort of didn't really hit until December. So, yeah I think the tools are kind of there.

They're not really being used yet. So, yeah, I think certainly with video, hasn't hit in 2024. Really. So yeah I think that Part of the prediction is right. So, I think I'm gonna give it, I'm gonna give you the point again for this one. Nice. So, next one next prediction was that AI hype is gonna fade and AI tools will just be seen as software.

Jonathan Cornelissen: Yeah, I think I could go either way on this one. I think if you go back to the summer, a bunch of articles claiming this and saying, hey, the hype is over, the profits are not there. There was this article in The Economist a couple months ago. And I quote, they were saying the Silicon Valley tech bros are having difficult weeks.

There's a growing number of investors that worry that AI will not deliver the vast profits. So, there's this PR narrative or press narrative that the hype is over. At the same time, I like data. so I did some research in preparation for this yesterday. And if you look at just at Google trends, The number of searches for AI, the number of searches for chat GPT, Gemini, and those types of models, they actually tell a different story.

There's a continued increase in demand. so I would more lean towards the hype is definitely not over. The actual usage seems to be very much increasing and most of these companies think about perplexity and so on still on an exponential kind of adoption curve. obviously biased and I'm kind of a tech optimist, but I would lean towards the hype is not over yet.

What do you think, Martijn?

Martijn Theuwissen: the same, actually, like I'm referring to my mailbox again. But I don't think the hype is over at this point I don't think we're there yet where it's just seen as software, it still has that magical feel to it. Like Microsoft Word is Microsoft Word, Microsoft Excel is Microsoft Excel.

But at the same time, like when you talk with people about AI, when you talk about check TPT, like they talk about it in a different way, like clearly it's not like in the software category for them. Like it's in this other undefined, maybe yet tool category.

Richie Cotton: Yeah, so I think that this is a bit of magic particularly for people who are I've not been like using this constantly for the last few years for things like the AI generated like Google search results that you mentioned before. Does that still count as as magical? Or do you think that's just going to become, oh, well, that's just what a Google search shows you.

Martijn Theuwissen: I think it will become like, that's just what a search will show you. probably argue that people are probably already used to it by now. They don't think of it as like, okay, this is like some kind of AI wizardry. It's like, okay, this is now how I get my search results.

Jo Cornelissen: Yeah, so I think some parts are getting normalized, even with the broader public, but at the same time, if you look at the announcement of Google's new video generation model, I think that's still blowing a lot of people away in terms of its power. So there's continuous innovation that's still generates additional hype in my mind. So I would actually say we got this one wrong. The AI hype is still here.

Richie Cotton: It's cool that AI is outrunning its own hype, I guess, and still staying cool. or, oh, outrunning it on Wii, anyway. But yeah okay I think you've, you've talked yourself out of a point on this. So, okay, failure there. I think our first failure of the day. AI is still hyped and cool.

Nice. Alright so. Next prediction is that narrow AI agents are going to become popular. Actually, do you want to talk us through what is a narrow AI agent first? Yeah Martin?

Martijn Theuwissen: Well, in short, I would say a narrow AI agent is an AI agent that's used for a very specific task. So let's say, for example, that I would like to run some performance marketing on a new course by Datacamp. And I want everything to be set up in the well known platforms like Facebook, Google, and so on.

Like, I give it to this. AI agents specialized in performance marketing on Google, if you want to go very specific, and like, they do the task for me as an example. And then to come back to the prediction, like, I don't have such an agent yet. So I think we have a miss on this one as well.

Do you think that talk about AI agents, like is more popular than ever? And again, it definitely like to me picked up like in the last couple of months, like it's one of those use cases that like entices people, it makes you excited, like this idea of like, Oh, I have all these boring tasks and I'm going to be able to like outsource them to this magic agents that, It's going to do it all for me.

So it's like, definitely like a fun thing to talk about. And advances that keep coming up, like people, this hype is building, like, are we close yet? Like, are we there yet of having such an agent? So, yeah, maybe also like some of the biggest stories this year, like involves like the semi agents, like if you think of Devon, like the AI software engineer so we're not there yet, not even like in the narrow point, but it's like.

The hype keeps building. It's like fun subject to talk about. So well, maybe next year 

Richie Cotton: yeah, certainly AI agents have been a recurring theme on DataFramed over the last few months. A lot of people getting very excited about it. It's not quite hit mainstream popularity yet. So, yeah I think This one's a miss, again, I'm afraid because yeah, it yeah, like you said, agents don't exist yet.

We haven't got our own personal AI butler. All right. So, next prediction was around investments in data governance have a higher ROI. I'm not sure how you even go about measuring this, but Jonathan, do you want to take a stab at this?

Jonathan Cornelissen: Yeah, this is a tricky one. should think about how we're going to measure things for next year's predictions. Because this is probably true. But it's, hard to know. And I haven't seen really good research on it, at least not recently. So the general theme here seems to be data governance, data quality has always been important, but it's even more important as you have Gen AI and the ability to do more with that data.

So think it just logically has to be true, but it's hard to measure. But if you look at some, some some implementations within larger organizations and how they're starting to leverage the existing data they have. It has to be true, right?

Richie Cotton: Absolutely. No one seems to want to publish. numbers on, hey, I invested in this data quality initiative, and this is how much money I got back from it. It does seem to vary a lot from one project to another, but I agree that because there are a load of generative AI applications and features out there, those wouldn't work unless you had decent data quality.

There must be some useful stuff happening there.

Martijn Theuwissen: one anecdotal thing I can share from the data cam site is that. For sure in the last year, I had more conversations around our data governance curriculum and everything around that compared to like in the years before that, so definitely is like a team and it became more of a team in the last 12 months, at least if you think about upskilling people in the wise of data governance.

Richie Cotton: That's very interesting that, people are actually taking this seriously now, because I know Back years ago when I was creating courses for Datacamp, it was always very difficult to get people excited about things like data quality because it's not the cool part of data science, but it's just incredibly useful.

Okay so I think I want to give you the point, just even though we can't measure it, it does feel like data governance, data quality, these things are incredibly important and they're only getting more so. All right we have got another prediction. So, we talked a little bit about this before, but R is going to decline in popularity as a language, and then JavaScript's going to challenge Python for AI supremacy.

So, talked about JavaScript and Python before, but yeah, what's happening with R? Is it really declining as fast as we think?

Martijn Theuwissen: Maybe this is a good one to like, again, go back to you, Richie, like you're a very like, our evangelist what do you say? Like, do you feel like we got this one right or wrong?

Richie Cotton: I've had so few conversations about R recently, and it does sadden me because it's still my favorite programming language. I did take a look at some of the programming language indexes to see how, the popularity is changing. So, the Tyobi index is maybe the most popular index for how popular languages are.

It's actually, it's fairly flat on this. So it's not at its sort of peak. I think it peaked somewhere around like sort of 2018, 2019. It's been a slow decline. It's not quite as slow as you'd think. There's another index from the IEEE, the Institute of something, something engineering, I can't remember what the other A's stand for, but anyway.

So that index. Wait job adverts and community values a bit more strongly and has been declining faster on that. So that the number of people who are getting excited about are for basically careers that's dropping. So yeah, it's a slow decline, but it's going to be hanging around for a good decade or so.

I'm making corporate environments.

Martijn Theuwissen: Yeah. So, okay. So, so that part of the prediction, I mean, right. I think the, JavaScript one, like JavaScript challenges, Python that's like harder to judge. Like if you look at documentation, like it's probably like a 50, 50. At the same time, like if you look at education element and like, okay, courses, do they use Python or do they use JavaScript?

I think I see more Python than JavaScript. So, but that one I actually find like harder to judge, where we stand there, I don't think it's like a clear win for JavaScript.

Jonathan Cornelissen: No, but think it actually, it links to what we were talking about earlier, where is extremely popular amongst software engineers in the web dev space. So as these two communities merge, are starting to see more JavaScript interfaces to data and AI models and use cases.

So does seem like that there is a trend whether, whether or not JavaScript is already challenging Python. I think that's debatable, but it's on its way to do that for sure.

Richie Cotton: Yeah, I suppose it's different parts of the stack. So, if you're creating an application that uses AI, then you're probably going to be accessing things via an API. And then, JavaScript's the natural thing to do. But you wouldn't necessarily want to create your neural network in JavaScript.

Like, there's PyTorch, but there's no JTorch or JSTorch or whatever. So yeah, that sort of sounds a bit crazy, a bit far fetched. But yeah, Different parts of the stack. complementary rather than competing in many cases, I think.

Jonathan Cornelissen: thing that shocked me that you just mentioned, Richie, I looked this up over the weekend is that indeed, The peak of people learning R was around beginning of 2018. and that actually shocked me that it's that long ago. Because R is still around, it's still fairly popular, but the decline started way back longer ago than I expected.

Richie Cotton: Yeah hanging in there. And I think with that, there's a big difference between individual learners and corporate learners on Datacamp. I don't how closely you look at these statistics, but certainly for individuals. That generally motivates you to try and find a job and so they go for whichever has the most jobs, which is Python, Corporation, you use whatever your colleagues use, and they're ours hanging on a lot longer.

Jonathan Cornelissen: Yep, yep.

Richie Cotton: so, give you that point. And the next one was, the next prediction was that data science notebooks become AI enabled. Do you want to talk through has this happened?

Jonathannathan Cornelissen: Yes, think this is a clear win on the prediction side. Every data science notebook now becoming AI enabled or already is AI enabled. Our own data lab very much has a lot of AI features. Google CoLab a bunch of basic features. And I think it's a trend that that will continue.

 Typically, you actually see most of the innovation for happening for software engineers and the tools they use. And for software engineers and kind of the I. D. E. They live in. There's Cursor, there's the Repl. it agent, there's Devin, there's a bunch of other companies that are trying to get software engineers use really AI first IDEs.

I haven't seen that happen for Datafolks yet, but typically we follow a similar path for the Datafolks what you're seeing for software engineers, it's just like one to two years later. And it very much seems that that's the next step here.

Richie Cotton: Absolutely. I guess where data lab is going, all the other competitors, the other creators of data science notebooks, they're kind of, they're joining in on it. All right. Yes. We'll give you the point for that. Okay. So, two more predictions to go. So, the next one was that MLOps becomes more important.

Martin, do you want to take this one?

Martijn Theuwissen: Yeah, I think we're like a little bit the same situation as the prediction on data governance. Like, okay, how, how do we measure this one? and it also suffers a little bit the same issues, like, how do you make someone excited about it? So I think to a degree. It became more important, but at the same time, like, okay, I think given how important it is, like the world hasn't called up with the idea.

So, I definitely wouldn't give us a full points on this one.

Richie Cotton: Yeah, it's another one where it's very tricky to measure. But yeah getting people excited about MLOps is another tricky one. It's a bit like data quality where people like, well, that sounds very important, but it's not there, the exciting bit in some cases. All right.

So we'll put that one as a maybe. We'll give you half a point for that one. All right. final prediction from last year is that prompt engineering won't become a real job. So, are the prompt engineers out there?

Jonathan Cornelissen: I think generally, this is true in the sense that prompt engineering is a skill. think there are actually prompt engineers out there, but it's like a very, very niche role. I listened to a podcast of somebody who does that at Anthropic recently. So like there are people who for most of what they do are, prompting a model, but it's, it's very rare.

think generally prompt engineering is. Or AI literacy, you could call it, is similar to computer literacy 40 years ago. and to us or people from our generation, it comes naturally because we grew up with computers. and so we didn't quite have to learn it, but I think if you go back 40 years ago, most people who were part of the workforce actually had to actively invest in learning how to work with a computer.

I think it's similar for AI and AI literacy where it comes natural to young people. It will come natural to our kids. For the existing workforce, there's quite an investment needed ensuring you get the most out of these models. So I think that prediction came true, even though there's maybe a few exceptions where it's not entirely true.

What do you think?

Richie Cotton: Yeah. So, I agree with that. And I think maybe when chat GPT first came out you had to write a lot of context in your prompts to make sure it gave the right answer. Like if you want to do anything mathematical there's all these tips about, oh, you have to tell it to think step by step and you want to give it context on like, oh, you like, don't make things up and things like that.

And now the models are smart enough to just do that automatically. So you don't need to provide all these like convoluted phrases just to make sure you get the right answer. So yeah all these sort of skills about like writing good prompts is like, well, you just write normal English and usually it does the right thing now.

So yeah, agree. Prompt engineering hasn't become job unless you're working at one of these foundation model companies or one of these sort of niche AI companies. Okay nice. All right. So, I'll give you the point for that. I'm not sure what the total is. I think you got two wrong and there were a couple of maybes.

So we're, we're hovering around something like seven or eight out of 11. Nice. a decent score. I'm going to try and beat that this year. That wraps up our recap of last year's predictions. Overall, I think Jean Martin did pretty well. Of course, I had to dock him a couple of points so they don't get cocky. That prediction on AI agents was a tricky one. It seems like we've been talking about them forever, but the delivery hasn't quite happened yet.

In our next episode, we'll be sharing our predictions for 2025, so please do tune in for that. And also, I don't get to chat to all you dear listeners often enough, so I'd love to hear your predictions too. Please post your ideas on your favorite social platform, tag in at Datacamp, I'm also RichieRocks most places on the internet.

And yeah, let's have a conversation about what you think is gonna happen. Alright, thanks for listening.