Industry Roundup #2: AI Agents for Data Work, The Return of the Full-Stack Data Scientist and Old languages Make a Comeback

Adel:  All right. Richie. How are you?

Richie:  Life is good. Great to be recording with you again, Adele.

Adel: Great, indeed. So this is our second industry roundup episode. And what are going to be. What are we going to be talking about today, Richie?

Richie:  Well, last time we spent a lot of time, chatting about I. Since the title of the show is data DataFrame today. I figured we'd talk about data. We should probably try and, not talk too much about AI, but I'm sure I'll creep in somewhere. This should be a drinking game. Every time we accidentally make, exciting, every time we accidentally talk about AI, you know, you have to take a drink.

Adel:  I'll think we'll be. I think we'll be drunk in the first ten minutes if that happens. Now we're going to talk about three stories today. One. You know, no surprise. We're going to mention a bit. I just a bit we're going to talk about coding agents for data science specifically, and what they could mean for the future of the data team.  We're going to talk about the return of the full stack data scientist. Could that be happening in three hour old programing languages making a comeback? We'll see. I don't know. So maybe first story I'll tell you, this is something I've been looking at and particularly excited about. But also, you know, intrigued us, right? This year we saw the release of Devin.  The coding agent code agent was released. Google has been experimenting with coding agents, right? Even, Lang BI has a data science agent specific for snowflake, for example. Right. So we're seeing more coding agents and more research is focusing now as well on like building data science focused coding agents. I think there was a Chinese paper that you sent me a few, a week ago, I think, on how, these Chinese researchers will be able to build a data science agent to automatically fit machine learning models.  Other fractal data is working on Kaggle competitions, right? Maybe based off of what? You've seen Rishi here as well, right. What are the particular use cases of coding agents in data science specifically? What are you excited about?

Richie:  Sure. So I think agents coming soon has been a persistent theme over the last few months. It really has like, a lot of data. Very much has been talking about it. And a lot of the use cases they give are like, oh, there'll be something that can help you book your flights and automatically add a calendar.  But that's not very exciting to me. Maybe if you're an executive who flies every week. Brilliant. I fly like 2 or 3 times a year. Not that useful for me, but data science agents. I am looking forward to these. And because there are so many different data roles, I think agents can be more useful in some places than others.  So, machine learning is perhaps the most amenable area here. So, you think about Datarobot has been pushing automatic machine learning for nearly a decade now. So we know that automatic machine learning is a really good idea. It's not entirely clear how much value the large language models add to this process. So, there's been, a machine learning agent from Lang Dot AI, which basically is designed to participate in Kaggle competitions automatically.  I suppose great, if it wins, you know, you could win some money without doing much effort. But the idea of, having some sort of bot doing machine learning in general, I'm in favor of it. Yeah. Oh, I would say data science is harder, though. So data science involves a lot of communication skills as well as the technical skills is very broad.  I don't think broad data science bots are going to be a thing in the near future, but we might see some for narrow tasks.

Adel:  Okay, I see what that was the last point that I wanted to focus on, actually. Ben Stansell, we recorded an episode this time last year, right? Where he talks about, okay, coding agents or, you know, generative AI will be able to work a lot on, you know, SQL code or boilerplate code. But what he was excited about, actually, is the use of LMS to, streamline, potentially automate, the creative parts of data science, like the problem thinking, right.  Have you seen, you know, agents whether, you know, coding agents focus on data science or traditional. No. Or, say, traditional more like software development, coding agents being able to also work well on the creative side of, you know, this type of work from problem solving. How do you scope a solution? What are the different avenues you to look at to solve a particular problem?  So on and so forth.

Richie:  Absolutely. So there are a lot of, attempts at this coming, coming soon or just being released. So Google has this experimental notebook based AI agent that specifically for data science. And so, the idea is that it can do exploratory data analysis and it can do some interpretation of your results. I don't know how well it performs yet, because reasoning about data and coming up with good conclusions is like, we know a LMS can do a bit of this.  Can it completely replace a human? Not yet really, unless you spend a lot of time, working on specific problems. I've also seen, there's an AI agent, built into, snowflake. So, this is interesting because rather than trying to do all of data science is quite limited in scope, which hopefully means it performs okay, what it does.  And the big problem with the data science, on a corporate, data set is, oh, you know, you've got this giant corporate database, hundreds or thousands of tables, and the agent doesn't know which bits of data to use to solve problems. So the idea with this, you create a snowflake for you, giving it just the data it needs.  And then it can answer specific problems like, giving you weekly updates on your time series. So limited stuff that works well okay.

Adel:  So so limited. Not yet ready for prime time. The creative parts of data science. Maybe. What are the for those who are building these agents, what are the nuances they need to take into account when building a data science specific coding agent versus a general coding agent?

Richie:  Yeah. So I think, it's much the same as any other software project is. You need to be really clear on like, what the scope is, what you're trying to solve, because the broader you go, the harder it's going to be. You have to write infinite number of tests to see whether it works or not. So, I think, perhaps the most, important thing is start with the stuff where it doesn't require creativity and you can automate things.  So streamline stuff that's going to give you the like instant productivity boost. And then you work on it. Well, can we do reasoning for a limited number of use cases and then expand and iterate?

Adel:  Okay. And how do you imagine, you know, two companies even need data science agents, right. Like do to you. How do you how do you imagine the implementation of a data science agent and data teams today as well?

Richie:  Yeah. I mean, so, if we think about, data analytics, maybe rather than data science that are really like a finite number of common business problems that people want to solve. And so that's much more amenable to, for so particularly if you all have something where you do weekly or monthly reporting, then creating an agent to answer questions about the report, there's going to be so many variations.  So it's probably a lot easier to create something that works really well for that.

Adel:  Yeah. And we've been talking about this theme of AI is coding for a while now on DataFrames in our webinars and so on and so forth. We've talked quite a bit about how AI assisted coding will change the data role, but if you have an a genetic workflow or an a genetic tool that, you know, takes open editors and starts tackling them, how do you see those changing the role as well?

Richie:  Yeah. That's really interesting. I think, that probably leads nicely into our next story about how is the data science role changing? So, this is where we had Cassie on the show last year and showed how there are two kinds of data scientists now. So, maybe we just, played the clip from last year there.

Cassie:  I introduced the the concepts of the and the data scientist versus the or data scientist. So the end data scientist would be somebody who is a statistician and they're an analyst and they're an AI engineer and they're a machine learning specialist. And, and and somebody who thinks that data science is the everything of data and that you should be expected to be maximally hardcore.  You should be paid a lot because you are this amazing unicorn who can do everything. And the end data scientist is incredibly offended by the author. Data scientist, just somebody who dares to wear the exact same job title as them. That is a job title that's supposed to come with money and is an or data scientist as. And they just do one of these things, maybe not even that.  Well, they touched a regression at some point. They made a scatterplot. Now they call themselves data scientist. You can imagine the horror from somebody's point of view. I spent 20 years becoming this good. How dare you? How dare you do this? And. Okay, I feel I feel for both sides. It is horrible to have had the impression that in order to practice in the first place, you have to have a level of quality.  That's insane. And you put yourself through that very difficult schooling and you show up and you're like, right, this is why I should be paid these dollars, which are rare and highly qualified. And then you see other people jumping onto the scene to pretend to be you, to take your salary and, and dilute the market. But how horrifying.  I get that. On the other hand, the and data scientist is a threat to getting things done because there are just so few of them. Like, if everybody has to be that truly hardcore level, nothing is going to get done. And it makes much more business sense to specialize. And even if you are an and data scientist, you will not be doing all of them every day or every week or every month.  Even right? Projects come in phases, and during some phases it's mostly statistics. And during other phases a whole lot of programing, you know, so on. And does that really all need to be housed in one person if it's not used all the time? Maybe it does make sense to specialize. Like if you have this expanding universe of data science, professional roles, data science adjacent roles, then maybe it's okay if people aren't the everything of data, and maybe it's okay if they only do some part of it.  That means that you can have more people in it and you can get more done with data.

Adel:  Okay. Thanks for dropping the wisdom.

Richie:  Yeah. And so, Cassie's point is that there were sort of two different generations of data scientists of the first generation. People were like, well, I need to hire, a statistician. I need to hire a programmer. I need to hire a data analyst. And I can't afford to hire three people. So we'll say one person. And that data scientist role that was born.  So it was really, quite a technical, high profile job. You know, you need a lot of different skills. And then people realized there aren't many people who can do all these things together. And so, data science seems to sort of change to have a lot more junior people. You have multiple people with different skills.  And so this these two breezed like, I can do everything. Data scientist. And we got that. I can just do something specific to my job, data scientist. And so, I don't know whether that's going to continue. So, basically to begin with, do you think we're going to have more of these unicorns, data scientists in the future?  Now, AI is changing things, or economic conditions are changing. Things are going to have more, specialized data scientists.

Adel:  It's hard to answer that question. From one end, you can make the case for either, kind of either scenarios happening. And I see, probably it depends on the maturity of the data team. I think, startups will most likely or smaller data teams will most likely go for a unicorn data scientist moving forward, because they need someone who can probably plug in the data pipeline.  Right. Works on some analytics engineering, right. Use AI to be able to kind of fill gaps that they may not have. And kind of, be able to approach problems from a holistic perspective. Right. But I do think as data teams mature, you will need, even if you have a, even if you have full stack data scientists on your team, you will still need data engineering expertise, for example.  Right? You will still need functional analysts to join your business teams. Right? Maybe I will be able to, you know, streamline the use or the need of a functional analyst. But I don't see it as well, happening in the foreseeable future. Right. So, I do see that there will be a place for both in the future.  I don't think so. Maybe not the best answer. Right. But I do think that there will be a place for both full stack data scientist and a more, you know, more kind of specialized data teams. But I wouldn't be surprised to see more and more so full stack data scientists, given, how AI allows you to fill the gaps that required multiple people before and a team absolutely.

Richie:  Is, very and it depends. And I suppose it depends on the who you've got and the rest of your team who you've got another teams and trying to make things play out, as a whole. Now, you mentioned the term full stack data scientist. Now, we did an episode earlier this year with, seven Goyal from Outer Banks.  So he was done about full stack data science being some regular data scientist, plus, that kind of engineering knowledge to put models into production. But I think a lot of people define it differently. So do you have a sense of, what a full stack data scientist would look to you? Like what skill set for that involve?

Adel:  I think a full stack data scientist is someone who's able to approach problems, a variety of problems, that are, you know, that literally refers to the full stack of data within your organization and able to tackle them. Right? So, you know, I'll give you an example of a good full stack data scientists, back when Data Camp was early in its days, Ramnath Vijay Nathan, who's no longer in our, data camp, right, moved on to another organization.  Right. But was a data camp for about five, six years. He was, you know, a full on, full stack data scientist. He would work on data pipelines. He would build machine learning models. He would deploy them. He would run analysis. Any data problem, you can put him on it. Right. And now we have a much more mature analytics team with different functional experts.  You know, for our marketing team, for our, business development team, so on and so forth. Right. So Ramnath is that example of a full stack data scientist, right? Because he's able to be effective on 90% of possible data problems you may encounter on your data team. Right? And this is what I mean. Ramnath was was extremely valuable during his first five years of data camp because it was a really small organization.  Right. And this comes back to that startup point. So for me, a full stack data scientist in a nutshell, is someone who is able to tackle, a large majority of problems that a data function may encounter.

Richie:  Okay. Yeah. I agree with Ramnath. It's brilliant. It also highlights the problem that Ramnath not very reproducible. So yeah. So maybe the flow here is like if you're making your first data high, you want someone senior with a broad variety of skills. And as your team does that you you gradually get people who are more specialized.  Okay. Sorry. You want to drop the.

Adel:  No, please. Not a agree.

Richie:  So I think, the key thing here is that in addition to data skills, you maybe need some other skill just to make you stand out, especially as data literacy becomes more popular. Like, everyone's a data worker now, at least in, like, white collar jobs. So, you need data plus something else. What should that something else be?

Adel:  I think so this something that we talked about on our data camp radar conference, a few weeks back. The importance of product sense, communication skills and project management cannot be understated, especially as you mature more, as a data function. Right. So what is product sense? You know, a lot of data science is probably one of the biggest trap data science fall into, right, is resume driven development.  You want to build the coolest, sexiest, shiniest toy, right. That uses the most cool advanced algorithms, right. Without necessarily having in mind what you're trying to do. Right. So product sense essentially is how do I use my data skill set to affect a business problem, right. Not how do I fit my data skillset neatly? Because I want to experiment with these certain tools and algorithms and products to fit that business problem however I can.  And then you end up having stakeholders are unhappy with your solution, right? So it's always working backwards from the problem. That's, product sense, communication skills. You know, if you're a data analyst or a data scientist and you're presenting to, a C-level executive on, your ROC curves, you're probably making a mistake here, right? You need to be able to also communicate clearly and have that kind of a language that your audience speaks.  Right? So that's where communication skills and I think data storytelling is a big part of that. Right. And thirdly, project management right. Do you keep your stakeholders involved? Do you get reviews often? Do you make sure that there are milestones within the delivery that you're working on? Right. This is, I think, kind of the key business skills that will define a great data scientist from a good data scientist in the future.

Richie:  I like this. There's quite a wide variety of well, I mean they're not quite soft skills, things like project management, but they're not sort of hard technical skills. In the same way that coding is. And I use hard and soft in the sort of traditional sense of how technical is, I always feel like, soft skills are one of those, joke phrases because the soft skills are the ones that are hardest to learn.   Anyway, yeah. So, you got communication skills. Project skills. And then I suppose, the other stuff that we talked about where it really gets a sort of technical add on, like, stuff like putting, having enough engineering skills to put, code into production. So you've got a real choice there for your secondary skill to go to.  Go alongside data.

Adel:  Yeah. I couldn't agree more. And, and in a lot of ways, you're painting here a kind of, a two visions of a full stack data scientist, right? Like you have a full stack data scientist who's really good. Technically, they're able to, you know, build models, deploy them into production, so on and so forth.  But then you also have a full stack data scientist who was able to, you know, build models, do analysis, but also manage a project, communicate with stakeholders. Right. So there's also different visions of what a full stack data scientist can be in this context. Does that make sense?

Richie:  Absolutely. So, it does seem the future is fairly bright in that you do have a bit of choose your own career, and there's quite a lot of leeway to do the things that you're interested in. Like if you don't care about project management, then don't do that. If you do care about product management, then, you know, go for it and there's going to be a role for you.

Adel:  Yeah. So maybe to sum up, the full stack data scientist is making a comeback, but not in the same. Not in the way that we thought it would. Right. At least that's how I posited. And, you know, speaking of comebacks, I think this also makes a great segue to our third story. This was not pre-planned at all. Which is old languages make a comeback and Python keeps on getting better.  Right? So just to get some context here, right. And the latest job index, for programing language popularity, Python remains number one. Right. We'll put it up on screen here. SQL is in the top ten, right. So that was also, pretty interesting. However, there are two shocks, right? One for trend is the eighth most popular programing language on the planet today.  And it's been on the rise since the end of 2022. And similarly Matlab is also on the rise. Again, I always thought Matlab is only exists and grad school labs and that's it. And it's never used in any, any other place. So, I'm surprised that it's ranking number 13 as the number 13 most used programing language on the planet.  So yeah, speaking of comebacks are, out of favor. What was previously thought as out of favor programing languages making a comeback, Richie?

Richie:  Yeah. So, those results, Fortran becoming more popular, Matlab becoming more popular. Don't make an awful lot of sense to me. I had to, ask the internet. Try and find some answers to this one. So, I asked on the Fortran subreddit, why is Fortran making a comeback? And there are kind of a couple of answers.  Some suggest it's a real thing, some suggest it's not. So the top answer is, well, it's because PyTorch is too slow. So if you are doing, any kind of foundation model building any kind of deep learning, PyTorch is the standard tool, but it's also written Python. I think there's, a bit of C underneath. But, it also it's designed to be a general framework.  It's not optimized for performance. So if you are building a giant model, you get some benefits in just rewriting some of the important bits of code in Fortran. But I can't believe that's like an awful lot of people. I feel like that's probably thousands of people, maybe tens of thousands at most doing that. So I'm not totally sure whether that's, that's a real trend or not.

Adel:  And what about Matlab? Why is it why is Matlab making a comeback?

Richie:  Yeah. So Matlab is even, is even less clear. And I think, it might be the, the same reason. I mean, I do love Matlab. I have to say, I used to program Matlab, a lot earlier in my career. It's got this nice mix of like, it's got a great idea, but also like you can write code as well.  So you've got like, good, good sort of halfway point between, writing code or I'm pointing and clicking, but I suspect the reason might be just a quirk with the Toby index. So it's very, heavily based on search results. So if you've got great documentation, great examples then of how to use the code, then your pitch be more popular.  If a lot.

Adel:  Of students are using your tool in grad school and they're searching for homework answers, does.

Richie:  That that that's something.

Adel:  That.

Richie:  So I think it might just be because Matlab has very good documentation rather than those, like a lot of people using, I've not seen a sharp uptick in Matlab. Matlab job, jobs or like people talking about Matlab in the wild. And it's the same with Fortran. Apparently they've had a big effort to try and make their documentation clearer.  So that might partially explain why it's appearing to be more popular.

Adel:  Interesting. And, you know, you mentioned kind of the, deep learning angle for the rise of, Fortran one or other potential reasons why Fortran may be also going up in popularity.

Richie:  Yeah. I mean, the other stuff Fortran tends to be used for, it's just it's high performance computing. So a lot of, scientific simulations, like it's really big in like weather forecasting. That's the only people I know who like super, keen on Fortran. It's obviously like the language, like 70 years old now at this point.  So yeah, anything you, you're do engineering simulation, things like that for data science. I'm not sure that many people are using Fortran directly. Like, I think if you want to get fast code now, it's all like it's rust underneath the hood. Yeah.

Adel:  Yeah. And what about other, languages? Richie, I'm going to ask you a sensitive question. Maybe. How's are doing these days?

Richie:  This makes me very sad. So, yeah, my background is in the L community and is still my favorite programing language by far. It is sliding down the rankings, which is a shame. It's just pythons eating everything. And I think, the reason I really love R is like, for data analysis tasks, it's just, it's just nicer to write them.  Python. It just works. Yeah, yeah. Like, the tidyverse stock is just nicer than pandas. GGplot2 is just nice. Any of the plotting libraries Python has. And for anyone who's like, well, with Python, there's Plotly. You can also do plotly from Owl. So yeah, it's great stuff. But is falling out of favor. And it is something we've seen on Datacamp as well.  But there's a big difference between B2C and B2B audiences. So if you're an individual learner, nobody's using, r that no one wants to learn R because there aren't as many jobs. But on a business level, programing libraries have a much longer lifespan, so it's going to be around in industry use for the next decade at least, I would think.  But for individual learners, they tend to prefer Python.

Adel:  Yeah. I also started off as an R learner. Right. That's how I broke in to, the data science space. I do enjoy working in Python much more. I do feel like it's, more intuitive to me, Python syntax than R syntax at this point in time. Right. But yeah, you know, it's interesting to have seen, like, the kind of the R versus Python debate, has subsided.  Let's just say, in the past year, and there's a clear winner, maybe one third kind of, programing languages that was always thought to be a dark horse in the data space, but I don't see a lot talked about today's Julia. How's Julia doing as well? Richie?

Richie:  Yeah, it's sort of. It's never quite managed to catch, hold, of, the my chat is, as it, as it should have done. Like, there was a lot of hype for it a decade ago when it was first launched. It's never managed to get those sort of big industry sponsors that, that Python has had.  It's not quite managed to, get the sort of academic base that I had. So is it still ticking away? I think the language is getting better, but it's not quite taken over as a thing that everyone must learn.

Adel:  Yeah, and I feel like rust is taking that spot. That is, at least in the ether, based on what I've seen data scientists talk about and kind of people who are, we try to be at the leading edge of the conversation. Do you see rust taking over Julia here as. Yes, kind of that third language in the data space.

Richie:  Well, so rust is one of the rising stars of the programing language space. It's just gaining a an awful lot of popularity. But it's not really replacing trying to replace like Python of SQL. It's a much lower level language. So what it's doing is it's especially it's competing with C plus Plus and Fortran for the lower level code.  So one of the big, sort of exciting things happening with rust in the data space is polars, which is a, it's a pandas replacement. So you write your data manipulation code, but it runs much faster because it's built on top of rust rather than whatever. Pandas is built on layers and layers of Python, I think.

Adel:  And we do have a really detailed, comparison article that compares Pandas and Polar, so I highly recommend that you check it out. We're going to leave it in the show notes. And maybe a couple of final questions here as well. Richie, what are your expectations for the programing language space in the next year? Who's still going to be on top?  Will there be any surprising, you know, any surprise darkhorse that will come out? What are your predictions?

Richie:  Oh, man. I don't think I want to bet against Python. I think it's just become too dumb to start. It is the the quirky keyboard that you will never get rid of. That sounds too negative. I do actually like Python, in places I just, you know, I, I miss our, So. Yeah. Python's going to stay strong.  The big thing I'm most excited about is the SQL language, improving. So, I mean, SQL has been around for 50 years, but the language is still evolving slightly. And I think as a particularly with DB, a lot of that has took a lot of influence from the R language. So there's a lot of, ways of making SQL syntax easier for people to learn, easier for people to write.  So I'm excited to see changes in the SQL language itself.

Adel:  Yeah. And it's incredible that SQL is still going so strong. How many years has it been at this point?

Richie:  Yeah. So it's just a 50th anniversary.

Adel:  50th? Yeah.

Richie:  Early this year. Yeah.

Adel:  Yeah. We had, Don Chamberlain on the podcast. Who was the inventor of SQL. I'd recommend everyone listen to that episode. And maybe as we wrap up, Richie here, you know, I asked you, what are your predictions for the programing languages? Right? But this is going to be our last industry roundup episode of the year, right?  And we are recording during Thanksgiving. Right. So the episode will be released, I think, on December 6th, if I'm not mistaken. Maybe. What are you grateful for? And what are you looking forward for? For it.

Richie:  To. Oh, what am I grateful for? I mean, you know, I, I've a very privileged life. I have good health. I have, good family life. I have, good job. So lots to be grateful for there, stuff I'm looking forward to. I mean, I'm hoping like this, and I stuff this, data takes off of it, so I can automate a lot of my job, and, you know, just,

Adel:  Trust me, you won't be going to the beach. Just find new stuff.

Richie:  I'm sure you'll find more things for me to do. Going. How about yourself? What are you grateful for? And, what do you,

Adel:  I'm also grateful for, one, having amazing colleagues. This is not just, I'm not just saying this because you're here, but having amazing colleagues. Yeah. Also have a very privileged life. Couldn't be more grateful for my family. My friends, my partner, my my colleagues. Great job. So, yeah, grateful for everything and everyone, in my life.  And then what am I looking forward to in 2025? So also, I'm kind of excited for a bit of a correction to happen in the I space, right? I'm very excited about the potential for AI, but I also see, a risk of the, maybe I'll say it this way, the Bitcoin ification of AI, right, where you see a lot of hype, around the AI space.  That tends to create, either false expectations or, negative emotion around AI, right? I'm excited for that. To subside slightly as we mature more with the technology. And, you know, it does seem like we're going to reach some form of plateau, at least in the intelligence of the models. The product experience will be get better.  The agent capabilities will get better. Right? And then, yeah, what that means as well is that, you know, maybe certain startups will take a hit, but at least will have a much more, grounded industry as well. So that's what I'm looking forward to in 2025. Seeing less AI hype and seeing more AI value.

Richie:  Essentially less hype, more value. But yeah, talking about, AI capabilities plateauing, that's a very controversial, take that. Lots of arguments around this is AI, particularly generative AI going to scale or not? Much further. Yeah. That sounds like,

Adel:  Something I'll cover in the next industry around the. Yeah, something to cover. And then this round up. And with that, I think, we'll cover it here. If it's, it will end the here for today. We will be taking a break on data framed. I think starting the for the last two weeks of the summer. And we'll come back with new episodes for, the start of the year.  Very excited to kick off the new year, with your Richie.

Richie:  All right. Likewise. Cool.