Effective Data Engineering with Liya Aizenberg, Director of Data Engineering at Away

Adel Nehme: Hello everyone, I'm Adel, Data Evangelist and Educator at DataCamp and if you're new here, DataFramed is a weekly podcast in which we explore how individuals and organizations can succeed with data and AI. In a lot of ways, the unsung heroes of the data functions are the data engineers. Data engineers act as a force multiplier, enabling everyone who needs data to have proper data access. So how do you build a high performing data engineering function? How do you make sure that data engineering teams are closely aligned with business value? Enter Leah Eisenberg. Leah is director of data engineering at away. com. She has more than 20 years of experience managing data engineering teams and shared a lot of her wisdom on today's discussion. Throughout the episode, we spoke about what makes a data engineering team effective, how to be agile as a data engineering team, how to best approach aligning the data engineering team's work with business value, and a lot more. Also, just a note, the audio on this one is not the quality you're generally used to on DataFramed.

But I hope you're still able to get value from it. We'll do a better job next time. If you enjoyed this episode and the DataFrame podcast, make sure to rate it wherever you get your podcasts. And now, on today's episode.

Liya Aizenberg, it's great to have you on the show.

Liya Aizenberg: I'm glad to be here, Adel.

Adel Nehme: So you are the director of data engineering at awaytravel.com and have been managing data engineering teams for a while now. So maybe set the stage for our conversation. What makes an effective data engineering team?

Liya Aizenberg: There are a few things I highlight for myself that I find important to be data engineering successful. It's trust, collaborative work, not afraid to make mistakes transparency. And now it's opportunities. People have an opportunity to exchange the knowledge within the team. I can go deep, deep dive into each and every thing I identified.

Adel Nehme: So we'll definitely deep dive into those. What we want to focus on first maybe is, building a data engineering team from scratch, right? If we take a step back, you know, you mentioned building the right team, knowledge exchange, building trust, collaboration, if you're building a data team from scratch, there's tons of data leaders listening here on the episode, Trying to build their own data engineering teams as well. maybe first on the type of roles that you would hire, of profiles do you look for in an early data engineering team? Whereas an early hire in a data engineering team.

Liya Aizenberg: I think I identify the important traits for myself. I'm looking for a spark, a spark and passion for the data and passion to learn new things. Good understanding and knowledge of data engineering principles are very important. and good personality open minded eager to learn and help others.

I find this important. And the passion, passion on data and empower your,

Adel Nehme: And then, you know, you mentioned here kind of the personality. If you deep dive into it a bit more, what are the kind of the cultural traits that you look for in an early data engineering hire?

Liya Aizenberg: I'm looking for the friendly and open minded people. I value people who take ownership of their work. who always finish what they start. You know, sometimes people get sidetracked, not finishing things, starting one thing, then sidetracked to do something else. So I, value the people who finish what they started not easily distracted.

Also good communicators people who speak out, it's very important to speak out, also the people who are not afraid to change. Because our industry evolves all the time, there are a lot of changes even within the one company can be a lot of changes. People who are not afraid and easily adapt to the change are very important.

And also result driven folks are very important. It's an amazing trait. So you want to see the result. You want to make sure that whatever you do, you produce the result in the outcomes organization.

Adel Nehme: Yeah, that's really great. I couldn't agree more on results driven, especially, you know, we're going to talk about, how to focus on the right projects that matter, because that's a big trap. Data teams can fall in and maybe, you know, we talked about the cultural traits, but and early data engineering hire, I'm sure you know, the technical skill set is so wide.

The data engineering ecosystem is so fragmented. there's so many tools, so many skills to adopt here. What are kind of the technical skills you also look for in early data engineering team?

Liya Aizenberg: It's very important. Like today I'm looking for the foundation of the Everyday engineers should have a good knowledge of a Python. TCL and the relation and non relationship databases. This is foundational all in knowledge of relation and non relational database as well as Python.

Also there are a lot of side items today on the market. It's very good to have a knowledge of virtualization tools such as Looker and Tableau because you build all your data marts. You build your data, but you actually need to serve this data to the business. So visualization tools are important.

That's the tools you're serving your data with. Like name a few. Looker, Tableau there are very good integration tools that are available today. Stage, Fivetran, Matillion, are currently in demand. Knowledge of DBT is important. This is the tool that you build your SQL models. I want to highlight SQL and Python.

Adel Nehme: I mean, SQL is the lingua franca of, data, whether you're, you're a data engineer or a data scientist or a data analyst. And when you mentioned kind of these canonical tools that you mentioned, kind of DB t Matt SQL Skills, what stands out for you as today? Kind of like the must have tool knowledge, especially in a modern data engineering team that you need to have, you know, outside of, traditional tools.

Like, here I see a traditional quote unquote between Python and sql.

Liya Aizenberg: I think any knowledge of IPaaS solution, it's integration platform as a service. It's a good knowledge integration tools, knowledge of airflow. knowledge of any cloud, AWS, GCP Azure, any of this knowledge of one of these clouds or multiple clouds is very important to today.

Adel Nehme: And you know, as the team grows and becomes more complex, at what point do you decide when it's time to bring in specialists? Like what kind of roles do you start looking for? What are those specialist skills looking like? And yeah, walk me through that.

Liya Aizenberg: First, you always want to see if you have a potential specialist already within your team, or if you can rise one up. You want to empower the growth of your immediate team first. As a leader, I have a periodical check ins with my team members. So I have a good understanding and good knowledge of what people want to do how they're going to move forward in their career.

but there are cases when you project is required very specific expertise and you don't have it within your team and you have no time to train your folks. in this situation, I usually source the expertise outside of the company to make sure the project is moved forward.

We're not blocking anyone. So everything goes plan. However, it's important to pair up this. expert, external expert together with your team members. So now it stays. Internally, and your folks are learning something new, So, it's very important to pair them up, so they can work together.

Your team members can learn new things while working along the side of this external SME.

Adel Nehme: Yeah, that's really great. And you mentioned here bringing up people to become specialists, And that, growing people and training them. What does that look like in practice? I'd love to know how you've approached upskilling data engineering teams so that they can specialize into, you know, their, respective specialist roles.

Liya Aizenberg: As you grow your data engineering team Once you have your periodical check ins, you have an understanding. Some people would like to be technical managers. Some people would like to be people managers. Not everyone want to be people manager. Not everyone want to be technical manager. So you work with your folks to see what's the interest of theirs.

And you also identify the strengths of your team members. And based on the strengths, you assign them to the project, give them the work to do and based on their interest. It's very important to grow your team and keep their interest and keep them excited. of the work they do.

Adel Nehme: And, you know, we've been really focused on the talent side of building a data engineering team, but I think a big question that a lot of data engineering teams, especially, you know, new ones have to face is what is the tech stack that we want to invest in, we mentioned like, between AWS, Google Cloud, Azure, right?

There's a cloud service. There's so many different technologies, right? data pipelining tools that one can use, Can you walk us through how you make a decision of what tech stack to adopt as a data engineering team and what are the factors that you look in to guide these decisions?

Liya Aizenberg: I'm a true believer you don't need to have a ton of different tech to be successful. believe less is more. I choose the tool based on their ability to scale future proof they are. Also, what kind of level of expertise I have within my team, right? I don't want to bring too much. Really something that new that my team has no idea how to do.

And also the price tag of this tool is also very important. You want to make sure that you're not overpaying and you're staying within your budget and cost efficient.

Adel Nehme: You know, oftentimes the data engineering leader when you're, making these decisions on a tech stack, You mentioned simplicity of tools and like not having a lot of tools. Why do you think a lot of data engineering teams tend to fall in the trap of buying so many tools and getting so many tools and like, how do you avoid that as a data engineering leader?

Liya Aizenberg: It's very important to choose the stack that's easy to maintain and adapt and also easy to find the talent to actually support moving forward. So you don't want to have that tool, like multiple tools that, doing the same thing. because it's just waste of money, right? So things to highlight here.

The tools that easily maintain, easy to adopt, the tools you can easy to find the talent, and you don't want to have the tools that overlap, like doing the same thing. As

Adel Nehme: So let me, let's switch gears here and talk about what makes a data engineering team value driven, right? You mentioned earlier in our discussion when talking about what makes a, data engineering team successful, It is that it focuses on value and is results driven, big risk that we discussed behind the scenes is that data teams can generally fall into the trap of building shiny toys, that generate little business value, but are really exciting to put on a resume, Think like, you know, it's deep learning model or this, you know, machine learning pipeline that doesn't necessarily drive a lot of business value. Maybe you walk us through why this dynamic still exists today, and how do you avoid that as a data engineering leader?

Liya Aizenberg: a leader, I'm responsible for bringing visibility to the data engineering across an organization. I build partnerships I usually build a partnership across the organization with various business teams and functions like marketing, finance, analytics, product. We work closely together on the company's strategy and roadmap.

Eventually, this team has become my team's stakeholders, This engineering work gets prioritized and aligned, based their business objectives. This approach allows data engineers to understand the bigger and also helps to identify company wide challenges and the goals.

It also helps my team prioritize its initiatives. And that's how I usually I ensure that data engineering team produces value driven outcomes.

Adel Nehme: And you're talking here about building interlocks, with the finance team, with the revenue operations team, with, different stakeholders within the organization. a good successful interlock look like? What does successful collaboration look like with other teams here?

Liya Aizenberg: You want your business stakeholders to be data savvy, thinking the resource that can be used in many different ways. That's why it is important to educate and show to your business partners what is possible from the data standpoint. A product manager plays a critical role, actually an effective collaboration between data engineering and business.

Hickadys is in the middle between technical and business. It's important that feedback loop has to be established between business and data engineering. Feedback is super important. as well as a regular check in. So when you have a regular check in to see how things are going and figuring out what could be done better, that's definitely improved the collaboration between a business and the engineering teams.

And it's improved how they work over time. the last thing I want to highlight It's alignment on the goals, right? So we all align data team and business. So we all aligned and on the same page what are we trying to achieve together? This means the understanding how the data can help the business.

make sure the data team's goals is to make a business better and to be a good partner across the company.

Adel Nehme: you're talking here about building partnerships with all of the company. I'm sure as a data engineering leader, it becomes really hard to prioritize what is the most valuable thing I can do for the company right now. So how do you quantify different projects, that will deliver business value.

Like, how do you prioritize the roadmap?

Liya Aizenberg: Usually the projects that get prioritized first and the one who actually address critical business needs, Projects that also contribute into revenue generation, cutting cost, or other KPIs. It's always good to figure out if there are any low hanging fruits. If there are any something we can do with a relatively low level of effort, but deliver a measurable bang.

Any quick wins out there, any quick wins we can get. if we find any type of quick win, it's a good thing to prioritize that.

Adel Nehme: When you look at certain projects, you know, a data engineering team does, sometimes a lot of that work is invisible, right? Like, for example, building a data platform or, you know, improving the integration of one dataset to another. How do you look at the value of these projects? How do you quantify the ROI of these types of projects?

Liya Aizenberg: Let's say we implemented a feature that recommends a different product for the customer to purchase. Let's say you have something in your cart and we build a project that recommends the product based on the content of your shopping cart. This feature plans to improve the conversion rate. So if you're right, so like you have a product recommendation and we are committing you to buy another product.

So we're trying to improve the conversion rate. In this case, we would be monitoring a percentage. Increase in conversion rate using like various A B testing functionality. That, how would you quantify? So if we build a product and this project drove increase of conversion rate, that it's considered successful.

there are some projects that actually doesn't have intangible benefits. It's very hard to measure, like for example, let's say data team develops a new feature, That provides the customer this visibility to their order step by step progress. Like, I'm in retail, so I'm talking to orders in shopping carts.

So let's say you place an order, but you don't have, you don't see the visibility how it progresses. Now you have an opportunity to see like, that your order has been received, it's been processed, the payment between. So you see the detail, the progression of your order, We cannot measure the benefits of that project, but it's actually, but we know that it will drive a high customer certification, right?

and we can obtain this information through like using different service different customer feedbacks, but it's actually not intangible and cannot be measured, like conversion rate, for example.

Adel Nehme: Yeah, that's great. Because this is what often time when I think about is that, you know, our data engineering team makes a data set available, a data camp to analyze by business users. how do you measure the success of the ROI of, a feature that is dependent on other teams using it, for example.

So, how do you ensure that as a data engineering leader, that your partners are leveraging the newly built data that is now available by data engineering teams? Right?

Liya Aizenberg: If a customer data platform has been built, and the requirement was to build a unified customer data platform. So working closely with marketing, we can help them build a customer segmentation. For example, we give them a visualization tools that they can use to actually segment their customer base based on the customer data platform we build.

You empower business folks Through education of the data of showcasing what possible and again working together to ensure data is used. Visualization tools are a good place to start.

Adel Nehme: we've been talking a lot about how data engineering teams can drive value and prioritize the roadmap and focus on value driven projects, but I think an important thing to think about here is how you can be iterative and agile as a data engineering team.

So being agile is something data teams across the board need to, you know, adopt more or like are slightly less mature than software engineering in general. So maybe what does agile look like for data engineering teams?

Liya Aizenberg: I think it's important to be flexible. Adapt quickly. If something, unplanned comes up, the team adjust and moves forward without creating friction and roadblocks. Collaboration teamwork, really important. So we need to rely on each other to do their part similars in team sports.

So you have to be relied that your team. Mates will be doing what they need to do. Business strategy changes frequently. Ability to adapt is important and flexibility. Ability to pivot to different direction if need be.

Adel Nehme: And then, we're talking about here, being able to be flexible and to pivot. But sometimes when looking at massive projects that data engineering teams undergo, such as, building a data platform or, building an entirely new data collection pipeline, how are data engineering teams able to kind of piecemeal these types of projects in a way that incrementally drives value? How have you approached this in the past?

Liya Aizenberg: If you have a big project on your plate I usually look at this as a building the house, instead of trying to build the entire house, At once, you start with basic, a solid foundation, walls, and the roof. This is your MVP, the minimum version of the house that's functional and serves the purpose of providing shelter. Same goes to the project. As the project progresses additional features and fresh out there are added the solution in small, incremental steps, right? It's like adding rooms, furniture, and decorations. To the house over time, make it more comfortable and functional as you go. First would be like, first would be old the required MVP minimum valuable product.

And after MVP is delivered we start to grind. We start to prioritize additional features. based on the value that will be, that they will be driving for organization.

Adel Nehme: So, you know, when you're building an MVP how do you ensure that the MVP delivers value, right? Like, how do you define what an MVP looks like depending on the project, for example?

Liya Aizenberg: This MVP have to be defined together with the stakeholders. So we, we collaborate to identify what is MVP should look like for a given project. And identify what features are required and what features are actually nice to have. It can be delivered later. So they get prioritized. The foundation has to build regardless.

We need the walls, we need the roof, all of this stuff. But then you prioritize the features based on the way they drive the value for the business. Start small, win big. Don't try to overhaul, everything at once. Pick a manageable project and break it down into the small, short sprints, like mini projects.

so it lets you see the benefits of Agile quickly and make adjustments as you progress.

Adel Nehme: And, given the above, like, what is the advice that you would give data engineering leaders here looking to adopt agile methodologies? How would you recommend that they start small on their next big project, for example?

Liya Aizenberg: Work closely with your product. Product manager is your, I would say your business voice, right? Work closely with your product manager, work closely with your business and define what is the important for organization at this particular time, And like I said, start small, win big one step at a time. And prioritization and value is defined together with business stakeholders and product managers.

Adel Nehme: Okay, that's really great. you know, we've been talking about how data engineering teams can be agile and can drive a lot of value. And we also talked about how data engineering teams can be results driven, right? And focus on Projects that matter, we alluded to this earlier in our discussion about the risk of pursuing shiny toys, And I know now with generative AI we see a lot of hype today about, the importance of building generative AI tools and generative AI product features, Do you find there's an increased risk of building something shiny and not necessarily useful here? How do you balance that as a data engineering leader?

Liya Aizenberg: It's important to create vibe and atmosphere where everyone in the data team feels like they can come up with a cool new ideas and try them out. But at this point in time we need to make sure the team knows that it's important to focus on ideas that actually help the business. It's like, hey, let's try new stuff.

and see what works, but let's also make sure it's useful and get the job done. goes back to result driven. you always want to bring the results. You don't want to just steal your wheels for no reason. You want to bring the results. you want to bring the valuable outcome.

Let's try, let's experiment, but let's concentrate on tools and functionalities that actually bring the value to the business we are in currently.

Adel Nehme: And with the context of generative AI, what is the role of the data engineering team necessarily in building these tools? Like, how do you see data engineering teams building generative AI, what's the role of the data engineering team in building generative AI use cases?

Liya Aizenberg: Primarily I see data engineering team responsible for actually bringing the data over from all different sources. Generative would require a lot of data from a lot of different sources to be right in the one spot. for actually models to work and use this data. Primarily focus of data engineering actually to bring this data over.

possibly stitch it together so the models can be used as data for learning and training. So responsibility actually to bring this data over and stitch it together for models and data science folks to use.

Adel Nehme: And do you find that the data engineering skill set will have to evolve, for example, to build, retrieval augmented generation pipelines or to build, really specific type of pipelines unique to generative AI use cases? Like how do you see the skill set evolving? Data engineering teams evolve over the next few years as generative AI becomes more prevalent.

Liya Aizenberg: what I see is that a volume of data changing, so we will need to adapt the skills and tools that can process all of this. large amounts of data. And like I said, that space and skills are evolving all the time. And we, whatever we did 10 years ago, it doesn't make sense today. And honestly, I see the same thing happening in 10 years from now.

So we need to adapt, we need to learn, we need to stay on top of what's going on in today's world. Data world, again, a lot of data, data sets we'll need a lot of storage and a lot of horsepower to process all of that and build actually something valuable. Uh,

Adel Nehme: 100%. I think we see a lot of use cases about not a lot of value yet. And then maybe as we close out our conversations, Leah, what are, trends that you're seeing in the data engineering space? You know, as we're talking about generative AI, we're talking about different tools. What are trends that you look in the data engineering space that you're excited about?

You're looking forward to seeing in the next few years or, next year is going to be maybe hard to predict by the next 12 months.

Liya Aizenberg: I see a lot of chat bots being used. I see a lot of changes in adopting the a lot of I, like, I don't think we're there yet I think we've just taken a small incremental steps towards ai. I don't think like everybody, a hundred percent understand what's going on.

It can be like not a hundred percent oriented to what's happening in the field. I think we just take again like small steps and learn and try different things. See what sticks, what works. And try to be creative and don't try to reinvent the wheel, right? If there is something exists, let's reuse it.

Let's use it. Let's see what we can adopt a lot of unknown and a lot to learn still for everyone in this space.

Adel Nehme: I couldn't agree more. And maybe as we close out our chat Leah, do you have any final notes to share with the audience?

Liya Aizenberg: I would say experiment, don't afraid to fail, learn from your mistake and don't forget to have fun with it. it's unknown for everyone. There are not a lot of experts in the fields at this moment, so we're all learning together. And ask questions, Ask questions, reach out to your teammates, reach out to your partners and keep learning.

And don't forget to have fun. That's important.

Adel Nehme: Definitely have fun. That is very, very important. Thank you so much Leah for coming on DataFriend.

Liya Aizenberg: Thank you, Adel, for having me. It was a pleasure.