Building High Performing Data Engineering Teams

Summary

Successful data engineering teams are key to driving business success, and the secret to managing such teams lies in encouraging collaboration, ensuring transparency, and aligning technical goals with business objectives. Leah Eisenberg, Director of Data Engineering at Away.com, shared insights from her experience in leading data teams. She stressed the importance of balancing technical proficiency with cultural traits such as trust, accountability, and adaptability. Leah also pointed out the role of internal partnerships in improving team productivity and discussed the need to prioritize projects that increase revenue and reduce costs. Moreover, she considered the necessity of keeping up with evolving technologies while maintaining a focus on value-driven work. The session emphasized the importance of strategic planning, cross-functional communication, and the efficient use of technology in forming successful data engineering teams.

Key Takeaways:

• Successful data engineering teams require a combination of technical skills and cultural traits such as trust and accountability.
• Aligning data engineering goals with business objectives is key for driving impactful projects.
• Internal partnerships and visibility are important to improving the productivity of data engineering teams.
• Striking a balance between experimenting with new technologies and focusing on value-driven outcomes is essential.
• Standardizing processes and using templates can reduce downtime and improve efficiency.

Deep Dives

Forming Effective Data Engineering Teams

Leah Eisenberg stressed the importance of both technical skills and cultural traits in forming successful data engineering teams. She identified trust, collaboration, transparency, and the freedom to make mistakes as essential components. "If the team has these traits, half of the success is already there," she noted. Successful teams work like a sports team, where each member trusts and supports one another. Transparency, especially from leaders, ensures everyone is aligned with the project's goals. Moreover, encouraging an environment where team members are not afraid to make mistakes promotes experimentation and innovation. Knowledge exchange among team members also plays a key role in forming strong teams. In addition to these cultural elements, technical skills such as proficiency in SQL, Python, and understanding of both relational and non-relational databases are fundamental. Leah highlighted the importance of ongoing skill development, suggesting that managers should facilitate mentorship programs and encourage participation in industry conferences and certifications.

Aligning Data Engineering with Business Goals

Aligning data engineering goals with business objectives is key for maximizing the team's impact. Leah discussed the importance of being part of strategic planning sessions to understand the company's priorities and align the engineering roadmap accordingly. She mentioned that data engineering teams often work in partnership with other departments such as marketing, finance, and product teams. This collaboration ensures that engineering efforts are directed toward projects that increase revenue and reduce costs. Leah also pointed out the role of data engineering teams in identifying quick wins for the business. By showcasing the team's ability to simplify processes and reduce redundancies, data engineers can demonstrate their value to the organization.

Improving Team Productivity through Internal Partnerships

Internal partnerships are crucial for improving the productivity of data engineering teams. Leah stressed the importance of bringing visibility to the team's work and educating business partners about the possibilities that data can offer. Rather than waiting for problems to arise, proactive communication can prevent negative attention and create a positive reputation for the team. Leah shared an example where her team quickly addressed a data integration issue during a website launch, demonstrating their value to the business. By actively participating in discussions and translating business problems into data solutions, data engineering teams can build strong partnerships and drive data-driven decision-making within the organization.

Striking a Balance between Experimentation and Value-Driven Work

Data engineering teams must find a balance between experimenting with new technologies and focusing on value-driven work. Leah acknowledged the excitement engineers feel when exploring new tools but stressed the importance of delivering outcomes that drive business success. Experimentation is encouraged, but it should not detract from projects that provide tangible benefits to the organization. Leah suggested using a minimum viable product (MVP) approach to incrementally deliver features and drive value. By focusing on foundational work and prioritizing features that offer the most significant impact, teams can ensure that their efforts are aligned with business objectives. She also mentioned the importance of reducing the tech stack to avoid redundancy and facilitate easier learning and support.