The New Data Engineering Team

Summary

Data engineering has quietly become one of the most consequential roles in any company betting on AI.

In this session of DataCamp's Data Science and Engineering Week, host Richie Cotton spoke with three leaders about how the job is shifting: Pradnesh Patil, CEO and co-founder of the agentic data engineering platform Ultimate AI; Neha Tharani, data foundation lead at the reinsurer SCOR; and Lisa Mirkovic, an independent data and AI strategy adviser who was previously VP of data engineering at Capital Group.

The consumption layer has moved from dashboards to conversation. Data now arrives as PDFs, images, and transcribed meeting notes rather than clean tables. Teams that invested early in solid data foundations are unlocking generative AI use cases that siloed teams cannot reach. Hiring still rewards the fundamentals of data modeling, distributed systems, and architecture, and now expects working knowledge of embeddings, vector search, and RAG. Interviews are being rebuilt around short exercises because AI-polished resumes all read the same. Governance has become a daily job as autonomous agents run queries, resize infrastructure, and occasionally drop production databases.

The panel stays candid about what works and what breaks. The full recording covers the specific tactics each leader uses to hire data engineers, validate AI output, and win management support for data quality before the AI spend starts.

Key Takeaways

• The primary customer for data teams has shifted from analysts building dashboards to business users and AI agents asking questions in plain language.
• Multimodal, unstructured data such as PDFs, images, and transcribed meeting notes is now standard input for data engineering and analytics work.
• Treating data as a product, packaged with metadata, tags, and lineage, is what lets generative AI understand context with little extra effort.
• Core data engineering fundamentals still matter most, but employers now expect working knowledge of embeddings, vector search, and RAG architectures.
• AI-generated resumes have pushed teams toward short, exercise-based interviews where candidates are told to use as much AI as they want.
• One Fortune 500 team cut infrastructure costs 30 to 40 percent by letting agents autonomously tune instance sizes and configuration.
• Autonomous agents create new governance risks, from thousand-dollar runaway queries to dropped production databases, which is why teams need guardrails that span tools.
• To win support for data quality work, run AI against messy siloed data and against a curated data product, then compare the cost and the accuracy.

Deep Dives

From dashboards to conversation

The biggest change the panel described is who data teams now serve. For years the customer was the analytics team, which turned clean tables into dashboards people filtered. That has flipped. "The consumption layer has shifted fundamentally," Neha Tharani said. Users now expect to ask a question and get an answer, and they want it to look ahead rather than only report the current state. As she put it, "users don't really care about dashboards... they actually want to put the questions in a chat, and actually get their response in real time."

The inputs changed at the same time. Tharani's team took on a problem that has long frustrated insurers and reinsurers: reading contractual data and comparing it across years to judge whether a risk is worth taking. Instead of the tidy tables her team was used to, the source material was documents. Her team parses everything from the PDFs, then combines those PDFs, images, and meeting notes into a central vector space that client managers can query directly.

Some of that source data is stranger than it sounds. One input stream was meeting notes captured automatically, side conversations included. "It's also meeting notes, where we just discussed about, like, some nice wine, and those were captured because GitHub Copilot was transcribing," Tharani said. The wider point held across the hour: multimodal, unstructured data is now normal input for data engineering, and running analytics on top of it is a recent capability. Tharani was also clear that many organizations are still getting the basics of data quality, lineage, and master data management right, and that the teams who benefit most will combine those fundamentals with the new AI work rather than choosing between them.

Data as a product, not just a table

Lisa Mirkovic's definition of success had nothing to do with a flashy demo. It was a decision her team made to build a solid data foundation, which she described as a unified platform to pull data from and publish data to. The part that matters most is the mindset. She talked about "treating data as a product and not just a table," meaning every dataset ships with metadata, tags, and lineage attached.

Her analogy was shopping. "Just like when you go to Amazon... you don't just look at the product itself. You look at, like, how many stars it had and what's the price and what's the origin," she said. That surrounding context is what tells you whether to trust what you are about to use. Package a dataset the same way, and a person or a model can judge it before relying on it.

The payoff showed up when generative AI arrived. Teams that had already packaged their data this way got working AI use cases with little extra effort, because the model could read the context that came with the data. Teams whose data sat split across silos got much less out of it. Her conclusion was that the foundation work pays for itself: "it unlocks scale and speed," which is what most teams are chasing.

Cotton connected it to a familiar frustration. Anyone who has worked with data has opened a random table and had no idea what a field meant. Applying product-quality standards, with clear documentation, owned metadata, and visible lineage, is what stops that from happening, and it is what makes the data usable by AI systems in the first place.

What data teams are hiring for now

The requirements are changing, though less dramatically than people assume. Tharani screens for the fundamentals first: engineers who are good at data modeling, distributed systems, and data architecture. What is new is a second layer on top. Her strong candidate understands "few shot learning, embeddings, vector search, RAG architectures." She drew a clear line on how far that needs to go: "I don't necessarily need someone who can train a Frontier model, but I would need someone who understands how to connect enterprise data... to the AI systems, responsibly and efficiently."

Mirkovic sees the roles themselves blending. When handoffs between product, engineering, and analytics slow a team down, companies collapse them, the same way they did during the move to cloud. She hires for range over specialization, "this generalist or what we like to call, like, the m shape," someone with broad end-to-end knowledge of how to deliver a data solution who is also deep in one or two areas. For senior candidates she looks for the ability to spot where AI went wrong and explain it up and down the organization. For juniors she looks for agency: when something breaks, do they call someone else or dig in and take charge.

Patil described the same problem from the hiring side, where AI has made screening harder. "Now every resume is perfect because people view job description into... some ChatGPT and Claude," he said, and video interviews are easy to game with live prompting tools. His team rebuilt the process around work instead of conversation. "We give exercises to people," he said, capped at sixty to ninety minutes, "and you can use as much AI as you want." A short discussion follows. He noted that many startups around the San Francisco Bay Area are changing how they interview for the same reason.

The soft skills that decide who succeeds

Every panelist put critical thinking near the top and gave it a concrete shape. Mirkovic's worry is misplaced trust: "AI looks like and sounds very confident," she said, so an engineer who accepts its output without question becomes a liability. Her habit is the five whys, asking why a result was produced and how a decision was reached until the reasoning holds up. She paired it with the ability to keep working when nobody, including the frontier labs, knows exactly what the role will look like next year, and her shortlist for that came down to adaptability, resilience, and critical thinking.

Patil turned critical thinking into a workflow. Because large language models are probabilistic, he argued, the validation layer around their output is the thing you actually build. His example was SQL. When AI rewrites or optimizes a query, the test is simple: "is it producing the same data with a new query?" You do not need another model to check that, only an old-fashioned data diff tool that compares the two result sets directly.

He also flagged a collaboration cost that AI created. A teammate can generate a six-page document in five minutes and then expect a colleague to spend forty-five minutes reviewing it. His rule is to edit AI output down before passing it on, because "all these LLM models are text hungry. If you ask it to do something instead of writing five lines, it's going to write five pages." Tharani added the skill of translation: a data engineer sits between business stakeholders, data teams, AI teams, and governance teams, and increasingly writes prompts the way you would explain a task to a five-year-old. Mirkovic's aside drew agreement from the panel: "I think we need a critical thinking class on data camp."

Governance, guardrails, and enabling the business

Governance is where the new risks concentrate, and Tharani was happy to own it: "It's my favorite topic." The responsibilities she listed are recognizable, including data provenance, cataloging, quality, access controls, and auditability, all the way to keeping traceability from source systems through to AI outputs. What is new is the failure modes AI amplifies. Models have an instinct to answer confidently even when they are wrong, which leaves a human to catch the error. On top of hallucination sit shadow AI, where employees paste proprietary data into their own tools, plus bias amplification and IP risk. Her reframing of the whole problem: the question is no longer whether AI can answer, but "what actions is AI able to take."

Patil has watched agents do real damage without guardrails. He pointed to public stories of an agent running a query that cost more than five thousand dollars, and of production databases being dropped at larger companies. "If you let it scale without guardrail, it's going to do crazy things," he said. His answer is a governance layer that spans tools rather than per-tool settings, and his team open-sourced a rules-and-permissions framework, Ultimate Core, under an MIT license as one starting point.

Agents also produce wins when they are fenced in well. Patil described a Fortune 500 company that built agents to manage its data infrastructure, resizing instances and adjusting configuration automatically as workloads shifted. Infrastructure utilization climbed and the bill fell 30 to 40 percent. As he put it, "as humans, we can't change infrastructure configurations, thousand times a day, but a machine can."

The other job is getting a team's tacit knowledge out of people's heads. Mirkovic described how much of it stays "hidden in Wikis and Slack channels and in a senior developer who is there twenty seven years, and only that person knows quirks that are not even in a GitHub repo." Patil sees teams working to democratize that knowledge so business users can self-serve the simple questions and bring only the hardest 20 percent back to the data team. That is where both agreed data engineering is heading: less doing the work by hand, more making the work repeatable for AI and for everyone else.