Navigating the Challenges of Product Integrations with Gil Feig, Co-Founder and CTO at Merge

 But it's really hard for them to go into Brex every time someone joins, add a new employee, and then if they get terminated, revoke that access. But also make sure the rules are set up correctly, the manager approvals are set up. It's really complex, but all of that data is contained in a company's HR system.

Who needs to approve, what their job title is, so rules can be set. So instead of actually having someone manually go add all this information to Ramp and Brex, Instead, they now offer native integrations with the HR systems of customer platforms. The problem being, every company uses a different HR system.

Workday, ADP, you have Rippling, Justworks, there's so many. And when you want to launch something like this, you have to integrate with all of them. That could be 50, 100 integrations. And so, essentially, it has become just a really big problem and a really unmanageable web of connections that companies are having to manage these days.

Richie Cotton: Okay that certainly makes sense. I mean, I think about how expenses worked like a decade ago and it's like, well, you weren't sure whether you were going to get approvals. You had to, like, drop an email to your manager before you bought something and then once it did go through, it's like, well, I, Take a photo of the receipt and upload it and all these like platforms for managing expenses.

Very, very cool. Like it does save a lot of boring admin work, but then, you need to make sure it hooks up into all your other systems for you to actually be able to take advantage of these sorts of things. So, yeah, I can certainly see how that's a very cool use case. Has something changed in the sort of software landscape or the product landscape that means integrations are more important now than they used to be?

Gil Feig: Oh, yeah, I think a lot has changed over recent years. So you kind of started to see maybe about 10, 10 years ago, the breakdown of these software suites, the, the workdays and the, net suites, and they just do so much. And you had all these startups popping up that specialized and were like, We're going to do time and attendance, but we're going to do it incredibly well.

We're going to build it modern with AI. It's going to be deeply connected to other platforms. so because of that, these new startups needed to integrate with these bigger tech companies to pull in that data. So they started building integrations with all of them. And then consumers or business buyers.

Saw that, hey, all of our software is starting to integrate with all of our other software. We're starting to see these, upstarts that do things really well. We want to make sure it connects to our existing systems. So they're buying things, it was doing it. Then they started demanding actually that their existing systems connected with everything else.

They're like, hey, all these startups have integrations with everyone else. Why don't you? And so that's where you kind of saw this just really take off. Customers really demanding integrations is what drove companies to do it. And now you even see the big software suites actually integrating with one another, integrating with their own competitors.

 And there's a lot of reasons for it. It makes them more sticky. It has a lot of positive effects in their business. And I think in recent years that that became very clear. And that's why you're seeing more of a trend, even among the companies that were originally really hesitant and lock their data down.

Richie Cotton: So, I like this is a customer driven approach and certainly think about, the tech stack, at least at DataGamut, I'm sure almost everywhere else, it seems to have grown a lot. Like every year, there's more and more different software products to buy. And once you start thinking about how you connect them, then I guess if you've got like N products, it's going to be like N squared connections between all these products, or that order of magnitude anyway.

So I can certainly see how the number of integrations is going to have dramatically increased. I guess, leads on to like, how do you scale this then? How do you go from, well, I want to build a software integration to I need hundreds or thousands of these things.

Gil Feig: so I think everyone ends up going there, but I think the problem is you don't realize it up front. I can give you an example from my last company. We were in recruiting and we had to integrate with applicant tracking system. So places you go to leave feedback on candidates. So first we built an integration with greenhouse, then we had customers say, we use lever, integrate with that.

So we did but we never built because we didn't really research. We didn't know it was early. But we never really built a system that was meant to scale to all the integrations our customers were using So ultimately we ended up just kind of frankensteining one on top of the next until we hit this point where we were like, this has to be already done.

But also we don't have the resources to even add one more integration because we're maintaining that we're managing that we're supporting customer requests that are coming in about Bugs or issues or even just someone not knowing how to onboard like it was just it was insane and it was killing us. It was killing our business.

We're turning down really big deals because we couldn't build those integrations. so ultimately that's really what led to the idea of merge in general was company should not be building this in house. Companies used to think it was right to build their own data entire warehouses with with all their own servers and racks.

And was pretty clear you shouldn't do that move to AWS. It's the same thing with integrations. Do not build 50 connections. It just makes no sense.

Richie Cotton: Okay. I like the idea that it's kind of the same as moving to the cloud. It's like, you just want someone else to manage the things that aren't part of your core business.

Gil Feig: Exactly. And it's ongoing. It's, you know, a lot of people, I think, think about these integrations as a one time, build it, set up, done, because we're all proactive engineers who write a ton of tests. And then, as we see in the data world a lot, the data doesn't follow the contracts, the docs are wrong, and we end up getting data that Should be impossible to get, and we have to be able to handle that.

So there's a lot of maintenance and management on merges. And we have a lot of tooling and, you know, we're not built out to manage that, but otherwise that does fall on on the business who builds those integrations and at scale, it's just not possible to maintain and manage all of that.

Richie Cotton: Okay, yeah, I think the idea of terrible data quality and impossible data actually occurring in real life, that's gonna be something very familiar to a lot of our listeners. So yeah it's, it's nice for someone else to handle it, I suppose. All right. So, talk me through who needs to care about this stuff.

Like who is normally responsible for building product integrations?

Gil Feig: so one thing that we like to emphasize at Merge is obviously building is part of it. But integrations are a full company strategy. You know, this isn't, some post a notification on Slack when something happens. You know, we're talking what, people listening to this podcast are familiar with.

Millions or potentially billions of rows of data moving daily. And so obviously, of course, data engineers are involved in software. Engineers are involved. Product managers actually tend to be the buyer because the value there is at a ton of integrations at once. They're the ones really working on that data and software engineers are going to be the sort of evaluators, the ones who get the technical buy in.

But you also have partnerships, design, you have technical customer support, you have pre sales and then any other post sales roles involved. You know, imagine you launch 50 integrations and a customer's like, hey, my API key is showing an error when I go to try to generate it. How does your team know how to go handle that for one out of the 50 integrations you just launched?

So that's why we say it's a full company effort. But of course, data engineers and engineers are responsible for that build.

Richie Cotton: Okay, so I kind of like the engineers at the core of this, both data and software. But yeah, obviously you do need all that kind of supporting infrastructure around, particularly on the product side and then support side as well. so, maybe some motivation then. Have you come across any success stories where companies have built a load of these integrations and then it's paid off with some sort of business benefit?

Gil Feig: Yeah, absolutely. I gave the example of Ramp and Brex. I think, a really exciting one. I can give another example here, which is Guru. So Guru essentially provides enterprise search for businesses, so they effectively come in, they log into Guru, they connect all of their business systems.

So that'd be their file storage system, whether that's Google Drive then they would connect their ticketing system, Asana, Jira, those sorts of things. HR, you have Workday, you have ADP, JustWorks, and then you know, a bunch of other platforms too. I mentioned ticketing. we also offer accounting, CRM.

So continues to go on. Companies connect basically everything they can that all gets indexed into Guru's search. And then they provide AI powered enterprise search for businesses. They need as much data as possible they're working within the confines of the best models that are available to everybody.

And so having that data is giving them a real competitive advantage there. They've onboarded thousands of customers onto their platform onto merges integration, which is really exciting.

Richie Cotton: Okay, so it does seem like basically if your customer is using like these other business software where the integration is required, that's going to be like a really important thing just for actually winning the business in the first place. And then also, I guess, once you get them set up, then it's going to make your product a little bit more sticky because, you know, they're then using two bits of software at once.

And so, yeah, a little bit more locked in.

Gil Feig: firmly believe that integrations are actually sticky in both directions. So a lot of times the system that's having mostly data pulled out of it those companies tend to fear people are extracting everything out of my system. We're losing value. It's not worth it for us to keep this API open.

But what they're actually finding is similar to Salesforce. People end up building a real ecosystem around those platforms. So, effectively, the other platforms that are integrating now depend on the data from that source system. And that customer can never get off that source system because their whole business is now running on it.

Richie Cotton: Okay so yeah, definite business benefits there. I'm wondering how it's going to change the engineering side of things. So, if you're living in a world where product integrations are easy. How does that change how the engineers work? Maybe we'll start with the data engineer since that's pretty close to the heart of 

Gil Feig: I think when data engineers, when we think about building integrations, The big one is this, I need to ETL a massive amount of data. Let's focus on the retrieving case. There's also writing data back. But let's just say for the ETL case here, you're pulling a ton of data out of the system. You have to build new ETLs for every platform you integrate with.

So, let's say your PM's like, we want to launch HR integrations to facilitate auto, you know, sending credit cards. All right, let's go build 50 ETL. Actually, it's a lot more. It's, it's maybe 10 ETLs per integration. So you're talking 500 ETLs. You're talking just a mass amount of maintenance. all these problems that are kind of, you don't have to deal with any of that with a unified API because you just build one.

You build one per piece of data and the unified API handles the actual differences between each platform. So that's, I would say, probably one of the biggest ones. The next one is, yeah, you really don't have to deal with the bugs. You don't have to deal with the data cleanup. You don't need platforms like great expectations or other, other things to make sure that your data is really in order here.

You can really just rely on Merge's normalization or whatever platform. And then also observability. So Merge, for example, offers a whole suite of observability tools. So. These could be things like data, connection issues you know, rate limits, anything that's popping up is all clearly documented and spelled out.

So it's just really easy to figure out what's going on with any integration. No need to build anything to monitor that.

Richie Cotton: that's kind of interesting. There's three different stages there of the data engineering workflow that's affected. So it's like your original, well, how do I do the ETL, so how do I get the data into my data warehouse? And then how do I check the quality of that? And then how do I observe that it's continuing to run as intended?

And all three stages are kind of basically, want those automating mostly anyway, I guess. All right, nice. And so, just related to this one idea that's becoming very popular in the data world is the idea of data products. So the idea that you treat data like a software product and you have like, guaranteed uptime and all this sort of stuff and APIs for like how it works.

This sounds very relevant to the sort of stuff you're working on. Have you thought about data products at all?

Gil Feig: In a lot of ways, we are a data product, right? Like our, API is just a way of retrieving data for your customers. Uptime, reliability, data quality, all of these things matter, and not just, you know, us saying it, we need our SOC 2 and our ISO audits to cover that, and it's not just security, it's, it's accuracy, it's, it's reliability we get audited for all of that.

So yes, it's critically important. And then among our customers to we're really helping power a lot of these use cases because a lot of them are, you know, things like sales dashboards pulling all my data from my CRM and display it in a way that gives me real insights over that data. They also have serious requirements around providing a great product in a lot of ways.

They're just a wrapper around data. So think it's a couple approaches. That one's a little bit more of a stretch, but absolutely, this is the ecosystem that we're in.

Richie Cotton: So I guess beyond the data engineers, like how is it changing the workflows for other people around the company?

Gil Feig: So I mentioned post sales before, you know, that's a really, really relevant one. It's anyone who gets involved with the deal after it's closed, but it actually pre sells as well. You need to be able to talk about these integrations. You need to be able to support them. And when something that a data engineer software engineer works on becomes customer facing, but it's still a little too technical, you Support can't really handle it.

You end up having to rope engineers back in, and when you have engineers directly connected to customer requests, that's the death of your business, right? That is so expensive. It's impossible to maintain. And so that's where, you know, these these observability tooling being non technical, making it so that someone in customers success can actually handle any incoming issues without looping and engineering critical to launching successful integration strategy overall.

Richie Cotton: So my understanding of this is normally if it's pre sales, then your account executive is going to sort of try and handle questions if it gets too technical, they're open a sales engineer. And I guess the idea here is that maybe you don't need that expensive sales engineer cause their time is always precious.

And then I, guess on the post sales side, it's like normally the customer service manager is going to be helping out until it gets too technical. Then you get a solution architect and you can avoid the need for those people being involved as

Gil Feig: Exactly, exactly. But, but ideally, you know, when you think about these integrations, it's like you want every single one of your customers connecting most of their platforms. And even a solutions architect like having anyone involved can be expensive. And so productizing the onboarding experience itself is very, very important you just don't want anyone else involved.

 And if you do have people involved, that's where you want customer support to be able to provide that frontline support using tooling that engineers built. That's, very clear about what's going wrong.

Richie Cotton: So, in terms of trying to implement this how do you go about it? Is it, start off with like, you try and build a product integration yourself and see where you're falling over and then, and then try and scale it once you've worked out the kinks? Or, what's the flow here?

Gil Feig: It's funny, Richie, we always say that businesses that have built at least one integration are our best customers because they have felt the horrendous pain that comes with this. I personally love getting on a call with like a, no offense to anyone, like a know it all brand new engineer who's like, well, I built a Slack integration in college and you know, it sent notifications and it's just not that it's, so complicated.

so much deeper than that.

Richie Cotton: I don't know whether you can name names, like, what are the worst sorts of product integrations to build then? so I mean, there are a lot, but data rich integrations with any core system of record is going to be tough. So look at a system like HR. when we think about data volume, so HR, you have all the people in a company. That's not going to be too high, right? Like even at Boeing, a million people, maybe, you know, 10, I don't know, whatever million if you include terminated.

Gil Feig: But then for each one of those, for every week that they were there, they got a paycheck. That's pretty bad. Then when you look at something like an accounting system, you're talking about, okay, every invoice that's ever been created every account that they've ever transacted with, the data starts to go up a little bit more.

But worse, you know, you look at something like a CRM system, where not only is every customer they've ever talked to in there and every deal they've ever run in there, but they've also imported massive data sets of just millions of companies from around the world, millions of contacts. They have this auto connected to scrapers and things that are just pulling in so much data.

That's where you really see struggle. So core systems of record in general, a lot of data, but systems like CRM file storage tend to have a lot of data. And then lastly, the always the other thing we're always looking at is how old are these APIs when you have a really ancient API? It has bad access patterns, and there's no way to really pull out the data you need.

 So you often make know, this is a, query term, n plus one queries where, first you have to query for the overall set of IDs, and then for each ID, you have to kind of look up the data, which is really inefficient. You want to kind of look up everything at once. You run into that problem a lot with API integrations, and it actually can make it physically impossible to extract all the data in a reasonable amount of time.

So you have to work really hard within those limits and figure out what you can do and productize. Hey, how can I best build this integration given the physical constraints that exist here?

Richie Cotton: yeah, I can certainly see how, like, your CRM system or something like that, it can get huge and unwieldy because there's lots of different types of data in there. It's going to get complex. I'm curious, most of those things sound like text data. Is it mostly text data you'd be working with or can you go full on multimedia?

Gil Feig: Yeah, multimedia. So we integrate like when you think about a recruiting system, an applicant tracking system. Every single candidate who's applied for a job or almost everyone has uploaded a resume, potentially a CV. that's all pulled, that's all available.

 When you think about a file storage system, there's metadata, the file structure, but otherwise it's just pure files. That's actually incredibly complex because of the rate limits on these systems. You're trying to sync the file structure while people are also trying to pull files. So yeah, I can get very messy there as well.

Richie Cotton: Okay, yeah, I suppose like at the very least you can have things like headshots of candidates in your, in your HR integration, all that kind of stuff. Yeah, I suppose like, a resume is just like a, excuse PDF or whatever. all right. So for people who are interested in building these things what sort of skills do you need?

Gil Feig: So I think just experience ETLing a massive amount of data is very important. You know, need to distribute the workloads. You need to make sure that you know, you're pulling the data efficiently. You're not, doing it in a way that would leave gaps data. But Merge has a lot of guides.

Us personally have a lot of guides on how to actually sync that data, how to set it up properly. would just recommend really doing your research before diving in. Because it's easy to think about these, and it's easy to test them when you're on some small test instance. But the second you onboard a customer with an amount of data, where if you took the amount of data and divided it by the rate limit, it would take, like, the old Windows screens, like, 475 years to transfer.

 That's where you really need to go deeper.

Richie Cotton: That's interesting. Yeah, certainly I can see how waiting 475 years for your data to transfer. That's not a great customer experience.

Gil Feig: There you go.

Richie Cotton: But it sounds like mostly what you need to know is just being able to understand how to work with APIs. Is that kind of the crux of it or are there other things as well?

Gil Feig: Yeah, I think knowing basics of APIs and understanding the importance of data transfer in terms of, Both an initial sync, so like data grab, and then ongoing syncs. How you can efficiently only pull changed data as opposed to repulling full data sets every time. So basics of, like ETL syncing.

Richie Cotton: And are there sort of product management skills you need here as well? Or is it basically just engineering skills?

Gil Feig: So I think product management skills are important because I mentioned this before, you need to rope in your whole company. And so it depends what resources you have available to you. If you're at a larger company, right, you can be that data engineer who really focuses on the edge build.

But if you're somewhere smaller, you do need to think about that full company, build and what happens. What happens when customers start reaching out for support? Does our team know what to say? that's why I think unified API platforms like merge, you know, they really provide a lot of that.

So data engineers can do what they want to do best and PMs can do what they want to do best. and no one has to kind of take on every role at the company.

Richie Cotton: So know you've run engineering teams before, so do you have any career advice for aspiring engineers?

Gil Feig: think this advice would probably not come off as, you know, extremely original, but I think it's so important and people tend to brush it off. You have to work hard. It's your time to learn when you're early career. You just One, you'll never have as much time to code again in your career, and you'll never have as much mentorship available to you.

It's really important that you're sucking up all the information you can. and when people say in your first three months of work, you'll learn more than your four years of college, it's absolutely true, as long as you put in the effort for it. I think in my first month of work, I learned more than, four years of college.

And so I, I just, you know, work hard, invest really heavily in your own learning. Don't take a job that's paying 10, 000 more because, you expect that, the money is going to make all the difference when it could be a slower growing company. You know, it doesn't have the same number of coworkers to learn from.

 All those things are just, temporary. And I think taking the learning will make you a lot more than 10, 000 a year later in your career.

Richie Cotton: Yeah, it's a trade off between short term gains and long term gains. And certainly I like the idea that learning throughout your career is incredibly important. feel like, you know, sort of mid career and still learning a lot. Okay, nice. Actually, do you have any career disaster stories you can tell us about? lessons learned. I always, I always get that wrong. Lessons learned.

Gil Feig: Okay, so I actually I have a I have a couple lessons learned. I think my biggest lesson learned has to be grass is always greener in tech. I think that every company always looks appealing. You always want to move and you always see the new hottest thing on TechCrunch or the new hottest trend like AI, which I'm obsessed with too, right?

 think all that being said, Every company has its thing going on on the inside. Every company is going through a lot, and I think you really need to optimize for yourself, for your learning, and for growth there.

Richie Cotton: Yeah, do like that idea. Just optimize for learning and growth. And of course, since you mentioned generative AI, it seems like we always end up talking about generative AI on the show, like whatever the topic is. And I think we've done quite well in avoiding it so far. is there a relationship between product integrations and gerontive AI, or is that one stretch too far?

Gil Feig: so I think, I think actually integrations and just the data engineering role in general in Gen AI, what I think is really cool is we are the picks and shovels for, for AI companies. Which is a reference. If you haven't heard this back to the gold rush when everyone moved out west and it was the people selling the picks and shovels that made the real money.

They were the winners. And ultimately that's us in the era. You have merge as a data provider. You've data engineers who are actually making the data flow. any individual business product that can give people a competitive advantage right now. When again, everyone has access to the same models.

Makes you a pick and shovel who's really going to succeed. 

Richie Cotton: Okay, that is fascinating. I think there's a lot of people going, well, you know, gerontive AI is where the money is, it's very exciting stuff. But actually going a few layers down into the data engineering, into the kind of fundamental infrastructure, that's like, a much more reliable source of value, I suppose.

Gil Feig: Yeah, absolutely. And as much as AI continues to change, whatever happens, it's going to need people like, all the listeners here and, products like merge.

Richie Cotton: All right. So, what are you most excited about in the world of product integrations?

Gil Feig: Well, a lot of things, I think one growth, really excited about it. I'm going to repeat AI because I think that, It's going to also change how the world of product integrations works. it remains to be seen, but right now building API integrations is incredibly cumbersome.

It's complex. It's tough to get the data that you want. And I'm excited to see how AI changes that landscape just to make data exchange a lot easier.

Richie Cotton: I do like the idea of just making things much easier for developers, for engineers, and simplifying a lot of the stuff where it's, I guess, integrating different products. it's a niche idea of a good time. So I guess the less time you can spend doing that and the more time you can spend on just like scaling things out, helping your customers, that's going to be a definite win for a lot of businesses.

Gil Feig: Yeah, and the truth is this fragmentation is just going to continue to grow. The problem is going continue to get worse. You see more and more vendors offering integrations, which is a snowball effect because now more and more buyers are expecting integrations. And you're just seeing more and more of that demand.

And also with that, merge is seeing more and more demand for unified APIs.

Richie Cotton: All right. Do you have any final advice then for people who are interested in building these integrations?

Gil Feig: Absolutely. I just think before you dive in, you should definitely check out unified API providers out there to see just what it takes off your plate. I do also recommend try building one for fun on your own first. I think you tend to see what happens when the documentation is bad.

The APIs don't function as expected. And when you finally onboard your first customer, it's, It's a huge pain, and it was so strong that it's what led me and my co founder to leave our companies two months into COVID to go start Merge. It was a very scary time, but I think the pain was so acute that it was just obvious we had to do it.

Richie Cotton: Okay. Pain driven development is more common than I think a lot of people realize. So yeah, I like the idea that it was so bad that you went off and started your own company to solve the problem. All right. Nice. Okay. Super. Thank you for your time, Gil.

Gil Feig: Thank you so much. Appreciate it, Richie.