Embedded Machine Learning on Edge Devices

Richie Cotton: Hi, everyone. As you may be able to tell from the accent, I'm not Adele. I am Richie a data evangelist at DataCamp. This week, I'm filling in for Adele, and I'll shortly begin co-hosting the data-framed podcast. I've worked in data science and data science education for nearly two decades.

And I've watched the field go from being a niche, nerdy thing to, well, a mainstream nerdy thing. And I've spent much of my career providing data science support to biologists, chemists, and business people. And my experience is that even brilliant people sometimes get terrified by having to analyze data. So I care about making data feel a little less scary to work with in the coming episodes and revisiting the traditional DataFramed theme of helping organizations become data-driven. I'd also like to provide some inspiration to you all about the joys and the possibilities of working with data.

Once you get over the fear, there is a lot of fun. Today, we'll discuss machine learning on low-powered devices, like bones and internet of things sensors. So there's a fascinating emerging field, and I'm very much looking forward to finding out about it from Daniel Situnayake. Dan is head of machine learning at edge impulse, and he's the author of the book Tiny ML and the forthcoming AI at the edge.

He previously worked at Google on TensorFlow light. So all this is to say he's a real expert in the field. Hi, Dan; thanks for joining us today. We're going to be talking about machine learning on edge devices, and that term's relatively unfamiliar to me. So perhaps for a bit of context, you could just start talking to us about, well, what is an edge device?

Daniel Situnayake: Yeah, great question Richie. And I'm, very excited to be here today. So an Edge device refers to a computer at the edge of the network. And so that sort of makes sense. If you think of the internet as this big sprawling set of interconnected devices, but it has an edge where there are devices that are sort of connected to the greater network on one side.

And then, on the other side, they're connected to the real world through sensors and other sorts of hardware. So a typical edge device might be for example, an internet-connected camera, or it could be a personal fitness wearable that you wear around your wrist, that logs, when you're doing exercise and uploads data about it to the cloud.

And there are tons of different applications for edge devices from everywhere through consumer electronics through to industrial systems, the kinds of devices that monitor industrial processing factories, and even the ultimate sort of edge device is a spacecraft. The Mars rovers that are up there driving around on Mars and collecting data, uploading it, and sending it back to earth. Those are, are furthest edge devices.

Machine Learning on an EDGE Device

Richie Cotton: That's pretty impressive, though. It's such a broad range of applications, but can you tell me why you have to do machine learning on an edge device? So I think the standard thing, if I'm doing machine learning, is that I'm gonna be doing something on my laptop. So that's going to be processing the model, or maybe I'm working in the cloud.

So is going to be some server somewhere processing things. Why would you want to do machine learning on a device that hasn't got much power?

Daniel Situnayake: If you think about sensor data and data about the real world, we sort. Have this idea that we, when we deal with data as data scientists, we're dealing with data sets that have been collected. And a lot of the time we're dealing with data sets that are accrued by our big backend systems and the systems that are generating the data already within our infrastructure basically. But if we're collecting sensor data, so we're collecting data about the real world. There's essentially a bottleneck where the real world has an almost infinite amount of data. It's just this endless source of new information. Imagine you have a, a device out there that's collecting temperature and pressure and all sorts of other stuff that's related to climates or the weather.

It is like an infinite stream of data out there for this thing to collect, but it might not have a very good connection with which to send that data to another system that can analyze it. So typically, edge devices, because they're on the edge of the network, might suffer from unreliable internet connections or low bandwidth, or maybe they need to operate under constrainted energy regime. So they're running off a battery or something like that, or solar power and they can't afford to be burning loads of energy with network communication all the time. If you think about your mobile phone, the thing that uses the most battery apart from the phone's screen is. Using a radio to broadcast information.

So if you are in a situation where your device is constrained in some way and is unable to send all of that data back at the rate that you are receiving it, then it makes sense to put some of this intelligence on the device, rather than depending on having intelligence running on a cloud service.

The inspiration behind Edge Computing

Richie Cotton: Yeah, I can. See how with a Mars Rover, you might not want to send all your data across space from Mars to somewhere on earth and then try to have all the results and send them back. So it feels like a slightly novel field. And I'm curious as to how you got into thinking about edge.

Daniel Situnayake: Yeah, so edge ML and sort of edge AI more broadly, which is the, the idea of doing all sorts of artificial intelligence on edge devices has only existed for a relatively short time. And essentially the thing that's. Propagated it and allowed it to develop is the fact that these edge devices have gotten powerful enough that they're able to run some of these heavy lifting algorithms, like deep learning models on the device. But, I got involved with this because I was working at Google on the TensorFlow team and TensorFlow, as you're probably aware, is Google's big, deep learning framework. And there's a part of TensorFlow called TensorFlow light, which is basically designed around deploying models down to mobile phones, because mobile phones are edge devices, right?

They're these little portable computers on the edge of the network with lots of sensors on. And some folks at Google realized it would be really cool to be able to run models on phones, to do, do all sorts of cool stuff. And then, The sort of next logical step from that was to think about like, okay, what kind of devices are even smaller than mobile phones and more, more edgy and more possible and lower powered.

And is it possible to run deep learning models on those? And I was lucky enough to be working at Google at the time where a new team started by a guy called Pete Warden was starting to build and investigate. And I basically lucked into being around during this awesome new technology being launched.

And I realized it was the most exciting thing I'd come across in the tech industry. So far, I have combined my fascination with embedded software engineering and edge devices with my fascination with data science and machine.

Richie Cotton: That's exciting. and so with the work you've been doing, like with TensorFlow and things like that, how is it different to just doing regular ML? How is it constrained or what makes it hard?

Daniel Situnayake: It's really interesting, actually, because when you're dealing with this type of real time sensor data, that by itself is a very different paradigm to the source of work that you do. On data sets, which are more tabular. So for example, like we're all very familiar with this idea of feature engineering for tabular data.

So maybe we are looking at analyzing the business performance of something or some government data about some kind of phenomena that are going on in our world. Our typical workflow for identifying good features to use from that data uses a, a set of tools, which occupy one paradigm. But when you think about real time live sensor data.

So this is basically like little samples of information that are coming in at very high frequency and essentially are quite challenging to feed raw into a machine learning model. There's this whole extra field. That has existed for a long time called digital signals processing, which is based around the idea that there are algorithms that can make sense of these kinds of real time streams of sensor data and distill them down and get the most interesting components most visible.

So. Whatever you're trying to do downstream. Isn't swamped with all of this raw data. And these algorithms look very different from anything that you do with tabular data. For example, with audio data, you basically at a certain frequency, get a long stream of numbers at a very high frequency, which represents the amount of pressure almost that's happening. The amount that your microphone's diaphragm is flexing at any given point. And to turn that into something that a deep learning model can understand, one of the best ways of doing it is through building something called a spectrogram, which is basically a representation of signals or series of values in the frequency domain.

So instead of having. A single column with these amplitude values, you end up with a 2d matrix, which represents time on the X axis and frequency on the Y axis that's bucketed into the individual buckets. And that is essentially like an image. And that's what you feed into. Deep learning model.

So you can use a convolution neural network to understand it. So it's all about transforming signals from their raw form, which often isn't very convenient to work with into a maybe higher dimensionality but lower frequency form, which is more convenient to use these machine learning.

Richie Cotton: That's fascinating. So I have distant memories of learning about Fourier transforms and wavelets. And things like that for doing signal processing from university. So it's not fresh in my mind. And I think it's something I've not used in a data science or machine learning context before.

So that sort of engineering-type stuff is always very separate. And so it's interesting to hear about how that sort of audio engineering and, with machine learning, the two of these have come together.

Daniel Situnayake: The thing I love about it is that there's this huge field of embedded engineering and signal processing that's existed for a fairly long time at this point. And people who are Absolut. Incredible experts in this type of stuff. And what we're finding is that there's this amazing fusion between the things that are coming from machine learning and deep learning and data science side with this signal processing expertise.

And when you get people from both sides together, you can figure out really amazing ways of interpreting signals and training models. It's really exciting.

Impact of Device Variable on Machine Learning

Richie Cotton: When you've got a very low-powered device and maybe you haven't got a great internet connection, you're limited. Battery life or things like that. How does that affect the machine learning side of things? So do you have to choose different models, train things differently, or how does it affect the actual models themselves?

Daniel Situnayake: So one thing that I'd, first of all, point out is that when we're talking about machine learning on embedded devices, the vast majority of the time we're talking about inference, we're not necessarily talking about training. So nothing is stopping you from training models on these edge devices. And in some cases, this does happen, but one of the biggest constraints of edge devices is that you've got limited memory to work with.

And so often when we're talking about training models in a supervised approach, that just isn't feasible. If you don't have. Enough space to store data. Maybe you've only got tens of kilobytes, which obviously isn't enough to store much of a data set. So if you're doing some kind of unsupervised approach or you've got some kind of interesting approach to how you fine tuning a model on device that can work sometimes, but a lot of the time we're just talking about running inference.

And then when we are, we are talking about running inference. There are really three main constraints. The first constraint is. And connected with that is performance like the performance of the device. Because if you are trying to get a device to run on a battery, for example, you're not gonna have this like two gigahertz processor, multicore CPU, or like a GPU running in there.

There are edge devices that are very powerful, but typically you're gonna see more constrained clock speeds than you will in. Server side infrastructure for running machine learning models. So the issue there is it's quite slow to run models. And so that means that you are gonna be biased towards training smaller models.

Like a smaller model is gonna be able to run more quickly on a device, use less energy and fits this set of constraints on the edge. And then the other two things are read only memory and. So the distinction there is that read only memory is where you're gonna be storing the weights of your model, plus the program that runs on your edge device.

So obviously there's a, a range always. In computing hardware, but at the smallest end, you might only have tens or hundreds of kilobytes to store your model's weights. So if we're thinking about typical, deep learning models for vision, even something like a mobile net model designed for running on a mobile phone is going to be too big to run on some of these smallest edge devices.

So you need to think about how do we make these models smaller, and then. RAM parts. The working memory is just as important because you need someone to store your signal when it comes in and you need to store the, for deep learning, for example, the intermediate activations of your models, layers while you're working.

And if you don't have enough Ram, you can't do that. So these are the three most important constraints and it typically. Forces you to think about smaller models and models, which are designed in a way to be efficient in terms of RAM usage. And it also encourages you to do as much signal processing as you can upfront because there's often this trade off between, you can extract features through signal processing and use a certain amount of computing.

Throw things at your deep learning model, for example, and make it figure out the feature extraction part two. And for any given device, there's going to be like an ideal set point for a particular data set, where there's a, a perfect trade-off between those two sections where it's the most efficient, but that's something you have to find on a per device and per data set basis.

So there's this whole extra constraint that comes. But you don't have to think about when you are thinking about data science on big, powerful developer workstations or on servers in the back end. And for me, that's what makes it really exciting. It's always nice to have a constraint because it forces you to be creative and think of new ways of doing stuff.

Trade-off in RAM vs Predictability

Richie Cotton: That's incredible. And so I get the sense that there's some sort of trade-off where you sacrifice maybe a little bit of predictive power because you're using. Blunt model, but that's necessary to be able to run it at all because you don't have the computing power or the RAM available. Is that about right?

Daniel Situnayake: Yeah. I mean, that can be true. So if we're, if we are thinking about larger, more, highly dimensional data, so images, for example, typically if you are compressing a model to get it, to run on an embedded device that might involve, first of all, just training a smaller model, like coming up with an architecture that has less weights and training that.

And. So that's one way, and, obviously, there's a trade off there with fewer weights, the less representative power the model has. And so potentially you are gonna get performance. This isn't as good as you would on a larger model. The second thing we have at our disposal is this idea of model compression, where you can take a trained model.

In a typical manner, as if it was gonna run on, on any device. And then you can do certain things to it that reduce it. Precision reduce its accuracy potentially, but make it smaller. So one of those things is called quantization. So quantization, which you might have come across in, is also in the back-end world.

It's a way of getting models to be smaller by reducing the precision of their weights in a typical deep learning model. For example. The weights are usually trained in float 32 formats. So we've got these 32 bit floating point numbers. They're very expressive. They can represent a very large range of values, but to get them to run on an embedded device, if you.

Reduce the number of bits available to store each weight. So you reduce that down to eight bits. So you're now storing the numbers that make up your model as eight-bit numbers. That's four times smaller model. So if you can do that and get away with it without screwing up the model's accuracy too much, that can be great.

And there are actually really smart ways of doing that where you. Really lose barely any accuracy, but reduce the model by four times or eight times, or even the smallest precision models are actually. B neural networks, BNS, which have one-bit weights. So literally the weights of the model are binary, and there are some special regimes you need in order to train those, but they, they can work really well.

So all of that comes with trade-offs, and it's about managing the trade-offs and keeping the amount of performance that you need for your application.

Getting Started in Edge ML

Richie Cotton: I think many people listening will want to know, how do I get involved in this? It sounds exciting. So how do you get started?

Daniel Situnayake: There are many different angles to this topic. Some people come at it from an embedded engineering angle where they. Worked in embedded engineering before working with microcontrollers or embedded Linux boards, like raspberry PI, for example, and they've hit a wall where they want to make sense of the world and the sense of data that they're pulling in on their device.

But there's only so much you can do with hand coded algorithms and basic signal processing. So they, they want to embed a bit more intelligence in devices. And then there are people coming from the other end where maybe they have a background in data science. Machine learning. And they'd like to see, like, how could I take some of this intelligence that we're able to encapsulate in models and push it down onto these tiny devices so that it's available in places it wasn't available before.

So your journey might be a little bit different depending on which side you're coming from. Because if you are a data scientist, you are gonna be a lot more familiar with these. Data science tools and less familiar with the embedded side. And if you're an embedded engineer, then it'll be vice versa.

Let's assume that you are a data scientist and you are excited about getting started with this kind of thing. I think a good place to begin. You don't necessarily have to have embedded devices to start thinking about how do you train smaller models. It can be interesting to. Start building an awareness of the relationship between models' size and the amount of resources they consume.

And just start trying to get into that mindset and tinker around with things and start learning like, oh, if you change this, what's the impact on performance. And you can start to learn about things like model compression and quantization. And begin your journey that way. And the easiest place to start is probably with the larger edge devices, things like a raspberry pie, because you can just run Python on there so you can train a TensorFlow model and then run it directly on the device. And there's not that much extra work involved if you're going down to microcontrollers. So these are things like. Tiny little processes that live inside of household gadgets and, really are super pervasive throughout our worlds and our built environments. The workflow for programming with those is a lot more complex.

So they generally require you to write. Code in C or C plus plus, and then use a device specific set of compilation tools that allow you to compile your code to run on that particular device. So a, a very easy to use set of devices in that area made by a company called Arno and they make.

Microcontrollers that are designed to be easy to program and easy to use, and they can be really good starting point for embedded machine learning development, but it can be easy to get lost because there's so much going on in this field. So what I, I would recommend I'm a bit biased because I work for a company called.

Edge impulse. And we have this amazing end-to-end suite of tools, which allows you to integrate with devices, collect data sets, experiment with combinations of signal processing and machine learning models, and then deploy to devices. And. I would recommend personally heading over to our documentation, just search for edge impulse, or go to edge impulse.com and dig in.

And we've got loads of tutorials, designed to make it easy, to jump in and start learning about how all these different moving parts relate to each other and how you can train a model. And.

Richie Cotton: Is there a sort of standard easy first project? Like what's the hello world equivalent for ML?

Daniel Situnayake: So I wrote a book along with Pete Warden, from the TensorFlow, like the micro team a few years back. And the hello world project that we came up for tiny ML was. Basically, can you train a tiny model just to do some arbitrary things? So for us, it was to predict the Y value given the X value for a sign, wave function.

So. Train the simplest possible model. And then just go through the process of converting it into a form that will run on embedded devices and then getting it deployed on there and having all of the parts of this process are quite difficult. Training an effective model is difficult. Collecting a good data set is difficult. Signal processing is difficult. Deploying the device so that it's good just to take something very easy and deploy it to an embedded device and get familiar with the entire flow there. that's sort of, if you're approaching things from scratch and doing everything yourself, but what I'd recommend is going to a platform like edge impulse and following the steps to do something that trains a model that runs on, uh, simple time series.

So for example, a lot of microcontroller development boards will have a built-in accelerometer. And a really good thing to do is to train a model. Recognizes different types of activity based on data from an accelerometer. So accelerometer basically returns a time series of three values. The three values are the amount of acceleration on the X axis to the Y axis and the Z axis.

So you get this idea of motion in 3d space, and you can train a model to recognize different types of activity. So like jogging versus doing pushups or something like. You collect a little data, set with data for each of these motions and then train, use some signal processing to extract the pertinent information, train a little model, and then deploy it to the device.

And then you can like wear this thing on your arm and it will classify your activities. And that's really cool because. That's how a smartwatch that people wear for fitness works that can understand when you are running versus swimming or something like that. And you, you can start to see how this all fits together end to end.

If you don't have any embedded experience, it becomes difficult even to grab that initial data from a device. So that's why these sort of end-to-end frameworks end-to-end platforms are so powerful because they come with all the software that you need in order to grab the data from there.

Common Mistakes 

Richie Cotton: Making your own fitness watch does sound like a pretty cool project, but I can see how there are a lot of different steps where you need to know a bit, about a lot of things in

Order to make that. So maybe talk about when people are trying to do this, what's a good idea. And what's a bad idea. So, perhaps can you tell me a bit about common mistakes people make when trying to do this sort of thing?

Daniel Situnayake: One of the things that I often come across is there's a lot bigger of a difference between test data and training data and real world data. If we're using sensor data, versus if we're. Training something based on stuff out of a database that came out of a, a piece of cloud software. So the thing with tabular data that's based on, like, I don't know the, the behavior of users in a social network or something like that is that the data that gets measured in the same way, and that gets collected in the same way during data collection and during production. It's just that we took a snapshot at a certain moment in time. We've got now a bunch of, of training and testing data. And after that time, all the real world data's generated in exactly the same way. But with embedded machine learning, you have more of a challenge when collecting data.

Imagine we're gonna design a new product. We're designing a new kind of smartwatch. That's supposed to help. I don't know, climbers and, and give them some feedback on their climbing. So this device doesn't actually exist yet. And before we get that device built, we need to. Design and train this machine learning system that can give the climbs their feedback, but because we don't have any devices built yet, we've got to collect somehow a data set without having our, our real world actual device available.

And so. To begin with  we'll have to develop a prototype that just collects data from some sensors. Maybe it's roughly the same size and shape as the real thing will be and have some people wear that device, do some climbing, and label the data in some way. Maybe it's like, oh, that was a difficult climb versus an easy climb or something like that.

And collect this whole data set. Once we've done that we can train a model and we can test it and we can see, okay, this model can classify accurately, whether this was an easy climb or a difficult climb, but the problem is. When we then get further through our product development process and we've got our finished hardware design, maybe it's a slightly different shape and size of watch.

Maybe it has a slightly different sensor on it. That's like it's still an accelerometer, but it's a different model of accelerometer because that one was cheaper and we needed to go with the cheaper one to hit our target for how much the device costs. So. When it gets deployed into the real world, the device is slightly different, meaning the data coming off will be slightly different.

And that means our model might not necessarily work very well. And the real challenge is that all the data is trapped at the edge of the network and the device is the thing that's interpreting it. We don't necessarily have a way to check whether it was working well or not. Right. We can't just compare the real-world performance versus the predicted because we don't have that data and we don't have a way to label it.

So there's this big risk that you can end up training a model that looks really good. Before it hits production. And then when it hits production and gets deployed to a real device, it doesn't work very well. And that's a risk in all of data science, but it's especially amplified here because we've got less feedback loops between development and production. So that's probably the biggest scariest thing that we had to work around.

Possible Solutions

Richie Cotton: Oh, wow. Uh . Yeah, I'm sure that could be a problem. And so how do companies deal with this sort of thing?

Daniel Situnayake: Yeah. I mean, it's, it's tricky and it's, it's an ongoing thing that we are evolving systems and workflows to deal with. But the most obvious thing is just keeping things as similar as possible between production and development and wherever possible, making sure you use the same sensors and ensure you are collecting data in the same way, but obviously that's not always possible. So then you've got to think about how do you set up some ongoing processes to continue testing and making sure that the system still works as the real world changes, because there's also, obviously this concept of drift where the real world is always changing and your model represents one frozen moment of that.

And that's even more amplified in. This field, because we are dealing directly with real world data that's captured from sensors. So one of the things that you can do is periodically send people out to capture more data and test your model against that. So that can be a helpful signal. So it's not as good as directly.

Looking at what's happening in the real world, but it's gonna give you a periodic snapshot of what's happening and you can see, okay, with this new data I've collected, the model's still working. Another thing you can do is look at the output of your model. So presumably these devices or smart watches that people are wearing for climbing.

They would be reporting data back into some system. So if they determine that the climb was challenging, that gets logged in the user's account somewhere so they can see it later. So one thing you can do is look at the distributions of the outputs of the model over time, because even if you can't afford to save all of the raw sensor data, you can potentially afford to save the output of the model. And if the distributions of the output. Are changing a lot over time, or if they're very different to what they were on your test data that you originally captured. For example, maybe that's a sign that some drift has occurred or that the real world is occupying a different distribution than the initial data set. So that could signal that you need to collect more data and do more evaluation.

Richie Cotton: Just going back to the climbing watch example. That sounds like it might just be a case of you get your climbs back in and get them to climb the same route they did before using the production version of the watch rather than prototype.

Daniel Situnayake: Exactly. Yeah. And maybe you made the mistake of initially collecting data from professional climbers, and then you found out that the people buying your watch are in. The public are just amateurs. And so maybe you need to get some amateurs to come in and collect data from them. And that gives you a better representation of what's out there in the real world. So all this stuff is things you have to be careful about in your sort of experiment design.

Best Practises for working with Edge

Richie Cotton: Yeah, the amateurs fall off the rock face. So just flipping this around. Are there any particular best practices for working with edge?

Daniel Situnayake: It's a big, big field. So there are lots of different individual places. But I think that the most important thing for me is trying to figure out, because this is such a big area. There is so, so many steps in the workflow of going from a. Data set to a, a model deployed on a device, coming up with a good understanding of all of the different moving parts at the very beginning is very important.

So understanding what is the hardware we're gonna be deploying this to so much of these things are based around like figuring out. A good compromise based on the hardware that you'll be deploying to. And so is that known ahead of time? do we get to choose the hardware based on what kind of model we end up training? Or is it more the other way around that we've got a device that already exists and we need to fit the model in its spare memory. 

So getting that pin down at the beginning is really, really important and that will inform the entire rest of the project. The second thing that's really important, I think is just having an understanding of what are we trying to achieve here? What's the minimum thing we can do that will result in a useful product, because sometimes it's good to aim for that basic. Improvement rather than saying, like, we want to do all these different things and make this sort of all singing, all dancing, magical AI product, and then trying and failing.

It's better to choose one simple, straightforward thing, which is going to be helpful to your customers and then deliver on that and then work up from there and gradually add more features and improve things, but try and bite off something simple to begin with, because this can be really challenging.

More Examples of Edge ML

Richie Cotton: I'd like to talk a little bit more about some of the use cases. So you mentioned that high-tech thing with the Mars Rover, and then there's a consumer product idea with this climbing watch. And then you talked a bit about sensors and things. So can you just give me some more examples of where this might get used?

Daniel Situnayake: We have customers across pretty much any space you can imagine. It covers everything from consumer products. Use a little bit of intelligence to do something smarter in the home, all the way through to industrial systems that understand what's going on in a factory, and maybe analyze products for quality after they've been manufactured.

So you can understand whether. There are flaws in your manufacturing process. Healthcare is an interesting area, interpreting bio signals, using machine learning to understand how someone's body is functioning or the, the progression of a disease. Agriculture's a really interesting one understanding like how plants are growing or how animal health is or understanding climate.

So anywhere where there's Rex connection. The real world and tangible things that are going on, and we need to know more about it. There are applications for embedded machine learning in any field you can think of.

Richie Cotton: Yeah, it's amazing. Uh, machine learning is just sort of invading every part of our lives. So yeah, it's interesting.

Daniel Situnayake: That that's something I've really find exciting about this is that until now, machine learning has been something that's existed in a constrained. Set of places that it will fit. Right? We've got these big, huge models running on powerful computers, but now we are able. Push machine learning down to smaller and smaller devices, so it can fit more neatly into little corners of our lives.

It democratizes things a little bit, rather than needing these one size fits all giant models that are running in the cloud. At some big tech companies data center, we can have little tiny models trained by people working. In little corners of the world, whether in some niche field or a place with particular problems and challenges, we can address with machine learning.

And it's a lot more feasible to get good performance out of these smaller models in little constrained contexts, rather than trying to create these big giant models that do everyth.

Richie Cotton: Lots of tiny, tiny things on tiny machines. So that's cool. this is very much, an emerging field and I get the feeling that there's lots of progress being made, but maybe there are some sorts of limitations at the moment. So perhaps you can talk a bit about where the sticking points are. particularly with tiny ML where I imagine you've got even stronger constraints.

Daniel Situnayake: So one of the biggest things is around tooling. And essentially if you are working in embedded. Software or hardware design chances are, if you are building something in agriculture, for example, your team is going to have a lot of experience and a lot of expertise around agriculture. And you're using that to design and build an effective product that works in that domain. So if you are working in that domain and you are, you are already a deep expert in your. Domain. And you're a deep expert in embedded engineering. You probably don't have the bandwidth to become a deep machine learning expert. If you want to use machine learning as part of a project to begin with, this huge, really difficult learning curve, where you've got to become an expert in ML and then apply that knowledge to the problem you're trying to solve.

The big problem. And the big challenge is how do we make it so that people who are domain experts in embedded engineering and in whatever field they're working in, can use machine learning without having to become a machine learning expert without having to take years to study the field and, and learn the ins and outs of data science. So that's where there's a huge amount of opportunity in this field is building tools that. Empower people not from machine learning and data science background, and obviously, building those tools requires data science and machine learning expertise. So there's, a huge opportunity and a huge challenge in adjusting and adapting, these toolchains so that people who are experts in one little subject area, that's very important can make use of.

Richie Cotton: That agriculture example really resonates with me. So my father-in-law was a sheep farmer and when he had decades and decades and decades, Of farming experience. I don't think he ever touched a computer in his life. And so getting . Him to think about machine learning would very much be a no go area, but trying to get all that domain knowledge he had into some kind of model would be a very interesting challenge. So I can see how that would be a difficult thing.

Daniel Situnayake: Yeah, absolutely. And that's even something we've directly seen people using edge AI models to recognize when sheer. In different parts of a farm or walking across the road or devices that are, are worn around an animal's neck that keep track of their biosignals and their activity so that you can understand how healthy they are.

Most Important Challenges

Richie Cotton: How are things progressing at the moment? What are the most important challenges that are being worked?

Daniel Situnayake: So right now, I think some of the biggest. Most important work is in this field of opening up access to people and allowing people to understand the machine learning workflow without being a machine learning expert. Basically building tools to understand sensor data and be able to look at how good a data set is of sensor data and understand quality and give people feedback around like, oh, you should be collecting more examples of this type of thing.

One of the biggest difficulties is that collecting label data can be very difficult. If you think about sensor data. Essentially it's like big, massive time series of numbers and it can be quite inscrutable. You can't necessarily look at a table of all these rows, like a thousand rows per second.

Some arbitrary value gets read from a sensor and makes any sense of it as a person. So we need to come up with techniques that allow people to take a big data set of like hundreds of hours of unlabeled, high frequency sensor data, and label it, and then be able to use it for training or perhaps be able to do some kind of training that doesn't depend on labels.

So that's, that's a really big thing. Another big thing is understanding. The performance of models on this type of data and explaining how models are performing and where they're working well and where they aren't across a, a big data set of this type of difficult to interpret data. One thing that's quite surprising in this field that we found is that the device constraints around memory and latency and so on aren't as big an obstacle, as you would think.

It turns out if you use some intelligent signal processing and you train a small model, you can often do a lot with very minimal hardware. We used to turning things up to the max and going with the biggest model that you can get away with without overfitting and, and all of that type of thing. But if you're thinking about fitting models onto these devices, there's a surprising amount you can do with what little space you have.

There are a lot of people in the field looking into. Compression and how can you fit bigger and bigger models onto smaller and smaller devices. But it's quite rare for us that we have a customer who needs to do something that requires a model that's too big to fit on an embedded device. I it's more often that they don't have a good enough data set to be able to train that model in the first.

Richie Cotton: That seems to be a common thing with. Learning so's that the hard part is getting good data to run the model rather than the actual modeling part itself.

Daniel Situnayake: Absolutely. Yeah. And especially with sensor data, where you look at things like natural language. Image data. There are huge data sets out there that have been scraped from the internet that people used to be able to use for creating and benchmarking new types of models. But with sensor data, there just isn't that much out there.

And people who have access to sensor data. Are quite wary about sharing it because it represents significant intellectual property on their part. If you have a, a company that manufactures some kind of widget and you've got all sorts of sensors on all of your equipment, that monitor how. This manufacturing is going, which represents valuable data that your company can use to potentially improve its processes.

And you don't necessarily want your competitors getting a hold of it. So that's a, a challenge as well is like the availability of data sets and the issues around privacy and sharing of data.

Future of Edge ML

Richie Cotton: I can see how that would be a problem. Maybe if there are listeners who have sensor data sets that they can publicly share, then I didn't encourage you to do that. if you can do it. Causing any privacy problems or IP problems. So this whole thing seems like it's at a point where it might be about to explode in popularity.

It feels like the tooling's coming together and the techniques are coming together and some of the processes. So what do you thinks gonna drive adoption of Edge ML over the next few years?

Daniel Situnayake: Yeah. The way we think about this is sort of there's this massive untapped potential. Based on sensor data that we've got these, these huge. Incredibly valuable pools of data exist in the real world and are completely discarded right now. Most data that's connected, collected by IOT devices.

For example, isn't really used for anything. And we've got all this technology. Around sensors and embedded devices that can collect lots of data about real world situations, but we're basically not making any use of it yet because of these bandwidth and latency constraints. So what's gonna drive that?

There's huge untapped value in this data throughout industry, consumer devices, and everywhere. And suddenly we've got the tools that allow us to make use of it, build better products, do things more intelligently, be more efficient, and create higher quality products on a manufacturing line, for example, or to monitor our natural world to understand how ecologies and the environment are doing and how the climate is changing.

So there's a huge need to interpret all of this data so we can use it. And this is the technology that allows us to do that.

Edge ML for Beginners

Richie Cotton: Exciting times. So, for people who want to give this a try, but maybe don't necessarily want to go it alone. Are there any good communities for people who want to get started?

Daniel Situnayake: There's a, a really awesome group called the tiny ML foundation, which has existed since quite early on in this field who run a series. Meetup groups across the world. There's dozens of them where you can go to local meetups with other interested people in this field. So if you go to tiny ml.org, you can find some links to those.

And the tiny ML foundation also run a conference or cut a few conferences every year. In different parts of the world. And we do these weekly or biweekly talks as well that you can watch that are broadcast live. And they're also on YouTube and so on. There's also a few forums around. So edge impulse forum is a really good place to go.

If you go to the edge, impulse.com website, there's a link there and a ton of people are hanging out, talking about embedded machine learning. Another good thing you can do is there are a few courses available, so there's a course error course. Some folks put this together at edge impulse, which gives you an introduction to how to start working with this technology.

There's also a Harvard edX course, which goes into the. More sort of in depth theory behind things. So if you are excited about the behind the scenes part, the more like mathy stuff, the Harvard course is a really good place to go. If you're excited about the application stuff and how do I just as quickly as possible start building things with this technology, then the Coursera course is really.

Richie Cotton: I should probably add that for anyone who's interested in analyzing sensor data, we do have, a DataCamp course as well on analyzing the internet of things, data in Python.

Daniel Situnayake: That sounds like a good place to start.

Richie Cotton: it's a very gentle introduction, with lots of time series analysis and a bit of machine learning. So before we finish, do you have any final advice for people who want to try this themselves?

Call to Action

Daniel Situnayake: Yeah. So don't be scared. Like this is a big field. There are a lot of moving parts, a lot of complexity, but it's definitely possible to. Focus on the bit you care about and ignore the rest of the complexity if you use the right tools. So I wouldn't recommend most people starting from scratch and trying to learn all these little pieces and all the moving parts simultaneously.

If you are already interested in data science and machine. Then find a platform that automates the embedded side of it for you. So it makes it easy to collect. Some data sets make it easy to deploy to a device, and then you can focus on machine learning. Otherwise, if you are an embedded engineer and you're excited about doing that part and don't necessarily want to spend years becoming a data science expert, the same platforms will help you.

Focus on the embedded part in building the product you want without having to go back to school. So definitely don't be afraid to use tools to help you, and don't try and start everything from scratch because it's very difficult.

Richie Cotto: Okay, thank you very much for answering all the questions. It's been a real pleasure having you on the podcast. Thank you very much, Dan.

Daniel Situnayake: Likewise. Thank you, Richie. It's been awesome chatting with you today.