Transcript:
Hey, there, it’s! Joseph from Robo flow today. I’m gonna show you how you can train your own Yolo V for object detection model to any data set to recognize any objects that you may have in your images and annotations now, for my example today. I’m going to be working with chest data before I tell you more about the data that I’m working on. Let me show you that. I have a working model, so you know what we’re working towards, okay, so in my example here. I have this image as you can see on the right in my collab notebook, and I have all these pieces and they’re not annotated, so let me go ahead and run inference on that exact same image file path in my notebook boom. Just like that, you see in the output, an annotated image and all the labels are correct. There’s the white King, the white pawn black cons, black bishop, etc. Now, for my example, my data set is totally and freely available and I’ll show you where to find it, but let me first show you what’s in our data. Set, okay, so inside my data set here. I have a bunch of chess images, so it’s just chess pieces on a chess board in various different positions. I also happen to have my images already annotated with bounding boxes, defining where on the chess board. Each of these pieces are my bounding boxes happen to be made in vaak? XML format. But chances are if you’ve already annotated in Cocoa Json or create a Mel Json or even yellow text files, you’ll be able to follow along with today’s tutorial, and if your images aren’t yet annotated, you can either have us a trouble flow. Annotate them for you or you can use an open source tool like image label to get things labeled for free. Okay, now let’s dive in the first thing that I want us to note is the tools we’ll be using today are all completely free that is creating a free. Rubble flow account and using Google Collab, which provides free GPU compute. So the first thing that I need to do is get my data in two row. Whoa, and to do that? I just need a visit app dot Robo flow. Ai, now I happen to already be logged in so my dip. My account opens with the empty state. Here you’ll be directed to create your own account using your email or Github. Whatever works best. I’m gonna go up here and I’m gonna create a data set. I’m gonna call my data set chess example. Yolo, it is a object detection problem. I have bounty boxes and my annotation class is pieces, so I go ahead and create that data set. Now as I mentioned. This data set is already annotated with my images and annotations all in this folder, so I can just take this folder and drop it directly into rubber flow and at this point. Robo Flow performs a series of automated checks, matching my annotations against the images and I get to visually inspect and see that everything looks OK. Some things that Robo flow is doing for me at this point is like one checking for zero pixel bounding box annotations, for example, an annotation where I may have accidentally clicked or a label or accidentally clicked on my behalf. It’s also checking to see if any my annotations exist outside a frame that is outside of the image that I wanted to be annotated in that case. Robo flow trains the annotation that it aligns up exactly with the edge of the image now. I see that my images are all annotated all 289 here, so I’m gonna go ahead and start the upload and create it to my own private Robo flow account when I’m asked to upload, you’ll notice that Robo flow asks me to create a trained valid test split. This is best practice from when we’re training and evaluating our models. I’ll follow a 70-20-10 split just as rebel flow advises now as these images upload. I want to note that. If you want to get access to this data set, you can visit Public Dot rebel flow dot Ai, where we have a series of object detection and classification datasets free to use. You can do things like detecting raccoons or working on thermal images they did is that the ironwork is here called chess pieces data set. And if I open that up, you’ll note that I could fork this data set, which means add it to my Robo flow account or I could work with one of the existing downloads now. I happen to already have this data downloaded, and I just added it to my Robo flow account, but the process by which I added this data to my Robo flow account works for any data set. If you have a problem, adding your images and your annotations, maybe they’re not matching up. Maybe there’s some sort of problem. You can always reach out. Write us a note. We’re happy to support new annotation formats. Okay, now that my data is in. Robo flow. I can perform a series of checks. Things like checking to see what my class balance is and seeing the size of my images understanding how many objects I have annotated across each of these images, and that’s what we call. The data set health check and so we’re below is performing these series of checks and then it surfaces for me. Generally how my classes are distributed. You’ll notice that predictably, because it’s a chess problem. I have a lot more pawns than I do. Queens or king’s pieces that only appear once on a given chess board the other thing. I’ll note is that I have approximately 10 annotations per image, Which means you know. Approximately 10 pieces were on the board. I have one Null example, which means one board that doesn’t have any pieces on it and I have zero missing annotations meaning. All of my images had a matching image file. I can also see down here things that, like the all my images are the same size to eight this the size here and the annotation heat map that shows me generally where my annotations are across each of my images. You’ll notice that in the chessboard, my white. King piece really didn’t move around enough. Maybe my training data, and that could cause a problem for me. Same with my white green piece, but alas. I’ll continue on the next thing I need to do in. Robo flow is I need to create a generated version of my images That means I need to go for my images in their raw form to a format that allows my model to learn from them things like resizing and removing Auto Orient data from the exif data. If you’re unfamiliar with that, you’ll note that if you check out this step, we have this link here, that says, when should I Auto Orient my images and you can click and visit that, and it rocks through in greater detail on our blog. Why that matters now I’m gonna leave that turned on because I don’t want my images to be improperly rotated in ways that I didn’t anticipate when I captured them the other thing that I’m going to do is I’m gonna add a resize. Option by default row flow has this stretch to 416 by 416 but there’s various different implementations of resize. You might want to consider. I could do things like, make sure that all my images are maintaining the same aspect ratio and then add black padding to the top and bottom or I could do a center crop where it zooms in to the dimensions of whatever I desire for my example here, I’m gonna go ahead and I will stretch too, and I’ll do 500 by 500 were actually 416 by 416 The reason for that number is Yolo. The way that it performs its convolutions is it actually expects multiples of 32 and 416 is large enough where there’s enough pixel clarity to understand but small enough that my model can learn more or learn more quickly now. The next thing is that I can. A documentation steps directly in Robo flow. If I want so things like rotating my images now this might make sense If I’m building a mobile application, Perhaps my images, the camera could come at the chess board from various different angles. So it might make sense to to do that. Do the hang on the context to your problem for my case. I’ll say that I will do say a little bit of random rotation up to 10 degrees in either direction, and you’ll notice that I get an image preview here and Rulfo shows me that my images can be randomly increased or decreased clockwise or counterclockwise by 10 degrees. There’s a series of other augmentation steps that Robo flow allows me to provide, but for the sake of my example, I’m just going to do random rotation and we’ll say an increase in decrease of 25% brightness. Now the last thing. I’ll note here is that Robo Flow allows me to generate additional training data so I could three different versions for each of these chess images that means that while my original data set only had 289 images, each image will undergo the pre-processing step, so it’ll be resized and then a tog mminton. It’ll be randomly rotated from positive to negative 10 degrees and then randomly made 10 or 25 percent brighter, a 25% darker so one image is going to undergo those three different random rotations. Maybe one time it gets rotated 80 degree or 8 degrees clockwise the next time, maybe two degrees counterclockwise, brighter or darker. You get the idea, so go ahead and generate my export and I can give it a name, so I’ll just say I don’t know, I’ll just call it one. Maybe because I’m just gonna call this version one and so this, then robofoot then creates and generates each of these individual images for me while those images generate. I’m gonna show you that the model that we’re gonna work on is available in the Robo Flow Model Library. So if I go to models dot Robo flow da AI? I have a series of different models that I might want to use to train. My data set on now, in my example. I said that we’re going to be doing a Yolo via 4 Pi torch today. So that’s the model that I want to use, so I click in, and I get a bit more information now. I’m gonna be making use of Google Collab, so I click this go pull a blink, And I’m presented with BA, a Jupiter notebook that has all of what I need to train a Yolo v4 model custom to my data now. The difference is that this? Jupiter notebook is hosted on top of Google Collab, which means I’m able to use GPU compute for free. Now there are some limitations of collab, and I’ll mention those in a moment now. Now that I have this notebook from the Robo Flow model library. I have to make a copy of it to my own count to my own. Google account Because this is Google Collab. So go ahead and go up here. Click file. Save a copy to my own. Drive and instead of calling this copy of I’m gonna call it. I don’t know JN Che’s. Yellow v4 ruble Flo PI torch So that I know that this notebook is mine in my version. OK, now let’s go back to our Robo flow data our export has generated, and I am now confronted with the ability to create an export link or download the files directly to my computer now, for my example. I’m gonna do, I’m gonna ask for a download code because I want to see the link that I use in my Jupiter notebook. The format that we need to use for our yolo via 4 Pi Torch implementation is a dot text format and it’s listed here as yo Avi 4 Pi Torch, So I select that format and then I click continue and Rofl creates a zipped file of all of those images. Ready for me to use and provides me with that link that I might need now. I quickly X out of that link here, and there’s a reason for that. The reason is that link should be private to your own account. Do not share your. Robo Flo. Download link with anyone else now. I do want to note that I actually have the ability to download those locally if I want as well and that means I could download and look at all. My individual Images recall that we generated 867 images from our original numbers and I can actually see and look at these, and I see that some are randomly brighter. Some of my randomly darker. They were rotated a little bit and this natural variation creates more examples for my model to learn from and not only more examples to learn from, but in different contexts. Some of the rooms that my chessboard could be in in production can be randomly brighter or they could be darker or the orientation could be a little bit different relative to the camera. So Robo flow simulates that with the augmentations that we selected. I also want to note that this version is frozen in so all of my settings Here are archived so that I know which settings I used for each of my given exports. Now let’s return back to our collab notebook. We’ll note that you’ll see a chess image when you open it up because this describes the everything we need and from here, it’s pretty smooth sailing. I mostly just need to run cells that call code from various places that I need the first cell that I run does a clone of the Robo flow Github implementation, Which is just this repo here, roll away. I Pi towards Yellow v4 and so we’re bringing all the contents of that repository directly into our Jupiter notebook or Google Collab notebook. And then I Pip. Install all the requirements text required to run this given series of of code and libraries. So I mean, some of those requirements, for example, are things like a specific version of Pi Torch, a specific version of numpy, a specific version of psychic image, specific pillow, etc, etc. And that will allow me to make advantage or take advantage of the rest of my notebook here while those install. I’m gonna scroll ahead a little bit and show a few things. The next thing that we’ll do is download the yo v4 weights that have already been converted to and ready to use for the format that we need so this. G down command downloads from Google. Drive on my behalf. So this cell will run right after the cell above it runs, and I want to know if you’re unfamiliar with Jupiter notebooks, the way that you can run a cell is you can just put your mouse inside and then do ctrl enter. It runs that cell. It’s important that you run these cells in order for the purposes of having your data ready now. The next thing now that everything is all set up inside our notebook. That means we have all the basics to run our code. We need to get our data, so we need to get our data from. Robo flow. Now this provides a little bit of instruction of things that we all did. We already imported all of our data to rubble flow and we applied a little bit of augmentation to increase our data set size. The next thing that we can also do is Unzip is create that export from rubber flow and then export the data from a row flow account to anywhere in this case. We’re going to export it to our collab notebook, and you’ll notice here that this is meant to be the export link we generated just a moment ago, so that export link was from our. Robo flow account here. This export yellow, yellow v4 Hi, torch was the one that we want for this notebook. C select yellow, v4 Type Pi Torch show, download code. And then I’m gonna move this out of frame, just so you can’t see while I copy my link [Music] and with this link, you should copy your link as well and drop it directly into your Jupiter notebook, so I’m gonna move my Jupiter notebook. How to frame for a moment while I drop my link in [Music] and then I’ll bring it back so you can see that. I ran this cell so I wouldn’t hadn’t ran that cell now. My link was here, but I replaced my link with the word done so as to not allow you to see it, but you’ll see that all my files are now in my Collab notebook. If I click the file symbol over here on the left and they cook in the Jupiter notebook hosted on Collab. I can see train valid test and those are the formats that came from Robo flow. If I open up any of those folders like this train folder here, it populates with all these images in the analyst and annotations, so this JPEG here is there you go, It’s that darkened black winged Queen image, which looks great The next thing that I do is I click. Create a couple of directories also gonna copy everything from train and put it directly into train dot text, copy everything from annotations and put it into train text. Do the same for validation as well the next thing. I’m going to do is I’m gonna run a function That tells my notebook. How many classes my problem has and that’s important because I need to pass the number of classes to the script that we’re using for the purposes of training and that’s. What this cell is, OK? So this cell does a lot of things. It calls the Train dot. I script Train Dot Pi script from the code of this Github repo. That we brought into our notebook so its calls this script and this script has a series of arguments. These flags that we pass, so lets. Talk a little bit about those flags. So B is the batch size. S is the number of subdivisions that we may want in the batch -. L is the learning rate. G is the direct to Train on the GPU device and that’s already built in so we can just say 0 because it’s going to use the only compute available, which is GPU. Oh, lab, and then I’m gonna call out any of the pre-trained specific weights that I have, and in this case, I’m gonna use the weights that I downloaded just a moment ago. I’m gonna call out the number of classes which num classes is this variable so 12 and I’m gonna call out a directory for where everything should be trained. The weight should be exported to, and then the last thing is passing the number of epochs that. I want to train. Okay, so I run that. And this is where things will take a little. While the training isn’t the fastest process, but this is free GPU compute from Google Co-op, and so you’ll see that the training process then begins to recite where it is and it will train along the way. This is where? I need to warn you that Google Collab it is a free resource, however, it will not give you GPU forever and it may not give you the best GPU available either, so some tests have shown that the GPU. Google Collab gives you goes to sleep after 12 hours without any questions asked and after 45 minutes if there’s no activity in the notebook in other words, if you have some important work saved after 12 hours, it needs to be saved elsewhere, And if you just leave a browser tab running like this, and you don’t check on it for about 45 minutes or your computer goes to sleep or something like that, You’ll lose training. You have to start over. So I recommend either using Robo flow train, which is a service that we offer. Will we will train your models on your behalf and give you the ability to do this or you can upload or upgrade to Coop. Pro or you can set up your own GPU. Using all the requirements that text now as that finishes training. I have a little movie magic where I can fast-forward and show you that this is actually already finished training, and so that’s where I have I notebook over here where everything indeed finished training and here. I’m gonna tell you a little bit about what’s being recorded. So every epoch we’re getting a few reports of how far along our model is, and we generally see loss going down over time. That’s good that means our model is learning our pieces better and better, and then at the very end, we can list out all those checkpoints, so all of those we actually checkpoint at every epoch, so there’s 50 different checkpoints in the notebook. We then actually grab a random image from the test directory. We passed that image to the model that you just trained using the weights that you just trained in here while we wait for inference so here, it actually found a series of all these pieces, and then we can visualize that inference below and look at that. We’re right back to where we started at the beginning of the tutorial. Now, what’s next after you have your custom model trained? What should you do well? You might want to export your trained weights meaning. Use your model somewhere else you can export it to. Oh, an annex and use that to run. Actually, you can export it to own the next two tensorflow – or own the next to run on device whether that say maybe. GPU enabled nano or something like this depends on your context, or maybe you’re going to train and run it on a mobile application. Regardless, this allows you to export your model and go about your way, OK? So we just walk through end to end how to upload data. Prepare it, make it be ready. Use the rofo model library to grab a lot of our interest and train our own custom. Yolo v4 model to any dataset of any interest and you’ll note that their performance is quite strong. Now don’t hesitate to reach out note that Robo flow is free to get started. So your first thousand images, which is plenty enough trained a successful model and you can reach out to us If you have any questions along the way help at Robbo flow. We can’t wait to see what you build on top of. Robo flow thanks.