WHT

Image Recognition In Python | How To Make Advanced Image Recognition Bots Using Python

Kian Brose

Subscribe Here

Likes

24,943

Views

531,369

How To Make Advanced Image Recognition Bots Using Python

Transcript:

Hey, so today I’ll teach you how to make some bots. We’ll be covering how to make a simple piano telespot how to make an aim booster bot that can track an object. Okay, so let’s get started. Step one download python and remember to check the add to path checkbox when installing step two open the console with administrator privileges and run the following commands. You can find them in the video description. These are all python libraries that we are going to use throughout the tutorial. I’ll explain what they do a bit more into the video. Step three now that we have completed This setup, create a new folder and create a text file named Botpy. You can name it to whatever you want. As long as it ends with the dot, py extension now begins the interesting part of the tutorial right, click your newly created Python file edit with idle edit with idle and paste this into the Code Editor. You can find this in the description. These are the libraries we downloaded a second ago. Let’s find piano tiles online. This one should do for this game. We have four rows where black squares will come down from each row. The fastest and easiest way to about this game is to check a single pixel from each row. We will check if that pixel is black, and if it is, we’re gonna click it. Okay, so first, we need to know the RGB value of black. Which is the tile’s color for this open, a new python shell by pressing windows search Python and open write Import Pi Auto Gui and then Pi Auto Dot Gui Dot display mouse position the shell will display the pixel you’re pointing at and its RGB value hover over any black part of the screen. You can see the color for black is zero red, zero, green and zero blue, go back to the shell and press ctrl C to stop it from running. We now want to figure out the X AK horizontal position for each row, open the python shell again and type import Pioto Gui and then pay auto guide on display most position hover over the first second third and fourth row, go back to the shell and copy the data from each respective row. We will only be using one Y vertical position, so I’ll put mine to around 400 which is around the middle of the game for me. The first house pixel is X, 597 second X 676 and so on. It will be different for you, but don’t worry, open the Podpi file. There are two ways to perform a click. The first is by using Pyota. Go, I click function which looks like this. The the downside for this is that it’s pretty slow so instead we will use the WIN32 API, which runs much faster to do this. We start the script by creating a click function like so, this uses the WIN32 API library to move the mouse to XY position with set cursor pause, pressing down, holding for 0.01 seconds and releasing. We want the 0.01 second delay since clicks sometimes fail to execute. If you click too fast, we then write while keyboard is press Q equal equal false. This line checks. If the key queue is pressed down, it returns 2 if it spreads down and false. If it’s not, we’re going to use this to stop the bot when we want to stop it. Okay, so now what we’ll do is check the RGB value of the first row’s pixel to do this. We use the if Pi Auto Gui Dot pixel and then your pixel’s location 0 equal equals 0 this will return the RGB value of the pixel located in X 597 and Y 400 the zero checks for the R value. Whereas if you were to switch the zero for a one, it would check for the G value, and if you were to switch it for two, it would check the B value. There’s no need to check for more than one in our case. So the R value is enough now that the script can tell when the pixel is black, we want to click it immediately when it does for this, we call the click function we created and make it click the first tile’s position, which is the same position we scanned. We do this by writing, click and then your pixel’s position. Alright, So repeat this process for the remaining three tiles and don’t forget to add the correct pixel location. And there you go. Press run run module, switch back to piano tiles and press start. So as you can see, yay! It works! [MUSIC] [Applause] [Music] [Applause]. Now that you know how to make a piano tile spot, Let’s move on to the aim. Booster Bot start by going to aimboosterco’m enable flash player if it’s not enabled and press reload in this game. You have to press the circles that appear randomly in this area of the screen. There are two approaches to solving this game. The first is to take a picture of the circle and locating it on the screen. This approach would be too slow so instead we will take a screenshot of the game area and check every one out of five pixels to see if that pixel is the color of the center of the target before doing this. However, I’ll demonstrate how to do image recognition. Okay, so through my amazing, artistic talent I have made what appears to be a stickman and paint! What, I’ll teach you now is how to detect if the stickman is on the screen or not. Go back to the folder You created with the Botpython file. Create a new file and name it stickmanpy edit the file without it like we did before and paste in the same libraries. As in the piano tile spot. Go back to the stick menu drew in my case. This one on Windows Press shift windows S. This will open up a screenshot tool that allows you to select a part of the screen to screenshot drag over the stick manager to save the screenshot open a new paint file and paste the stickman. You now want to trim it as small as possible by selecting the area that contains the stickman and cutting excess like so once you have cut the excess press file, save as and save it as stickmanpng in the same folder as the script, saving it in the same location as the script is important open the stickmanscript you just created, we will now create a while loop with while one inside this loop will write the script that is actually going to find the image. If piotrgylocateonscreen stickmanpng is not equal, none this line uses pi outer Gui to locate the stickman on the screen. If it doesn’t find anything, the output of this line will be none, so if it’s not none, it means it found it. So if you write the following script if it finds the image on the screen, it will print out. I can see it and then wait for 0.5 seconds. The delay so that the script doesn’t spam the console, and if it doesn’t find the image, it writes, I am unable to see it run. The script by pressing run run module. And as you can see, the script can see the image. But if you move the window over the stickman, it stops working now. One of the questions you might have is what if the background of the image I am trying to detect changes, for example. If I were to put this disturbance here, The script wouldn’t detect the stickman since it’s not an identical match to fix this. We simply add the confidence parameter and the locate on screen like this. Let me explain when the bot looks at the screen, it checks every pixel and gives a percentage chance probability of finding a match to the image you’re asking to search for by default. Pyro Gui only gives an output when it’s 100 sure of having a match but by using confidence 0.8 we tell it that if it’s 80 sure or more to have found the image, it finds it you can, of course, adjust the value to whatever you need, but wait, there’s more now. You might be asking yourself. How can I make the script run faster? The first and most obvious way is by checking a smaller region of the screen so instead of searching for the image on the entire screen. We have to tell it where to look. Finding the correct region is a bit of a process. But I’ll explain as best as I can. Create a new python script in the folder. Copy the path to the folder by clicking here and copying this open the script and write the following code. It’s in the description as well, but remember to change this path to your folder’s path. You copied a second ago. This script saves the image on the region 0 0 100 100 as savedimagepng. The first number is the X position that the screenshot starts on the second number is the Y position that the screenshot starts on the third number is the width and the fourth is the height, so let’s say to region 100 200 500 500 The script will take a screenshot starting 100 pixels from the right 200 pixels down and have a width and height of 500 pixels. Now we just need to tweak the values manually. Run the script look at the output until we find the region of the stickman. So as you can see running. The script saves an image with the region. I gave it. I found that the region for the stickman in my case is around 150 175 350 and 600 Keep in mind. It doesn’t have to be too close to the object. It’s fine to check the area around the stickman as well open the stickman script again and write region equal U region And then grayscale equals true. The grayscale parameter gives a speed boost of around 30 so it’s always a nice thing to have do not use grayscale if you’re getting fake results, but in our case, it’s fine, run the newly modified script and as you can see, it can detect the stickman, even though parts of it are hidden and or modified. Alright, so back to the aimboosterbot create a new python file in the same folder as all the other files we made name it aimboosterpy paste in the libraries we used in the two last examples and then rewrite the same click function. We used in piano tiles. Remember to add a timesleep2 to give you two seconds to switch over to aimbooster? When starting the script, so as I explained a bit earlier, we’ll be taking a screenshot of this region of the screen where the circles appear and then check every one out of five pixels to see if the color of that pixel matches the color of the center of the target. So first we want to figure out what the region for the game area is to do this open. The script you created during the stickman demo that takes a screenshot of a region and saves it and tweak the region values until you get this region, which is the region where the circles are going to be appearing. Now that you know the game region, let’s start. The script with the while keyboard is pressed equally glue, false so that we can stop the bot from moving the mouse when we want it to stop, write the following line. This line uses payout guy screenshot function to take a screenshot of the screen and store it in the pick variable. In our case, it takes a screenshot of the region. We give it which for me is 660 350 600 400 Your region will be different, so don’t mind it If it’s not exactly the same. We then use the size function to get the width and height of the pick variable like so this will store the width and the variable width and height in the variable height. Pretty self-explanatory now. We want to check what the RGB value of. The center of the target is for this. I took a screenshot of the game while playing pasted the picture in Microsoft paint open the Python, Shell wrote Import Pi out of Gui and Pi Auto Gy the display mouse position I then hover over the center of the target, and the console will print out the RGB value of the pixel. I am pointing to the center of the target. Has an RGB value of 255 219 and 195. Copy The RGB values go back to the script and paste it anywhere as a comment note to make a comment in Python. Use the hashtag symbol, OK? So since checking every single pixel in our screenshot for a match is a bit overkill. We will instead check for the first pixel, then the fifth, then the tenth etc, etc. But first let me explain how a for loop works. Let’s take this example for X in range 0 10 1. Print X. This will output every single number, starting from 0 up to 9. This means that the script will set X to 0 print out X so 0 and then increase X by 1. Which is the third value here. It will do this while X is smaller than 10 The maximum value discount output is 9. If we were to run 4x in range 0 15 1. Print X, the script will print out every number from 0 to 14. If we were to increase the third value here to 2 for X in range 0 15 to print X, We will instead get increments of 2 instead of 1. So the output would be 0 2 4 6 8 10 12 14. So as you might have already guessed, we will be using increments of 5 for our image. So for example, let’s say that our image has a width of 500 pixels and a height of 600 pixels we write for X in range 0 500 5 for Y in range, 0 605 RGB equal pick dot, get pixel X Y. The script will save the RGB values of the pixel and position X Y of our picture pick, thanks to the get pixel function, So for example, the script will first check the pixel x0 y0 then Y 5 then X 0 Y 10 then X 0 y 15 etc, etc, until it hits the maximum value of Y 595 since we have increments of 5 and Y has to be smaller than 600 and store the red value in R green value in G and a blue value in B. Alright, so your script should now look like this. Since my game has a region width of 600 and a height of 400 I will use the width variable here and the height variable here instead of hard-coded numbers. You can use hard-coded numbers if you want. But for simplicity, I will use the variables so now that we can successfully check the color of every fifth pixel, we want to perform it click. If the color matches the center of the targets color, you can code this in two different ways in my case. The center of the color of the target is always going to be the same, so we can add a if B, which is the blue value equal equal 195. This checks. If the blue value of the pixel located in X Y is equal equal to the value 195. Which is the blue value of the center of our target to check for the red and green values, Change it to either R for red and G for green. But in this case, blue is more than enough, but if you wanted to create a bot similar to this one where the center is not always going to be the same color, then you can use range again. If you were to use range, the loop would look like this. If B in range 180 210. This will check if the blue value stored in B is between 180 and 210. In my case. Checking for a range is unnecessary, so I will just stick for if B equal equal 195. Okay, so now that the script can tell when the color of the pixel is the color, we want it to be. We want to click for this simply. Call the Click X plus 660 y plus 350 If you wonder why I have these values remember that the click function will click a pixel based on your screen resolution, so a click 0 0 will click at the top left of the screen, but in our case we want to perform a click relative to the screenshots region since our region’s first pixel, aka pixel 0 0 of our screenshot starts at 660 pixels to the right and 350 pixels down. We want the click function to also be 660 pixels, right and 350 pixels down since the script runs a bit too fast, it might click the circles a bit too fast, so we added time Dot sleep after the click to prevent Miss Misclicks. I’ll put mine to 0.05 We then end the script with a break to stop the script from clicking the same place again. Save the script, run the script tab over to aim booster and press challenge. And voila there! You have it a working aim, Booster bot. Remember that you have to hold the cue to stop the bot? If you experience accuracy, issues just slightly tweak the delay after the click. So the bot doesn’t have the time to click twice so before ending the video. I just wanted to ask if the fast pace of this video was okay, or if it was way too hard to follow, and if you want a more detailed in-depth video about Botting. Alright, so I hope you enjoyed. This fast paced tutorial. If it helped you out, please consider subscribing and leaving like if you have any tips on how I can improve the video. Do leave them in. The video description? Have a great day.

0.3.0 | Wor Build 0.3.0 Installation Guide

Transcript: [MUSIC] Okay, so in this video? I want to take a look at the new windows on Raspberry Pi build 0.3.0 and this is the latest version. It's just been released today and this version you have to build by yourself. You have to get your own whim, and then you...

read more