Transcript:
Welcome to this neural network programming series. In this episode, we will learn how to use Tensor Board to visualize metrics of our CNN. During the neural network training process without further ado, let’s get started at this point in the series. We’ve just finished running our network through the training process now. We want to get more metrics about this process to better understand What’s going on under the hood to do this. We’re going to learn how to use tensor board with Pi Torch and this episode will cover the basics and in the next episode, we’ll see the true power of Tensor Board with how it allows us to rapidly experiment with the different hyper parameters for now, though let’s kick things off with the basics Tensor board is tensor flows visualization toolkit that was designed as a visualization tool for machine learning experimentation As of Pi Torch version, 1.1 0 P Torch has added a tensor board utility package that enables us to use Tensor Board with Pi Torch since the plight or support for Tensile board is very new and the steps for installation may change. Check the blog post for this episode on deep Blizzard. Calm for the Tensor Board install steps, basically, as long as we have the correct versions of both tensor board and Pi Torch. Were good to go. So let’s briefly Talk about what Tensor board actually is, and then we’ll see some code. Tensor board is a front-end web interface that essentially reads data from a file and then displays it. It allows us to track and visualize metrics like our loss and our accuracy. Over time. It also allows us to visualize our network’s graph, and it allows us to view histograms of values like our weights, biases and any other tensors that we can think of to use tensor board. All we need to do is write the data. We want Tensor board to use to a file that tensile board can read and to make this easy for us. My torch has created a utility class called summary writer before we look at the code. Let’s start tense board and see what the interface looks like right out of the box. Alright, the first thing we’re gonna do before we run. Tenser board is just check the version that I’m running on my system. We check the version by calling Tensor Board and passing in the version flag. All right, so we can see that. I’m running version 1.1 4.0 and this is the nightly build to know whether or not you need to install the nightly build check with a blog post for this video on deep lasercom, where any version related updates will be posted for now. Let’s run Tensor board to run Tensor Board. We just need to run the Tencel Board program and pass in the log der parameter value by default with Pi Torch data files are written into a runs folder that exists in the local directory. So we’re gonna specify runs here and we run Tensor board. We just wait a few moments and we’ll get a message back That tells us that Tensor Board version, 1.1 4 is up and running at the hostname of the Machine Port 6006 so essentially we have a server that is up and running and waiting for requests to come in on Port 6006 in order to pull up Tensor board. We just need to open our Web browser and then browse to port 6006 and this is what Tensor board looks like right out of the box with no data and so what is telling us here is that there are no dashboards because there basically is no data, And if we just go back to the terminal, I will will kill that session and Ill. Just show you that in my current directory. I have the runs folder. And if we list the items that are in that folder, we’ll see that there’s nothing there, but after we run our PI torch code, we’ll see that there are some directories that are getting written into the runs directory, and after we do this tensor board, we’ll see the new data and it will consume it and display it to us. So let’s jump over into the notebook and get some. Pi Torch code written, so we can actually have some data to see inside our tensor board. All right, we’re here in the notebook. We’ve been working with throughout this series now. These are our standard imports that we need to pull in the fashion in this data set and build a neural network with Pi Torch and train it now the only additional thing that we have here from what we’ve seen in the past. Is this new import statement at the bottom from Torch Util’s Dot Tensor Board import summary writer. So this is the class that’s gonna allow us to easily write out or send the data to the tensor board files, so let’s go ahead and run this and get our imports. Alright, the next thing we’re gonna do is check our torch and torch vision versions. You can see here that. I’m running Pi Torch version 1.1 and for Torch Vision running version 0.3 now for Pi Torch 1.1 is the required version to have access to Torch Util’s Tensor Board. So make sure you update your version of Pi Torch to at least 1.1 If you run into any issues that you think are version related, be sure to check the blog post for this video in deep. Blizzard comm. That’s where all the version related updates will be sad thing about the videos is that we can’t update on that easily, but we can’t update the website. So just be sure to check there. Alright, next! This is the function that we’ve been using to calculate the number of correct values given a prediction tensor and a label tensor next. We have our neural network and this is just a convolutional neural network that has two con flares. Coughlin’s Con Two and then three linear layers, SC 1 SC 2 and the output layer those are defined as class attributes inside the class constructor and then down in the forward method is where we actually make use of all of these layers. We pass the tensor in it transforms its way through the network and we return the transformed tensor as output. OK, next we’re going to pull in our training set using the torch vision package. This is the fashion in this data set that we have access to through torch vision data sets next. We’re going to use this training set and create a data loader so in this data loader we’re passing in the training set we’re specifying a batch size of 100 and we’re also saying that we want the batches to be shuffled and now we’re set up. We’re ready to train and use tensor board, so let’s see exactly how we use Tensor board, given a simple example, and then we’ll look at how we use it in the training loop we’re first going to create a summary writer instance, And we’re gonna call it. TB for Tensor Board. Then we create an instance of our network after we have our network, we need to get some images and labels, so we grab a batch from our train loader, and we unpack it into two variables and images tensor and a label’s tensor, then we’re gonna use the make grid utility function that lives inside torch vision to create a grid of images that will ultimately see inside Tensor Board. We’re gonna pass this grid to the summary writer. So what we do is we call on our summary router object. TB, we say TB dot add image we pass what is called a tag and the data, so we’re saying images and then we’re passing in the grid of images. There’s a hundred images that have come from our batch, and then they’ve been transformed into a grid. We’re gonna see these intensive board now. Next, we say TV add graph, we pass our network and the images tensor. And as a result of this goal, we’ll be able to see a graph or a visualization of our network inside Tensor Board. Now, Lastly, we need to close the summary writers we call. TB close. So let’s run this really quickly and then check what happens inside Tensor Board. The grid of images and the network were both written to the file system. So lets. Go take a look at what this looks like, okay. I’m here back in my terminal, and if we just do LS on the runs directory. Now we can see that we have a folder here and inside that folder if we just do. Ls on runs in this folder. We can see that we have a file that has the data that we want to be displayed inside Tensor Board, so now we’re ready to run Tensor Board, So let’s just go ahead and run. Tensor board again will say Tensor board. – that’s logged. Our runs now attends our board is up and running on the Localhost listening at Port 6006 so we get back in our Web browser and we can just see here that now. I’d already refreshed, but basically before we had nothing to show. Now we have our images here. This is at the batch that we passed in the grid and then we also have a graph tab so here. This graph is of our network. So if we just double click, we can actually get a visualization here of our network, and we can see that we have one conf layer, another con flayer and then three linear layers. Now this isn’t necessarily the most interesting thing in the world, but it is here and so go ahead and check it out and dig in. You can kind of double click on some of these things and get some more information, but since we build this network, we know what this network is like. There’s really no need for us to dig in and do any additional analysis. This is just getting us going with tensor board. What we really want to see is data that we collect or emit from our training process, so lets. Go see how to do that now. Let’s go over the training process really quickly. And then we’ll be able to better understand where we need to put the tensor board calls in order to get data out of our training process, so we get an instance of our network. Then we get a data loader. Then we define our optimizer. If you’re not familiar with these things, be sure to check the previous videos in the series. Well, it comes next is the loop of Epochs, so we say for epoch in some range right now we have the range defined as one, so we’re only going to do a single epoch in this particular loop. Once we get in -. The Epoch we’re going to track the total loss and the total correct predictions as we iterate over all the batches within this epoch. So we define variables here initialize at zero, then for each batch. We follow these steps one. We procure a batch from the data loader two. We unpack this batch into an images tensor in a label’s tensor three. We pass the images tensor to the network for we take the predictions tensor, along with the label’s tensor and pass these two to the cross-entropy loss function 5 once the loss Tensor is obtained, We zero our gradients using the OPTIMIZER. Then we call the backward method on the lost tensor six. This is the point when everything is ready. We ask the OPTIMIZER to step in the direction of the loss functions minimum to do this. The OPTIMIZER updates the network’s weights using the gradients in the learning rate. Now we have a network that is smarter than before, okay, so once we’ve updated the weights, we take our loss and we track it, and then we also get our number of correct predictions and we track that as well. We do this for every single batch, and then we print the statistics for each epoch. We have a total number correct of 46,000 434 and a loss value of 357 So that’s a review of the training process. Now, let’s see how we can add any calls to tensor board to get this data out of our training loop and into tensor board. Alright, so here’s. Our new training loop with tensor board calls added, so lets. Just go over where these calls are at it. The first calls are what we already saw before We get the images and labels tensors and we create a grid and then we pass in the grid and the images. This is the part that we’ve already done, but we basically create our summary rider instance call it. TB for Tensor Board and then we’re ready to make the calls to add the data. So we add image. Then we add a graph. Then we begin our training loop, so everything in the training loop is exactly the same, except for down here. When we’ve completed a full pass over all batches in our training set, and we want to take these statistics and sit down here where we used to just print. The statistics we’re going to make some additional calls to the summary writer. So we’re gonna call. TB add scalar and a scalar is just a number, so we’re just essentially adding a number and then we’re also gonna call. TB add histogram, which is a set of values that will be used to create a histogram. So let’s look at the add scaler calls we pass in three arguments to this call we pass in what is called the tag and that’s just the name of the value, so we call add scalar, and then we say for which one for the loss and so if we call that multiple times, the intensive world will know to group that data together, so we say we want to add a scalar with a vowel with a tag of loss, then we pass the actual value total loss, and then we pass the epoch for which this is occurring and we do that for loss number correct and accuracy. All three of those, so this is gonna allow us to visualize all three of those across all of our epochs. Next we call add histogram and then for this, we can pass sets of values our cough, one bias tensor, our comp one weight tensor and then we’re gonna pass our comp one weight gradient tensor, so these are also tags, so we name them out comp one bias. Conf one weight, conf one weight grad. Those are the tags, and then we also pass the values, which is just Network comp wind-up bias and so on and we also pass the epoch, so that’s all there is to it to actually get this data out of our training loop and up into tensor board. Let’s go ahead and run this training loop and then see how this data looks in Tensor Board. [MUSIC] All right, so our training process has completed for these ten epochs. Let’s flip over to Tensor board and see how this looks. All right. We’re here intensive board, and I’m going to just refresh, and now what you can see is before. We only had images and graphs tabs at the top. Now we have Scalars distributions and histograms, so let’s take a quick look at the scalars. Each of these By the way correspond to the calls that were making ad scalar add images add graph and add histogram so we can see here that we have tabs for the accuracy, loss and number, correct? These are the tags that we sent in to the calls that we made. So for the accuracy we have that graphed the loss and number of correct, and if we just filter, we can say Dot, and it will display another tab here that has for all scalars that match this search, so we can see that our accuracy went up and then our loss went down and our number of correct went up and also we can hover over this and we can see the name of the run more on this in just a second, that’s super important, and then we can see the value and the step, so the steps are the epochs, and as we move, we can see each value and the corresponding step. All right, so let’s go take a look at the histograms now. Now for the histograms, this is where we passed in our cough. One bias tensor, our colorful and weight tensor and our comp one weight. Grad tensor. Then we can see each of these over time here. So at the back, that’s the first time step or the first epoch, and then as we move forward, you can see one. Two, three, four, five, six, seven, eight nine and this is showing us the frequency of occurrences for values inside of our bias Tensor. Same thing for the weights here we can start at the back. Which is the first epoch and we can move forward and same thing for the gradients and we can also see the distributions of each one of these graphs now the distributions, along with the calls to add histogram. So if we add a histogram, we also get distribution data. Now, let’s hop right back over here to the scalars, and I want you to just notice these two runs right here, so we have runs and we can search through all the runs by writing a filter and this is what we’re going to talk about in the next video because this is the real where the real power of tensor board comes in the ability to make multiple runs and then compare them side-by-side because when we have additional runs here and we check and uncheck them, We will see that those runs show come on and come off of the graph and so we can compare and see like which run had the best accuracy. That’s gonna be the topic of the next video. It is helpful to see the loss and accuracy values over time, however, we might need to admit to ourselves that tensor board isn’t really needed for this. We could just as easily done this inside of Jupiter notebook. The real power of tents. Our board is its out-of-the-box capability of comparing multiple runs. This allows us to rapidly experiment by varying our parameter values between runs and then comparing the outcomes to see which parameter values are working best in the next episode. When we cover hyper parameter experimentation. We’ll see how this is done for now, we should have a good understanding of what Tensor Board is and how we can use it with Pi Torch. If you haven’t already be sure to check out deep lizard, calm weather, there’s blog posts for each episode. There’s even quizzes now that you can use to test your understanding of the content and don’t forget about the deep lizard hivemind, where you can get exclusive perks and rewards thanks for contributing to collective intelligence. I’ll see in the next one. I want to talk to you about hill. Climbing and valley crossing these two concepts represent two types of intelligence. The first is concerned with reaching a goal. This is like the idea of optimization. The second is concerned with deciding what the goal should be. It’s like finding the most meaningful goals. These ideas correspond to the concepts of problem solving and creativity as humans, it’s important to realize that being intelligent is about more than just problem solving an optimization of some pre-existing goal. We must realize that being intelligent is also about finding the most meaningful goals. Can you think of examples of goals from these two categories? Look for hills and valleys and put your examples in the comments? I’ll see you in the valley beyond [Music].