Tabular Data Analysis | 13.5: Tabular Data – Processing Tutorial

The Coding Train

Subscribe Here





13.5: Tabular Data - Processing Tutorial


Hi, so before. I get into the main content in this video, which will be tabular data. I wanted to mention these videos here. Intro to processing for database They are made by Mathew Epler. I will include a link below. I recommend you watch these videos over the one You’re currently watching. Cuz They’re actually quite a bit in many ways more comprehensive, but certainly in particular, if you want to look for different algorithms for drawing different ways of setting pixels and drawing shapes on the screen that visualize the data. I would recommend these. I’m in the videos. I’m making them much more focused on. How do you actually get the data? Parse the data, and and the next step is really left to you of what what you’re gonna do with that and how you’re gonna draw and display that data so. I recommend these take a look at them. Let’s let’s talk about tabular data in processing. So you might find, you know, you wake up one morning and you decide. I need to find some data at somewhere and you just like look around the world and most likely you find the data on your computer in this case, although it, you know, you should find data in other places too. How do you get that data into processing so it might be that? The data is meant for you to get its open. It’s there’s an there’s an API for in an application programming interface. I think that’s what it stands for this means. There is an engine that’s designed to hand data from one application, a web server over to some other application. Like the one you’re making in processing. If that’s the case, you’re in a good place. Another form that another possibility might be. Oh, the data is available for download its downloadable as calm as a spreadsheet. This is actually what? I want to focus on in this video. This is also a great thing because processing has a table class a table class, which is an object that load tabular data spreadsheet data comma separated values data that’s in a tabular format directly to processing, and it will be very easy for you to parse in and go through that data. Another format might be. XML extensible markup language. I don’t know somebody correct the comments loop. That’s not right. Another format might be JSON Javascript object notation, so it makes sense to have standard format for data if they’re if a format is standard, we can always parse it very easily since we know exactly how the data will be how the data is presented to us and we’ll look at these standardized formats. XML and JSON in the next video for now I want to stick to table tabular data. So what is tabular data? So let’s come back over here and I’m going to close this browser window and come over here. So this is the Citibike website. A city bike is a bike sharing program in New York City. In case you’re not familiar with it and city bike. Just it has a site where all the data you can just download all the data and it has trip histories and there’s all sorts of it’s just some grocery. There’s all sorts of things so. I’ve already downloaded this data and I have it now here in as a spreadsheet that it’s a CSV file comma separated values that I’ve opened up here in libreoffice, so you can see this is tabular data. There are columns. Each column has a heading the duration of the trip. The start time the stop time. There’s lots of other things here, too. The station name the station latitude longitude, so and then there are rows. The rows are individual records much like a database. A database has fields and then records. A database of about of people might be first-name Lastname, Age, Gender. And there’s lots of records. Lots of people with first names, last names, age gender. The you can think of these, the column as headers or keys for the that particular column. So this is the kind of thing that we could work with very easily easily in a piece of software like Libre Office, and in fact, it even has like some automatic ways of graphing that data, But what I want to look at is. How do we get this data into processing so first? Let’s just take a moment and look at. How is this data actually stored? So if I close this and I’m gonna go into sublime here. Sublime text, which is a text editor and now I have that same file open, but I’m looking at its raw format, so you can see here that the first A line of text in this text file actually has all the headers in quotes separated by commas, and the next line has all of that, then all the other rows are records and that if data are just separated by commas, so this is a little harder for us to look at visually as a human being, but this is very easy for prop for libre office to to parse and also for processing to parse. So how do we do that? So let’s say we have this file, um. I don’t know what I’m doing next something. Hopefully that will be helpful to you. So let’s say we have this file, okay. Let’s let’s come over here. We know we could load a text file and a CSV is just a text file with load strings and load strings would get this data and in fact, there was a tithe many years ago where this was your only option in processing load strings and then after you split and have to chop it up and do all this stuff and it’s a big pain in the neck. Life is good these days and I can say this table. T equals load table data dot CSV. So I can take this particular file and load it into a table object. Load strings, give me as an array of strings load table gives me a table object and we’re going to see shortly load XML and load JSON as well so table XML and JSON. Those are the three standardized data formats that processing knows about can grab. And and you can parse them very easily, and another video will look at what if the data is not in any of these formats, what do we do? You know, we will come up with a solution. Okay, so lets. Come back over here. Okay, so remember, this example from a video or two ago. What was in this example, we have a string of comma separated values. We split that string up. We converted to two integers and we draw some stuff. I want to do exactly the same thing. Only instead of a hard-coded string, I want to load those comma separated values from a file and instead of loading it as a string and splitting it up and doing all that stuff manually. I want to load it into a table object, so I can parse it automatically. So now if I go to the data folder here, we can see. There is a CSV which just Mac preview by the way will visualize it for you as a grid, which is kind of useful and then, but if I actually open up that file, what’s in that file? It is just these four numbers, so these four numbers. I want to load this in as a table object, so let’s actually get rid of everything and instead of having a string. I’m gonna say load a table load table Data Dot Csv. Now I have the data as a table object rather than just as a raw string. So what’s the next thing that I need to do so here? We’re here right now. We have a table object. And what is the table object? A table object essentially in memory now is all of the information in that table and it’s kind of stored in a grid-like format Now another object that we need is a table row object, so if the table object refers to the entire table, a table row object refers to an individual row, so if we now have a reference to an individual row we can start asking for. Give me this piece of data in that row. Give me this piece of data in that row. Give me this piece of data in that row, so lets. Look at how we might do that so here now. I can say, give me a table row table. Dot, get row 0 This is a very simple. UPS, this is a very simple case where there is only a single row so now. I have a reference to this row and I can start doing something like, OK? Give me an X, Which is row dot, get. What is it it’s an integer get to get the the integer in the first column or Y is the integer in the second column, and maybe there’s a width and a height, which are the numbers in the second, the fourth, the third and fourth column, and by the way, get float so depending on what’s in that table? I can ask for that piece of data as a certain data type. Oh, is it? Is there an integer you know? I have to know this in advance. But is it an integer in that column? Give me that number. Is it a floating-point number? Is it a string? And we’ll see as we get to another example. We’ll see a more complex, but for right now, it’s simplest for me to just say, Give me all the integers. Whoops, give me the integers zero One two and three, and then what could I do? I could draw a rectangle and there we go now. Let’s let’s confirm that this is really really working. I mean I. This seems like the right rectangle. It’s a rectangle associated with these numbers, But what if I were to say 300 200 120 75 So that’s another row. I’ve added another row to this table, and if I run this again, we still see that first row, and if I run this again now. I’m going to look at the next row. Look, I’ve got the date a rectangle for that next row, so you can see. I can load add a table. I can look at any row. I can pull out the individual pieces of data in that row and I can use them. The truth of the matter is while this works, this isn’t exactly the scenario that will come up most commonly. And there’s a there’s a difference here. So coming back over here, right. I had a row and I was able to get each piece of data out of that. Road cameras are always going off. Sorry, okay. So am I back? Yes, so how did? I look at each one of those pieces of data. I looked at it by the index. The number of the row. But what you might have is actually a name for each row, And in fact in your file, the name for each row might actually actually be in the file itself as asked the first row itself. I lost my train of thought with the stupid camera. Growing up, okay, lets. Get back on track here. So what did we do? We got each piece of data as its index, but a more convenient way of working with tabular data in processing is actually to have a file that happens to have header information in it so now. I’ve added headed for header information. This file if you go back and look at that citibike data, it’s already in there. Most files will have this, And I could just say all right now. Let’s see what happens now when I run it. Oh, look, I don’t get anything. Why am I not getting anything because the first row in this file has strings in it? I can’t get those strings as numbers to draw something so one of the things. If my file has a Henner, I need to tell processing that there’s a header and I do that by when I load table. I had a second argument, the string header and now when I run, it, we’re gonna see. Ah, the the first row now is the actual data. The header is ignored as data and once. I’ve done this. I can reference the values by their header name and the reason why this is so much more convenient. Is, you know, in this case, there’s just four, and it’s very easy for me to track, but if you have a huge data set like the city bike data. I don’t know which number which column number is the latitude? But I know it’s just called latitude, so I can say getfloa’t latitude for when I want the latitude of a particular record, so this is a much more convenient way of working and this kind of wraps it up in a way. This is really the functionality that we need. We need to be a low to table deal with its header, figure out how we get individual rows out and then get the individual pieces of data from each column of that row, so let’s look at a different another scenario that has a bit more involved in it. There’s a few missing pieces here that. I think we could look at, so I’m gonna save this and I’m going to to have it open already. No, I’m going to open. I had it open recently, so I’m gonna open another example. This is from the learning processing book. Its example 18.3 of course, this is from the next edition of learning processing book, which doesn’t exist. But anyway, if you go to the Github repository, the examples are there and we can see, okay, so this particular example is is going to load a file. I mean, I’m gonna look at the data file for this example, and we can see it here and we can see. We have another table, so there’s a few things that are a little different one. This table each column is a different data type so there. X is an integer Y is an integer. The diameters of float and name is a string, so lets. Find the place in the code where that data is being loaded. And you can see it right here and actually. I just want to change something really quickly. Hopefully this will still work so you can see here. This is where for any given row, Give me the X integer, giving the Y integer. Give me the Diana man or float and give me the name as a string. So this is where we can see. Now that a table can have different data types and each column. And you can pull out that data. If you know the data type now, something else you’ll see here is what is each row. I have a loop. I starts at 0 I goes to row. Count, give me every row index. I so this is just like looping through an array. Give me all the elements of the array. This is just give me all the rows of the table, and so this is what you’ll see in a lot of this is more likely what you’re going to have and your project that you’re going to make. You’re gonna want to loop through all the rows of the table. Now there’s something else kind of really interesting about this particular example. Let me run this. What’s on the screen? There are these circles each one that I hover over. It has like a label and you can see. These circles are made. These bubbles are made from the data in that file and there’s a bubble object. My audio is still working yet. There’s a bubble object right here. Look at this bubble object as an X, A Y A diameter, a name two floats, add another float and a string. This is back what we did with object-oriented programming. I need to have this thing. This object that has a bunch of properties and some functionality. And we have this questions like. Well, Let’s say I have a lot of these objects. How do I initialize them? Maybe I make them all randomly or maybe I arrange them in a grid pattern or a spiral pattern, But here’s another way of doing it. What’s in that file? Look at this data file. This data file has all of the initial values for a bunch of objects, so this is mapping a CSV to the data in an object is a really useful way of just keeping track of information or even doing data visualization. You have this, you know, if I if I want to have load the city bike data, I might make a trip object and that trip object would have a start station and end station a duration, so that so this mapping I think is a really useful way of working with data from a file in processing, so let’s look at a couple other pieces of functionality here, so I’m going to go back to the sketch and I’m going to start clicking in the sketch click-click-click. Now I added three bubbles with the labels blah into this particular file. And I’m going to do this particular sketch. I’m going to close this and I’m going to run it again. New bubbles! I added, are still there, even though I quit the sketch and opened it again. How did that happen? Let’s go back and look at the data file again. I probably have to close it. Yeah, sorry to close it, lets. Look at it again! Look at this! There are new records in the file, so in addition to loading data from a file from a tape tabular data from a CSV file. You can write a file out. So this particular example is demonstrating the fact that you can use a table object for persistence If I wanted to keep track of the list of high scores for a game, I could always every time I run my processing sketch. I could save out the current high scores, so they could access them The next time I run the sketch. And if we look at the code down here, we will see. How is this happened? Look every time. I click the mouse. I make a new row and I set the values of that row. I set the X to Mouse. X is set the Y to Mouse way. I said the diameter, something random, and I give it a label and you can see here. Then what happens I can now save. I can now save that A table object out back to the same file, so there’s a lot going on here in this example and what we’ve talked about, but just to sum it all up. We have a CSV file. You can load that. See us about fee file and you can read through every row and pull out all the data you can make objects from that data. You could visualize those objects, and then if those objects change or you add more, you could save that file back out, So what I would suggest to you? There’s a there’s a couple different things you could try as an exercise. One is you might work with this. Try to make the bubbles just have like a color and see if you can like, load the color from a file. Save colors out to the file. Another thing you might do is find some tabular data. Make your own spreadsheet. Download that citibike data. Find something else and see if you could at least parse it and do some very basic drawing with it. You know, maybe just you just get started with. You know, a simple bar graph or something very basic, so give that a try and in the next video we’re gonna look at this exact same scenario, but instead of the data being a tabular format, we’re gonna look at the data as XML or as JSON. Okay, and this is the end.

0.3.0 | Wor Build 0.3.0 Installation Guide

Transcript: [MUSIC] Okay, so in this video? I want to take a look at the new windows on Raspberry Pi build 0.3.0 and this is the latest version. It's just been released today and this version you have to build by yourself. You have to get your own whim, and then you...

read more