Transcript:
Hello everyone My name is Alex Today I want to introduce Eager Execution to everyone You should have heard this name in the first two lectures. But I will give you a better introduction This is necessary and object-oriented Pythonic new way to use TensorFlow We want to introduce to you today This is part of the core of TensorFlow You are here, or you already know it or because you are watching the live broadcast I hope TensorFlow Has become an image execution engine for machine learning Let you execute images on a large scale And other great things right? Why did we choose images in the first place? Now I want to introduce Eager Execution to everyone This allows us to go beyond what is made with images I think it’s a good idea to review what is bothering us This is a good reason you want your calculations As a platform independent image Once you have this it will be easy to distinguish the image When I went to graduate school Before auto diff was the standard for machine learning toolkits I did not wish to be able to do this It’s like… life is better now, trust me In addition, if you have a platform independent Abstract representation of computation You just need to deploy almost anything you want Can run it on TPU can run it on GPU You can put it on your phone or you can put it on the Raspberry Pi It’s like all kinds of great deployment scenarios You will hear today This platform-independent view is really valuable In addition, the compiler can also process the data flow graph internally They know how to handle all kinds of good optimizations Rely on your global vision of computing For example, constant folding of common common sub-expression elimination Things like data layout and similar Many optimizations are really deep learning specific We can choose how to arrange your channels and your height correctly Width and other things Your convolution speed can be faster Finally, this is a very important key reason for us at Google I hope it’s important to you too Once you have a platform independent Calculation representation You can deploy and distribute it to hundreds of machines Or TPU-pod, as shown earlier It is a seamless process Since the quality of the image is very good This makes us think it’s a good idea to go beyond them now And let you execute eager execution ? A good start is You don’t actually have to give up automatic differentiation I believe you will all be familiar with other frameworks such as Pythons Autograph … Autograd is right Let you distinguish dynamic code… You don’t need Distinguished computational prior Just build a trajectory Then walk back along the trajectory to calculate the gradient Besides, if you don’t stop building a platform… Calculate the image like this You can repeat faster When you build, you can also play with your model Check it Poke it and stab it This can make you more efficient When you want to make all these changes In addition, you can also run your model on the debugger and analyzer Add various analysis tools to them To really understand how they do what they are doing Finally, if we don’t force You do calculations in different ways and continue to use the ones you are using You can use All host programming language machines to handle control flow and data flow And complex data structures For some models this makes it crucial to make the model work I hope you don’t think, “How can I use it?” In fact, the way to use it is very simple First you have to import TensorFlow Call tf.enable_eager_execution After doing this Every time you want to run a TensorFlow operation Just like in this case.matmul Instead of TensorFlow building the image later Will run that matrix multiplication when executed We will run that matrix multiplication immediately And give you results Then you can print the result It can also be sliced or dice Can do whatever you want Because it happened immediately You can have highly dynamic control flow It depends on the actual value of the calculation being run This is just one I wrote Simple example of Wolfe condition line search It’s okay, it’s just important. It’s kind of like while loops It depends on the complex value This is a computer based calculation It will run well on any device And with this enable_eager_execution thing We also bring you a few symbols of TensorFlow This makes it easier for you to write code that can run Whether it’s creating images or eagerly running We also bring you a new way of making gradients I believe you are now familiar with our normal TensorFlow How to make a gradient effect Where did you just create the variable to create your loss function I hope you guys can think of a better loss function than this Then you have to call tf.gradients to distinguish it But when you have eager execution, we will try our best to improve efficiency If you plan to… there is one thing to consider If you want to distinguish a calculation You need to keep track of the information in memory Everything that has happened so far Such as your activation and such things When you are not calculating the gradient I don’t want you to pay for this tracking because the performance is real Like all the reasons we did it Because we want to use these big and beautiful hardware Train those models super fast When eager execution is enabled Use this context manager when you want to calculate the gradient Keep the reel active while the reel is recordable All the operations performed when you calculate the gradient It can be played repeatedly, otherwise, the API is still the same In addition, write a training loop in eager As Derrick pointed out a lot of… Very simple and straightforward You only need to use Python for loop It’s good to repeat your data set eager ’s data set Their performance and image execution machine Show the same high level Then you can just make predictions Calculate your gradient to provide for your gradient And do all the things I’m used to doing But what is really interesting about eager execution is When you finish writing this code It’s done We already know the procedure But when you want to do something like debugging while still writing Then… Any model code can be used when eager execution is enabled– I use my example here as a gradual example– Add some notes, like putting a Python debugger wherever you want Once in the Python debugger You can fully use debugging You can print any tensor tensor values can change any value Can run any operation on any tensor This will allow you Really understand what’s going on in the model And can really fix any problems You can also use eager execution code and analyze it Use any of your favorite and most familiar analysis tools Here i have a small model just finished matmul And bias_add Suppose I don’t know which of these operations Will become a bottleneck, which one is lower? I believe you all know that matmul is much more expensive Here you can run that code through the Python profiler Just like you use other programming And find out that the batch matmul for me here is 15 times more expensive Than added my bias And by the way The things that run in the Google Cooperative Lab This is it… It is completely open to share the GPU-supported interface It can also be run in Jupyter notebook just like hosted on Google Cloud This is really cool, I think we have an eager demo This is hosted there, you can try again later Or if you use live streaming If you can find the link, you can play now But with eager We have brought a lot of new APIs This makes it easier for you to build TensorFlow images And execute various models These APIs are Compatible with eager execution and image construction Has always been a frequent low-priority feature proposal How to customize the gradient in TensorFlow I believe you are also familiar with the tricks of other developers Such as stop gradients and functions or other similar things But we are launching a new API that can run in eager and image execution What I like about this example is that this is a question many people ask Many people, how to do it If I want to do positive push Then the inverse method is used on the gradient of a specific tensor Edit it, it’s norm to keep it small to prevent surge This only requires six lines of code Use the inverse method to make a version of tf.identity I cut the gradient of it, I think it’s really cool I look forward to seeing what you can do with this When you don’t just write six lines of code And solve all kinds of new and interesting problems An important change occurred when programming using eager instead of graph I really hope you guys stop and think about it We are trying to make everything as pythonic as possible And object-oriented The variables in TensorFlow are… Usually a complicated thing But it becomes simpler when eager execution is enabled A TensorFlow variable is just a Python object You create one and you have it You can write, you can change its value, you can read its value You can retrieve your memory when the last reference value disappears Even in GPU memory If you want to share variables, just reuse those objects Don’t worry about the scope of variables or any other complicated structures Because we have this object-based variable method We can look at some APIs in TensorFlow And rethink them to be more object-oriented And easier to use And very… What really caught our attention was The metrics API needs to be overhauled So we introduce this new tfe.metrics package to everyone There are two methods for each indicator One will update the value and one will give you the result Hope all developers can be familiar with using this API But please don’t try to compare this with other indicator APIs (laughter) We also give you a method to save the object-based TensorFlow model If you look at the TensorFlow checkpoint now You know these are dependent on variable names The variable name is not only dependent on the name you give to the variable And all the variables in the image This may be a bit difficult to save and load a subset of the model And really control what’s in the checkpoint So we introduced a completely object-oriented Python object primary save API Where you guys… Just like Python pickle is like any variable Can be obtained from the saved model and your model is also saved You can save any subset of the model Can load any subset of the model You can even use this tfe.checkpoint object to create what you want to save This is not only a model Here we have an optimizer and a global_step But you can really put anything in there The idea is that this object graph eventually becomes a variable It’s what you can save and load so you can have [GAN] Save and load your discriminator And separate from the generator Then you can take the discriminator and load the backup Just like another newer network you can use in another part of the model This should give you more control and get more TensorFlow checkpoint When I told other developers that I was working on eager execution Everyone will ask me if it is fast? Because of the image… With this high performance promise How fast can it run Python code every moment? The answer is we make it fast enough For computationally intensive models You hardly see any extra time spent using Python We are as fast as TensorFlow images for… Sometimes a little bit faster for reasons I can’t understand Even for highly dynamic models Its performance will not be worse than anything that can be found Please do not append these numbers There are more benchmarks in our code base We are very actively optimizing the performance of eager I hope the information you get is If your model keeps the GPU busy If you are doing large-scale convolution and large-scale matrix multiplication Use eager execution to do experiments for research and model construction Almost no cost But when you make something smaller It takes extra time. I want to review it again. Say it again, ignore them Because we are very active in optimizing If you just run no op in TensorFlow Like identity It only takes a few microseconds to execute If you use eager execution to open it Just a few extra microseconds If you are tracking the gradient 3 microseconds left If it’s just GPU stream queuing Just one digit microsecond If you can perform enough calculations to keep the GPU busy You won’t see any disadvantages of using eager execution Again these numbers are improving rapidly Don’t rely on it too much There is a code base for TensorFlow Large ecosystem of models, frameworks, checkpoints I don’t want anyone to give up If you want to use eager execution, I don’t want you to give up We also seriously think How do you interoperate between eager and graph One way is to call graphs from eager code You can use tfe.make_template to do this This creates the image function parameters When you build an image through it A small Python function written for you Then you can use it as a manipulate and call images from eager execution We can also do the opposite Call eager from image Suppose there is a big picture and you already know everything about it But a small piece in the calculation You really don’t know how to express Or you really don’t know Or you don’t want to use graph representation like TensorFlow Then you can wrap it in tfe.py_func What you get there is when the Python function is executed It is eager tensors, you can run any TensorFlow operation Including convolution and other things not available in my_py You can also look at other values And check and use the dynamic control flow there I hope these two things Make you really reuse eager and graph code The easiest way to make eager and graph compatible is Write some model code to run in two ways If you think about it Once your model is finished writing, debugging and testing I can’t tell you if you need to build an image Or execute eagerly Write, repeat and debug eager Then import the same code into the image Put it on the Estimator and deploy it on the TPU pod Deploy it on the GPU Distribute it and do whatever you want This is what we are going to do in the example model There is a link at the end of the speech So you don’t need to worry about writing it down Here are some practical suggestions Write some when executing eagerly and creating images Code that works well To do this, you can use Keras layers, they are really great They are object-oriented and pythonic They are easy to understand, manipulate and play with Use Keras model to stitch these layers together This will guide you to automatically save and load training with various things As long as you want but you will not be forced to use Use tf.contrib.summary instead of tf.summary They will soon be moved to the Tensor board open source package If you guys are watching the video Or it has been removed Use tfe.metrics instead of tf.metrics Because these are object-oriented and easy to use Easier to use in eager And use object-based save Better user experience anyway So I hope you will always want to be like this If you do It is very likely that your code It will be super awesome in eager execution and image construction Now i want to spend some time Tell you why you should enable eager execution You know it’s like good… The important reason drove us to establish this If you are new to machine learning Or new to TensorFlow but you want to learn Be able to use these objects and manipulate them directly Than build an image and interact in a later conversation Is a better experience This is more intuitive It’s much better to let you know what’s going on I have shown you everything and go directly to the place of execution Play with it And figure out how to get the image later If you are a researcher are repeating these models very quickly Modifying their internal properties… Compare them and try out some great models And our TensorFlow team Unexpectedly when designing TensorFlow Eager execution makes it easier for you to understand what is happening Debug what is happening Improve productivity in advance If you are a researcher this is something for you In addition, if your model fails to work, you want to understand why Able to enable eager execution and then start in the debugger Modify some values and run it Will understand that this is priceless, it saves me a lot of time Similarly, if you want to analyze a model use any tool you like To dissect Python eager execution is your friend In addition, there are some models such as recursive RNN In the static data stream image If you don’t need the whole calculation This is easier to express If you are working on one of these models eager execution is also one of the options I think the real reason you should enable this is It’s really interesting Interacting with TensorFlow is a very good intuitive way I hope you enjoy it Now I want to point out some things Now some of my colleagues are sitting there They will go to the demo room during the break Bring a laptop Just like jupyter notebooks Let you type and try the eager model Please try it out If you are watching live Type in that short link- Hope it will stay on the screen long enough for you to type Use it immediately It’s really great We have a TensorFlow getting started guide That should be online now programmers_guide/eager Tell you what you need to know about eager execution What you need to know to start using TensorFlow And the use of eager execution We also have many example models From RNNs to all kinds of things Just behind that link I encourage you to take a look How to easily write a model And reuse the same code from the image How easy it is to deploy We have image deployment of all models In addition to highly dynamic models It’s really hard to write in image format Try it If you guys try Let us know how it is We are very excited to be able to share with you And hope you all have a good time correct thank you all (applause) ♪ (music) ♪