🔥 PyTorch Course Summary

📝 Communication Standards

Tone & Language

Use clear, concise technical language
Maintain a professional yet approachable tone
Explain complex concepts with practical examples
Use appropriate mathematical notation when discussing algorithms
Include visual explanations for complex neural network architectures

Documentation & References

Reference PyTorch official documentation (pytorch.org)
Cite academic papers for advanced concepts
Link to relevant research papers on arXiv
Include references to standard machine learning textbooks
Document all data preprocessing steps

Response Structure

Begin with a high-level overview
Follow with detailed technical explanation
Include code examples when applicable
End with practical applications
Provide performance metrics and benchmarks

💡 Problem-Solving Approach

Analysis Methods

Data Exploration
- Statistical analysis
- Visualization techniques
- Distribution analysis
- Missing data handling
- Outlier detection
Model Selection
- Use case evaluation
- Architecture comparison
- Performance requirements
- Resource constraints
- Scalability considerations

Solution Development

Iterative Process
- Prototype development
- Validation methods
- Performance optimization
- Error analysis
- Model refinement
Implementation Strategy
- Environment setup
- Data pipeline creation
- Model architecture design
- Training workflow
- Deployment planning

🎯 Domain Expertise

Primary Focus Areas

Deep Learning Fundamentals
- Neural Network Architecture
- Optimization Algorithms
- Loss Functions
- Backpropagation
- Gradient Descent Variations
PyTorch Ecosystem
- Tensor Operations
- Autograd Mechanics
- Dataset Management
- Model Deployment
- Distribution Strategies

Key Skills

Technical Competencies
- Python Programming
- Mathematical Foundations
- Statistical Analysis
- GPU Computing
- Distributed Training
Applied Skills
- Model Design
- Hyperparameter Tuning
- Performance Optimization
- Debug & Troubleshooting
- Production Deployment

🛠 Technical Implementation

Code Standards

Style Guidelines
- Follow PEP 8 conventions
- Use type hints
- Implement proper error handling
- Write comprehensive docstrings
- Maintain consistent naming conventions
Best Practices
- Modular architecture
- Clean code principles
- Version control workflow
- Testing protocols
- Documentation standards

🎓 Educational Approach

Teaching Methods

Theoretical Foundation
- Mathematical concepts
- Algorithm explanation
- Architecture design
- Performance analysis
- Optimization techniques
Practical Application
- Hands-on exercises
- Real-world projects
- Case studies
- Performance profiling
- Deployment scenarios

Learning Resources

Core Materials
- Official PyTorch tutorials
- Research papers
- Technical blogs
- Video lectures
- Code repositories
Supplementary Resources
- Practice datasets
- Example projects
- Community forums
- Online courses
- Reference implementations

🤝 Interaction Guidelines

User Support

Technical Assistance
- Implementation guidance
- Debugging help
- Performance optimization
- Best practices advice
- Architecture reviews
Learning Support
- Concept clarification
- Resource recommendations
- Progress tracking
- Feedback provision
- Mentorship guidance

📊 Quality Standards

Code Quality

Implementation Standards
- Clean code principles
- Performance optimization
- Memory efficiency
- GPU utilization
- Scalability considerations
Testing Requirements
- Unit tests
- Integration tests
- Performance benchmarks
- Edge case handling
- Error recovery

🔒 Security & Safety

Security Practices

Data Protection
- Input validation
- Secure data handling
- Access control
- Environment security
- Dependency management
Model Security
- Adversarial defense
- Model robustness
- Privacy preservation
- Ethical considerations
- Bias mitigation

🔄 Continuous Improvement

Development Areas

Technical Enhancement
- Performance optimization
- Feature implementation
- Architecture updates
- Tool integration
- Documentation improvement
Learning Integration
- New research findings
- Community feedback
- User suggestions
- Industry trends
- Best practices updates

Topic Categories

Part 1: Fundamentals (Sections 1-32)

Python Basics
Linear Algebra
Calculus
Probability & Statistics
Machine Learning Basics

Part 2: PyTorch Foundations (Sections 33-64)

Tensor Operations
Autograd Mechanics
Neural Network Basics
Optimization Algorithms
Loss Functions

Part 3: Deep Learning (Sections 65-96)

Convolutional Networks
Recurrent Networks
Transformers
Generative Models
Transfer Learning

Part 4: Advanced Topics (Sections 97-128)

Distributed Training
Model Optimization
Custom Extensions
C++ Frontend
Mobile Deployment

Part 5: Production (Sections 129-160)

Model Serving
Performance Tuning
Scaling Strategies
Monitoring
Maintenance

Meta Information

Version: 2.0.0 Last Updated: 2025-03-05 19:37:43 UTC License: MIT

🚀 Quick Navigation

Section 1
Section 2
Section 3
Section 4
Section 5
Section 6
Section 7
Section 8
Section 9
Section 10
Section 11
Section 12
Section 13
Section 14
Section 15
Section 16
Section 17
Section 18
Section 19
Section 20
Section 21
Section 22
Section 23
Section 24
Section 25
Section 26
Section 27
Section 28
Section 29
Section 30
Section 31
Section 32
Section 33
Section 34
Section 35
Section 36
Section 37
Section 38
Section 39
Section 40
Section 41
Section 42
Section 43
Section 44
Section 45
Section 46
Section 47
Section 48
Section 49
Section 50
Section 51
Section 52
Section 53
Section 54
Section 55
Section 56
Section 57
Section 58
Section 59
Section 60
Section 61
Section 62
Section 63
Section 64
Section 65
Section 66
Section 67
Section 68
Section 69
Section 70
Section 71
Section 72
Section 73
Section 74
Section 75
Section 76
Section 77
Section 78
Section 79
Section 80
Section 81
Section 82
Section 83
Section 84
Section 85
Section 86
Section 87
Section 88
Section 89
Section 90
Section 91
Section 92
Section 93
Section 94
Section 95
Section 96
Section 97
Section 98
Section 99
Section 100
Section 101
Section 102
Section 103
Section 104
Section 105
Section 106
Section 107
Section 108
Section 109
Section 110
Section 111
Section 112
Section 113
Section 114
Section 115
Section 116
Section 117
Section 118
Section 119
Section 120
Section 121
Section 122
Section 123
Section 124
Section 125
Section 126
Section 127
Section 128
Section 129
Section 130
Section 131
Section 132
Section 133
Section 134
Section 135
Section 136
Section 137
Section 138
Section 139
Section 140
Section 141
Section 142
Section 143
Section 144
Section 145
Section 146
Section 147
Section 148
Section 149
Section 150
Section 151
Section 152
Section 153
Section 154
Section 155
Section 156
Section 157
Section 158
Section 159
Section 160
Section 161
Section 162
Section 163
Section 164
Section 165
Section 166
Section 167
Section 168
Section 169
Section 170
Section 171
Section 172
Section 173
Section 174
Section 175
Section 176
Section 177
Section 178
Section 179
Section 180
Section 181
Section 182
Section 183
Section 184
Section 185
Section 186
Section 187
Section 188
Section 189
Section 190
Section 191
Section 192
Section 193
Section 194
Section 195
Section 196
Section 197
Section 198
Section 199
Section 200
Section 201
Section 202
Section 203
Section 204
Section 205
Section 206
Section 207
Section 208
Section 209
Section 210
Section 211
Section 212
Section 213
Section 214
Section 215
Section 216
Section 217
Section 218
Section 219
Section 220
Section 221
Section 222
Section 223
Section 224
Section 225
Section 226
Section 227
Section 228
Section 229
Section 230
Section 231
Section 232
Section 233
Section 234
Section 235
Section 236
Section 237
Section 238
Section 239
Section 240
Section 241
Section 242
Section 243
Section 244
Section 245
Section 246
Section 247
Section 248
Section 249
Section 250
Section 251
Section 252
Section 253
Section 254
Section 255
Section 256

📚 Course Overview

This comprehensive PyTorch course covers:

🤖 Machine Learning fundamentals
🧠 Deep Learning with PyTorch
💻 Hands-on coding exercises
🛠️ Practical implementations

📋 Prerequisites

✅ 3-6 months Python coding experience
✅ Basic understanding of programming concepts

🔗 Additional Resources

📖 Course Content

Section 1: Main Topics

Key Topics:

This comprehensive course will teach you the foundations of machine learning and deep learning using PyTorch
PyTorch is a machine learning framework written in Python
You'll learn machine learning by writing PyTorch code

📄 Click to view detailed content

This comprehensive course will teach you the foundations of machine learning and deep learning
using PyTorch. PyTorch is a machine learning framework written in Python. You'll learn machine
learning by writing PyTorch code. So when in doubt, run the provided code and experiment.
Your teacher for this course is Daniel Bourke. Daniel is a machine learning engineer and popular
course creator. So enjoy the course and don't watch the whole thing in one sitting.
Hello, welcome to the video. It's quite a big one. But if you've come here to learn machine
learning and deep learning and PyTorch code, well, you're in the right place. Now, this video and
tutorial is focused for beginners who have got about three to six months of Python coding experience.
So we're going to cover a whole bunch of important machine learning concepts
by writing PyTorch code. Now, if you get stuck, you can leave a comment below or post on the course
GitHub discussions page. And on GitHub is where you'll be able to find all the materials that we cover,
as well as on learn pytorch.io. There's an online readable book version of this course there.
But if you finish this video and you find that, hey, I would still like to learn more PyTorch.
I mean, you can't really cover all the PyTorch in a day that video titles just apply on words of
the length of video. That's an aside. There is five more chapters available at learn pytorch.io,
covering everything from transfer learning to model deployment to experiment tracking.
And all the videos to go with those are available at zero to mastery.io. But that's enough for me.
Having machine learning and I'll see you inside.
Hello, my name is Daniel and welcome to the deep learning with
PyTorch course. Now, that was too good not to watch twice. Welcome to the deep learning with
cools at fire PyTorch course. So this is very exciting. Are you going to see that animation
quite a bit because, I mean, it's fun and PyTorch's symbol is a flame because of torch.
But let's get into it. So naturally, if you've come to this course, you might have already
researched what is deep learning, but we're going to cover it quite briefly.
And just in the sense of how much you need to know for this course, because we're going to be
more focused on, rather than just definitions, we're going to be focused on getting practical
and seeing things happen. So if we define what machine learning is, because as we'll see in a
second, deep learning is a subset of machine learning. Machine learning is turning things
data, which can be almost anything, images, text, tables of numbers, video, audio files,
almost anything can be classified as data into numbers. So computers love numbers,
and then finding patterns in those numbers. Now, how do we find those patterns? Well,
the computer does this part specifically a machine learning algorithm or a deep learning
algorithm of things that we're going to be building in this course. How? Code and math. Now,
this course is code focused. I want to stress that before you get into it. We're focused on
writing code. Now, behind the scenes, that code is going to trigger some math to find patterns in
those numbers. If you would like to deep dive into the math behind the code, I'm going to be
linking extra resources for that. However, we're going to be getting hands on and writing lots of
code to do lots of this. And so if we keep going to break things down a little bit more,
machine learning versus deep learning, if we have this giant bubble here of artificial
intelligence, you might have seen something similar like this on the internet. I've just
copied that and put it into pretty colors for this course. So you've got this overarching
big bubble of the topic of artificial intelligence, which you could define as, again, almost anything
you want. Then typically, there's a subset within artificial intelligence, which is known as machine
learning, which is quite a broad topic. And then within machine learning, you have another topic
called deep learning. And so that's what we're going to be focused on working with PyTorch,
writing deep learning code. But again, you could use PyTorch for a lot of different machine
learning things. And truth be told, I kind of use these two terms interchangeably. Yes, ML is the
broader topic and deep learning is a bit more nuanced. But again, if you want to form your
own definitions of these, I'd highly encourage you to do so. This course is more focused on,
rather than defining what things are, is seeing how they work. So this is what we're focused on doing.
Just to break things down, if you're familiar with the fundamentals of machine learning,
you probably understand this paradigm, but we're going to just rehash on it anyway. So if we
consider traditional programming, let's say you'd like to write a computer program that's enabled to,
or has the ability to reproduce your grandmother's favorite or famous roast chicken dish. And so we
might have some inputs here, which are some beautiful vegetables, a chicken that you've raised on the
farm. You might write down some rules. This could be your program, cut the vegetables, season the
chicken, preheat the oven, cook the chicken for 30 minutes and add vegetables. Now, it might not
be this simple, or it might actually be because your Sicilian grandmother is a great cook. So she's
put things into an art now and can just do it step by step. And then those inputs combined with
those rules makes this beautiful roast chicken dish. So that's traditional programming. Now,
a machine learning algorithm typically takes some inputs and some desired outputs and then

Section 2: Main Topics

Key Topics:

So where in traditional program, we had to hand write all of these rules, the ideal machine learning algorithm will figure out this bridge between our inputs and our idealized output
Now, in the machine learning sense, this is typically described as supervised learning, because you will have some kind of input with some kind of output, also known as features, and also known as labels
And the machine learning algorithm's job is to figure out the relationships between the inputs or the features and the outputs or the label

📄 Click to view detailed content

figures out the rules. So the patterns between the inputs and the outputs. So where in traditional
program, we had to hand write all of these rules, the ideal machine learning algorithm will figure
out this bridge between our inputs and our idealized output. Now, in the machine learning sense, this
is typically described as supervised learning, because you will have some kind of input with
some kind of output, also known as features, and also known as labels. And the machine learning
algorithm's job is to figure out the relationships between the inputs or the features and the outputs
or the label. So if we wanted to write a machine learning algorithm to figure out our Sicilian
grandmother's famous roast chicken dish, we would probably gather a bunch of inputs of ingredients
such as these delicious vegetables and chicken, and then have a whole bunch of outputs of the
finished product and see if our algorithm can figure out what we should do to go from these
inputs to output. So that's almost enough to cover of the difference between traditional programming
and machine learning as far as definitions go. We're going to get hands on encoding these sort
of algorithms throughout the course. For now, let's go to the next video and ask the question,
why use machine learning or deep learning? And actually, before we get there, I'd like you to
think about that. So going back to what we just saw, the paradigm between traditional programming
and machine learning, why would you want to use machine learning algorithms rather than
traditional programming? So if you had to write all these rules, could that get cumbersome?
So have a think about it and we'll cover it in the next video.
Welcome back. So in the last video, we covered briefly the difference between
traditional programming and machine learning. And again, I don't want to spend too much time
on definitions. I'd rather you see this in practice. And I left you with the question,
why would you want to use machine learning or deep learning? Well, let's think of a good reason.
Why not? I mean, if we had to write all those handwritten rules to reproduce Alsace and grandmother's
roast chicken dish all the time, that would be quite cumbersome, right? Well, let's draw a line
on that. Why not? What's a better reason? And kind of what we just said, right? For a complex
problem, can you think of all the rules? So let's imagine we're trying to build a self-driving car.
Now, if you've learned to drive, you've probably done so in maybe 20 hours, 100 hours. But now,
I'll give you a task of writing down every single rule about driving. How do you back out of your
driveway? How do you turn left and go down the street? How do you park a reverse park? How do
you stop at an intersection? How do you know how fast to go somewhere? So we just listed half a
dozen rules. But you could probably go a fair few more. You might get into the thousands.
And so for a complex problem, such as driving, can you think of all the rules? Well, probably not.
So that's where machine learning and deep learning come in to help. And so this is a beautiful comment
I like to share with you on one of my YouTube videos is my 2020 machine learning roadmap.
And this is from Yashawing. I'm probably going to mispronounce this if I even try to.
But Yashawing says, I think you can use ML. So ML is machine learning. I'm going to use that
a lot throughout the course, by the way. ML is machine learning, just so you know.
For literally anything, as long as you can convert it into numbers, ah, that's what we said before,
machine learning is turning something into computer readable numbers. And then programming it to find
patterns, except with a machine learning algorithm, typically we write the algorithm and it finds
the patterns, not us. And so literally it could be anything, any input or output from the universe.
That's pretty darn cool about machine learning, right? But should you always use it just because
it could be used for anything? Well, I'd like to also introduce you to Google's number one rule
of machine learning. Now, if you can build a simple rule based system such as the step of five
rules that we had to map the ingredients to our Sicilian grandmothers roast chicken dish,
if you can write just five steps to do that, that's going to work every time, well, you should
probably do that. So if you can build a simple rule based system that doesn't require machine
learning, do that. And of course, maybe it's not so very simple, but maybe you can just write some
rules to solve the problem that you're working on. And this is from a wise software engineer,
which is, I kind of hinted at it before, rule one of Google's machine learning handbook. Now,
I'm going to highly recommend you read through that, but we're not going to go through that in
this video. So check that out. You can Google that otherwise the links will be where you get links.
So just keep that in mind, although machine learning is very powerful and very fun and very
excited, it doesn't mean that you should always use it. I know this is quite the thing to be saying
at the start of a deep learning machine learning course, but I just want you to keep in mind,
simple rule based systems are still good. Machine learning isn't a solve all for everything.
Now, let's have a look at what deep learning is good for, but I'm going to leave you on a
clip hammock because we're going to check this out in the next video. See you soon.

Section 3: Main Topics

Key Topics:

In the last video, we familiarized ourselves with Google's number one rule of machine learning, which is basically if you don't need it, don't use it
And with that in mind, what should we actually be looking to use machine learning or deep learning for
But if you have a long, long list of rules, like the rules of driving a car, which could be hundreds, could be thousands, could be millions, who knows, that's where machine learning and deep learning may help

📄 Click to view detailed content

In the last video, we familiarized ourselves with Google's number one rule of machine learning,
which is basically if you don't need it, don't use it. And with that in mind,
what should we actually be looking to use machine learning or deep learning for?
Well, problems with long lists of rules. So when the traditional approach fails to
remember the traditional approach is you have some sort of data input, you write a list of rules for
that data to be manipulated in some way, shape, or form, and then you have the outputs that you
know. But if you have a long, long list of rules, like the rules of driving a car, which could be
hundreds, could be thousands, could be millions, who knows, that's where machine learning and
deep learning may help. And it kind of is at the moment in the world of self-driving cars,
machine learning and deep learning are the state of the art approach.
Continually changing environments. So whatever the benefits of deep learning is that it can
keep learning if it needs to. And so it can adapt and learn to new scenarios. So if you update the
data that your model was trained on, it can adjust to new different kinds of data in the future.
So similarly to if you are driving a car, you might know your own neighborhood very well.
But then when you go to somewhere you haven't been before, sure you can draw on the foundations
of what you know, but you're going to have to adapt. How fast should you go? Where should you
stop? Where should you park? These kinds of things. So with problems with long lists of rules,
or continually changing environments, or if you had a large, large data set. And so this is where
deep learning is flourishing in the world of technology. So let's give an example. One of my
favorites is the food 101 data set, which you can search for online, which is images of 101
different kinds of foods. Now we briefly looked at what a rule list might look like for cooking
your grandmother's famous Sicilian roast chicken dish. But can you imagine if you wanted to build
an app that could take photos of different food, how long your list of rules would be to differentiate
101 different foods? It'd be so long. You need rule sets for every single one. Let's just take
one food, for example. How do you write a program to tell what a banana looks like? I mean you'd
have to code what a banana looks like, but not only a banana, what everything that isn't a banana
looks like. So keep this in mind. What deep learning is good for? Problems with long lists of rules,
continually changing environments, or discovering insights within large collections of data.
Now, what deep learning is not good for? And I'm going to write typically here because,
again, this is problem specific. Deep learning is quite powerful these days and things might
change in the future. So keep an open mind, if there's anything about this course, it's not for
me to tell you exactly what's what. It's for me to spark a curiosity into you to figure out what's
what, or even better yet, what's not what. So when you need explainability, as we'll see,
the patterns learned by a deep learning model, which is lots of numbers, called weights and biases,
we'll have a look at that later on, are typically uninterpretable by a human. So some of the times
deep learning models can have a million, 10 million, 100 million, a billion, some models are getting
into the trillions of parameters. When I say parameters, I mean numbers or patterns in data.
Remember, machine learning is turning things into numbers and then writing a machine learning model
to find patterns in those numbers. So sometimes those patterns themselves can be lists of numbers
that are in the millions. And so can you imagine looking at a list of numbers that has a million
different things going on? That's going to be quite hard. I find it hard to understand
three or four numbers, let alone a million. And when the traditional approach is a better option,
again, this is Google's rule number one of machine learning. If you can do what you need to do with
a simple rule based system, well, maybe you don't need to use machine learning or deep learning.
Again, I'm going to use the deep learning machine learning terms interchangeably.
I'm not too concerned with definitions. You can form your own definitions, but just so you know,
from my perspective, ML and deep learning are quite similar. When arrows are unacceptable.
So since the outputs of a deep learning model aren't always predictable, we'll see that deep
learning models are probabilistic. That means they're when they predict something, they're making a
probabilistic bet on it. Whereas in a rule based system, you kind of know what the outputs are
going to be every single time. So if you can't have errors based on probabilistic errors,
well, then you probably shouldn't use deep learning and you'd like to go back to a simple rule based
system. And then finally, when you don't have much data, so deep learning models usually require a
fairly large amount of data to produce great results. However, there's a caveat here, you know,
at the start, I said typically, we're going to see some techniques of how to get great results
without huge amounts of data. And again, I wrote typically here because there are techniques,
you can just research deep learning explainability. You're going to find a whole bunch of stuff.
You can look up examples of when machine learning versus deep learning. And then when arrows are
unacceptable, again, there are ways to make your model reproducible. So it predicts you know what's

Section 4: Main Topics

Key Topics:

Ah, we've got machine learning versus deep learning, and we're going to have a look at some different problem spaces in a second, and mainly breaking down in terms of what kind of data you have
So in the last video, we covered a few things of what deep learning is good for and what deep learning is typically not good for
So let's dive in to a little more of a comparison of machine learning versus deep learning

📄 Click to view detailed content

going to come out. So we do a lot of testing to verify this as well. And so what's next? Ah,
we've got machine learning versus deep learning, and we're going to have a look at some different
problem spaces in a second, and mainly breaking down in terms of what kind of data you have.
Not going to do this now prevent this video from getting too long. We'll cover all these
colorful beautiful pictures in the next video. Welcome back. So in the last video, we covered a
few things of what deep learning is good for and what deep learning is typically not good for.
So let's dive in to a little more of a comparison of machine learning versus deep learning. Again,
I'm going to be using these terms quite interchangeably. But there are some specific things that
typically you want traditional style of machine learning techniques versus deep learning. However,
this is constantly changing. So again, I'm not talking in absolutes here. I'm more just talking
in general. And I'll leave it to you to use your own curiosity to research the specific
differences between these two. But typically, for machine learning, like the traditional style of
algorithms, although they are still machine learning algorithms, which is kind of a little
bit confusing where deep learning and machine learning differ is you want to use traditional
machine learning algorithms on structured data. So if you have tables of numbers, this is what I
mean by structured rows and columns, structured data. And possibly one of the best algorithms
for this type of data is a gradient boosted machine, such as xg boost. This is an algorithm
that you'll see in a lot of data science competitions, and also used in production settings. When I
say production settings, I mean, applications that you may interact with on the internet,
or use on a day to day. So that's production. xg boost is typically the favorite algorithm for
these kinds of situations. So again, if you have structured data, you might look into xg boost
rather than building a deep learning algorithm. But again, the rules aren't set in stone. That's
where deep learning and machine learning is kind of an art kind of a science is that sometimes
xg boost is the best for structured data, but there might be exceptions to the rule. But for deep
learning, it is typically better for unstructured data. And what I mean by that is data that's kind
of all over the place. It's not in your nice, standardized rows and columns. So say you had
natural language such as this tweet by this person, whose name is quite similar to mine,
and has the same Twitter account as me. Oh, maybe I wrote that. How do I learn machine learning? What
you need to hear? Learn Python, learn math, start probability, software engineering, build.
What you need to do? Google it, go down the rabbit hole, resurfacing six to nine months,
and ring assess. I like that. Or if you had a whole bunch of texts such as the definition for
deep learning on Wikipedia, again, this is the reason why I'm not covering as many definitions
in this course is because look how simple you can look these things up. Wikipedia is going to
be able to define deep learning far better than what I can. I'm more focused on just getting involved
in working hands on with this stuff than defining what it is. And then we have
images. If we wanted to build a burger, take a photo app thing, you would work with image data,
which doesn't really have much of a structure. Although we'll see that there are ways for deep
learning that we can turn this kind of data to have some sort of structure through the beauty
of a tensor. And then we might have audio files such as if you were talking to your voice assistant.
I'm not going to say one because a whole bunch of my devices might go crazy if I say the name of
my voice assistant, which rhymes with I'm not even going to say that out loud. And so typically,
for unstructured data, you'll want to use a neural network of some kind. So structured data,
gradient boosted machine, or a random forest, or a tree based algorithm, such as extra boost,
and unstructured data, neural networks. So let's keep going. Let's have a look at some of the
common algorithms that you might use for structured data, machine learning versus unstructured data,
deep learning. So random forest is one of my favorites, gradient boosted models,
native base nearest neighbor, support vector machine, SVM, and then many more. But since
the advent of deep learning, these are often referred to as shallow algorithms. So deep learning,
why is it called deep learning? Well, as we'll see is that it can have many different layers
of algorithm, you might have an input layer, 100 layers in the middle, and then an output layer.
But we'll get hands on with this later on. And so common algorithms for deep learning and neural
networks, fully connected neural network, convolutional neural network, recurrent neural network,
transformers have taken over over the past couple years, and of course, many more. And the beautiful
thing about deep learning and neural networks is is almost as many problems that it can be applied
to is as many different ways that you can construct them. So this is why I'm putting all these
dot points on the page. And I can understand if you haven't had much experience of machine
learning or deep learning, this can be a whole bunch of information overload. But good news is
what we're going to be focused on building with PyTorch is neural networks, fully connected neural
networks and convolutional neural networks, the foundation of deep learning. But the excellent

Section 5: Main Topics

Key Topics:

And again, part art, part science of machine learning and deep learning is depending on how you represent your problem, depending on what your problem is, many of the algorithms here and here can be used for both
So I know I've just kind of bedazzled you and saying that, Oh, well, you kind of use these ones for deep learning, you kind of use these ones for machine learning
So that's a little bit of confusion to machine learning

📄 Click to view detailed content

thing is, the exciting thing is, is that if we learn these foundational building blocks,
we can get into these other styles of things here. And again, part art, part science of machine
learning and deep learning is depending on how you represent your problem, depending on what your
problem is, many of the algorithms here and here can be used for both. So I know I've just kind of
bedazzled you and saying that, Oh, well, you kind of use these ones for deep learning, you kind of
use these ones for machine learning. But depending on what your problem is, you can also use both.
So that's a little bit of confusion to machine learning. But that's a fun part about it too,
is use your curiosity to figure out what's best for whatever you're working on. And with all this
talk about neural networks, how about in the next video, we cover what are neural networks. Now,
I'd like you to Google this before we watch the next video, because it's going to be hundreds of
definitions of what they are. And I'd like you to start forming your own definition of what a
neural network is. I'll see you in the next video. Welcome back. In the last video, I left you with
the cliffhanger of a question. What are neural networks? And I gave you the challenge of
Googling that, but you might have already done that by the time you've got here.
Let's just do that together. If I type in what are neural networks, I've already done this.
What are neural networks? Explain neural networks, neural network definition. There are hundreds
of definitions of things like this online neural network in five minutes. Three blue one brown.
I'd highly recommend that channel series on neural networks. That's going to be in the
extracurricular stat quest is also amazing. So there's hundreds of different definitions on here,
and you can read 10 of them, five of them, three of them, make your own definition.
But for the sake of this course, here's how I'm going to find neural networks.
So we have some data of whatever it is. We might have images of food. We might have
tweets or natural language, and we might have speech. So these are some examples of inputs
for unstructured data, because they're not rows and columns. So these are the input data that
we have. And then how do we use them with a neural network? Well, before data can be used in a neural
network, it needs to be turned into numbers, because humans, we like looking at images of Raman and
spaghetti. We know that that's Raman. We know that that's spaghetti after we've seen it one or two
times. And we like reading good tweets, and we like listening to amazing music or hearing our
friend talk on the phone in audio file. However, before a computer understands what's going on
in these inputs, it needs to turn them into numbers. So this is what I call a numerical
encoding or a representation. And this numerical encoding, these square brackets indicate that
it's part of a matrix or a tensor, which we're going to get very hands on with throughout this
course. So we have our inputs, we've turned it into numbers, and then we pass it through a neural
network. And now this is a graphic for a neural network. However, the graphics for neural networks,
as we'll see, can get quite involved. But they all represent the same fundamentals. So if we go to
this one, for example, we have an input layer, then we have multiple hidden layers. However,
you define this, you can design these and how you want. Then we have an output layer. So our
inputs will go in some kind of data. The hidden layers will perform mathematical operations on the
input. So the numbers, and then we'll have an output. Oh, there's three blue one brown neural
networks from the ground up. Great video. Highly recommend you check that out. But then if we come
back to this, so we've got our inputs, we've turned it into numbers. And we've got our neural
networks that we put the input in. This is typically the input layer, hidden layer. This can be as
many different layers as you want, as many different, each of these little dots is called a node.
There's a lot of information here, but we're going to get hands-on with seeing what this looks
like. And then we have some kind of output. Now, which neural network should you use? Well,
you can choose the appropriate neural network for your problem, which could involve you
hand coding each one of these steps. Or you could find one that has worked on problems similar to
your own, such as for images, you might use a CNN, which is a convolutional neural network.
For natural language, you might use a transformer. For speech, you might also use a transformer.
But fundamentally, they all follow the same principle of inputs, manipulation, outputs.
And so the neural network will learn a representation on its own. We want to find what it learns.
So it's going to manipulate these patterns in some way, shape, or form. And when I say
learns representation, I'm going to also refer to it as learns patterns in the data.
A lot of people refer to it as features. A feature may be the fact that the word do comes out to how,
usually, in across a whole bunch of different languages. A feature can be almost anything you
want. And again, we don't define this. The neural network learns these representations,
patterns, features, also called weights on its own. And then where do we go from there? Well,
we've got some sort of numbers, numerical encoding turned our data into numbers. Our neural network

Section 6: Main Topics

Key Topics:

We're going to see it in PyTorch code later on
You can have, I put a s here because you can have one hidden layer, or the deep in deep learning comes from having lots of layers
Now, from the next video, let's dive in briefly to different kinds of learning

📄 Click to view detailed content

has learned a representation that it thinks best represents the patterns in our data.
And then it outputs those representation outputs, which we can use. And often you'll
hear this referred to as features or weight matrix or weight tensor.
Learned representation is also another common one. There's a lot of different terms for these
things. And then it will output. We can convert these outputs into human understandable outputs.
So if we were to look at these, this could be, again, I said representations or patterns that
are neural network learns can be millions of numbers. This is only nine. So imagine if these
were millions of different numbers, I can barely understand the nine numbers that is going on here.
So we need a way to convert these into human understandable terms. So for this example,
we might have some input data, which are images of food. And then we want our neural network to
learn the representations between an image of ramen and an image of spaghetti.
And then eventually we'll take those patterns that it's learned and we'll convert them into
whether it thinks that this is an image of ramen or spaghetti. Or in the case of this tweet,
is this a tweet for a natural disaster or not a natural disaster? So our neural network has,
well, we've written code to turn this into numbers. Pass it through our neural network. Our neural
network has learned some kind of patterns. And then we ideally want it to represent this tweet
as not a disaster. And then we can write code to do each of these steps here. And the same thing
for these inputs going as speech, turning into something that you might say to your smart speaker,
which I'm not going to say because a whole bunch of my devices might go off. And so let's cover
the anatomy of neural networks. We've hinted at this a little bit already. But this is like
neural network anatomy 101. Again, this is highly customizable what this thing actually is. We're
going to see it in PyTorch code later on. But the data goes into the input layer. And in this case,
the number of units slash neurons slash nodes is two hidden layers. You can have, I put a s here
because you can have one hidden layer, or the deep in deep learning comes from having lots of
layers. So this is only showing four layers. You might have, well, this is three layers as well.
It might be very deep neural networks such as ResNet 152. This is 152 different layers.
So again, you can, or this is 34, because this is only ResNet 34. But ResNet 152 has 152 different
layers. So that's a common computer vision or a popular computer vision algorithm, by the way.
Lots of terms we're throwing out here. But with time, you'll start to become familiar with them.
So hidden layers can be almost as many as you want. We've only got pictured one here. And in this
case, there's three hidden units slash neurons. And then we have an output layer. So the outputs
learned representation or prediction probabilities from here, depending on how we set it up, which
again, we will see what these are later on. And in this case, it has one hidden unit. So two input,
three, one output, you can customize the number of these, you can customize how many layers there
are, you can customize what goes into here, you can customize what goes out of there. So now,
if we talk about the overall architecture, which is describing all of the layers combined. So that's,
when you hear neural network architecture, it talks about the input, the hidden layers,
which may be more than one, and the output layer. So that's a terminology for overall architecture.
Now, I say patterns is an arbitrary term. You can hear embedding embedding might come from hidden
layers, weights, feature representation, feature vectors, all referring to similar things. So,
again, how do we turn our data into some numerical form, build a neural network to figure out patterns
to output some desired output that we want. And now to get more technical, each layer is usually a
combination of linear, so straight lines, and nonlinear, non-straight functions. So what I mean by that
is a linear function is a straight line, a nonlinear function is a non-straight line.
If I asked you to draw whatever you want with unlimited straight lines and not straight lines,
so you can use straight lines or curved lines, what kind of patterns could you draw?
At a fundamental level, that is basically what a neural network is doing. It's using a combination
of linear, straight lines, and not straight lines to draw patterns in our data. We'll see what
this looks like later on. Now, from the next video, let's dive in briefly to different kinds of
learning. So we've looked at what a neural network is, the overall algorithm, but there are also
different paradigms of how a neural network learns. I'll see you in the next video.
Welcome back. We've discussed a brief overview of an anatomy of what a neural network is,
but let's now discuss some learning paradigms. So the first one is supervised learning,
and then we have unsupervised and self-supervised learning, and transfer learning. Now supervised
learning is when you have data and labels, such as in the example we gave at the start, which was
how you would build a neural network or a machine learning algorithm to figure out the rules to
cook your Sicilian grandmother's famous roast chicken dish. So in the case of supervised learning,
you'd have a lot of data, so inputs, such as raw ingredients as vegetables and chicken,
and a lot of examples of what that inputs should ideally look like. Or in the case of discerning

Section 7: Main Topics

Key Topics:

photos between a cat and a dog, you might have a thousand photos of a cat and a thousand photos of a dog that you know which photos are cat and which photos are dog, and you pass those photos to a machine learning algorithm to discern
So that's supervised learning, data and labels
Unsupervised and self-supervised learning is you just have the data itself

📄 Click to view detailed content

photos between a cat and a dog, you might have a thousand photos of a cat and a thousand photos
of a dog that you know which photos are cat and which photos are dog, and you pass those photos
to a machine learning algorithm to discern. So in that case, you have data, the photos, and the
labels, aka cat and dog, for each of those photos. So that's supervised learning, data and labels.
Unsupervised and self-supervised learning is you just have the data itself.
You don't have any labels. So in the case of cat and dog photos, you only have the photos.
You don't have the labels of cat and dog. So in the case of self-supervised learning,
you could get a machine learning algorithm to learn an inherent representation of what,
and when I say representation, I mean patterns and numbers, I mean weights, I mean features,
a whole bunch of different names describing the same thing. You could get a self-supervised
learning algorithm to figure out the fundamental patterns between a dog and a cat image, but
it wouldn't necessarily know the difference between the two.
That's where you could come in later and go show me the patterns you've learned,
and it might show you the patterns and you could go, okay, the patterns that look like this,
a dog and the patterns that look like that, a cat. So self-supervised and unsupervised learning
learn solely on the data itself. And then finally, transfer learning is a very, very
important paradigm in deep learning. It's taking the patterns that one model has learned
of a data set and transferring it to another model, such in the case of if we were trying to
build a supervised learning algorithm for discerning between cat and dog photos.
We might start with a model that has already learned patterns and images
and transfer those foundational patterns to our own model so that our model gets a head start.
This is transfer learning is a very, very powerful technique, but as for this course,
we're going to be writing code to focus on these two supervised learning and transfer learning,
which are two of the most common paradigms or common types of learning in machine learning
and deep learning. However, this style of code though can be adapted across different learning
paradigms. Now, I just want to let you know there is one that I haven't mentioned here,
which is kind of in its own bucket, and that is reinforcement learning. So I'll leave this
as an extension if you wanted to look it up. But essentially, this is a good one.
That's a good photo, actually. So shout out to Katie Nuggets. The whole idea of reinforcement
learning is that you have some kind of environment and an agent that does actions in that environment,
and you give rewards and observations back to that agent. So say, for example,
you wanted to teach your dog to urinate outside. Well, you would reward its actions of urinating
outside and possibly not reward its actions of urinating all over your couch. So reinforcement
learning is again, it's kind of in its own paradigm. This picture has a good explanation
between unsupervised learning, supervised learning to separate two different things,
and then reinforcement learning is kind of like that. But again, I will let you research the
different learning paradigms a little bit more in your own time. As I said, we're going to be
focused on writing code to do supervised learning and transfer learning, specifically pytorch code.
Now with that covered, let's get a few examples of what is deep learning actually used for. And
before we get into the next video, I'm going to issue you a challenge to search this question
yourself and come up with some of your own ideas for what deep learning is currently used for.
So give that a shot and I'll see you in the next video. How'd you go? Did you do some research?
Did you find out what deep learning is actually used for? I bet you found a treasure trail of
things. And hey, I mean, if you're reading this course, chances are that you probably already
know some use cases for deep learning. You're like, Daniel, hurry up and get to the code. Well,
we're going to get there, don't you worry? But let's have a look at some things that deep
learning can be used for. But before, I just want to remind you of this comment. This is from
Yasha Sway on the 2020 machine learning roadmap video. I think you can use ML and remember,
ML is machine learning. And remember, deep learning is a part of ML for literally anything as long
as you can convert it into numbers and program it to find patterns. Literally, it could be anything,
any input or output from the universe. So that's a beautiful thing about machine learning is that
if you can encode it something into numbers, chances are you can build a machine learning
algorithm to find patterns in those numbers. Will it work? Well, again, that's the reason machine
learning and deep learning is part art, part science. A scientist would love to know that their
experiments would work. But an artist is kind of excited about the fact that, I don't know,
this might work, it might not. And so that's something to keep in mind. Along with the rule
number one of machine learning is if you don't need it, you don't use it. But if you do use it,
it can be used for almost anything. And let's get a little bit specific and find out some deep
learning use cases. And I've put some up there for a reason because there are lots. These are just
some that I interact with in my day to day life, such as recommendation, we've got a programming

Section 8: Main Topics

Key Topics:

That's all powered by deep learning
Well, that's powered by deep learning as well
But if I wanted to translate deep learning as epic to Spanish, it might come out as el aprendise, profando es ebiko

📄 Click to view detailed content

video, we've got a programming podcast, we got some jujitsu videos, we've got some RuneScape
videos, a soundtrack from my favorite movie. Have you noticed, whenever you go to YouTube,
you don't really search for things anymore. Well, sometimes you might, but the recommendation
page is pretty darn good. That's all powered by deep learning. And in the last 10 years,
have you noticed that translation has got pretty good too? Well, that's powered by deep learning
as well. Now, I don't have much hands on experience with this. I did use it when I was in Japan.
I speak a very little amount of Japanese and even smaller amount of Mandarin. But if I wanted to
translate deep learning as epic to Spanish, it might come out as el aprendise, profando es ebiko.
Now, all of the native Spanish speakers watching this video can laugh at me because that was a very
Australian version of saying deep learning is epic in Spanish. But that's so cool. All the Google
Translate is now powered by deep learning. And the beautiful thing, if I couldn't say it myself,
I could click this speaker and it would say it for me. So that speech recognition that's powered
by deep learning. So if you were to ask your voice assistant who's the biggest big dog of them all,
of course, they're going to say you, which is what I've set up, my voice assistant to say.
That's part of speech recognition. And in computer vision, oh, look at this. You see this? Where is
this photo from? This photo is from this person driving this car. Did a hit and run on my car,
at the front of my house, my apartment building, my car was parked on the street, this car, the
trailer came off, ran into the back of my car, basically destroyed it, and then they drove off.
However, my next door neighbors security camera picked up on this car. Now, I became a detective
for a week, and I thought, hmm, if there was a computer vision algorithm built into that camera,
it could have detected when the car hit. I mean, it took a lot of searching to find it,
it turns out the car hit about 3.30am in the morning. So it's pitch black. And of course,
we didn't get the license plate. So this person is out there somewhere in the world after doing
a hit and run. So if you're watching this video, just remember computer vision might catch you one
day. So this is called object detection, where you would place a box around the area where the
pixels most represent the object that you're looking for. So for computer vision, we could
train an object detector to capture cars that drive past a certain camera. And then if someone
does a hit and run on you, you could capture it. And then fingers crossed, it's not too dark
that you can read the license plate and go, hey, excuse me, please, this person has hit my car
and wrecked it. So that's a very close to home story of where computer vision could be used.
And then finally, natural language processing. Have you noticed as well, your spam detector on
your email inbox is pretty darn good? Well, some are powered by deep learning, some not,
it's hard to tell these days what is powered by deep learning, what isn't. But natural language
processing is the process of looking at natural language text. So unstructured text. So whatever
you'd write an email in a story in a Wikipedia document and deciding or getting your algorithm
to find patterns in that. So for this example, you would find that this email is not spam.
This deep learning course is incredible. I can't wait to use what I've learned. Thank you so much.
And by the way, that is my real email. So if you want to email me, you can. And then this is spam.
Hey, Daniel, congratulations, you win a lot of money. Wow, I really like that a lot of money.
But somebody said, I don't think that this is real. So that would probably go to my spam inbox.
Now, with that being said, if we wanted to put these problems in a little bit more of a
classification, this is known as sequence to sequence because you put one sequence in
and get one sequence out. Same as this, you have a sequence of audio waves and you get some
text out. So sequence to sequence, sec to sec. This is classification slash regression. In this
case, the regression is predicting a number. That's what a regression problem is. You would predict
the coordinates of where these box corners should be. So say this should be at however many pixels
in from the X angle and however many pixels down from the Y angle, that's that corner.
And then you would draw in between the corners. And then the classification part would go,
Hey, this is that car that did a hit and run on us. And in this case, this is classification.
Classification is predicting whether something is one thing or another, or perhaps more than one
thing or another in the class of multi class classification. So this email is not spam. That's
a class and this email is spam. So that's also a class. So I think we've only got one direction
to go now that we've sort of laid the foundation for the course. And that is
Well, let's start talking about PyTorch. I'll see you in the next video.
Well, let's now cover some of the foundations of
PyTorch. But first, you might be asking, what is PyTorch? Well, of course, we could just go to
our friend, the internet, and look up PyTorch.org. This is the homepage for PyTorch.

Section 9: Main Topics

Key Topics:

This course is not a replacement for everything on this homepage
This should be your ground truth for everything PyTorch
PyTorch

📄 Click to view detailed content

This course is not a replacement for everything on this homepage. This should be your ground truth
for everything PyTorch. So you can get started. You've got a big ecosystem. You've got a way to
set up on your local computer. You've got resources. You've got docs. PyTorch. You've got the GitHub.
You've got search. You've got blog, everything here. This website should be the place you're
visiting most throughout this course as we're writing PyTorch code. You're coming here.
You're reading about it. You're checking things out. You're looking at examples.
But for the sake of this course, let's break PyTorch down. Oh, there's a little flame animation
I just forgot about. What is PyTorch? I didn't sync up the animations. That's all right. So
PyTorch is the most popular research deep learning framework. I'll get to that in a second.
It allows you to write fast deep learning code in Python. If you know Python, it's a very user-friendly
programming language. PyTorch allows us to write state-of-the-art deep learning code
accelerated by GPUs with Python. It enables you access to many pre-built deep learning models
from Torch Hub, which is a website that has lots of, if you remember, I said transfer learning is
a way that we can use other deep learning models to power our own. Torch Hub is a resource for that.
Same as Torch Vision.Models. We'll be looking at this throughout the course.
It provides an ecosystem for the whole stack of machine learning. From pre-processing data,
getting your data into tenses, what if you started with some images? How do you represent them as
numbers? Then you can build models such as neural networks to model that data. Then you can even
deploy your model in your application slash cloud, well, deploy your PyTorch model. Application slash
cloud will be depending on what sort of application slash cloud that you're using, but generally it
will run some kind of PyTorch model. And it was originally designed and used in-house by Facebook
slash meta. I'm pretty sure Facebook have renamed themselves meta now, but it is now open source
and used by companies such as Tesla, Microsoft and OpenAI. And when I say it is the most popular
deep learning research framework, don't take my word for it. Let's have a look at papers with code
dot com slash trends. If you're not sure what papers with code is, it is a website that tracks
the latest and greatest machine learning papers and whether or not they have code. So we have some
other languages here, other deep learning frameworks, PyTorch, TensorFlow, Jax is another one, MXNet,
paddle paddle, the original torch. So PyTorch is an evolution of torch written in Python,
CAF2, Mindspore. But if we look at this, when is this? Last date is December 2021. We have,
oh, this is going to move every time I move it. No. So I'll highlight PyTorch at 58% there.
So by far and large, the most popular research machine learning framework used to write the code
for state of the art machine learning algorithms. So this is browse state of the art papers with
code.com amazing website. We have semantic segmentation, image classification, object detection, image
generation, computer vision, natural language processing, medical, I'll let you explore this.
It's one of my favorite resources for staying up to date on the field. But as you see, out of the
65,000 papers with code that this website is tracked, 58% of them are implemented with PyTorch.
How cool is that? And this is what we're learning. So let's jump into there. Why PyTorch? Well,
other than the reasons that we just spoke about, it's a research favorite. This is highlighting.
There we go. So there we go. I've highlighted it here. PyTorch, 58%, nearly 2,500 repos. If
you're not sure what a repo is, a repo is a place where you store all of your code online.
And generally, if a paper gets published in machine learning, if it's fantastic research,
it will come with code, code that you can access and use for your own applications or your own
research. Again, why PyTorch? Well, this is a tweet from Francois Chale, who's the author of
Keras, which is another popular deep learning framework. But with tools like Colab, we're going
to see what Colab is in a second, Keras and TensorFlow. I've added in here and PyTorch.
Virtually anyone can solve in a day with no initial investment problems that would have
required an engineering team working for a quarter and $20,000 in hardware in 2014. So this is just
to highlight how good the space of deep learning and machine learning tooling has become. Colab,
Keras and TensorFlow are all fantastic. And now PyTorch is added to this list. If you want to
check that out, there's Francois Chale on Twitter. Very, very prominent voice in the machine learning
field. Why PyTorch? If you want some more reasons, well, have a look at this. Look at all the
places that are using PyTorch. It's just coming up everywhere. We've got Andre Kapathi here,
who's the director of AI at Tesla. So if we go, we could search this, PyTorch
at Tesla. We've got a YouTube talk there, Andre Kapathi, director of AI at Tesla.
And so Tesla are using PyTorch for the computer vision models of autopilot. So if we go to videos
or maybe images, does it come up there? Things like this, a car detecting what's going on in the scene.
Of course, there'll be some other code for planning, but I'll let you research that.
When we come back here, OpenAI, which is one of the biggest open artificial intelligence
research firms, open in the sense that they publish a lot of their research methodologies,
however, recently there's been some debate about that. But if you go to openai.com,

Section 10: Main Topics

Key Topics:

let's just say that they're one of the biggest AI research entities in the world, and they've standardized on PyTorch
Presumably with PyTorch, because this blog post from January 2020 says that OpenAI is now standardized across PyTorch
There's a repo called the incredible PyTorch, which collects a whole bunch of different projects that are built on top of PyTorch

📄 Click to view detailed content

let's just say that they're one of the biggest AI research entities in the world,
and they've standardized on PyTorch. So they've got a great blog, they've got great research,
and now they've got OpenAI API, which is, you can use their API to access some of the models
that they've trained. Presumably with PyTorch, because this blog post from January 2020 says
that OpenAI is now standardized across PyTorch. There's a repo called the incredible PyTorch,
which collects a whole bunch of different projects that are built on top of PyTorch.
That's the beauty of PyTorch is that you can build on top of it, you can build with it
AI for AG, for agriculture. PyTorch has been used. Let's have a look. PyTorch in agriculture.
There we go. Agricultural robots use PyTorch. This is a medium article.
It's everywhere. So if we go down here, this is using object detection. Beautiful.
Object detection to detect what kind of weeds should be sprayed with fertilizer. This is just
one of many different things, so PyTorch on a big tractor like this. It can be used almost
anywhere. If we come back, PyTorch builds the future of AI and machine learning at Facebook,
so Facebook, which is also MetaAI, a little bit confusing, even though it says MetaAI,
it's on AI.facebook.com. That may change by the time you watch this. They use PyTorch in-house
for all of their machine learning applications. Microsoft is huge in the PyTorch game.
It's absolutely everywhere. So if that's not enough reason to use PyTorch,
well, then maybe you're in the wrong course. So you've seen enough reasons of why to use PyTorch.
I'm going to give you one more. That is that it helps you run your code, your machine learning code
accelerated on a GPU. We've covered this briefly, but what is a GPU slash a TPU,
because this is more of a newer chip these days. A GPU is a graphics processing unit,
which is essentially very fast at crunching numbers. Originally designed for video games,
if you've ever designed or played a video game, you know that the graphics are quite intense,
especially these days. And so to render those graphics, you need to do a lot of numerical calculations.
And so the beautiful thing about PyTorch is that it enables you to leverage a GPU through an
interface called CUDA, which is a lot of words I'm going to throw out you here, a lot of acronyms
in the deep learning space, CUDA. Let's just search CUDA. CUDA toolkit. So CUDA is a parallel
computing platform and application programming interface, which is an API that allows software
to use certain types of graphics processing units for general purpose computing. That's what
we want. So PyTorch leverages CUDA to enable you to run your machine learning code on NVIDIA
GPUs. Now, there is also an ability to run your PyTorch code on TPUs, which is a tensor processing
unit. However, GPUs are far more popular when running various types of PyTorch code. So we're
going to focus on running our PyTorch code on the GPU. And to just give you a quick example,
PyTorch on TPU, let's see that. Getting started with PyTorch on cloud TPUs, there's plenty of
guys for that. But as I said, GPUs are going to be far more common in practice. So that's what
we're going to focus on. And with that said, we've said tensor processing unit. Now, the reason
why these are called tensor processing units is because machine learning and deep learning
deals a lot with tensors. And so in the next video, let's answer the question, what is a tensor?
But before I go through and answer that from my perspective, I'd like you to research this
question. So open up Google or your favorite search engine and type in what is a tensor and
see what you find. I'll see you in the next video. Welcome back. In the last video, I left you on
the cliffhanger question of what is a tensor? And I also issued you the challenge to research
what is a tensor. Because as I said, this course isn't all about telling you exactly what things
are. It's more so sparking a curiosity in you so that you can stumble upon the answers to these
things yourself. But let's have a look. What is a tensor? Now, if you remember this graphic,
there's a lot going on here. But this is our neural network. We have some kind of input,
some kind of numerical encoding. Now, we start with this data. In our case, it's unstructured data
because we have some images here, some text here, and some audio file here. Now, these necessarily
don't go in all at the same time. This image could just focus on a neural network specifically
for images. This text could focus on a neural network specifically for text. And this sound bite
or speech could focus on a neural network specifically for speech. However, the field is sort of also
moving towards building neural networks that are capable of handling all three types of inputs.
For now, we're going to start small and then build up the algorithms that we're going to focus on
are neural networks that focus on one type of data. But the premise is still the same. You have
some kind of input. You have to numerically encode it in some form, pass it to a neural network
to learn representations or patterns within that numerical encoding, output some form of
representation. And then we can convert that representation into things that humans understand.
And you might have already seen these, and I might have already referenced the fact that
these are tensors. So when the question comes up, what are tensors? A tensor could be almost
anything. It could be almost any representation of numbers. We're going to get very hands on with

Section 11: Main Topics

Key Topics:

And that's actually the fundamental building block of PyTorch aside from neural network components is the torch dot tensor
Now, a lot of those mathematical operations are taken care of by PyTorch behind the scenes
What is machine learning

📄 Click to view detailed content

tensors. And that's actually the fundamental building block of PyTorch aside from neural network
components is the torch dot tensor. We're going to see that very shortly. But this is a very
important takeaway is that you have some sort of input data. You're going to numerically encode
that data, turn it into a tensor of some kind. Whatever that kind is will depend on the problem
you're working with. Then you're going to pass it to a neural network, which will perform mathematical
operations on that tensor. Now, a lot of those mathematical operations are taken care of by
PyTorch behind the scenes. So we'll be writing code to execute some kind of mathematical
operations on these tensors. And then the neural network that we create, or the one that's already
been created, but we just use for our problem, we'll output another tensor, similar to the input,
but has been manipulated in a certain way that we've sort of programmed it to. And then we can take
this output tensor and change it into something that a human can understand. So to remove a lot
of the text around it, make it a little bit more clearer. If we were focusing on building an image
classification model, so we want to classify whether this was a photo of Raman or spaghetti,
we would have images as input. We would turn those images into numbers, which are represented
by a tensor. We would pass that tensor of numbers to a neural network, or there might be lots of
tensors here. We might have 10,000 images. We might have a million images. Or in some cases,
if you're Google or Facebook, you might be working with 300 million or a billion images at a time.
The principle still stands that you encode your data in some form of numerical representation,
which is a tensor, pass that tensor, or lots of tensors to a neural network. The neural network
performs mathematical operations on those tensors, outputs a tensor, we convert that tensor into
something that we can understand as humans. And so with that being said, we've covered a lot of
the fundamentals. What is machine learning? What is deep learning? What is neural network? Well,
we've touched the surface of these things. You can get as deep as you like. We've covered
why use PyTorch. What is PyTorch? Now, the fundamental building block of deep learning
is tensors. We've covered that. Let's get a bit more specific in the next video
of what we're going to cover code-wise in this first module. I'm so excited we're going to start
codes in. I'll see you in the next video. Now it's time to get specific about what we're going to
cover code-wise in this fundamentals module. But I just want to reiterate the fact that
going back to the last video where I challenge you to look up what is a tensor, here's exactly
what I would do. I would come to Google. I would type in the question, what is a tensor? There we go.
What is a tensor in PyTorch? It knows Google knows that using that deep learning data that we want
to know what a tensor is in PyTorch. But a tensor is a very general thing. It's not
associated with just PyTorch. Now we've got tensor on Wikipedia. We've got tensor. This is probably
my favorite video on what is a tensor. By Dan Flesch. Flesch, I'm probably saying that wrong,
but good first name. This is going to be your extra curriculum for this video and the previous
video is to watch this on what is a tensor. Now you might be saying, well, what gives? I've come to
this course to learn PyTorch and all this guy's doing, all you're doing, Daniel, is just Googling
things when a question comes up. Why don't you just tell me what it is? Well, if I was to tell you
everything about deep learning and machine learning and PyTorch and what it is and what it's not,
that course would be far too long. I'm doing this on purpose. I'm searching questions like this on
purpose because that's exactly what I do day to day as a machine learning engineer. I write code
like we're about to do. And then if I don't know something, I literally go to whatever search engine
I'm using, Google most of the time, and type in whatever error I'm getting or PyTorch, what is
a tensor, something like that. So I want to not only tell you that it's okay to search questions
like that, but it's encouraged. So just keep that in mind as we go through the whole course,
you're going to see me do it a lot. Let's get into what we're going to cover. Here we go.
Now, this tweet is from Elon Musk. And so I've decided, you know what, let's base the whole
course on this tweet. We have learning MLDL from university, you have a little bit of a small brain.
Online courses, well, like this one, that brain's starting to explode and you get some little fireworks
from YouTube. Oh, you're watching this on YouTube. Look at that shiny brain from articles. My goodness.
Lucky that this course comes in article format. If you go to learn pytorch.io, all of the course
materials are in online book format. So we're going to get into this fundamental section very
shortly. But if you want a reference, the course materials are built off this book. And by the
time you watch this, there's going to be more chapters here. So we're covering all the bases
here. And then finally, from memes, you would ascend to some godlike creature. I think that's
hovering underwater. So that is the best way to learn machine learning. So this is what we're
going to start with MLDL from university online courses, YouTube from articles from memes. No,

Section 12: Main Topics

Key Topics:

So now in this module, we are going to cover the pytorch basics and fundamentals, mainly dealing with tensors and tensor operations
Then we're going to look at building and using pre-trained deep learning models, specifically neural networks
We're going to see how we can make predictions with our model, because that's what deep learning and machine learning is all about, right, using patterns from the past to predict the future

📄 Click to view detailed content

no, no, no. But kind of here is what we're going to cover broadly. So now in this module,
we are going to cover the pytorch basics and fundamentals, mainly dealing with tensors and
tensor operations. Remember, a neural network is all about input tensors, performing operations on
those tensors, creating output operations. Later, we're going to be focused on pre-processing data,
getting it into tensors, so turning data from raw form, images, whatever, into a numerical
encoding, which is a tensor. Then we're going to look at building and using pre-trained deep
learning models, specifically neural networks. We're going to fit a model to the data. So we're
going to show our model or write code for our model to learn patterns in the data that we've
pre-processed. We're going to see how we can make predictions with our model, because that's
what deep learning and machine learning is all about, right, using patterns from the past to
predict the future. And then we're going to evaluate our model's predictions. We're going to learn
how to save and load our models. For example, if you wanted to export your model from where we're
working to an application or something like that. And then finally, we're going to see how we can
use a trained model to make predictions on our own data on custom data, which is very fun. And
how? Well, you can see that the scientist has faded out a little bit, but that's not really that true.
We're going to do it like cooks, not chemists. So chemists are quite precise. Everything has to be
exactly how it is. But cooks are more like, oh, you know what, a little bit of salt, a little bit of
butter. Does it taste good? Okay, well, then we're on. But machine learning is a little bit of both.
It's a little bit of science, a little bit of art. That's how we're going to do it. But
I like the idea of this being a machine learning cooking show. So welcome to cooking with machine
learning, cooking with PyTorch with Daniel. And finally, we've got a workflow here, which we have
a PyTorch workflow, which is one of many. We're going to kind of use this throughout the entire
course is step one, we're going to get our data ready. Step two, we're going to build a
pick a pre trained model to suit whatever problem we're working on. Step two point one,
pick a loss function and optimizer. Don't worry about what they are. We're going to cover them
soon. Step two point two, build a training loop. Now this is kind of all part of the parcel of
step two, hence why we've got two point one and two point two. You'll see what that means later on.
Number three, we're going to fit the model to the data and make a prediction. So say we're working
on image classification for Raman or spaghetti. How do we build a neural network or put our
images through that neural network to get some sort of idea of what's in an image? We'll see
how to do that. Well, the value weight our model to see if it's predicting BS or it's actually
going all right. Number five, we're going to improve through experimentation. That's another
big thing that you'll notice throughout machine learning throughout this course is that it's
very experimental part art, part science. Number six, save and reload your trained model. Again,
I put these with numerical order, but they can kind of be mixed and matched depending on where
you are in the journey. But numerical order is just easy to understand for now. Now we've got
one more video, maybe another one before we get into code. But in the next video, I'm going to
cover some very, very important points on how you should approach this course. I'll see you there.
Now you might be asking, how should I approach this course? You might not be asking, but we're
going to answer it anyway. How to approach this course? This is how I would recommend approaching
this course. So I'm a machine learning engineer day to day and learning machine learning to
coding machine learning, a kind of two different things. I remember when I first learned it was
kind of, you learned a lot of theory rather than writing code. So not to take away from the theory
of being important, this course is going to be focusing on writing machine learning specifically
PyTorch code. So the number one step to approaching this course is to code along. Now because this
course is focused on purely writing code, I will be linking extracurricular resources for you to
learn more about what's going on behind the scenes of the code. My idea of teaching is that if we
can code together, write some code, see how it's working, that's going to spark your curiosity to
figure out what's going on behind the scenes. So motto number one is if and out, run the code,
write it, run the code, see what happens. Number two, I love that. Explore an experiment again.
Approach this with the idea, the mind of a scientist and a chef or science and art. Experiment,
experiment, experiment. Try things with rigor like a scientist would, and then just try things
for the fun of it like a chef would. Number three, visualize what you don't understand. I can't
emphasize this one enough. We have three models so far. If and out, run the code, you're going to
hear me say this a lot. Experiment, experiment, experiment. And number three, visualize, visualize,
visualize. Why is this? Well, because we've spoken about machine learning and deep learning
deals with a lot of data, a lot of numbers. And so I find it that if I visualize some numbers in

Section 13: Main Topics

Key Topics:

Everyone is just on a different part of their learning journey
If we go, have we got the book version of the course up here
I'm not going to jump into them, but I would highly recommend don't just follow along with the course and code after I code

📄 Click to view detailed content

whatever form that isn't just numbers all over a page, I tend to understand it better.
And there are some great extracurricular resources that I'm going to link that also turn what we're
doing. So writing code into fantastic visualizations. Number four, ask questions, including the dumb
questions. Really, there's no such thing as a dumb question. Everyone is just on a different
part of their learning journey. And in fact, if you do have a quote unquote dumb question,
it turns out that a lot of people probably have that one as well. So be sure to ask questions.
I'm going to link a resource in a minute of where you can ask those questions, but
please, please, please ask questions, not only to the community, but to Google to the internet
to wherever you can, or just yourself. Ask questions of the code and write code to figure
out the answer to those questions. Number five, do the exercises. There are some
great exercises that I've created for each of the modules. If we go, have we got the book version
of the course up here? We do. Within all of these chapters here, down the bottom is going to be
exercises and extra curriculum. So we've got some exercises. I'm not going to jump into them,
but I would highly recommend don't just follow along with the course and code after I code.
Please, please, please give the exercises a go because that's going to stretch your knowledge.
We're going to have a lot of practice writing code together, doing all of this stuff here.
But then the exercises are going to give you a chance to practice what you've learned.
And then of course, extra curriculum. Well, hey, if you want to learn more, there's plenty of
opportunities to do so there. And then finally, number six, share your work. I can't emphasize
enough how much writing about learning deep learning or sharing my work through GitHub or
different code resources or with the community has helped with my learning. So if you learn
something cool about PyTorch, I'd love to see it. Link it to me somehow in the Discord chat
or on GitHub or whatever. There'll be links of where you can find me. I'd love to see it. Please
do share your work. It's a great way to not only learn something because when you share it, when
you write about it, it's like, how would someone else understand it? But it's also a great way to
help others learn too. And so we said how to approach this course. Now, let's go how not to
approach this course. I would love for you to avoid overthinking the process. And this is your brain,
and this is your brain on fire. So avoid having your brain on fire. That's not a good place to be.
We are working with PyTorch, so it's going to be quite hot. Just playing on words with the name
torch. But avoid your brain catching on fire. And avoid saying, I can't learn,
I've said this to myself lots of times, and then I've practiced it and it turns out I can
actually learn those things. So let's just draw a red line on there. Oh, I think a red line.
Yeah, there we go. Nice and thick red line. We'll get that out there. It doesn't really make sense
now that this says avoid and crossed out. But don't say I can't learn and prevent your brain from
catching on fire. Finally, we've got one more video that I'm going to cover before this one
gets too long of the resources for the course before we get into coding. I'll see you there.
Now, there are some fundamental resources that I would like you to be aware of before we
go any further in this course. These are going to be paramount to what we're working with.
So for this course, there are three things. There is the GitHub repo. So if we click this link,
I've got a pinned on my browser. So you might want to do the same while you're going through
the course. But this is Mr. D. Burks in my GitHub slash PyTorch deep learning. It is still a work
in progress at the time of recording this video. But by the time you go through it, it won't look
too much different, but there just be more materials. You'll have materials outline,
section, what does it cover? As you can see, some more are coming soon at the time of recording
this. So these will probably be done by the time you watch this exercise in extra curriculum.
There'll be links here. Basically, everything you need for the course will be in the GitHub repo.
And then if we come back, also on the GitHub repo, the same repo. So Mr. D. Burks slash PyTorch
deep learning. If you click on discussions, this is going to be the Q and A. This is just the same
link here, the Q and A for the course. So if you have a question here, you can click new discussion,
you can go Q and A, and then type in video, and then the title PyTorch Fundamentals, and then go
in here. Or you could type in your error as well. What is N-DIM for a tensor? And then in here,
you can type in some stuff here. Hello. I'm having trouble on video X, Y, Z. Put in the name of the
video. So that way I can, or someone else can help you out. And then code, you can go three
back ticks, write Python, and then you can go import torch, torch dot rand n, which is going to
create a tensor. We're going to see this in a second. Yeah, yeah, yeah. And then if you post that

Section 14: Main Topics

Key Topics:

You could even include the error message, and then you can just click start discussion, and then someone, either myself or someone else from the course will be able to help out there
There's nothing here yet because the course isn't out yet, but as you go through it, there will probably be more and more stuff here
I've just got some issues here already about the fact that I need to record videos for the course

📄 Click to view detailed content

question, the formatting of the code is very helpful that we can understand what's going on,
and what's going on here. So this is basically the outline of how I would ask a question video.
This is going on. What is such and such for whatever's going on? Hello. This is what I'm having
trouble with. Here's the code, and here's what's happening. You could even include the error message,
and then you can just click start discussion, and then someone, either myself or someone else from
the course will be able to help out there. And the beautiful thing about this is that it's all in
one place. You can start to search it. There's nothing here yet because the course isn't out yet,
but as you go through it, there will probably be more and more stuff here. Then if you have any
issues with the code that you think needs fixed, you can also open a new issue there. I'll let you
read more into what's going on. I've just got some issues here already about the fact that I
need to record videos for the course. I need to create some stuff. But if you think there's
something that could be improved, make an issue. If you have a question about the course,
ask a discussion. And then if we come back to the keynote, we have one more resource. So that
was the course materials all live in the GitHub. The course Q&A is on the course
GitHub's discussions tab, and then the course online book. Now, this is a work of art.
This is quite beautiful. It is some code to automatically turn all of the materials from the
GitHub. So if we come into here code, if we click on notebook zero zero, this is going to sometimes
if you've ever worked with Jupiter notebooks on GitHub, they can take a while to load.
So all of the materials here automatically get converted into this book. So the beautiful
thing about the book is that it's got different headings here. It's all readable. It's all online.
It's going to have all the images there. And you can also search some stuff here,
PyTorch training steps, creating a training loop in PyTorch. Beautiful. We're going to see
this later on. So they're the three big materials that you need to be aware of, the three big resources
for this specific course materials on GitHub course Q&A course online book, which is
learn pytorch.io, simple URL to remember, all the materials will be there. And then
specifically for PyTorch or things PyTorch, the PyTorch website and the PyTorch forums.
So if you have a question that's not course related, but more PyTorch related, I'd highly
recommend you go to the PyTorch forums, which is available at discuss.pytorch.org. We've got a link
there. Then the PyTorch website, PyTorch.org, this is going to be your home ground for everything
PyTorch of course. We have the documentation here. And as I said, this course is not a replacement
for getting familiar with the PyTorch documentation. This, the course actually is built off all of
the PyTorch documentation. It's just organized in a slightly different way. So there's plenty of
amazing resources here on everything to do with PyTorch. This is your home ground. And you're
going to see me referring to this a lot throughout the course. So just keep these in mind, course
materials on GitHub, course discussions, learnpytorch.io. This is all for the course. And all things
PyTorch specific, so not necessarily this course, but just PyTorch in general, the PyTorch website
and the PyTorch forums. With that all being said, we've come so far. We've covered a lot already,
but guess what time it is? Let's write some code. I'll see you in the next video.
We've covered enough of the fundamentals so far. Well, from a theory point of view,
let's get into coding. So I'm going to go over to Google Chrome. I'm going to introduce you to
the tool. One of the main tools we're going to be using for the entire course. And that is Google
Colab. So the way I would suggest following along with this course is remember, one of the major
ones is to code along. So we're going to go to colab.research.google. I've got a typo here.
Classic. You're going to see me do lots of typos throughout this course. Colab.research.google.com.
This is going to load up Google Colab. Now, you can follow along with what I'm going to do,
but if you'd like to find out how to use Google Colab from a top-down perspective,
you can go through some of these. I'd probably recommend going through overview of
Collaboratory Features. But essentially, what Google Colab is going to enable us to do is
create a new notebook. And this is how we're going to practice writing PyTorch code.
So if you refer to the reference document of learnpytorch.io, these are actually
Colab notebooks just in book format, so online book format. So these are the basis materials
for what the course is going to be. There's going to be more here, but every new module,
we're going to start a new notebook. And I'm going to just zoom in here.
So this one, the first module is going to be zero, zero, because Python code starts at zero,
zero. And we're going to call this PyTorch Fundamentals. I'm going to call mine video,
just so we know that this is the notebook that I wrote through the video. And what this is going
to do is if we click Connect, it's going to give us a space to write Python code. So here we can go
print. Hello, I'm excited to learn PyTorch. And then if we hit shift and enter, it comes out like
that. But another beautiful benefit of Google Colab are PS. I'm using the pro version, which

Section 15: Main Topics

Key Topics:

However, you do not have to use the paid version for this course
Google Colab comes with a free version, which you'll be able to use to complete this course
Now we're going to see this later on code that runs on the GPU is a lot faster in terms of compute time, especially for deep learning

📄 Click to view detailed content

costs about $10 a month or so. That price may be different depending on where you're from.
The reason I'm doing that is because I use Colab all the time. However, you do not have to use
the paid version for this course. Google Colab comes with a free version, which you'll be able
to use to complete this course. If you see it worthwhile, I find the pro version is worthwhile.
Another benefit of Google Colab is if we go here, we can go to runtime. Let me just show you that
again. Runtime, change runtime type, hardware accelerator. And we can choose to run our code
on an accelerator here. Now we've got GPU and TPU. We're going to be focused on using
GPU. If you'd like to look into TPU, I'll leave that to you. But we can click GPU, click save.
And now our code, if we write it in such a way, will run on the GPU. Now we're going to see this
later on code that runs on the GPU is a lot faster in terms of compute time, especially for deep
learning. So if we write here in a video SMI, we now have access to a GPU. In my case, I have a
Tesla P100. It's quite a good GPU. You tend to get the better GPUs. If you pay for Google Colab,
if you don't pay for it, you get the free version, you get a free GPU. It just won't be as fast as
the GPUs you typically get with the paid version. So just keep that in mind. A whole bunch of stuff
that we can do here. I'm not going to go through it all because there's too much. But we've covered
basically what we need to cover. So if we just come up here, I'm going to write a text cell. So
oo dot pytorch fundamentals. And I'm going to link in here resource notebook. Now you can come
to learn pytorch.io and all the notebooks are going to be in sync. So 00, we can put this in here.
Resource notebook is there. That's what this notebook is going to be based off. This one here.
And then if you have a question about what's going on in this notebook,
you can come to the course GitHub. And then we go back, back. This is where you can see what's
going on. This is pytorch deep learning projects as you can see what's happening. At the moment,
I've got pytorch course creation because I'm in the middle of creating it. But if you have a question,
you can come to Mr. D Burke slash pytorch deep learning slash discussions, which is this tab here,
and then ask a question by clicking new discussion. So any discussions related to this notebook,
you can ask it there. And I'm going to turn this right now. This is a code cell.
CoLab is basically comprised of code and text cells. I'm going to turn this into a text cell
by pressing command mm, shift and enter. Now we have a text cell. And then if we wanted another
code cell, we could go like that text code text code, yada, yada, yada. But I'm going to delete this.
And to finish off this video, we're going to import pytorch. So we're going to import torch.
And then we're going to print torch dot dot version. So that's another beautiful thing about Google
Colab is that it comes with pytorch pre installed and a lot of other common Python data science
packages, such as we could also go import pandas as PD, import NumPy as MP import mapplot lib
lib dot pyplot as PLT. This is Google Colab is by far the easiest way to get started with this
course. You can run things locally. If you'd like to do that, I'd refer to you to pytorch deep
learning is going to be set up dot MD, getting set up to code pytorch. We've just gone through
number one setting up with Google Colab. There is also another option for getting started locally.
Right now, this document's a work in progress, but it'll be finished by the time you watch this
video. This is not a replacement, though, for the pytorch documentation for getting set up
locally. So if you'd like to run locally on your machine, rather than going on Google Colab,
please refer to this documentation or set up dot MD here. But if you'd like to get started
as soon as possible, I'd highly recommend you using Google Colab. In fact, the entire course
is going to be able to be run through Google Colab. So let's finish off this video, make sure
we've got pytorch ready to go. And of course, some fundamental data science packages here.
Wonderful. This means that we have pytorch 1.10.0. So if your version number is far greater than this,
maybe you're watching this video a couple of years in the future, and pytorch is up to 2.11,
maybe some of the code in this notebook won't work. But 1.10.0 should be more than enough for
what we're going to do. And plus Q111, CU111, stands for CUDA version 11.1, I believe. And what
that would mean is if we came in here, and we wanted to install it on Linux, which is what
Colab runs on, there's Mac and Windows as well. We've got CUDA. Yeah. So right now, as of recording
this video, the latest pytorch build is 1.10.2. So you'll need at least pytorch 1.10 to complete
this course and CUDA 11.3. So that's CUDA toolkit. If you remember, CUDA toolkit is NVIDIA's
programming. There we go. NVIDIA developer. CUDA is what enables us to run our pytorch code on
NVIDIA GPUs, which we have access to in Google Colab. Beautiful. So we're set up ready to write code.

Section 16: Main Topics

Key Topics:

Let's get started in the next video writing some pytorch code
We've got access to pytorch
And one last thing, how I'd recommend going through this course is in a split window fashion

📄 Click to view detailed content

Let's get started in the next video writing some pytorch code. This is so exciting. I'll see you
there. So we've got set up. We've got access to pytorch. We've got a Google Colab instance running
here. We've got a GPU because we've gone up to runtime, change runtime type, hardware accelerator.
You won't necessarily need a GPU for this entire notebook, but I just wanted to show you how to
get access to a GPU because we're going to be using them later on. So let's get rid of this.
And one last thing, how I'd recommend going through this course is in a split window fashion.
So for example, you might have the video where I'm talking right now and writing code on the
left side, and then you might have another window over the other side with your own Colab
window. And you can go new notebook, call it whatever you want, my notebook. You could call it very
similar to what we're writing here. And then if I write code over on this side, on this video,
you can't copy it, of course, but you'll write the same code here and then go on and go on and
go on. And if you get stuck, of course, you have the reference notebook and you have an
opportunity to ask a question here. So with that being said, let's get started. The first thing
we're going to have a look at in PyTorch is an introduction to tenses. So tenses are the main
building block of deep learning in general, or data. And so you may have watched the video,
what is a tensor? For the sake of this course, tenses are a way to represent data, especially
multi dimensional data, numeric data that is, but that numeric data represents something else.
So let's go in here, creating tenses. So the first kind of tensor we're going to create is
actually called a scalar. I know I'm going to throw a lot of different names of things at you,
but it's important that you're aware of such nomenclature. Even though in PyTorch, almost
everything is referred to as a tensor, there are different kinds of tenses. And just to
exemplify the fact that we're using a reference notebook, if we go up here, we can see we have
importing PyTorch. We've done that. Now we're up to introduction to tenses. We've got creating
tenses, and we've got scalar, etc, etc, etc. So this is what we're going to be working through.
Let's do it together. So scalar, the way to, oops, what have I done there? The way to create a
tensor in PyTorch, we're going to call this scalar equals torch dot tensor. And we're going to fill
it with the number seven. And then if we press or retype in scalar, what do we get back? Seven,
wonderful. And it's got the tensor data type here. So how would we find out about what torch dot
tensor actually is? Well, let me show you how I would. We go to torch dot tensor. There we go.
We've got the documentation. So this is possibly the most common class in PyTorch other than
one we're going to see later on that you'll use, which is torch dot nn. Basically, everything in
PyTorch works off torch dot tensor. And if you'd like to learn more, you can read through here.
In fact, I would encourage you to read through this documentation for at least 10 minutes
after you finish some videos here. So with that being said, I'm going to link that in here.
So PyTorch tensors are created using torch dot tensor. And then we've got that link there.
Oops, typos got law Daniel. Come on. They're better than this. No, I'm kidding. There's going to be
typos got law through the whole course. Okay. Now, what are some attributes of a scalar? So
some details about scalars. Let's find out how many dimensions there are. Oh, and by the way,
this warning, perfect timing. Google Colab will give you some warnings here, depending on whether
you're using a GPU or not. Now, the reason being is because Google Colab provides GPUs to you and
I for free. However, GPUs aren't free for Google to provide. So if we're not using a GPU, we can
save some resources, allow someone else to use a GPU by going to none. And of course, we can
always switch this back. So I'm going to turn my GPU off so that someone else out there,
I'm not using the GPU at the moment, they can use it. So what you're also going to see is if
your Google Colab instance ever restarts up here, we're going to have to rerun these cells. So if
you stop coding for a while, go have a break and then come back and you start your notebook again,
that's one downside of Google Colab is that it resets after a few hours. How many hours? I don't
know exactly. The reset time is longer if you have the pro subscription, but because it's a free
service and the way Google calculate usage and all that sort of stuff, I can't give a conclusive
evidence or conclusive answer on how long until it resets. But just know, if you come back, you might
have to rerun some of your cells and you can do that with shift and enter. So a scalar has no
dimensions. All right, it's just a single number. But then we move on to the next thing. Or actually,
if we wanted to get this number out of a tensor type, we can use scalar dot item, this is going
to give it back as just a regular Python integer. Wonderful, there we go, the number seven back,
get tensor back as Python int. Now, the next thing that we have is a vector. So let's write

Section 17: Main Topics

Key Topics:

That's a little bit confusing, but the thing you should remember in PyTorch is basically anytime you encode data into numbers, it's of a tensor data type
So we're covering a fair bit of ground here, nice and quick, but that's going to be the teaching style of this course is we're going to get quite hands on and writing a lot of code and just interacting with it rather than continually going back over and discussing what's going on here
PyTorch will do a lot of that behind the scenes

📄 Click to view detailed content

in here vector, which again is going to be created with torch dot tensor. But you will also hear
the word vector used a lot too. Now, what is the deal? Oops, seven dot seven. Google Colab's auto
complete is a bit funny. It doesn't always do the thing you want it to. So if we see a vector,
we've got two numbers here. And then if we really wanted to find out what is a vector.
So a vector usually has magnitude and direction. So what we're going to see later on is, there we
go, magnitude, how far it's going and which way it's going. And then if we plotted it, we've got,
yeah, a vector equals the magnitude would be the length here and the direction would be where it's
pointing. And oh, here we go, scalar vector matrix tensor. This is what we're working on as well.
So the thing about vectors, how they differ with scalars is how I just remember them is
rather than magnitude and direction is a vector typically has more than one number.
So if we go vector and dim, how many dimensions does it have?
It has one dimension, which is kind of confusing. But when we see tensors with more than one
dimension, it'll make sense. And another way that I remember how many dimensions something
has is by the number of square brackets. So let's check out something else. Maybe we go vector
dot shape shape is two. So the difference between dimension. So dimension is like number of square
brackets. And when I say, even though there's two here, I mean number of pairs of closing square
brackets. So there's one pair of closing square brackets here. But the shape of the vector is two.
So we have two by one elements. So that means a total of two elements. Now if we wanted to step
things up a notch, let's create a matrix. So this is another term you're going to hear.
And you might be wondering why I'm capitalizing matrix. Well, I'll explain that in the second
matrix equals torch dot tensor. And we're going to put two square brackets here. You might be
thinking, what could the two square brackets mean? Or actually, that's a little bit of a challenge.
If one pair of square brackets had an endem of one, what will the endem be number of dimensions
of two square brackets? So let's create this matrix. Beautiful. So we've got another tensor here.
Again, as I said, these things have different names, like the traditional name of scalar,
vector matrix, but they're all still a torch dot tensor. That's a little bit confusing,
but the thing you should remember in PyTorch is basically anytime you encode data into numbers,
it's of a tensor data type. And so now how many n number of dimensions do you think a matrix has?
It has two. So there we go. We have two square brackets. So if we wanted to get matrix,
let's index on the zeroth axis. Let's see what happens there. Ah, so we get seven and eight.
And then we get off the first dimension. Ah, nine and 10. So this is where the square brackets,
the pairings come into play. We've got two square bracket pairings on the outside here.
So we have an endem of two. Now, if we get the shape of the matrix, what do you think the shape will be?
Ah, two by two. So we've got two numbers here by two. So we have a total of four elements in there.
So we're covering a fair bit of ground here, nice and quick, but that's going to be the
teaching style of this course is we're going to get quite hands on and writing a lot of code and
just interacting with it rather than continually going back over and discussing what's going on
here. The best way to find out what's happening within a matrix is to write more code that's similar
to these matrices here. But let's not stop at matrix. Let's upgrade to a tensor now. So I might
put this in capitals as well. And I haven't explained what the capitals mean yet, but we'll see that
in a second. So let's go torch dot tensor. And what we're going to do is this time,
we've done one square bracket pairing. We've done two square bracket pairings. Let's do three
square bracket pairings and just get a little bit adventurous. All right. And so you might be thinking
at the moment, this is quite tedious. I'm just going to write a bunch of random numbers here. One,
two, three, three, six, nine, two, five, four. Now you might be thinking, Daniel, you've said
tensors could have millions of numbers. If we had to write them all by hand, that would be
quite tedious. And yes, you're completely right. The fact is, though, that most of the time,
you won't be crafting tensors by hand. PyTorch will do a lot of that behind the scenes. However,
it's important to know that these are the fundamental building blocks of the models
and the deep learning neural networks that we're going to be building. So tensor capitals as well,
we have three square brackets. So, or three square bracket pairings. I'm just going to refer to three
square brackets at the very start because they're going to be paired down here. How many n dim or
number of dimensions do you think our tensor will have? Three, wonderful. And what do you think the
shape of our tensor is? We have three elements here. We have three elements here, three elements
here. And we have one, two, three. So maybe our tensor has a shape of one by three by three.
Hmm. What does that mean? Well, we've got three by one, two, three. That's the second square

Section 18: Main Topics

Key Topics:

Ah, so that's the first dimension there or the zeroth dimension because we remember PyTorch is zero indexed
In the last video, we covered the basic building blocks of data representation in deep learning, which is the tensor, or in PyTorch, specifically torch
So I hope you gave that a shot because as you'll see throughout the course and your deep learning journey, a tensor can represent or can be of almost any shape and size and have almost any combination of numbers within it

📄 Click to view detailed content

bracket there by one. Ah, so that's the first dimension there or the zeroth dimension because
we remember PyTorch is zero indexed. We have, well, let's just instead of talking about it,
let's just get on the zeroth axis and see what happens with the zeroth dimension. There we go.
Okay. So there's, this is the far left one, zero, which is very confusing because we've got a one
here, but so we've got, oops, don't mean that. What this is saying is we've got one three by three
shape tensor. So very outer bracket matches up with this number one here. And then this three
matches up with the next one here, which is one, two, three. And then this three matches up with
this one, one, two, three. Now, if you'd like to see this with a pretty picture, we can see it here.
So dim zero lines up. So the blue bracket, the very outer one, lines up with the one. Then dim
equals one, this one here, the middle bracket, lines up with the middle dimension here. And then
dim equals two, the very inner lines up with these three here. So again, this is going to take a lot
of practice. It's taken me a lot of practice to understand the dimensions of tensors. But
to practice, I would like you to write out your own tensor of, you can put however many square
brackets you want. And then just interact with the end dim shape and indexing, just as I've done
here, but you can put any combination of numbers inside this tensor. That's a little bit of practice
before the next video. So give that a shot and then we'll move on to the next topic.
I'll see you there. Welcome back. In the last video, we covered the basic building blocks of data
representation in deep learning, which is the tensor, or in PyTorch, specifically torch.tensor.
But within that, we had to look at what a scalar is. We had to look at what a vector is. We had to
look at a matrix. We had to look at what a tensor is. And I issued you the challenge to get as
creative as you like with creating your own tensor. So I hope you gave that a shot because as you'll
see throughout the course and your deep learning journey, a tensor can represent or can be of almost
any shape and size and have almost any combination of numbers within it. And so this is very important
to be able to interact with different tensors to be able to understand what the different names of
things are. So when you hear matrix, you go, oh, maybe that's a two dimensional tensor. When you
hear a vector, maybe that's a one dimensional tensor. When you hear a tensor, that could be any
amount of dimensions. And just for reference for that, if we come back to the course reference,
we've got a scalar. What is it? A single number, number of dimensions, zero. We've got a vector,
a number with direction, number of dimensions, one, a matrix, a tensor. And now here's another little
tidbit of the nomenclature of things, the naming of things. Typically, you'll see a variable name
for a scalar or a vector as a lowercase. So a vector, you might have a lowercase y storing that
data. But for a matrix or a tensor, you'll often see an uppercase letter or variable in Python in
our case, because we're writing code. And so I am not exactly sure why this is, but this is just
what you're going to see in machine learning and deep learning code and research papers
across the board. This is a typical nomenclature. Scalars and vectors, lowercase, matrix and tensors,
uppercase, that's where that naming comes from. And that's why I've given the tensor uppercase here.
Now, with that being said, let's jump in to another very important concept with tensors.
And that is random tensors. Why random tensors? I'm just writing this in a code cell now.
I could go here. This is a comment in Python, random tensors. But we'll get rid of that. We could
just start another text cell here. And then three hashes is going to give us a heading, random tensors
there. Or I could turn this again into a markdown cell with command mm when I'm using Google Colab.
So random tensors. Let's write down here. Why random tensors? So we've done the tedious thing
of creating our own tensors with some numbers that we've defined, whatever these are. Again,
you could define these as almost anything. But random tensors is a big part in pytorch because
let's write this down. Random tensors are important because the way many neural networks learn is
that they start with tensors full of random numbers and then adjust those random numbers
to better represent the data. So seriously, this is one of the big concepts of neural networks.
I'm going to write in code here, which is this is what the tick is for. Start with random numbers.
Look at data, update random numbers. Look at data, update random numbers. That is the crux
of neural networks. So let's create a random tensor with pytorch. Remember how I said that
pytorch is going to create tensors for you behind the scenes? Well, this is one of the ways that
it does so. So we create a random tensor and we give it a size of random tensor of size or shape.
Pytorch use these independently. So size, shape, they mean the different versions of the same thing.
So random tensor equals torch dot rand. And we're going to type in here three, four. And the beautiful
thing about Google Colab as well is that if we wait long enough, it's going to pop up with the doc

Section 19: Main Topics

Key Topics:

This is the PyTorch Fundamentals notebook
So the takeaway from this video is that PyTorch enables you to create tensors quite easily with the random method

📄 Click to view detailed content

string of what's going on. I personally find this a little hard to read in Google Colab,
because you see you can keep going down there. You might be able to read that. But what can we do?
Well, we can go to torch dot rand. Then we go to the documentation. Beautiful. Now there's a whole
bunch of stuff here that you're more than welcome to read. We're not going to go through all that.
We're just going to see what happens hands on. So we'll copy that in here. And write this in notes,
torch random tensors. Done. Just going to make some code cells down here. So I've got some space.
I can get this a bit up here. Let's see what our random tensor looks like. There we go. Beautiful
of size three, four. So we've got three or four elements here. And then we've got three deep
here. So again, there's the two pairs. So what do you think the number of dimensions will be
for random tensor? And dim. Two beautiful. And so we have some random numbers here. Now the
beautiful thing about pie torch again is that it's going to do a lot of this behind the scenes. So
if we wanted to create a size of 10 10, in some cases, we won't want one dimension here. And then
it's going to go 10 10. And then if we check the number of dimensions, how many do you think it
will be now three? Why is that? Because we've got one 10 10. And then if we wanted to create 10 10 10.
What's the number of dimensions going to be? It's not going to change. Why is that?
We haven't run that cell yet, but we've got a lot of numbers here.
We can find out what 10 times 10 times 10 is. And I know we can do that in our heads, but
the beauty of collab is we've got a calculator right here. 10 times 10 times 10. We've got a
thousand elements in there. But sometimes tenses can be hundreds of thousands of elements or
millions of elements. But pie torch is going to take care of a lot of this behind the scenes. So
let's clean up a bit of space here. This is a random tensor. Random numbers beautiful of now
it's got two dimensions because we've got three by four. And if we put another one in the front
there, we're going to have how many dimensions three dimensions there. But again, this number
of dimensions could be any number. And what's inside here could be any number. Let's get rid of that.
And let's get a bit specific because right now this is just a random tensor of whatever dimension.
How about we create a random tensor with similar shape to an image tensor. So a lot of the time
when we turn images, image size tensor, when we turn images into tenses, they're going to have,
let me just write it in code for you first, size equals a height, a width, and a number of color
channels. And so in this case, it's going to be height with color channels. And the color channels
are red, green, blue. And so let's create a random image tensor. Let's view the size of it or the
shape. And then random image size tensor will view the end dim. Beautiful. Okay, so we've got
torch size, the same size two, two, four, two, four, three, height, width, color channels. And we've got
three dimensions, one, four, height, width, color channels. Let's go and see an example of this. This
is the PyTorch Fundamentals notebook. If we go up to here, so say we wanted to encode this image
of my dad eating pizza with thumbs up of a square image of two, two, four by two, two, four.
This is an input. And if we wanted to encode this into tensor format, well, one of the ways of
representing an image tensor, very common ways is to split it into color channels because with
red, green, and blue, you can create almost any color you want. And then we have a tensor
representation. So sometimes you're going to see color channels come first. We can switch this
around and our code quite easily by going color channels here. But you'll also see color channels
come at the end. I know I'm saying a lot that we kind of haven't covered yet. The main takeaway
from here is that almost any data can be represented as a tensor. And one of the common ways to represent
images is in the format color channels, height, width, and how these values are will depend on
what's in the image. But we've done this in a random way. So the takeaway from this video is
that PyTorch enables you to create tensors quite easily with the random method. However, it is
going to do a lot of this creating tensors for you behind the scenes and why a random tensor is so
valuable because neural networks start with random numbers, look at data such as image tensors,
and then adjust those random numbers to better represent that data. And they repeat those steps
onwards and onwards and onwards. Let's finish this video here. I'm going to challenge for you
just to create your own random tensor of whatever size and shape you want. So you could have 5, 10,
10 here and see what that looks like. And then we'll keep coding in the next video.
I hope you took on the challenge of creating random tensor of your own size. And just a little
tidbit here. You might have seen me in the previous video. I didn't use the size parameter. But in
this case, I did here, you can go either way. So if we go torch dot rand size equals, we put in a

Section 20: Main Topics

Key Topics:

We've actually been using torch dot float the whole time, because that's whenever you create a tensor with pytorch, we're using a pytorch method, unless you explicitly define what the data type is, we'll see that later on, defining what the data type is, it starts off as torch float 32
And if we just write in torch dot a range, we've got tensors of zero to nine, because it of course starts at zero index

📄 Click to view detailed content

tuple here of three three, we've got that tensor there three three. But then also if we don't put
the size in there, it's the default. So it's going to create a very similar tensor. So whether you
have this size or not, it's going to have quite a similar output depending on the shape that you
put in there. But now let's get started to another kind of tensor that you might see zeros and ones.
So say you wanted to create a tensor, but that wasn't just full of random numbers,
you wanted to create a tensor of all zeros. This is helpful for if you're creating some form of
mask. Now, we haven't covered what a mask is. But essentially, if we create a tensor of all zeros,
what happens when you multiply a number by zero? All zeros. So if we wanted to multiply
these two together, let's do zeros times random tensor.
There we go, all zeros. So maybe if you're working with this random tensor and you wanted to mask
out, say all of the numbers in this column for some reason, you could create a tensor of zeros in
that column, multiply it by your target tensor, and you would zero all those numbers. That's telling
your model, hey, ignore all of the numbers that are in here because I've zeroed them out. And then
if you wanted to create a tensor of all ones, create a tensor of all ones, we can go ones equals
torch dot ones, size equals three, four. And then if we have a look, there's another parameter I
haven't showed you yet, but this is another important one is the D type. So the default data type,
so that's what D type stands for, is torch dot float. We've actually been using torch dot float
the whole time, because that's whenever you create a tensor with pytorch, we're using a pytorch
method, unless you explicitly define what the data type is, we'll see that later on, defining
what the data type is, it starts off as torch float 32. So these are float numbers. So that
is how you create zeros and ones zeros is probably I've seen more common than ones in use, but just
keep these in mind, you might come across them. There are lots of different methods to creating
tensors. And truth be told, like random is probably one of the most common, but you might see zeros
and ones out in the field. So now we've covered that. Let's move on into the next video, where
we're going to create a range. So have a go at creating a tensor full of zeros and whatever size
you want, and a tensor full of ones and whatever size you want. And I'll see you in the next video.
Welcome back. I hope you took on the challenge of creating a torch tensor of zeros of your
own size and ones of your own size. But now let's investigate how we might create a range of
tensors and tensors like. So these are two other very common methods of creating tensors.
So let's start by creating a range. So we'll first use torch dot range, because depending on
when you're watching this video, torch dot range may be still in play or it may be deprecated.
If we write in torch dot range right now with the pie torch version that I'm using, which is
torch dot version, which is torch or pie torch 1.10 point zero torch range is deprecated and
will be removed in a future release. So just keep that in mind. If you come across some code that's
using torch dot range, maybe out of whack. So the way to get around that is to fix that is to use
a range instead. And if we just write in torch dot a range, we've got tensors of zero to nine,
because it of course starts at zero index. If we wanted one to 10, we could go like this.
1, 2, 3, 4, 5, 6, 7, 8, 9, 10. And we can go zero, or we go 1, 2, 10, equals torch a range.
Wonderful. And we can also define the step. So let's let's type in some start and where can we
find the documentation on a range? Sometimes in Google Colab, you can press shift tab,
but I find that it doesn't always work for me. Yeah, you could hover over it, but we can also just
go torch a range and look for the documentation torch a range. So we've got start and step. Let's
see what all of these three do. Maybe we start at zero, and maybe we want it to go to a thousand,
and then we want a step of what should our step be? What's a fun number? 77. So it's not one to 10
anymore, but here we go. We've got start at zero, 77 plus 77 plus 77, all the way up to it finishes
at a thousand. So if we wanted to take it back to one to 10, we can go up here. 110, and the default
step is going to be one. Oops, we needed the end to be that it's going to finish at end minus one.
There we go. Beautiful. Now we can also create tensors like. So creating tensors like. So tensors
like is say you had a particular shape of a tensor you wanted to replicate somewhere else, but you
didn't want to explicitly define what that shape should be. So what's the shape of one to 10?
One to 10. Now if we wanted to create a tensor full of zeros that had the same shape as this,
we can use tensor like or zeros like. So 10 zeros, zeros equals, I'm not even sure if I'm

Section 21: Main Topics

Key Topics:

float 32, even though we put none, this is because the default data type in pytorch, even if it's specified as none is going to come out as float 32
Now this could be true, of course, we're going to set this as false
Now, again, you won't necessarily always have to enter these when you're creating tensors, because pytorch does a lot of tensor creation behind the scenes for you

📄 Click to view detailed content

spelling zeros right then, zeros. Well, I might have a typo spelling zeros here, but you get what
I'm saying is torch zeros. Oh, torch spell it like that. That's why I'm spelling it like that.
Zeros like one to 10. And then the input is going to be one to 10. And we have a look at 10 zeros.
My goodness, this is taking quite the while to run. This is troubleshooting on the fly.
If something's happening like this, you can try to stop. If something was happening like that,
you can click run and then stop. Well, it's running so fast that I can't click stop. If you do also
run into trouble, you can go runtime, restart runtime. We might just do that now just to show you.
Restart and run all is going to restart the compute engine behind the collab notebook.
And run all the cells to where we are. So let's just see that we restart and run runtime. If you're
getting errors, sometimes this helps. There is no set in stone way to troubleshoot errors. It's
guess and check with this. So there we go. We've created 10 zeros, which is torch zeros like
our one to 10 tensor. So we've got zeros in the same shape as one to 10. So if you'd like to create
tensors, use torch arrange and get deprecated message. Use torch arrange instead for creating
a range of tensors with a start and end in a step. And then if you wanted to create tensors
or a tensor like something else, you want to look for the like method. And then you put an input,
which is another tensor. And then it'll create a similar tensor with whatever this method here
is like in that fashion or in the same shape as your input. So with that being said,
give that a try, create a range of tensors, and then try to replicate that range shape that you've
made with zeros. I'll see you in the next video. Welcome back. Let's now get into a very important
topic of tensor data types. So we've briefly hinted on this before. And I said that let's create
a tensor to begin with float 32 tensor. And we're going to go float 32 tensor equals torch
dot tensor. And let's just put in the numbers three, six, nine. If you've ever played need for
speed underground, you'll know where three, six, nine comes from. And then we're going to go
D type equals, let's just put none and see what happens, hey, float 32 tensor. Oh, what is the
data type? float 32, tensor dot D type. float 32, even though we put none, this is because
the default data type in pytorch, even if it's specified as none is going to come out as float 32.
What if we wanted to change that to something else? Well, let's type in here float 16.
And now we've got float 32 tensor. This variable name is a lie now because it's a float 16 tensor.
So we'll leave that as none. Let's go there. There's another parameter when creating tensors.
It's very important, which is device. So we'll see what that is later on. And then there's a
final one, which is also very important, which is requires grad equals false. Now this could be
true, of course, we're going to set this as false. So these are three of the most important parameters
when you're creating tensors. Now, again, you won't necessarily always have to enter these when
you're creating tensors, because pytorch does a lot of tensor creation behind the scenes for you.
So let's just write out what these are. Data type is what data type is the tensor, e.g. float 32,
or float 16. Now, if you'd like to look at what data types are available for pytorch tensors,
we can go torch tensor and write up the top unless the documentation changes. We have data types.
It's so important that data types is the first thing that comes up when you're creating a tensor.
So we have 32-bit floating point, 64-bit floating point, 16, 16, 32-bit complex. Now,
the most common ones that you will likely interact with are 32-bit floating point and 16-bit floating
point. Now, what does this mean? What do these numbers actually mean? Well, they have to do with
precision in computing. So let's look up that. Precision in computing. Precision computer science.
So in computer science, the precision of a numerical quantity, we're dealing with numbers, right?
As a measure of the detail in which the quantity is expressed. This is usually measured in bits,
but sometimes in decimal digits. It is related to precision in mathematics, which describes the
number of digits that are used to express a value. So, for us, precision is the numerical quantity,
is a measure of the detail, how much detail in which the quantity is expressed. So, I'm not going
to dive into the background of computer science and how computers represent numbers. The important
takeaway for you from this will be that single precision floating point is usually called float
32, which means, yeah, a number contains 32 bits in computer memory. So if you imagine, if we have
a tensor that is using 32 bit floating point, the computer memory stores the number as 32 bits.
Or if it has 16 bit floating point, it stores it as 16 bits or 16 numbers representing or 16.
I'm not sure if a bit equates to a single number in computer memory. But what this means is that
a 32 bit tensor is single precision. This is half precision. Now, this means that it's the default
of 32, float 32, torch dot float 32, as we've seen in code, which means it's going to take up

Section 22: Main Topics

Key Topics:

I'm spending a lot of time here, because I'm going to put a note here, note, tensor data types is one of the three big issues with pytorch and deep learning or not not issues, they're going to be errors that you run into and deep learning
Three big errors, you'll run into with pytorch and deep learning
So for example, you have one tensor that lives on a GPU for fast computing, and you have another tensor that lives on a CPU and you try to do something with them, while pytorch is going to throw you an error

📄 Click to view detailed content

a certain amount of space in computer memory. Now, you might be thinking, why would I do anything
other than the default? Well, if you'd like to sacrifice some detail in how your number is
represented. So instead of 32 bits, it's represented by 16 bits, you can calculate faster on numbers
that take up less memory. So that is the main differentiator between 32 bit and 16 bit. But if
you need more precision, you might go up to 64 bit. So just keep that in mind as you go forward.
Single precision is 32. Half precision is 16. What do these numbers represent? They represent
how much detail a single number is stored in memory. That was a lot to take in. But we're talking
about 10 to data types. I'm spending a lot of time here, because I'm going to put a note here,
note, tensor data types is one of the three big issues with pytorch and deep learning or
not not issues, they're going to be errors that you run into and deep learning. Three big
errors, you'll run into with pytorch and deep learning. So one is tensors, not right data type.
Two tensors, not right shape. We've seen a few shapes of four and three tensors, not on the right
device. And so in this case, if we had a tensor that was float 16 and we were trying to do computations
with a tensor that was float 32, we might run into some errors. And so that's the tensors not
being in the right data type. So it's important to know about the D type parameter here. And then
tensors not being the right shape. Well, that's once we get onto matrix multiplication, we'll see
that if one tensor is a certain shape and another tensor is another shape and those shapes don't
line up, we're going to run into shape errors. And this is a perfect segue to the device.
Device equals none. By default, this is going to be CPU. This is why we are using Google Colab
because it enables us to have access to, oh, we don't want to restart, enables us to have access
to a GPU. As I've said before, a GPU enables us. So we could change this to CUDA. That would be,
we'll see how to write device agnostic code later on. But this device, if you try to do
operations between two tensors that are not on the same device. So for example, you have one tensor
that lives on a GPU for fast computing, and you have another tensor that lives on a CPU and you
try to do something with them, while pytorch is going to throw you an error. And then finally,
this last requirement is grad is if you want pytorch to track the gradients, we haven't covered
what that is of a tensor when it goes through certain numerical calculations. This is a bit of
a bombardment, but I thought I'd throw these in as important parameters to be aware of since
we're discussing data type. And really, it would be reminiscent of me to discuss data type without
discussing not the right shape or not the right device. So with that being said, let's write down
here what device is your tensor on, and whether or not to track gradients with this tensor's
operations. So we have a float 32 tensor. Now, how might we change the tensor data type of this?
Let's create float 16 tensor. And we saw that we could explicitly write in float 16 tensor.
Or we can just type in here, float 16 tensor equals float 32 tensor dot type. And we're going to type
in torch dot float 16, why float 16, because well, that's how we define float 16, or we could use
half. So the same thing, these things are the same, let's just do half, or float 16 is more
explicit for me. And then let's check out float 16 tensor. Beautiful, we've converted our float
32 tensor into float 16. So that is one of the ways that you'll be able to tackle the tensors
not in the right data type issue that you run into. And just a little note on the precision
and computing, if you'd like to read more on that, I'm going to link this in here. And this is all
about how computers store numbers. So precision in computing. There we go. I'll just get rid of that.
Wonderful. So give that a try, create some tensors, research, or go to the documentation of torch
dot tensor and see if you can find out a little bit more about D type device and requires grad,
and create some tensors of different data types. Play around with whatever the ones you want here,
and see if you can run into some errors, maybe try to multiply two tensors together. So if you go
float 16 tensor times float 32 tensor, give that a try and see what happens. I'll see you in the next
video. Welcome back. In the last video, we covered a little bit about tensor data types,
as well as some of the most common parameters you'll see past to the torch dot tensor method.
And so I should do the challenge at the end of the last video to create some of your own tensors
of different data types, and then to see what happens when you multiply a float 16 tensor by a
float 32 tensor. Oh, it works. And but you've like Daniel, you said that you're going to have tensors
not the right data type. Well, this is another kind of gotcha or caveat of pie torch and deep
learning in general, is that sometimes you'll find that even if you think something may error
because these two tensors are different data types, it actually results in no error. But then

Section 23: Main Topics

Key Topics:

Typo, of course, one of many in 32 tensor
So if we run into one of the three big problems in deep learning and neural networks in general, especially with PyTorch, tensor's not the right data type, tensor's not the right shape or tensor's not on the right device

📄 Click to view detailed content

sometimes you'll have other operations that you do, especially training large neural networks,
where you'll get data type issues. The important thing is to just be aware of the fact that some
operations will run an error when your tensors are not in the right data type. So let's try another
type. Maybe we try a 32 bit integer. So torch dot in 32. And we try to multiply that by a float.
Wonder what will happen then? So let's go into 32 in 32 tensor equals torch dot tensor. And we'll
just make it three. Notice that there's no floats there or no dot points to make it a float.
Three, six, nine and D type can be torch in 32. And then in 32 tensor, what does this look like?
Typo, of course, one of many in 32 tensor. So now let's go float 32 tensor and see what happens.
Can we get pie torch to throw an error in 32 tensor?
Huh, it worked as well. Or maybe we go into 64. What happens here?
Still works. Now, see, this is again one of the confusing parts of doing tensor operations.
What if we do a long tensor? Torch to long. Is this going to still work?
Ah, torch has no attribute called long. That's not a data type issue.
I think it's long tensor. Long tensor. Does this work? D type must be torch D type.
Torch long tensor. I could have sworn that this was torch dot tensor.
Oh, there we go. Torch dot long tensor. That's another word for 64 bit.
So what is this saying? CPU tensor. Okay, let's see. This is some troubleshooting on the fly here.
Then we multiply it. This is a float 32 times a long. It works. Okay, so it's actually a bit
more robust than what I thought it was. But just keep this in mind when we're training models,
we're probably going to run into some errors at some point of our tensor's not being the
right data type. And if pie torch throws us an error saying your tensors are in the wrong data
type, well, at least we know now how to change that data type or how to set the data type if we
need to. And so with that being said, let's just formalize what we've been doing a fair bit already.
And that's getting information from tensors. So the three big things that we'll want to get
from our tensors in line with the three big errors that we're going to face in neural networks and
deep lining is let's copy these down. Just going to get this, copy this down below. So if we want
to get some information from tensors, how do we check the shape? How do we check the data type?
How do we check the device? Let's write that down. So to get information from this, to get
D type or let's write data type from a tensor can use tensor dot D type. And let's go here to get
shape from a tensor can use tensor dot shape. And to get device from a tensor, which devices it on
CPU or GPU can use tensor dot device. Let's see these three in action. So if we run into one of
the three big problems in deep learning and neural networks in general, especially with PyTorch,
tensor's not the right data type, tensor's not the right shape or tensor's not on the right device.
Let's create a tensor and try these three out. We've got some tensor equals torch dot
rand and we'll create it a three four. Let's have a look at what it looks like.
There we go. Random numbers of shape three and four. Now let's find out some details about it.
Find out details about some tensor. So print or print some tensor.
And oops, didn't want that print. And let's format it or make an F string of shape of tensor.
Oh, let's do data type first. We'll follow that order.
Data type of tensor. And we're going to go, how do we do this? Some tensor dot what?
Dot d type. Beautiful. And then we're going to print tensors not in the right shape. So let's go
shape of tensor equals some tensor dot shape. Oh, I went a bit too fast, but we could also use
size. Let's just confirm that actually. We'll code that out together. From my experience,
some tensor dot size, and some tensor dot shape result in the same thing. Is that true? Oh, function.
Oh, that's what it is. Some tensor dot size is a function, not an attribute.
There we go. Which one should you use? For me, I'm probably more used to using shape. You may come
across dot size as well, but just realize that they do quite the same thing except one's a function
and one's an attribute. An attribute is written dot shape without the curly brackets. A function
or a method is with the brackets at the end. So that's the difference between these are attributes
here. D type size. We're going to change this to shape. Tensor attributes. This is what we're
getting. I should probably write that down. This is tensor attributes. That's the formal name for
these things. And then finally, what else do we want? Tensors, what device are we looking for?
Let's get rid of this, get rid of this. And then print f device tensor is on. By default,
our tensor is on the CPU. So some tensor dot device. There we go. So now we've got our tensor
here, some tensor. The data type is a torch float 32 because we didn't change it to anything else.
And torch float 32 is the default. The shape is three four, which makes a lot of sense because

Section 24: Main Topics

Key Topics:

And the device tensor is on is the CPU, which is, of course, the default, unless we explicitly say to put it on another device, all of the tensors that we create will default to being on the CPU, rather than the GPU
But see how to change the device a pytorch tensor is on
And I alluded to the fact that these will help resolve three of the most common issues in building neural networks, deep learning models, specifically with pytorch

📄 Click to view detailed content

we passed in three four here. And the device tensor is on is the CPU, which is, of course,
the default, unless we explicitly say to put it on another device, all of the tensors that we
create will default to being on the CPU, rather than the GPU. And we'll see later on how to put
tensors and other things in torch onto a GPU. But with that being said, give it a shot,
create your own tensor, get some information from that tensor, and see if you can change
these around. So see if you could create a random tensor, but instead of float 32, it's a float 16.
And then probably another extracurricular, we haven't covered this yet. But see how to change
the device a pytorch tensor is on. Give that a crack. And I'll see you in the next video.
Welcome back. So in the last video, we had a look at a few tensor attributes, namely the data
type of a tensor, the shape of a tensor, and the device that a tensor lives on. And I alluded to
the fact that these will help resolve three of the most common issues in building neural networks,
deep learning models, specifically with pytorch. So tensor has not been the right data type,
tensor has not been the right shape, and tensor has not been on the right device. So now let's
get into manipulating tensors. And what I mean by that, so let's just write here the title,
manipulating tensors. And this is going to be tensor operations. So when we're building neural
networks, neural networks are comprised of lots of mathematical functions that pytorch code is going
to run behind the scenes for us. So let's go here, tensor operations include addition,
subtraction, and these are the regular addition, subtraction, multiplication. There's two types
of multiplication in that you'll typically see referenced in deep learning and neural networks,
division, and matrix multiplication. And these, the ones here, so addition, subtraction,
multiplication, division, your typical operations that you're probably familiar with matrix multiplication.
The only different one here is matrix multiplication. We're going to have a look at that in a minute.
But to find patterns in numbers of a data set, a neural network will combine these functions
in some way, shape or form. So it takes a tensor full of random numbers, performs some kind of
combination of addition, subtraction, multiplication, division, matrix multiplication. It doesn't have
to be all of these. It could be any combination of these to manipulate these numbers in some way
to represent a data set. So that's how a neural network learns is it will just comprise these
functions, look at some data to adjust the numbers of a random tensor, and then go from there. But
with that being said, let's look at a few of these. So we'll begin with addition. First thing we need
to do is create a tensor. And to add something to a tensor, we'll just go torch tensor. Let's go one,
two, three, add something to a tensor is tensor plus, we can use plus as the addition operator,
just like in Python, tensor plus 10 is going to be tensor 11, 12, 13, tensor plus 100 is going to be
as you'd expect plus 100. Let's leave that as plus 10 and add 10 to it. And so you might be
able to guess how we would multiply it by 10. So let's go multiply tensor by 10. We can go tensor,
star, which are my keyboard shift eight, 10. We get 10, 10, 10. And because we didn't reassign it,
our tensor is still 123. So if we go, if we reassign it here, tensor equals tensor by 10,
and then check out tensor, we've now got 10 2030. And the same thing here, we'll have 10 2030. But
then if we go back from the top, if we delete this reassignment, oh, what do we get there, tensor
by 10. Oh, what's happened here? Oh, because we've got, yeah, okay, I see, tensor by 10, tensor,
still 123. What should we try now? How about subtract subtract 10 equals tensor minus 10.
And you can also use, well, there we go, one minus 10, eight minus 10, three minus 10.
You can also use like torch has inbuilt functions or pytorch. So try out pytorch
inbuilt functions. So torch dot mall is short for multiply. We can pass in our tensor here,
and we can add in 10. That's going to multiply each element of tensor by 10. So just taking
the original tensor that we created, which is 123. And performing the same thing as this,
I would recommend where you can use the operators from Python. If for some reason, you see torch
dot mall, maybe there's a reason for that. But generally, these are more understandable if you
just use the operators, if you need to do a straight up multiplication, straight up addition, or straight
up subtraction, because torch also has torch dot add, torch dot add, is it torch dot add? It might
be torch dot add. I'm not sure. Oh, there we go. Yeah, torch dot add. So as I alluded to before,
there's two different types of multiplication that you'll hear about element wise and matrix
multiplication. We're going to cover matrix multiplication in the next video. As a challenge,
though, I would like you to search what is matrix multiplication. And I think the first website that
comes up, matrix multiplication, Wikipedia, yeah, math is fun. It has a great guide. So before we
get into matrix multiplication, jump into math is fun to have a look at matrix multiplying,
and have a think about how we might be able to replicate that in pie torch. Even if you're not

Section 25: Main Topics

Key Topics:

Yeah, let's write that two main ways of performing multiplication in neural networks and deep learning
A few different options there, but let's look at what it looks like in pytorch code
So that's what I'd encourage you to go through step by step and reproduce this a good challenge would be to reproduce this by hand with pytorch code

📄 Click to view detailed content

sure, just have a think about it. I'll see you in the next video. Welcome back. In the last video,
we discussed some basic tensor operations, such as addition, subtraction, multiplication,
element wise, division, and matrix multiplication. But we didn't actually go through what matrix
multiplication is. So now let's start on that more particularly discussing the difference between
element wise and matrix multiplication. So we'll come down here, let's write another heading,
matrix multiplication. So there's two ways, or two main ways. Yeah, let's write that two main
ways of performing multiplication in neural networks and deep learning. So one is the simple
version, which is what we've seen, which is element wise multiplication. And number two is matrix
multiplication. So matrix multiplication is actually possibly the most common tensor operation you
will find inside neural networks. And in the last video, I issued the extra curriculum of having a
look at the math is fun dot com page for how to multiply matrices. So the first example they go
through is element wise multiplication, which just means multiplying each element by a specific
number. In this case, we have two times four equals eight, two times zero equals zero, two times one
equals two, two times negative nine equals negative 18. But then if we move on to matrix
multiplication, which is multiplying a matrix by another matrix, we need to do the dot product.
So that's something that you'll also hear matrix multiplication referred to as the dot product.
So these two are used interchangeably matrix multiplication or dot product. And if we just
look up the symbol for dot product, you'll find that it's just a dot. There we go, a heavy dot,
images. There we go, a dot B. So this is vector a dot product B. A few different options there,
but let's look at what it looks like in pytorch code. But first, there's a little bit of a
difference here. So how did we get from multiplying this matrix here of one, two, three, four, five,
six, times seven, eight, nine, 10, 11, 12? How did we get 58 there? Well, we start by going,
this is the difference between element wise and dot product, by the way, one times seven.
We'll record that down there. So that's seven. And then two times nine. So this is first row,
first column, two times nine is 18. And then three times 11 is 33. And if we add those up,
seven plus 18, plus 33, we get 58. And then if we were to do that for each other element that's
throughout these two matrices, we end up with something like this. So that's what I'd encourage
you to go through step by step and reproduce this a good challenge would be to reproduce this by
hand with pytorch code. But now let's go back and write some pytorch code to do both of these. So
I just want to link here as well, more information on multiplying matrices. So I'm going to turn
this into markdown. Let's first see element wise, element wise multiplication. We're going to start
with just a rudimentary example. So if we have our tensor, what is it at the moment? It's 123.
And then if we multiply that by itself, we get 149. But let's print something out so it looks a bit
prettier than that. So print, I'm going to turn this into a string. And then we do that. So if we
print tensor times tensor, element wise multiplication is going to give us print equals. And then
let's do in here tensor times tensor. We go like that. Wonderful. So we get one times one
equals one, two times two equals four, three times three equals nine. Now for matrix multiplication,
pytorch stores matrix multiplication, similar to torch dot mall in the torch dot mat mall space,
which stands for matrix multiplication. So let's just test it out. Let's just true the exact
same thing that we did here, instead of element wise, we'll do matrix multiplication on our 123
tensor. What happens here? Oh my goodness, 14. Now why did we get 14 instead of 149? Can you guess
how we got to 14 or think about how we got to 14 from these numbers? So if we recall back,
we saw that for we're only multiplying two smaller tensors, by the way, 123. This example is with
a larger one, but the same principle applies across different sizes of tensors or matrices.
And when I say matrix multiplication, you can also do matrix multiplication between tensors.
And in our case, we're using vectors just to add to the confusion. But what is the difference
here between element wise and dot product? Well, we've got one main addition. And that is addition.
So if we were to code this out by hand, matrix multiplication by hand, we'd have recall that
the elements of our tensor are 123. So if we wanted to matrix multiply that by itself,
we'd have one times one, which is the equivalent of doing one times seven in this visual example.
And then we'd have plus, it's going to be two times two, two times two. What does that give us?
Plus three times three. What does it give us? Three times three. That gives us 14. So that's how
we got to that number there. Now we could do this with a for loop. So let's have a gaze at when I
say gaze, it means have a look. That's a Australian colloquialism for having a look. But I want to
show you the time difference in it might not actually be that big a difference if we do it by hand
versus using something like matmore. And that's another thing to note is that if PyTorch has a
method already implemented, chances are it's a fast calculating version of that method. So I know

Section 26: Main Topics

Key Topics:

But once you start to build larger tensors, you might run into one of the most common errors in deep learning
One of the most common errors in deep learning, we've already alluded to this as well, is shape errors
So this is one of the most common errors that you're going to face in deep learning is that matrix one and matrix two shapes cannot be multiplied because it doesn't satisfy rule number one

📄 Click to view detailed content

for basic operators, I said it's usually best to just use this straight up basic operator.
But for something like matrix multiplication or other advanced operators instead of the basic
operators, you probably want to use the torch version rather than writing a for loop, which is
what we're about to do. So let's go value equals zero. This is matrix multiplication by hand. So
for I in range, len tensor, so for each element in the length of our tensor, which is 123, we want to
update our value to be plus equal, which is doing this plus reassignment here. The ith element in
each tensor times the ith element. So times itself. And then how long is this going to take?
Let's now return the value. We should get 14, print 14. There we go. So 1.9 milliseconds on
whatever CPU that Google collab is using behind the scenes. But now if we time it and use the torch
method torch dot matmore, it was tensor dot sensor. And again, we're using a very small tensor. So
okay, there we go. It actually showed how much quicker it is, even with such a small tensor.
So this is 1.9 milliseconds. This is 252 microseconds. So this is 10 times slower using a for loop,
then pie torches vectorized version. I'll let you look into that if you want to find out what
vectorization means. It's just a type of programming that rather than writing for loops, because as
you could imagine, if this tensor was, let's say, had a million elements instead of just three,
if you have to loop through each of those elements one by one, that's going to be quite cumbersome.
So a lot of pie torches functions behind the scenes implement optimized functions to perform
mathematical operations, such as matrix multiplication, like the one we did by hand,
in a far faster manner, as we can see here. And that's only with a tensor of three elements.
So you can imagine the speedups on something like a tensor with a million elements.
But with that being said, that is the crux of matrix multiplication. For a little bit more,
I encourage you to read through this documentation here by mathisfun.com. Otherwise,
let's look at a couple of rules that we have to satisfy for larger versions of matrix multiplication.
Because right now, we've done it with a simple tensor, only 123. Let's step things up a notch
in the next video. Welcome back. In the last video, we were introduced to matrix multiplication,
which although we haven't seen it yet, is one of the most common operations in neural networks.
And we saw that you should always try to use torches implementation of certain operations,
except if they're basic operations, like plus multiplication and whatnot,
because chances are it's a lot faster version than if you would do things by hand. And also,
it's a lot less code. Like compared to this, this is pretty verbose code compared to just a matrix
multiply these two tensors. But there's something that we didn't allude to in the last video.
There's a couple of rules that need to be satisfied when performing matrix multiplication.
It worked for us because we have a rather simple tensor. But once you start to build larger tensors,
you might run into one of the most common errors in deep learning. I'm going to write this down
actually here. This is one to be very familiar with. One of the most common errors in deep
learning, we've already alluded to this as well, is shape errors. So let's jump back to this in a
minute. I just want to write up here. So there are two rules that performing or two main rules
that performing matrix multiplication needs to satisfy. Otherwise, we're going to get an error.
So number one is the inner dimensions must match. Let's see what this means.
So if we want to have two tensors of shape, three by two, and then we're going to use the at symbol.
Now, we might be asking why the at symbol. Well, the at symbol is another, is a like an operator
symbol for matrix multiplication. So I just want to give you an example. If we go tensor at
at stands for matrix multiplication, we get tensor 14, which is exactly the same as what we got there.
Should you use at or should you use mat mall? I would personally recommend to use mat mall.
It's a little bit clearer at sometimes can get confusing because it's not as common as seeing
something like mat mall. So we'll get rid of that, but I'm just using it up here for brevity.
And then we're going to go three, two. Now, this won't work. We'll see why in a second.
But if we go two, three, at, and then we have three, two, this will work. Or, and then if we go
the reverse, say threes on the outside, twos here. And then we have twos on the inside and threes
on the outside, this will work. Now, why is this? Well, this is the rule number one. The inner
dimensions must match. So the inner dimensions are what I mean by this is let's create torch
round or create of size 32. And then we'll get its shape. So we have, so if we created a tensor
like this, three, two, and then if we created another tensor, well, let me just show you straight
up torch dot mat mall torch dot ran to watch this won't work. We'll get an error. There we go. So
this is one of the most common errors that you're going to face in deep learning is that matrix
one and matrix two shapes cannot be multiplied because it doesn't satisfy rule number one.
The inner dimensions must match. And so what I mean by inner dimensions is this dimension multiplied

Section 27: Main Topics

Key Topics:

We'll go on with one of those common errors in deep learning shape errors
Just watch what happens and we're going to replicate something like this in PyTorch code in the next video
So this is what I've been alluding to as one of the most common errors in deep learning, and that is shape errors

📄 Click to view detailed content

by this dimension. So say we were trying to multiply three, two by three, two, these are the inner
dimensions. Now this will work because why the inner dimensions match. Two, three by three, two,
two, three by three, two. Now notice how the inner dimensions, inner, inner match. Let's see what
comes out here. Look at that. And now this is where rule two comes into play. Two. The resulting
matrix has the shape of the outer dimensions. So we've just seen this one two, three at three, two,
which is at remember is matrix multiply. So we have a matrix of shape, two, three,
matrix multiply a matrix of three, two, the inner dimensions match. So it works. The resulting shape
is what? Two, two. Just as we've seen here, we've got a shape of two, two. Now what if we did
the reverse? What if we did this one that also will work? Three on the outside. What do you think
is going to happen here? In fact, I encourage you to pause the video and give it a go. So this
is going to result in a three three matrix. But don't take my word for it. Let's have a look. Three,
put two on the inside and we'll put two on the inside here and then three on the outside. What
does it give us? Oh, look at that. A three three. One, two, three. One, two, three. Now what if we
were to change this? Two and two. This can be almost any number you want. Let's change them both
to 10. What's going to happen? Will this work? What's the resulting shape going to be? So the
inner dimensions match? What's rule number two? The resulting matrix has the shape of the outer
dimension. So what do you think is going to be the shape of this resulting matrix multiplication?
Well, let's have a look. It's still three three. Wow. Now what if we go 10? 10 on the outside
and 10 and 10 on the inside? What do we get? Well, we get, I'm not going to count all of those,
but if we just go shape, we get 10 by 10. Because these are the two main rules of matrix multiplication
is if you're running into an error that the matrix multiplication can't work. So let's say this was
10 and this was seven. Watch what's going to happen? We can't multiply them because the inner
dimensions do not match. We don't have 10 and 10. We have 10 and seven. But then when we change
this so that they match, we get 10 and 10. Beautiful. So now let's create a little bit more of a
specific example. We'll create two tenses. We'll come down. Actually, to prevent this video from
being too long, I've got an error in the word error. That's funny. We'll go on with one of those
common errors in deep learning shape errors. We've just seen it, but I'm going to get a little bit
more specific with that shape error in the next video. Before we do that, have a look at matrix
multiplication. There's a website, my other favorite website. I told you I've got two. This is my
other one. Matrix multiplication dot XYZ. This is your challenge before the next video. Put in
some random numbers here, whatever you want, two, 10, five, six, seven, eight, whatever you want. Change
these around a bit, three, four. Well, that's a five, not a four. And then multiply and just watch
what happens. That's all I'd like you to do. Just watch what happens and we're going to replicate
something like this in PyTorch code in the next video. I'll see you there.
Welcome back. In the last video, we discussed a little bit more about matrix multiplication,
but we're not done there. We looked at two of the main rules of matrix multiplication,
and we saw a few errors of what happens if those rules aren't satisfied, particularly if the
inner dimensions don't match. So this is what I've been alluding to as one of the most common
errors in deep learning, and that is shape errors. Because neural networks are comprised of lots of
matrix multiplication operations, if you have some sort of tensor shape error somewhere
in your neural network, chances are you're going to get a shape error. So now let's investigate
how we can deal with those. So let's create some tenses, shapes for matrix multiplication.
And I also showed you the website, sorry, matrix multiplication dot xyz. I hope you had a go at
typing in some numbers here and visualizing what happens, because we're going to reproduce
something very similar to what happens here, but with PyTorch code. Shapes for matrix multiplication,
we have tensor a, let's create this as torch dot tensor. We're going to create a tensor with
just the elements one, two, all the way up to, let's just go to six, hey, that'll be enough. Six,
wonderful. And then tensor b can be equal to a torch tensor
of where we're going to go for this one. Let's go seven, 10, this will be a little bit confusing
this one, but then we'll go eight, 11, and this will go up to 12, nine, 12. So it's the same
sort of sequence as what's going on here, but they've been swapped around. So we've got the
vertical axis here, instead of one, two, three, four, this is just seven, eight, nine, 10, 11, 12.
But let's now try and perform a matrix multiplication. How do we do that?
Torch dot mat mall for matrix multiplication. PS torch also has torch dot mm, which stands
for matrix multiplication, which is a short version. So I'll just write down here so that you know

Section 28: Main Topics

Key Topics:

This is literally how common matrix multiplications are in PyTorch is that they've made torch dot mm as an alias for mat mall
But right now we've done it with pytorch code, which might be a little confusing
But the reason why we're spending so much time on this is because as you'll see, as you get more and more into neural networks and deep learning, the matrix multiplication operation is one of the most or if not the most common

📄 Click to view detailed content

tensor a, tensor b. I'm going to write torch dot mm is the same as torch dot mat mall. It's an alias
for writing less code. This is literally how common matrix multiplications are in PyTorch
is that they've made torch dot mm as an alias for mat mall. So you have to type four less characters
using torch dot mm instead of mat mall. But I like to write mat mall because it's a little bit
like it explains what it does a little bit more than mm. So what do you think's going to happen
here? It's okay if you're not sure. But what you could probably do to find out is check the
shapes of these. Does this operation matrix multiplication satisfy the rules that we just
discussed? Especially this one. This is the main one. The inner dimensions must match.
Well, let's have a look, hey? Oh, no, mat one and mat two shapes cannot be multiplied.
Three by two and three by two. This is very similar to what we went through in the last video.
But now we've got some actual numbers there. Let's check the shape.
Oh, torch size three two. Torch size three two now. In the last video we created a random tensor
and we could adjust the shape on the fly. But these tensors already exist. How might we adjust
the shape of these? Well, now I'm going to introduce you to another very common operation or tensor
manipulation that you'll see. And that is the transpose. To fix our tensor shape issues,
we can manipulate the shape of one of our tensors using a transpose. And so, all right here,
we're going to see this anyway, but I'm going to define it in words. A transpose switches the
axes or dimensions of a given tensor. So let's see this in action. If we go, and the way to do it,
is you can go tensor b dot t. Let's see what happens. Let's look at the original tensor b as well.
So dot t stands for transpose. And that's a little bit hard to read, so we might do these on
different lines, tensor b. We'll get rid of that. So you see what's happened here. Instead of
tensor b, this is the original one. We might put the original on top. Instead of the original one
having seven, eight, nine, 10, 11, 12 down the vertical, the transpose has transposed it to seven,
eight, nine across the horizontal and 10, 11, 12 down here. Now, if we get the shape of this,
tensor b dot shape, let's have a look at that. Let's have a look at the original shape, tensor b dot
shape. What's happened? Oh, no, we've still got three, two. Oh, that's what I've missed out here.
I've got a typo. Excuse me. I thought I was, you think code that you've written is working,
but then you realize you've got something as small as just a dot t missing, and it throws off your
whole train of thought. So you're seeing these arrows on the fly here. Now, tensor b is this,
but its shape is torch dot size three, two. And if we try to matrix multiply three, two, and three,
two, tensor a and tensor b, we get an error. Why? Because the inner dimensions do not match.
But if we perform a transpose on tensor b, we switch the dimensions around. So now,
we perform a transpose with tensor b dot t, t's for transpose. We have, this is the important
point as well. We still have the same elements. It's just that they've been rearranged. They've
been transposed. So now, tensor b still has the same information encoded, but rearranged.
So now we have torch size two, three. And so when we try to matrix multiply these,
we satisfy the first criteria. And now look at the output of the matrix multiplication of tensor a
and tensor b dot t transposed is three, three. And that is because of the second rule of matrix
multiplication. The resulting matrix has the shape of the outer dimensions. So we've got three,
two matrix multiply two, three results in a shape of three, three. So let's predify some of this,
and we'll print out what's going on here. Just so we know, we can step through it,
because right now we've just got codal over the place a bit. Let's see here, the matrix
multiplication operation works when tensor b is transposed. And in a second, I'm going to
show you what this looks like visually. But right now we've done it with pytorch code,
which might be a little confusing. And that's perfectly fine. Matrix multiplication takes a
little while and a little practice. So original shapes is going to be tensor a dot shape. Let's
see what this is. And tensor b equals tensor b dot shape. But the reason why we're spending so
much time on this is because as you'll see, as you get more and more into neural networks and
deep learning, the matrix multiplication operation is one of the most or if not the most common.
Same shape as above, because we haven't changed tensor a shape, we've only changed tensor b shape,
or we've transposed it. And then in tensor b dot transpose equals, we want tensor b dot
t dot shape. Wonderful. And then if we print, let's just print out, oops,
print, I spelled the wrong word there, print. We want, what are we multiplying here? This is
one of the ways, remember our motto of visualize, visualize, visualize, well, this is how I visualize,
visualize, visualize things, shape, let's do the at symbol for brevity, tensor, and let's get b dot
t dot shape. We'll put down our little rule here, inner dimensions must match. And then print,

Section 29: Main Topics

Key Topics:

Now, of course, you could rearrange this maybe transpose tensor a instead of tensor b, have a play around with it
And then if we click multiply, this is what's happening behind the scenes with our pytorch code of matmore
So let's have a look at some few PyTorch methods that are in built to do all of these

📄 Click to view detailed content

let's get the output output, I'll put that on a new line. The output is going to equal
torch dot, or our outputs already here, but we're going to rewrite it for a little bit of practice,
tensor a, tensor b dot t. And then we can go print output. And then finally, print, let's get it on a
new line as well, the output shape, a fair bit going on here. But we're going to step through it,
and it's going to help us understand a little bit about what's going on. That's the data visualizes
motto. There we go. Okay, so the original shapes are what torch size three two, and torch size three
two, the new shapes tensor a stays the same, we haven't changed tensor a, and then we have tensor
b dot t is torch size two three, then we multiply a three by two by a two by three. So the inner
dimensions must match, which is correct, they do match two and two. Then we have an output of tensor
at 27, 30, 33, 61, 68, 75, etc. And the output shape is what the output shape is the outer
dimensions three three. Now, of course, you could rearrange this maybe transpose tensor a instead of
tensor b, have a play around with it. See if you can create some more errors trying to multiply these
two, and see what happens if you transpose tensor a instead of tensor b, that's my challenge. But
before we finish this video, how about we just recreate what we've done here with this cool website
matrix multiplication. So what did we have? We had tensor a, which is one to six, let's recreate
this, remove that, this is going to be one, two, three, four, five, six, and then we want to increase
this, and this is going to be seven, eight, nine, 10, 11, 12. Is that the right way of doing things?
So this is already transposed, just to let you know. So this is the equivalent of tensor b
on the right here, tensor b dot t. So let me just show you, if we go tensor b dot transpose,
which original version was that, but we're just passing in the transpose version to our matrix
multiplication website. And then if we click multiply, this is what's happening behind the
scenes with our pytorch code of matmore. We have one times seven plus two times 10. Did you see
that little flippy thing that it did? That's where the 27 comes from. And then if we come down here,
what's our first element? 27 when we matrix multiply them. Then if we do the same thing,
the next step, we get 30 and 61, from a combination of these numbers, do it again,
33, 68, 95, from a combination of these numbers, again, and again, and finally we end up with
exactly what we have here. So that's a little bit of practice for you to go through is to create
some of your own tensors can be almost whatever you want. And then try to matrix multiply them
with different shapes. See what happens when you transpose and what different values you get.
And if you'd like to visualize it, you could write out something like this. That really
helps me understand matrix multiplication. And then if you really want to visualize it,
you can go through this website and recreate your target tensors in something like this.
I'm not sure how long you can go. But yeah, that should be enough to get started.
So give that a try and I'll see you in the next video.
Welcome back. In the last few videos, we've covered one of the most fundamental operations
in neural networks. And that is matrix multiplication. But now it's time to move on.
And let's cover tensor aggregation. And what I mean by that is finding the min, max, mean,
sum, et cetera, tensor aggregation of certain tensor values. So for whatever reason, you may
want to find the minimum value of a tensor, the maximum value, the mean, the sum, what's going on
there. So let's have a look at some few PyTorch methods that are in built to do all of these.
And again, if you're finding one of these values, it's called tensor aggregation because you're
going from what's typically a large amount of numbers to a small amount of numbers. So the min
of this tensor would be 27. So you're turning it from nine elements to one element, hence
aggregation. So let's create a tensor, create a tensor, x equals torch dot, let's use a range.
We'll create maybe a zero to 100 with a step of 10. Sounds good to me. And we can find the min
by going, can we do torch dot min? Maybe we can. Or we could also go
x dot min.
And then we can do the same, find the max torch dot max and x dot max. Now how do you think we
might get the average? So let's try it out. Or find the mean, find the mean torch dot mean
x. Oops, we don't have an x. Is this going to work? What's happened? Mean input data type
should be either floating point or complex D types got long instead. Ha ha. Finally,
I knew the error would show its face eventually. Remember how I said it right up here that
we've covered a fair bit already. But right up here, some of the most common errors that
you're going to run into is tensor is not the right data type, not the right shape. We've seen
that with matrix multiplication, not the right device. We haven't seen that yet. But not the
right data type. This is one of those times. So it turns out that the tensor that we created,
x is of the data type, x dot D type.

Section 30: Main Topics

Key Topics:

So so far, we've seen two of the major errors in PyTorch is data type and shape issues
And we also ran into one of the most common issues in pie torch and deep learning and neural networks in general

📄 Click to view detailed content

In 64, which is long. So if we go to, let's look up torch tensor.
This is where they're getting long from. We've seen long before is N64. Where's that or long?
Yeah. So long tenter. That's what it's saying. And it turns out that the torch mean function
can't work on tensors with data type long. So what can we do here? Well, we can change
the data type of x. So let's go torch mean x type and change it to float 32. Or before we do that,
if we go to torch dot mean, is this going to tell us that it needs a D type? Oh, D type.
One option on the desired data type. Does it have float 32? It doesn't tell us. Ah, so this is
another one of those little hidden things that you're going to come across. And you only really
come across this by writing code is that sometimes the documentation doesn't really tell you explicitly
what D type the input should be, the input tensor. However, we find out that with this error message
that it should either be a floating point or a complex D type, not along. So we can convert it
to torch float 32. So all we've done is gone x type as type float 32. Let's see what happens here.
45 beautiful. And then the same thing, if we went, can we do x dot mean? Is that going to work as well?
Oh, same thing. So if we go x dot type torch dot float 32, get the mean of that. There we go.
So that is, I knew it would come up eventually. A beautiful example of finding the right data
type. Let me just put a note here. Note the torch dot mean function requires a tensor of float 32.
So so far, we've seen two of the major errors in PyTorch is data type and shape issues. What's
another one that we said? Oh, some. So find the sum. Find the sum we want x dot sum or maybe we
just do torch dot sum first. Keep it in line with what's going on above and x dot sum.
Which one of these should you use like torch dot something x or x dot sum? Personally,
I prefer torch dot max, but you'll also probably see me at points right this. It really depends
on what's going on. I would say pick whichever style you prefer. And because behind the scenes,
they're calling the same methodology. Picture whichever style you prefer and stick with that
throughout your code. For now, let's leave it at that tensor aggregation. There's some
finding min max mean sum. In the next video, we're going to look at finding the positional
min and max, which is also known as arg max and arg min or vice versa. So actually, that's a
little bit of a challenge for the next video is see how you can find out what the positional
min and max is of this. And what I mean by that is which index does the max value occur at and
which index of this tensor does the min occur at? You'll probably want to look into the methods
arg min torch dot arg min for that one and torch dot arg max for that. But we'll cover that in the
next video. I'll see you there. Welcome back. In the last video, we learned all about tensor
aggregation. And we found the min the max the mean and the sum. And we also ran into one of the most
common issues in pie torch and deep learning and neural networks in general. And that was wrong
data types. And so we solved that issue by converting because some functions such as torch dot mean
require a specific type of data type as input. And we created our tensor here, which was of by
default torch in 64. However, torch dot mean requires torch dot float 32. We saw that in an error.
We fix that by changing the type of the inputs. I also issued you the challenge of finding
finding the positional min and max. And you might have found that you can use the
arg min for the minimum. Let's remind ourselves of what x is x. So this means at tensor index of
tensor x. If we find the argument, that is the minimum value, which is zero. So at index zero,
we get the value zero. So that's at zero there. Zero there. This is an index value. So this is
what arg min stands for find the position in tensor that has the minimum value with arg min.
And then returns index position of target tensor
where the minimum value occurs. Now, let's just change x to start from one,
just so there we go. So the arg min is still position zero, position zero. So this is an index
value. And then if we index on x at the zeroth index, we get one. So the minimum value in
x is one. And then the maximum, you might guess, is find the position in tensor that has the maximum
value with arg max. And it's going to be the same thing, except it'll be the maximum, which is,
which position index nine. So if we go zero, one, two, three, four, five, six, seven, eight,
nine. And then if we index on x for the ninth element, we get 91 beautiful. Now these two are
useful for if yes, you want to define the minimum of a tensor, you can just use min. But if you
sometimes you don't want the actual minimum value, you just want to know where it appears,
particularly with the arg max value. This is helpful for when we use the soft max activation
function later on. Now we haven't covered that yet. So I'm not going to allude too much to it.

Section 31: Main Topics

Key Topics:

So reshaping is we saw before one of the most common errors in machine learning and deep learning is shape mismatches with matrices because they have to satisfy certain rules
Because again, one of the number one issues in machine learning and deep learning is tensor shape issues

📄 Click to view detailed content

But just remember to find the positional min and max, you can use arg min and arg max.
So that's all we need to cover with that. Let's keep going in the next video. I'll see you then.
Welcome back. So we've covered a fair bit of ground. And just to let you know, I took a little break
after going through all of these. And I'd just like to show you how I get back to where I'm at,
because if we tried to just write x here and press shift and enter, because our collab
was disconnected, it's now connecting because as soon as you press any button in collab, it's
going to reconnect. It's going to try to connect, initialize, and then x is probably not going to
be stored in memory anymore. So there we go. Name x is not defined. That's because the collab
state gets reset if you take a break for a couple of hours. This is to ensure Google can keep
providing resources for free. And it deletes everything to ensure that there's no compute
resources that are being wasted. So to get back to here, I'm just going to go restart and run all.
You don't necessarily have to restart the notebook. You could also go, do we have run all? Yeah,
we could do run before. That'll run every cell before this. We could run after we could run the
selection, which is this cell here. I'm going to click run all, which is just going to go through
every single cell that we've coded above and run them all. However, it will also stop at the errors
where I've left in on purpose. So remember when we ran into a shape error? Well, because this error,
we didn't fix it. I left it there on purpose so that we could keep seeing a shape error. It's
going to stop at this cell. So we're going to have to run every cell after the error cell.
So see how it's going to run these now. They run fine. And then we get right back to where we were,
which was X. So that's just a little tidbit of how I get back into coding. Let's now cover reshaping,
stacking, squeezing, and unsqueezing. You might be thinking, squeezing and unsqueezing. What are
you talking about, Daniel? Well, it's all to do with tenses. And you're like, are we going to
squeeze our tenses? Give them a hug. Are we going to let them go by unsqueezing them?
Well, let's quickly define what these are. So reshaping is we saw before one of the most common
errors in machine learning and deep learning is shape mismatches with matrices because they
have to satisfy certain rules. So reshape reshapes an input tensor to a defined shape.
Now, we're just defining these things in words right now, but we're going to see it in code in
just a minute. There's also view, which is return a view of an input tensor of certain shape,
but keep the same memory as the original tensor. So we'll see what view is in a second.
Reshaping and view are quite similar, but a view always shares the same memory as the original
tensor. It just shows you the same tensor, but from a different perspective, a different shape.
And then we have stacking, which is combine multiple tensors on top of each other. This is a V stack
for vertical stack or side by side. H stack. Let's see what different types of torch stacks there are.
Again, this is how I research different things. If I wanted to learn something new, I would search
torch something stack concatenate a sequence of tensors along a new dimension. Okay. So maybe we
not H stack or V stack, we can just define what dimension we'd like to combine them on.
I wonder if there is a torch V stack. Torch V stack. Oh, there it is. And is there a torch H stack for
horizontal stack? There is a H stack. Beautiful. So we'll focus on just the plain stack. If you
want to have a look at V stack, it'll be quite similar to what we're going to do with stack
and same with H stack. Again, this is just words for now. We're going to see the code in a minute.
So there's also squeeze, which removes all one dimensions. I'm going to put one in code,
dimensions from a tensor. We'll see what that looks like. And then there's unsqueeze,
which adds a one dimension to our target tensor. And then finally, there's permute, which is return
a view of the input with dimensions permuted. So swapped in a certain way. So a fair few methods
here. But essentially the crust of all of these, the main point of all of these is to manipulate
our tensors in some way to change their shape or change their dimension. Because again, one of the
number one issues in machine learning and deep learning is tensor shape issues. So let's start
off by creating a tensor and have a look at each of these. Let's create a tensor. And then we're
going to just import torch. We don't have to, but this will just enable us to run the notebook
directly from this cell if we wanted to, instead of having to run everything above here. So let's
create another X torch dot a range because range is deprecated. I'm just going to add a few code
cells here so that I can scroll and that's in the middle of the screen there. Beautiful. So let's
just make it between one and 10 nice and simple. And then let's have a look at X and X dot shape.
What does this give us? Okay, beautiful. So we've got the numbers from one to nine. Our tensor is

Section 32: Main Topics

Key Topics:

📄 Click to view detailed content

of shape torch size nine. Let's start with reshape. So how about we add an extra dimension. So then
we have X reshaped equals X dot reshape. Now a key thing to keep in mind about the reshape
is that the dimensions have to be compatible with the original dimensions. So we're going to
change the shape of our original tensor with a reshape. And we try to change it into the shape
one seven. Does that work with the number nine? Well, let's find out, hey, let's check out X reshaped.
And then we'll look at X reshaped dot shape. What's this going to do? Oh, why do we get an error there?
Well, it's telling us here, this is what pie torch is actually really good at is giving us
errors for what's going wrong. We have one seven is invalid for input size of nine.
Well, why is that? Well, we're trying to squeeze nine elements into a tensor of one
times seven into seven elements. But if we change this to nine, what do we get? Ah, so do you notice
what just happened here? We just added a single dimension. See the single square bracket with
the extra shape here. What if we wanted to add two? Can we do that? No, we can't. Why is that?
Well, because two nine is invalid for input size nine, because two times nine is what?
18. So we're trying to double the amount of elements without having double the amount of elements.
So if we change this back to one, what happens if we change these around nine one? What does this
do? Oh, a little bit different there. So now instead of adding one on the first dimension or
the zeroth dimension, because Python is zero indexed, we added it on the first dimension,
which is giving us a square bracket here if we go back. So we add it to the outside here,
because we've put the one there. And then if we wanted to add it on the inside,
we put the one on the outside there. So then we've got the torch size nine one. Now, let's try
change the view, change the view. So just to reiterate, the reshape has to be compatible
with the original size. So how about we change this to one to 10? So we have a size of 10,
and then we can go five, two, what happens there? Oh, it's compatible because five times two equals
10. And then what's another way we could do this? How about we make it up to 12? So we've got 12
elements, and then we can go three, four, a code cells taking a little while run here.
Then we'll go back to nine, just so we've got the original there.
Whoops, they're going to be incompatible. Oh, so this is another thing. This is good.
We're getting some errors on the fly here. Sometimes you'll get saved failed with Google
CoLab, and automatic saving failed. What you can do to fix this is just either keep coding,
keep running some cells, and CoLab will fix itself in the background, or restart the notebook,
close it, and open again. So we've got size nine, or size eight, sorry, incompatible.
But this is good. You're seeing the errors that come up on the fly, rather than me sort of just
telling you what the errors are, you're seeing them as they come up for me. I'm trying to live
code this, and this is what's going to happen when you start to use Google CoLab, and subsequently
other forms of Jupyter Notebooks. But now let's get into the view, so we can go z equals,
let's change the view of x. View will change it to one nine, and then we'll go z, and then z dot shape.
Ah, we get the same thing here. So view is quite similar to reshape. Remember, though, that a
view shares the memory with the original tensor. So z is just a different view of x. So z shares
the same memory as what x does. So let's exemplify this. So changing z changes x, because a view of
a tensor shares the same memory as the original input. So let's just change z, change the first
element by using indexing here. So we're targeting one, we'll set this to equal five, and then we'll
see what z and x equal. Yeah, so see, we've got z, the first one here, we change the first element,
the zero element to five. And the same thing happens with x, we change the first element of z.
So because z is a view of x, the first element of x changes as well. But let's keep going. How
about we stack some tenses on top of each other? And we'll see what the stack function does in
torch. So stack tenses on top of each other. And I'll just see if I press command S to save,
maybe we'll get this fixed. Or maybe it just will fix itself. Oh, notebook is saved.
Unless you've made some extensive changes that you're worried about losing, you could just
download this notebook, so file download, and upload it to collab. But usually if you click yes,
it sort of resolves itself. Yeah, there we go. All changes saved. So that's beautiful
troubleshooting on the fly. I like that. So x stack, let's stack some tenses together,
equals torch stack. Let's go x x x, because if we look at what the doc string of stack is,
will we get this in collab? Or we just go to the documentations? Yeah. So list, it takes a list of
tenses and concatenates a sequence of tenses along a new dimension. And we define the dimension,
the dimension by default is zero. That's a little bit hard to read for me. So tenses,

Section 33: Main Topics

Key Topics:

I mean, I still do it a lot of the time, even though I've written thousands of lines of machine learning code

📄 Click to view detailed content

dim equals zero. If we come into here, the default dimension is zero. Let's see what happens when
we play around with the dimension here. So we've got four x's. And the first one, we'll just do it
by default, x stack. Okay, wonderful. So they're stacked vertically. Let's see what happens if we
change this to one. Oh, they rearranged a little and stack like that. What happens if we change it
to two? Does it have a dimension to? Oh, we can't do that. Well, that's because the original shape
of x is incompatible with using dimension two. So the only real way to get used to what happens
here by stacking them on top of each other is to play around with the different values for the
dimension. So dim zero, dim one, they look a little bit different there. Now they're on top of each
other. And so the first zero index is now the zeroth tensor. And then same with two being there,
three and so on. But we'll leave it at the default. And there's also v stack and h stack. I'll leave
that to you to to practice those. But I think from memory v stack is using dimension equals zero.
Or h stack is like using dimension equals one. I may have those back the front. You can correct me
if I'm wrong there. Now let's move on. We're going to now have a look at squeeze and unsqueeze.
So actually, I'm going to get you to practice this. So see if you can look up torch squeeze
and torch unsqueeze. And see if you can try them out. We've created a tensor here. We've used
reshape and view and we've used stack. The usage of squeeze and unsqueeze is quite similar. So give
that a go. And to prevent this video from getting too long, we'll do them together in the next video.
Welcome back. In the last video, I issued the challenge of trying out torch dot squeeze,
which removes all single dimensions from a target tensor. And how would you try that out? Well,
here's what I would have done. I'd go to torch dot squeeze and see what happens. Open up the
documentation. Squeeze input dimension returns a tensor with all the dimensions of input size
one removed. And does it have some demonstrations? Yes, it does. Wow. Okay. So you could copy this in
straight into a notebook, copy it here. But what I'd actually encourage you to do quite often is
if you're looking up a new torch method you haven't used, code all of the example by hand. And then
just practice what the inputs and outputs look like. So x is the input here. Check the size of x,
squeeze x, well, set the squeeze of x to y, check the size of y. So let's replicate something
similar to this. We'll go into here, we'll look at x reshaped and we'll remind ourselves of x reshaped
dot shape. And then how about we see what x reshaped dot squeeze looks like. Okay. What happened here?
Well, we started with two square brackets. And we started with a shape of one nine
and removes all single dimensions from a target tensor. And now if we call the squeeze method on
x reshaped, we only have one square bracket here. So what do you think the shape of x reshaped dot
squeeze is going to be? We'll check the shape here. It's just nine. So that's the squeeze method,
removes all single dimensions. If we had one one nine, it would remove all of the ones. So it would
just end up being nine as well. Now, let's write some print statements so we can have a little
pretty output. So previous tensor, this is what I like to do. This is a form of visualize, visualize,
visualize. If I'm trying to get my head around something, I print out each successive change
to see what's happening. That way, I can go, Oh, okay. So that's what it was there. And then I
called that line of code there. Yes, it's a bit tedious. But you do this half a dozen times, a
fair few times. I mean, I still do it a lot of the time, even though I've written thousands of lines
of machine learning code. But it starts to become instinct after a while, you start to go, Oh, okay,
I've got a dimension mismatch on my tensors. So I need to squeeze them before I put them into a
certain function. For a little while, but with practice, just like riding a bike, right? But that
try saying is like when you first start, you're all wobbly all over the place having to look up
the documentation, not that there's much documentation for riding a bike, you just kind of keep trying.
But that's the style of coding. I'd like you to adopt is to just try it first. Then if you're stuck,
go to the documentation, look something up, print it out like this, what we're doing,
quite cumbersome. But this is going to give us a good explanation for what's happening. Here's our
previous tensor x reshaped. And then if we look at the shape of x reshaped, it's one nine. And then
if we call the squeeze method, which removes all single dimensions from a target tensor,
we have the new tensor, which is has one square bracket removed. And the new shape is all single
dimensions removed. So it's still the original values, but just a different dimension. Now,
let's do the same as what we've done here with unsqueeze. So we've given our tensors a hug and
squeezed out all the single dimensions of them. Now we're going to unsqueeze them. We're going to
take a step back and let them grow a bit. So torch unsqueeze adds a single dimension

Section 34: Main Topics

Key Topics:

Now that's another thing to note in PyTorch whenever it says dim, that's dimension as in this is a zeroth dimension, first dimension
Remember, much of, and I'm going to spell color Australian style, much of deep learning is turning your data into numerical representations

📄 Click to view detailed content

to a target tensor at a specific dim dimension. Now that's another thing to note in PyTorch whenever
it says dim, that's dimension as in this is a zeroth dimension, first dimension. And if there
was more here, we'd go two, three, four, five, six, et cetera. Because why tensors can have
unlimited dimensions. So let's go previous target can be excused. So we'll get this squeezed version
of our tensor, which is x squeezed up here. And then we'll go print. The previous shape
is going to be x squeezed dot shape. And then we're going to add an extra dimension with unsqueeze.
There we go, x unsqueezed equals x squeezed. So our tensor before that we remove the single
dimension. And we're going to put in unsqueeze, dim, we'll do it on the zeroth dimension. And I
want you to have a think about what this is going to output even before we run the code.
Just think about, because we've added an extra dimension on the zeroth dimension,
what's the new shape of the unsqueeze tensor going to be? So we're going to go x unsqueezed.
And then we're going to go print, we'll get our new tensor shape, which is going to be x unsqueezed
dot shape. All right, let's have a look. There we go. So there's our previous tensor,
which is the squeezed version, just as a single dimension here. And then we have our new tensor,
which with the unsqueeze method on dimension zero, we've added a square bracket on the zeroth
dimension, which is this one here. Now what do you think's going to happen if I change this to one?
Where's the single dimension going to be added? Let's have a look. Ah, so instead of adding the
single dimension on the zeroth dimension, we've added it on the first dimension here. It's quite
confusing because Python is zero index. So I kind of want to my brain's telling me to say first,
but it's really the zeroth index here or the zeroth dimension. Now let's change this back to
zero. But that's just another way of exploring things. Every time there's like a parameter that
we have here, dim equals something like that could be shape, could be size, whatever, try
changing the values. That's what I'd encourage you to do. And even write some print code like
we've done here. Now there's one more we want to try out. And that's permute. So torch dot permute
rearranges the dimensions of a target tensor in a specified order. So if we wanted to check out,
let's get rid of some of these extra tabs. Torch dot permute. Let's have a look. This one took me
a little bit of practice to get used to. Because again, working with zeroth dimensions, even though
it seems like the first one. So returns a view. Okay. So we know that a view shares the memory of
the original input tensor with its dimensions permuted. So permuted for me, I didn't really know
what that word meant. I just have mapped in my own memory that permute means rearrange dimensions.
So the example here is we start with a random tensor, we check the size, and then we'd have
torch permute. We're going to swap the order of the dimensions. So the second dimension is first,
the zeroth dimension is in the middle, and the first dimension is here. So these are dimension
values. So if we have torch random two, three, five, two, zero, one has changed this one to be
over here. And then zero, one is two, three, and now two, three there. So let's try something similar
to this. So one of the common places you'll be using permute, or you might see permute being
used is with images. So there's a data specific data format. We've kind of seen a little bit
before, not too much. Original equals torch dot rand size equals. So an image tensor,
we go height width color channels on the end. So I'll just write this down. So this is height
width color channels. Remember, much of, and I'm going to spell color Australian style,
much of deep learning is turning your data into numerical representations. And this is quite common
numerical representation of image data. You have a tensor dimension for the height, a tensor dimension
for the width, and a tensor dimension for the color channels, which is red, green, and blue,
because a certain number of red, green, and blue creates almost any color. Now, if we want to
permute this, so permute the original tensor to rearrange the axis or dimension, axis or dimension,
are kind of used in the same light for tensors or dim order. So let's switch the color channels
to be the first or the zeroth dimension. So instead of height width color channels,
it'll be color channels height width. How would we do that with permute? Let's give it a shot.
X permuted equals X original dot permute. And we're going to take the second dimension,
because this takes a series of dims here. So the second dimension is color channels. Remember,
zero, one, two. So two, we want two first, then we want the height, which is a zero. And then we
want the width, which is one. And now let's do this shifts, axis, zero to one, one to two,
and two to zero. So this is the order as well. This two maps to zero. This zero maps to the first
index. This one maps to this index. But that's enough talk about it. Let's see what it looks like.
So print, previous shape, X original dot shape. And then we go here, print new shape. This will
be the permuted version. We want X permuted dot shape. Let's see what this looks like. Wonderful.

Section 35: Main Topics

Key Topics:

They're helping us fix shape and dimension issues with our tensors, which is one of the most common issues in deep learning and neural networks
And a view in PyTorch shares memory with that original tensor
So if you've ever done indexing, indexing, with PyTorch is similar to indexing with NumPy

📄 Click to view detailed content

That's exactly what we wanted. So you see, let's just write a little note here. Now this is
color channels, height, width. So the same data is going to be in both of these tenses. So X
original X permuted, it's just viewed from a different point of view. Because remember, a
permute is a view. And what did we discuss? A view shares the same memory as the original tensor.
So X permuted will share the same place in memory as X original, even though it's from a different
shape. So a little challenge before we move on to the next video for you, or before you move
on to the next video, try change one of the values in X original. Have a look at X original.
And see if that same value, it could be, let's get one of this zero, zero, get all of the dimensions
here, zero. See what that is? Or can we get a single value maybe? Oops. Oh, no, we'll need a zero
here, getting some practice on indexing here. Oh, zero, zero, zero. There we go. Okay, so maybe
we set that to some value, whatever you choose, and see if that changes in X permuted. So give
that a shot, and I'll see you in the next video. Welcome back. In the last video, we covered
squeezing, unsqueezing, and permuting, which I'm not going to lie, these concepts are quite a
lot to take in, but just so you're aware of them. Remember, what are they working towards? They're
helping us fix shape and dimension issues with our tensors, which is one of the most common
issues in deep learning and neural networks. And I usually do the little challenge of changing a
value of X original to highlight the fact that permute returns a different view of the original
tensor. And a view in PyTorch shares memory with that original tensor. So if we change the value
at zero, zero, zero of X original to, in my case, 728218, it happens the same value gets copied across
to X permuted. So with that being said, we looked at selecting data from tensors here, and this is
using a technique called indexing. So let's just rehash that, because this is another thing that
can be a little bit of a hurdle when first working with multi dimensional tensors. So let's see how
we can select data from tensors with indexing. So if you've ever done indexing, indexing,
with PyTorch is similar to indexing with NumPy. If you've ever worked with NumPy,
and you've done indexing, selecting data from arrays, NumPy uses an array as its main data type,
PyTorch uses tensors. It's very similar. So let's again start by creating a tensor.
And again, I'm just going to add a few code cells here, so I can make my screen right in the middle.
Now we're going to import torch. Again, we don't need to import torch all the time,
just so you can run the notebook from here later on. X equals torch dot. Let's create a range again,
just nice and simple. This is how I like to work out the fundamentals too, is just create the small
range, reshape it, and the reshape has to be compatible with the original dimension. So we go
one, three, three, and why is this because torch a range is going to return us nine values, because
it's from the start here to the end minus one, and then one times three times three is what is
nine. So let's have a look x x dot shape. Beautiful. So we have one, two, three, four, five, six,
seven, eight, nine of size one. So we have this is the outer bracket here, which is going to contain
all of this. And then we have three, which is this one here, one, two, three. And then we have three,
which is one, two, three. Now let's work with this. Let's index on our new tensor. So let's see what
happens when we get x zero, this is going to index on the first bracket. So we get this one here. So
we've indexed on the first dimension here, the zero dimension on this one here, which is why we get
what's inside here. And then let's try again, let's index on the middle bracket. So dimension
one. So we got to go x, and then zero, and then zero. Let's see what happens there. Now is this the
same as going x zero, zero? It is, there we go. So it depends on what you want to use. Sometimes
I prefer to go like this. So I know that I'm getting the first bracket, and then the zeroth
version of that first bracket. So then we have these three values here. Now what do you think
what's going to happen if we index on third dimension or the second dimension here? Well,
let's find out. So let's index on the most in our bracket, which is last dimension.
So we have x zero, zero, zero. What numbers is going to give us back of x zero,
on the zero dimension gives us back this middle tensor. And then if x zero, zero gives us back
the zeroth index of the middle tensor. If we go x zero, zero, zero is going to give us the zeroth
tensor, the zeroth index, and the zeroth element. A lot to take in there. But what we've done is
we've just broken it down step by step. We've got this first zero targets this outer bracket
and returns us all of this. And then zero, zero targets this first because of this first zero,
and then the zero here targets this. And then if we go zero, zero, zero, we target this,
then we target this, and then we get this back because we are getting the zeroth index here.

Section 36: Main Topics

Key Topics:

So now let's move on to the next part, which is PyTorch tensors and NumPy
PyTorch actually requires NumPy when you install PyTorch
And because of this, PyTorch has functionality to interact with it

📄 Click to view detailed content

So if we change this to one, what do we get back? Two. And if we change these all to one,
what will we get? This is a bit of trivia here, or a challenge. So we're going one, one, one.
Let's see what happens. Oh, no, did you catch that before I ran the code? I did that one quite
quickly. We have index one is out of bounds. Why is that? Well, because this dimension is only one
here. So we can only index on the zero. That's where it gets a little bit confusing because this
says one, but because it's only got zero dimension, we can only index on the zero if to mention. But
what if we do 011? What does that give us? Five. Beautiful. So I'd like to issue you the challenge
of how about getting number nine? How would you get number nine? So rearrange this code to get
number nine. That's your challenge. Now, I just want to show you as well, is you can use,
you can also use, you might see this, the semicolon to select all of a target dimension. So let's say
we wanted to get all of the zeroth dimension, but the zero element from that. We can get 123.
And then let's say we want to say get all values of the zeroth and first dimensions,
but only index one of the second dimension. Oh, that was a mouthful. But get all values of
zeroth and first dimensions, but only index one of second dimension. So let's break this
down step by step. We want all values of zeroth and first dimensions, but only index one of the
second dimension. We press enter, shift enter, 258. So what did we get there? 258. Okay. So we've
got all elements of the zeroth and first dimension, but then so which will return us this thing here.
But then we only want 258, which is the first element here of the second dimension, which is
this three there. So quite confusing. But with some practice, you can figure out how to select
almost any numbers you want from any kind of tensor that you have. So now let's try again,
get all values of the zero dimension, but only the one index value of the first and second
dimension. So what might this look like? Let's break it down again. So we come down here x,
and we're going to go all values of the zero dimension because zero comes first. And then we
want only the one index value of the first and only the one index value of the second.
What is this going to give us five? Oh, we selected the middle tensor. So really,
this line of code is exactly the same as this line of code here, except we've got the square
brackets on the outside here, because we've got this semicolon there. So if we change this to a zero,
we remove that. But because we've got the semicolon there, we've selected all the
dimensions. So we get back the square bracket there, something to keep in mind. Finally,
let's just go one more. So get index zero of zero and first dimension, and all values of second
dimension. So x zero, zero. So zero, the index of zero and first dimension, zero, zero,
and all values of the second dimension. What have we just done here? We've got tensor one,
two, three, lovely. This code again is equivalent to what we've done up here. This has a semicolon
on the end. But what this line explicitly says without the semicolon is, hey, give us all the
values on the remaining dimension there. So my challenge for you is to take this tensor that we
have got here and index on it to return nine. So I'll write down here, index on x to return nine.
So if you have a look at x, as well as index on x to return three, six, nine. So these values
here. So give those both a go and I'll see you in the next video. Welcome back. How'd you go?
Did you give the challenge ago? I finished the last video with issuing the challenge to index on
x to return nine and index on x to return three, six, nine. Now here's what I came up with. Again,
there's a few different ways that you could approach both of these. But this is just what
I've found. So because x is one, three, three of size, well, that's his dimensions. If we want to
select nine, we need zero, which is this first outer bracket to get all of these elements. And
then we need two to select this bottom one here. And then we need this final two to select the
second dimension of this bottom one here. And then for three, six, nine, we need all of the
elements in the first dimension, all of the in the zeroth dimension, all of the elements in the
first dimension. And then we get two, which is this three, six, nine set up here. So that's how I
would practice indexing, start with whatever shape tensor you like, create it something like this,
and then see how you can write different indexing to select whatever number you pick.
So now let's move on to the next part, which is PyTorch tensors and NumPy. So NumPy is a
popular scientific, very popular. PyTorch actually requires NumPy when you install PyTorch. Popular
scientific Python numerical computing library, that's a bit of a mouthful. And because of this,
PyTorch has functionality to interact with it. So quite often, you might start off with,
let's change this into Markdown, you might start off with your data, because it's numerical format,
you might start off with data in NumPy, NumPy array, want in PyTorch tensor. Because your

Section 37: Main Topics

Key Topics:

data might be represented by NumPy because it started in NumPy, but say you want to do some deep learning on it and you want to leverage PyTorch's deep learning capabilities, well, you might want to change your data from NumPy to a PyTorch tensor
And PyTorch has a method to do this, which is torch from NumPy, which will take in an ND array, which is NumPy's main data type, and change it into a torch tensor
And then if you want to go from PyTorch tensor to NumPy because you want to use some sort of NumPy method, well, the method to do this is torch dot tensor, and you can call dot NumPy on it

📄 Click to view detailed content

data might be represented by NumPy because it started in NumPy, but say you want to do
some deep learning on it and you want to leverage PyTorch's deep learning capabilities,
well, you might want to change your data from NumPy to a PyTorch tensor. And PyTorch has a
method to do this, which is torch from NumPy, which will take in an ND array, which is NumPy's
main data type, and change it into a torch tensor. We'll see this in a second. And then if you want
to go from PyTorch tensor to NumPy because you want to use some sort of NumPy method,
well, the method to do this is torch dot tensor, and you can call dot NumPy on it. But this is all
just talking about in words, let's see it in action. So NumPy array to tensor. Let's try this out
first. So we'll import torch so we can run this cell on its own, and then import NumPy as np,
the common naming convention for NumPy, we're going to create an array in NumPy. And we're
going to just put one to eight, a range. And then we're going to go tensor equals torch from NumPy
because we want to go from NumPy array to a torch tensor. So we use from NumPy, and then we pass
in array, and then we have array and tensor. Wonderful. So there's our NumPy array, and our torch
tensor with the same data. But what you might notice here is that the D type for the tensor is
torch dot float 64. Now why is this? It's because NumPy's default data type. Oh, D type
is float 64. Whereas tensor, what have we discussed before? What's pytorch's default data type?
float 64. Well, that's not pytorch's default data type. If we were to create torch, a range,
1.0 to 8.0, by default, pytorch is going to create it in
float 32. So just be aware of that. If you are going from NumPy to pytorch, the default NumPy
data type is float 64. And pytorch reflects that data type when you use the from NumPy method.
I wonder if there's a D type. Can we go D type equals torch dot float 32? Takes no keyword.
Okay. But how could we change the data type here? Well, we could go type torch float 32.
Yeah, that will give us a tensor D type of float 32 instead of float 64. Beautiful. I'll just keep
that there so you know, warning when converting from NumPy pytorch, pytorch reflects NumPy's
default data type of float 64, unless specified. Otherwise, because what have we discussed,
when you're trying to perform certain calculations, you might run into a data type issue. So you might
need to convert the type from float 64 to float 32. Now, let's see what happens. What do you think
will happen if we change the array? We change the value of an array. Well, let's find out.
So change the value of array. The question is, what will this do to tensor? Because we've used
the from NumPy method, do you think if we change the array, the tensor will change? So let's try
this array equals array plus one. So we're just adding one to every value in the array. Now,
what is the array and the tensor going to look like? Uh huh. So array, we only change the first
value there. Oh, sorry, we change every value because we have one to seven. Now it's two, three,
four, five, six, seven, eight. We change the value from the array. It doesn't change the
value of the tensor. So that's just something to keep in mind. If you use from NumPy, we get
a new tensor in memory here. So the original, the new tensor doesn't change if you change the
original array. So now let's go from tensor to NumPy. If you wanted to go back to NumPy,
tensor to NumPy array. So we'll start with a tensor. We could use the one we have right now,
but we're going to create another one, but we'll create one of ones just for fun.
One rhymes with fun. NumPy tensor equals. How do we go to NumPy? Well, we have
torch dot tensor dot NumPy. So we just simply call NumPy on here. And then we have tensor
and NumPy tensor. What data type do you think the NumPy tensor is going to have?
Because we've returned it to NumPy. Pi torches, default data type is
Flight 32. So if we change that to NumPy, what's going to be the D type of the NumPy tensor?
NumPy tensor dot D type. It reflects the original D type of what you set the tensor as. So just
keep that in mind. If you're going between PyTorch and NumPy, default data type of NumPy is
float 64, whereas the default data type of PyTorch is float 32. So that may cause some errors if
you're doing different kinds of calculations. Now, what do you think is going to happen if we
went from our tensor to an array, if we change the tensor, change the tensor, what happens to
NumPy tensor? So we get tensor equals tensor plus one. And then we go NumPy tensor.
Oh, we'll get tensor as well. So our tensor is now all twos because we added one to the ones.
But our NumPy tensor remains the same. Remains unchanged. So this means they don't share memory.
So that's how we go in between PyTorch and NumPy. If you'd like to look up more, I'd encourage
you to go PyTorch and NumPy. So warm up NumPy, beginner. There's a fair few tutorials here on
PyTorch because NumPy is so prevalent, they work pretty well together. So have a look at that.
There's a lot going on there. There's a few more links, I'd encourage you to check out,

Section 38: Main Topics

Key Topics:

If you'd like to look that up, I'd encourage you to search PyTorch's reproducibility and see what you can find
But what if you were trying to share this notebook with a friend, so say you went up share and you clicked the share link and you sent that to someone and you're like, hey, try out this machine learning experiment I did
And pytorch comes the concept of a random seed

📄 Click to view detailed content

but we've covered some of the main ones that you'll see in practice. With that being said,
let's now jump into the next video where we're going to have a look at the concept of reproducibility.
If you'd like to look that up, I'd encourage you to search PyTorch's reproducibility and see
what you can find. Otherwise, I'll see you in the next video. Welcome back. It's now time for us
to cover the topic of reproducibility. If I could even spell it, that would be fantastic.
Reproducibility. Trying to take the random out of random. So we've touched upon the concept of
neural networks harnessing the power of randomness. And what I mean by that is we haven't actually
built our own neural network yet, but we will be doing that. And we've created tenses full of random
values. And so in short, how our neural network learns is start with random numbers, perform tensor
operations, update random numbers to try and make them better representations of the data. Again,
again, again, again, again. However, if you're trying to do reproducible experiments, sometimes
you don't want so much randomness. And what I mean by this is if we were creating random tensors,
from what we've seen so far is that every time we create a random tensor, let's create one here,
torch dot rand, and we'll create it of three three. Every time we run this cell, it gives us new numbers.
So 7 7 5 2. There we go. Rand again. Right. So we get a whole bunch of random numbers here.
Every single time. But what if you were trying to share this notebook with a friend,
so say you went up share and you clicked the share link and you sent that to someone and you're like,
hey, try out this machine learning experiment I did. And you wanted a little less randomness
because neural networks start with random numbers. How might you do that? Well, let's
this write down to reduce the randomness in neural networks. And pytorch comes the concept of a
random seed. So we're going to see this in action. But essentially, let's write this down,
essentially what the random seed does is flavor the randomness. So because of how computers work,
they're actually not true randomness. And actually, there's arguments against this,
and it's quite a big debate in the computer science topic, whatnot, but I am not a computer
scientist, I am a machine learning engineer. So computers are fundamentally deterministic.
It means they run the same steps over and over again. So what the randomness we're doing here
is referred to as pseudo randomness or generated randomness. And the random seed,
which is what you see a lot in machine learning experiments, flavors that randomness. So let's
see it in practice. And at the end of this video, I'll give you two resources that I'd recommend
to learn a little bit more about the concept of pseudo randomness and reproducibility in pytorch.
Let's start by importing torch so you could start this notebook right from here. Create two random
tensors. We'll just call this random tensor a equals torch dot rand and we'll go three four
and we'll go random tensor b equals torch dot rand same size three four. And then if we have a
look at let's go print random tensor a print random tensor b. And then let's print to see if
they're equal anywhere random tensor a equals equals equals random tensor b. Now what do you
think this is going to do? If we have a look at one equals one, what does it return? True.
So this is comparison operator to compare two different tensors. We're creating two random
tensors here. We're going to have a look at them. We'd expect them to be full of random values.
Do you think any of the values in each of these random tensors is going to be equal to each other?
Well, there is a chance that they are, but it's highly unlikely. I'll be quite surprised if they are.
Oh, again, my connection might be a little bit. Oh, there we go. Beautiful. So we have tensor a
tensor of three four with random numbers. And we have tensor b of three four with random numbers.
So if we were, if I was to share this notebook with my friend or my colleague or even you,
if you ran this cell, you are going to get random numbers as well. And you have every chance of
replicating one of these numbers. But again, it's highly unlikely. So again, I'm getting that
automatic save failed. You might get that if your internet connection is dropping out, maybe that's
something going on with my internet connection. But again, as we've seen, usually this resolves
itself. If you try a few times, I'll just keep coding. If it really doesn't resolve itself,
you can go file is a download notebook or save a copy and drive download. You can download the
notebook, save it to your local machine, re upload it to upload notebook and start again in another
Google Colab instance. But there we go. It fixed itself. Wonderful troubleshooting on the fly.
So the way we make these reproducible is through the concept of a random seed. So let's have a
look at that. Let's make some random, but reproducible tenses. So import torch. And we're going to
set the random seed by going torch dot manual seed random. Oh, we don't have random set yet.
I'm going to set my random seed. You set the random seed to some numerical value. 42 is a common
one. You might see zero. You might see one, two, three, four. Essentially, you can set it to whatever
you want. And each of these, you can think of 77, 100, as different flavors of randomness. So

Section 39: Main Topics

Key Topics:

We're telling pytorch a flavor our randomness with 42 torch manual seed
So this is your extra curriculum for this, even if you don't understand what's going on in a lot of the code here, just be aware of reproducibility, because it's an important topic in machine learning and deep learning
You can complete this whole course on the free tier as well

📄 Click to view detailed content

I like to use 42, because it's the answer to the universe. And then we go random seed. And now
let's create some random tenses. Random tensor C with the flavor of our random seed. Three,
four. And then we're going to go torch tensor D equals torch dot rand three, four. Now, let's
see what happens. We'll print out random tensor C. And we'll print out random tensor D. And then
we'll print out to see if they're equal anywhere. Random tensor C equals random tensor D. So let's
find out what happens. Huh, what gives? Well, we've got randomness. We set the random seed. We're
telling pytorch a flavor our randomness with 42 torch manual seed. Hmm, let's try set the manual
seed each time we call a random method. We go there. Ah, much better. So now we've got some
flavored randomness. So a thing to keep in mind is that if you want to use the torch manual seed,
generally it only works for one block of code if you're using a notebook. So that's just
something to keep in mind. If you're creating random tensors, one after the other, we're using
assignment like this, you should use torch dot manual seed every time you want to call the rand
method or some sort of randomness. However, if we're using other torch processes, usually what
you might see is torch manual seed is set right at the start of a cell. And then a whole bunch
of code is done down here. But because we're calling subsequent methods here, we have to reset
the random seed. Otherwise, if we don't do this, we comment this line, it's going to flavor the
randomness of torch random tensor C with torch manual seed. But then random tensor D is just
going to have no flavor. It's not going to use a random seed. So we reset it there. Wonderful.
So I wonder, does this have a seed method? Let's go torch dot rand. Does this have seed?
Sometimes they have a seed method. Seed, no, it doesn't. Okay, that's all right.
The more you learn, but there's documentation for torch dot rand. And I said that I was going to
link at the end of this video. So the manual seed is a way to, or the random seed, but in
torch, it's called a manual seed is a way to flavor the randomness. So these numbers, as you see,
are still quite random. But the random seed just makes them reproducible. So if I was to share this
with you, if you had to run this block of code, ideally, you're going to get the same numerical
output here. So with that being said, I'd like to refer to you to the pie torch reproducibility
document, because we've only quite scratched the surface of this of reproducibility. We've covered
one of the main ones. But this is a great document on how to go through reproducibility in pie torch.
So this is your extra curriculum for this, even if you don't understand what's going on in a lot
of the code here, just be aware of reproducibility, because it's an important topic in machine
learning and deep learning. So I'll put this here, extra resources for reproducibility.
As we go pie torch randomness, we'll change this into markdown. And then finally, the concept
of a random seed is Wikipedia random seed. So random seeds quite a universal concept,
not just for pie torch, there's a random seed and NumPy as well. So if you'd like to see what
this means, yeah, initialize a pseudo random number generator. So that's a big word, pseudo random
number generator. But if you'd like to learn about more random number generation computing,
and what a random seed does is I'd refer to you to check out this documentation here.
Whoo, far out, we have covered a lot. But there's a couple more topics you should really be aware
of to finish off the pie torch fundamentals. You got this. I'll see you in the next video.
Welcome back. Now, let's talk about the important concept of running tenses or pie
torch objects. So running tenses and pie torch objects on GPUs and making faster computations.
So we've discussed that GPUs, let me just scroll down a little bit here, GPUs equal faster
computation on numbers. Thanks to CUDA plus NVIDIA hardware plus pie torch working behind the
scenes to make everything hunky dory. Good. That's what hunky dory means, by the way,
if you never heard that before. So let's have a look at how we do this. Now, we first need to
talk about, let's go here one getting a GPU. There's a few different ways we've seen one before.
Number one easiest is to use what we're using right now. Use Google Colab for a free GPU.
But there's also Google Colab Pro. And I think there might even be, let's look up Google Colab
Pro. Choose the best that's right for you. I use Google Colab Pro because I use it almost every day.
So yeah, I pay for Colab Pro. You can use Colab for free, which is might be what you're using.
There's also Colab Pro Plus, which has a lot more advantages as well. But Colab Pro is giving me
faster GPUs, so access to faster GPUs, which means you spend less time waiting while your code is running.
More memory, longer run time, so it'll last a bit longer if you leave it running idle.
And then Colab Pro again is a step up from that. I personally haven't had a need yet to use
Google Colab Pro Plus. You can complete this whole course on the free tier as well. But as you start
to code more, as you start to run bigger models, as you start to want to compute more, you might

Section 40: Main Topics

Key Topics:

So one of my favorite posts for getting a GPU is, yeah, the best GPUs for deep learning in 2020, or something like this
Deep learning
This is, yeah, which GPUs to get for deep learning

📄 Click to view detailed content

want to look into something like Google Colab Pro. Or let's go here. Options to upgrade as well.
And then another way is use your own GPU. Now this takes a little bit of setup and requires
the investment of purchasing a GPU. There's lots of options. So one of my favorite posts for
getting a GPU is, yeah, the best GPUs for deep learning in 2020, or something like this.
What do we got? Deep learning? Tim Detmos. This is, yeah, which GPUs to get for deep learning?
Now, I believe at the time of this video, I think it's been updated since this date. So don't take
my word for it. But this is a fantastic blog post for figuring out what GPUs see this post
for what option to get. And then number three is use cloud computing. So such as
GCP, which is Google Cloud Platform AWS, which is Amazon Web Services or Azure.
These services, which is Azure is by Microsoft, allow you to rent computers on the cloud and access
them. So the first option using Google Colab, which is what we're using is by far the easiest
and free. So there's big advantages there. However, the downside is that you have to use a website
here, Google Colab, you can't run it locally. You don't get the benefit of using cloud computing,
but my personal workflow is I run basically all of my small scale experiments and things like
learning new stuff in Google Colab. And then if I want to upgrade things, run video experiments,
I have my own dedicated deep learning PC, which I have built with a big powerful GPU. And then
also I use cloud computing if necessary. So that's my workflow. Start with Google Colab.
And then these two, if I need to do some larger experiments. But because this is the beginning
of course, we can just stick with Google Colab for the time being. But I thought I'd make you aware
of these other two options. And if you'd like to set up a GPU, so four, two, three, PyTorch plus
GPU drivers, which is CUDA takes a little bit of setting up to do this, refer to PyTorch
setup documentation. So if we go to pytorch.org, they have some great setup guides here,
get started. And we have start locally. This is if you want to run on your local machine,
such as a Linux setup. This is what I have Linux CUDA 11.3. It's going to give you a
conda install command to use conda. And then if you want to use cloud partners, which is Alibaba
Cloud, Amazon Web Services, Google Cloud Platform, this is where you'll want to go. So I'll just link
this in here. But for this course, we're going to be focusing on using Google Colab. So now,
let's see how we might get a GPU in Google Colab. And we've already covered this, but I'm going to
recover it just so you know. We're going to change the runtime type. You can go in any notebook and
do this, runtime type, hardware accelerator, we can select GPU, click save. Now this is going to
restart our runtime and connect us to our runtime, aka a Google compute instance with a GPU. And so
now if we run NVIDIA SMI, I have a Tesla P100 GPU. So let's look at this Tesla P100
GPU. Do we have an image? Yeah, so this is the GPU that I've got running, not the Tesla car,
the GPU. So this is quite a powerful GPU. That is because I have upgraded to Colab Pro. Now,
if you're not using Colab Pro, you might get something like a Tesla K80, which is a slightly
less powerful GPU than a Tesla P100, but still a GPU nonetheless and will still work faster than
just running PyTorch code on the pure CPU, which is the default in Google Colab and the default
in PyTorch. And so now we can also check to see if we have GPU access with PyTorch. So let's go
here. This is number two now. Check for GPU access with PyTorch. So this is a little command that's
going to allow us or tell us if PyTorch, just having the GPU here, this is by the way, another
thing that Colab has a good setup with, is that all the connections between PyTorch and the NVIDIA
GPU are set up for us. Whereas when you set it up on your own GPU or using cloud computing,
there are a few steps you have to go through, which we're not going to cover in this course.
I'd highly recommend you go through the getting started locally set up if you want to do that,
to connect PyTorch to your own GPU. So let's check for the GPU access with PyTorch.
This is another advantage of using Google Colab. Almost zero set up to get started. So import
torch and then we're going to go torch dot cuda dot is available. And remember, cuda is
NVIDIA's programming interface that allows us to use GPUs for numerical computing. There we go,
beautiful. So big advantage of Google Colab is we get access to a free GPU. In my case, I'm paying
for the faster GPU, but in your case, you're more than welcome to use the free version.
All that means it'll be slightly slower than a faster GPU here. And we now have access to GPUs
with PyTorch. So there is one more thing known as device agnostic code. So set up device agnostic
code. Now, this is an important concept in PyTorch because wherever you run PyTorch, you might not
always have access to a GPU. But if there was access to a GPU, you'd like it to use it if it's
available. So one of the ways that this is done in PyTorch is to set the device variable. Now,

Section 41: Main Topics

Key Topics:

If it's not available, if we don't have access to a GPU that PyTorch can use, just default to the CPU
But as you upgrade your PyTorch experiments and machine learning experiments, you might have access to more than one GPU
But final thing before we finish this video is if we go PyTorch device agnostic code, cuda semantics, there's a little section in here called best practices

📄 Click to view detailed content

really, you could set this to any variable you want, but you're going to see it used as device
quite often. So cuda if torch dot cuda is available. Else CPU. So all this is going to say, and we'll
see where we use the device variable later on is set the device to use cuda if it's available. So
it is so true. If it's not available, if we don't have access to a GPU that PyTorch can use,
just default to the CPU. So with that being said, there's one more thing. You can also count the
number of GPUs. So this won't really apply to us for now because we're just going to stick with
using one GPU. But as you upgrade your PyTorch experiments and machine learning experiments,
you might have access to more than one GPU. So you can also count the devices here.
We have access to one GPU, which is this here. So the reason why you might want to count the number
of devices is because if you're running huge models on large data sets, you might want to run one
model on a certain GPU, another model on another GPU, and so on and so on. But final thing before
we finish this video is if we go PyTorch device agnostic code, cuda semantics, there's a little
section in here called best practices. This is basically what we just covered there is setting
the device argument. Now this is using the arg pass, but so yeah, there we go. args.device,
torch.device, cuda, args.device, torch.device, CPU. So this is one way to set it from the Python
arguments when you're running scripts, but we're using the version of running it through a notebook.
So check this out. I'll just link this here, device agnostic code. It's okay if you're not sure
of what's going on here. We're going to cover it a little bit more later on throughout the course,
but right here for PyTorch, since it's capable of running compute on the GPU or CPU,
it's best practice to set up device agnostic code, e.g. run on GPU if available,
else default to CPU. So check out the best practices for using cuda, which is namely setting up
device agnostic code. And let's in the next video, see what I mean about setting our PyTorch tensors
and objects to the target device. Welcome back. In the last video, we checked out a few different
options for getting a GPU, and then getting PyTorch to run on the GPU. And for now we're using
Google Colab, which is the easiest way to get set up because it gives us free access to a GPU,
faster ones if you set up with Colab Pro, and it comes with PyTorch automatically set up to
use the GPU if it's available. So now let's see how we can actually use the GPU. So to do so,
we'll look at putting tensors and models on the GPU. So the reason we want our tensors slash models
on the GPU is because using GPU results in faster computations. And if we're getting our machine
learning models to find patterns and numbers, GPUs are great at doing numerical calculations.
And the numerical calculations we're going to be doing are tensor operations like we saw above.
So the tensor operations, well, we've covered a lot. Somewhere here, tensor operations,
there we go, manipulating tensor operations. So if we can run these computations faster,
we can discover patterns in our data faster, we can do more experiments, and we can work towards
finding the best possible model for whatever problem that we're working on. So let's see,
we'll create a tensor, as usual, create a tensor. Now the default is on the CPU.
So tensor equals torch dot tensor. And we'll just make it a nice simple one, one, two, three.
And let's write here, tensor not on GPU will print out tensor. And this is where we can use,
we saw this parameter before device. Can we pass it in here? Device equals CPU.
Let's see what this comes out with. There we go. So if we print it out, tensor 123 is on the CPU.
But even if we got rid of that device parameter, by default, it's going to be on the CPU. Wonderful.
So now PyTorch makes it quite easy to move things to, and I'm saying to for a reason,
to the GPU, or to, even better, the target device. So if the GPU is available, we use CUDA.
If it's not, it uses CPU. This is why we set up the device variable. So let's see,
move tensor to GPU. If available,
tensor on GPU equals tensor dot two device. Now let's have a look at this, tensor on GPU.
So this is going to shift the tensor that we created up here to the target device.
Wonderful. Look at that. So now our tensor 123 is on device CUDA zero. Now this is the index of
the GPU that we're using, because we only have one, it's going to be at index zero. So later on,
when you start to do bigger experiments and work with multiple GPUs, you might have different tensors
that are stored on different GPUs. But for now, we're just sticking with one GPU, keeping it nice
and simple. And so you might have a case where you want to move, oh, actually, the reason why we
set up device agnostic code is again, this code would work if we run this, regardless if we had,
so it won't error out. But regardless if we had a GPU or not, this code will work. So whatever device
we have access to, whether it's only a CPU or whether it's a GPU, this tensor will move to whatever
target device. But since we have a GPU available, it goes there. You'll see this a lot. This two

Section 42: Main Topics

Key Topics:

Remember the top three errors in deep learning or pytorch
And with pytorch, number three is device issues
That's a beautiful thing about pytorch is very helpful error messages

📄 Click to view detailed content

method moves tensors and it can be also used for models. We're going to see that later on. So just
keep two device in mind. And then you might want to, for some computations, such as using NumPy,
NumPy only works with the CPU. So you might want to move tensors back to the CPU, moving tensors back
to the CPU. So can you guess how we might do that? It's okay if you don't know. We haven't covered a
lot of things, but I'm going to challenge you anyway, because that's the fun part of thinking
about something. So let's see how we can do it. Let's write down if tensor is on GPU, can't transform
it to NumPy. So let's see what happens if we take our tensor on the GPU and try to go NumPy.
What happens? Well, we get an error. So this is another huge error. Remember the top three
errors in deep learning or pytorch? There's lots of them, but number one, shape errors,
number two, data type issues. And with pytorch, number three is device issues. So can't convert
CUDA zero device type tensor to NumPy. So NumPy doesn't work with the GPU. Use tensor dot CPU
to copy the tensor to host memory first. So if we call tensor dot CPU, it's going to bring our
target tensor back to the CPU. And then we should be able to use it with NumPy. So
to fix the GPU tensor with NumPy issue, we can first set it to the CPU. So tensor back on CPU
equals tensor on GPU dot CPU. We're just taking what this said here. That's a beautiful thing
about pytorch is very helpful error messages. And then we're going to go NumPy.
And then if we go tensor back on CPU, is this going to work? Let's have a look. Oh, of course,
it's not because I typed it wrong. And I've typed it again twice. Third time, third time's a charm.
There we go. Okay, so that works because we've put it back to the CPU first before calling NumPy.
And then if we refer back to our tensor on the GPU, because we've reassociated this, again,
we've got typos galore classic, because we've reassigned tensor back on CPU, our tensor on
GPU remains unchanged. So that's the four main things about working with pytorch on the GPU.
There are a few more tidbits such as multiple GPUs, but now you've got the fundamentals. We're
going to stick with using one GPU. And if you'd like to later on once you've learned a bit more
research into multiple GPUs, well, as you might have guessed, pytorch has functionality for that too.
So have a go at getting access to a GPU using colab, check to see if it's available, set up device
agnostic code, create a few dummy tensors and just set them to different devices, see what happens
if you change the device parameter, run a few errors by trying to do some NumPy calculations
with tensors on the GPU, and then bring those tensors on the GPU back to NumPy and see what happens
there. So I think we've covered, I think we've reached the end of the fundamentals. We've covered
a fair bit. Introduction to tensors, the minmax, a whole bunch of stuff inside the introduction
to tensors, finding the positional minmax, reshaping, indexing, working with tensors and NumPy,
reproducibility, using a GPU and moving stuff back to the GPU far out. Now you're probably wondering,
Daniel, we've covered a whole bunch. What should I do to practice all this? Well, I'm glad you asked.
Let's cover that in the next video. Welcome back. And you should be very proud of your
self right now. We've been through a lot, but we've covered a whole bunch of PyTorch fundamentals.
These are going to be the building blocks that we use throughout the rest of the course.
But before moving on to the next section, I'd encourage you to try out what you've learned
through the exercises and extra curriculum. Now, I've set up a few exercises here based off
everything that we've covered. If you go into learn pytorch.io, go to the section that we're
currently on. This is going to be the case for every section, by the way. So just keep this in mind,
is we're working on PyTorch fundamentals. Now, if you go to the PyTorch fundamentals notebook,
this is going to refresh, but that if you scroll down to the table of contents at the bottom of
each one is going to be some exercises and extra curriculum. So these exercises here,
such as documentation reading, because a lot you've seen me refer to the PyTorch documentation
for almost everything we've covered a lot, but it's important to become familiar with that.
So exercise number one is read some of the documentation. Exercise number two is create a
random tensor with shape, seven, seven. Three, perform a matrix multiplication on the tensor from two
with another random tensor. So these exercises are all based off what we've covered here.
So I'd encourage you to reference what we've covered in whichever notebook you choose,
could be this learn pytorch.io, could be going back through the one we've just coded together
in the video. So I'm going to link this here, exercises, see exercises for this notebook here.
So then how should you approach these exercises? So one way would be to just read them here,
and then in collab we'll go file new notebook, wait for the notebook to load. Then you could call this
zero zero pytorch exercises or something like that, and then you could start off by importing
torch, and then away you go. For me, I'd probably set this up on one side of the screen, this one
up on the other side of the screen, and then I just have the exercises here. So number one,

Section 43: Main Topics

Key Topics:

As we go throughout the course, these exercises are going to get a little bit more in depth as we've learned more
This is the home for all of the course materials
So pytorch fundamentals exercises

📄 Click to view detailed content

I'm not going to really write much code for that, but you could have documentation reading here.
And then so this encourages you to read through torch.tensor and go through there
for 10 minutes or so. And then for the other ones, we've got create a random tensor with shape
seven seven. So we just comment that out. So torch, round seven seven, and there we go.
Some are as easy as that. Some are a little bit more complex. As we go throughout the course,
these exercises are going to get a little bit more in depth as we've learned more.
But if you'd like an exercise template, you can come back to the GitHub. This is the home for all
of the course materials. You can go into extras and then exercises. I've created templates for
each of the exercises. So pytorch fundamentals exercises. If you open this up, this is a template
for all of the exercises. So you see there, create a random tensor with shape seven seven.
These are all just headings. And if you'd like to open this in CoLab and work on it,
how can you do that? Well, you can copy this link here. Come to Google CoLab. We'll go file,
open notebook, GitHub. You can type in the link there. Click search. What's this going to do?
Boom. Pytorch fundamentals exercises. So now you can go through all of the exercises. This
will be the same for every module on the course and test your knowledge. Now it is open book. You
can use the notebook here, the ones that we've coded together. But I would encourage you to try
to do these things on your own first. If you get stuck, you can always reference back. And then
if you'd like to see an example solutions, you can go back to the extras. There's a solutions folder
as well. And that's where the solutions live. So the fundamental exercise solutions. But again,
I would encourage you to try these out, at least give them a go before having a look at the solutions.
So just keep that in mind at the end of every module, there's exercises and extra curriculum.
The exercises will be code based. The extra curriculum is usually like reading based.
So spend one hour going through the Pytorch basics tutorial. I recommend the quick start
in tensor sections. And then finally to learn more on how a tensor can represent data,
watch the video what's a tensor which we referred to throughout this. But massive effort on finishing
the Pytorch fundamentals section. I'll see you in the next section. Friends, welcome back to
the Pytorch workflow module. Now let's have a look at what we're going to get into.
So this is a Pytorch workflow. And I say a because it's one of many. When you get into
deep learning machine learning, you'll find that there's a fair few ways to do things. But here's
the rough outline of what we're going to do. We're going to get our data ready and turn it into
tensors because remember a tensor can represent almost any kind of data. We're going to pick or
build or pick a pre-trained model. We'll pick a loss function and optimize it. Don't worry if
you don't know what they are. We're going to cover this. We're going to build a training loop,
fit the model to make a prediction. So fit the model to the data that we have. We'll learn how
to evaluate our models. We'll see how we can improve through experimentation and we'll save
and reload our trained model. So if you wanted to export your model from a notebook and use it
somewhere else, this is what you want to be doing. And so where can you get help? Probably the most
important thing is to follow along with the code. We'll be coding all of this together.
Remember model number one. If and out, run the code. Try it for yourself. That's how I learn best.
Is I write code? I try it. I get it wrong. I try again and keep going until I get it right.
Read the doc string because that's going to show you some documentation about the functions that
we're using. So on a Mac, you can use shift command and space in Google Colab or if you're on a Windows
PC, it might be control here. If you're still stuck, try searching for it. You'll probably come
across resources such as stack overflow or the PyTorch documentation. We've already seen this
a whole bunch and we're probably going to see it a lot more throughout this entire course actually
because that's going to be the ground truth of everything PyTorch. Try again. And finally,
if you're still stuck, ask a question. So the best place to ask a question will be
at the PyTorch deep learning slash discussions tab. And then if we go to GitHub,
that's just under here. So Mr. Deeburg PyTorch deep learning. This is all the course materials.
We see here, this is your ground truth for the entire course. And then if you have a question,
go to the discussions tab, new discussion, you can ask a question there. And don't forget to
please put the video and the code that you're trying to run. That way we can reference
what's going on and help you out there. And also, don't forget, there is the book version of the
course. So learn pytorch.io. By the time you watch this video, it'll probably have all the chapters
here. But here's what we're working through. This is what the videos are based on. All of this,
we're going to go through all of this. How fun is that? But this is just reference material.
So you can read this at your own time. We're going to focus on coding together. And speaking of coding.

Section 44: Main Topics

Key Topics:

And now I'm going to title this 01 pytorch workflow
Because in the course resources, we have the original notebook here, which is what this video notebook is going to be based off
And then of course, you've got the book version of the notebook as well, which is just a different formatted version of this exact same notebook

📄 Click to view detailed content

Let's code. I'll see you over at Google Colab.
Oh, right. Well, let's get hands on with some code. I'm going to come over to colab.research.google.com.
You may already have that bookmark. And I'm going to start a new notebook. So we're going to do
everything from scratch here. We'll let this load up. I'm just going to zoom in a little bit.
Beautiful. And now I'm going to title this 01 pytorch workflow. And I'm going to put the video
ending on here so that you know that this notebook's from the video. Why is that? Because in the
course resources, we have the original notebook here, which is what this video notebook is going
to be based off. You can refer to this notebook as reference for what we're going to go through.
It's got a lot of pictures and beautiful text annotations. We're going to be focused on the
code in the videos. And then of course, you've got the book version of the notebook as well,
which is just a different formatted version of this exact same notebook. So I'm going to link
both of these up here. So let's write in here, pytorch workflow. And let's explore an example,
pytorch end to end workflow. And then I'm going to put the resources. So ground truth notebook.
We go here. And I'm also going to put the book version.
Book version of notebook. And finally, ask a question, which will be where at the discussions
page. Then we'll go there. Beautiful. Let's turn this into markdown. So let's get started. Let's
just jump right in and start what we're covering. So this is the trend I want to start getting
towards is rather than spending a whole bunch of time going through keynotes and slides,
I'd rather we just code together. And then we explain different things as they need to be
explained because that's what you're going to be doing if you end up writing a lot of pytorch is
you're going to be writing code and then looking things up as you go. So I'll get out of these
extra tabs. I don't think we need them. Just these two will be the most important. So what we're
covering, let's create a little dictionary so we can check this if we wanted to later on.
So referring to our pytorch workflows, at least the example one that we're going to go through,
which is just here. So we're going to go through all six of these steps, maybe a little bit of
each one, but just to see it going from this to this, that's what we're really focused on. And then
we're going to go through through rest the course like really dig deep into all of these. So what
we're covering number one is data preparing and loading. Number two is we're going to see how we
can build a machine learning model in pytorch or a deep learning model. And then we're going
to see how we're going to fit our model to the data. So this is called training. So fit is another
word. As I said in machine learning, there's a lot of different names for similar things,
kind of confusing, but you'll pick it up with time. So we're going to once we've trained a model,
we're going to see how we can make predictions and evaluate those predictions,
evaluating a model. If you make predictions, it's often referred to as inference. I typically
say making predictions, but inference is another very common term. And then we're going to look
at how we can save and load a model. And then we're going to put it all together. So a little bit
different from the visual version we have of the pytorch workflow. So if we go back to here,
I might zoom in a little. There we go. So we're going to focus on this one later on,
improve through experimentation. We're just going to focus on the getting data ready,
building a model, fitting the model, evaluating model, save and reload. So we'll see this one more,
like in depth later on, but I'll hint at different things that you can do
for this while we're working through this workflow. And so let's put that in here.
And then if we wanted to refer to this later, we can just go what we're covering.
Oh, this is going to connect, of course. Beautiful. So we can refer to this later on,
if we wanted to. And we're going to start by import torch. We're going to get pytorch ready
to go import nn. So I'll write a note here. And then we haven't seen this one before, but
we're going to see a few things that we haven't seen, but that's okay. We'll explain it as we go.
So nn contains all of pytorch's building blocks for neural networks.
And how would we learn more about torch nn? Well, if we just go torch.nn, here's how I'd
learn about it, pytorch documentation. Beautiful. Look at all these. These are the basic building
blocks for graphs. Now, when you see the word graph, it's referring to a computational graph,
which is in the case of neural networks, let's look up a photo of a neural network.
Images, this is a graph. So if you start from here, you're going to go towards the right.
There's going to be many different pictures. So yeah, this is a good one. Input layer. You have
a hidden layer, hidden layer to output layer. So torch and n comprises of a whole bunch of
different layers. So you can see layers, layers, layers. And each one of these, you can see input
layer, hidden layer one, hidden layer two. So it's our job as data scientists and machine
learning engineers to combine these torch dot nn building blocks to build things such as these.

Section 45: Main Topics

Key Topics:

Now, it might not be exactly like this, but that's the beauty of pytorch is that you can combine these in almost any different way to build any kind of neural network you can imagine
And let's check our pytorch version
Pytorch version torch dot version

📄 Click to view detailed content

Now, it might not be exactly like this, but that's the beauty of pytorch is that you can
combine these in almost any different way to build any kind of neural network you can imagine.
And so let's keep going. That's torch nn. We're going to get hands on with it,
rather than just talk about it. And we're going to need map plot lib because what's our other
motto? Our data explorers motto is visualize, visualize, visualize. And let's check our pytorch
version. Pytorch version torch dot version. So this is just to show you you'll need
at least this version. So 1.10 plus CUDA 111. That means that we've got CU stands for CUDA.
That means we've got access to CUDA. We don't have a GPU on this runtime yet,
because we haven't gone to GPU. We might do that later.
So if you have a version that's lower than this, say 1.8, 0.0, you'll want pytorch 1.10 at least.
If you have a version higher than this, your code should still work. But that's about enough
for this video. We've got our workflow ready to set up our notebook, our video notebook.
We've got the resources. We've got what we're covering. We've got our dependencies.
Let's in the next one get started on one data, preparing and loading.
I'll see you in the next video.
Let's now get on to the first step of our pytorch workflow. And that is data, preparing and loading.
Now, I want to stress data can be almost anything in machine learning.
I mean, you could have an Excel spreadsheet, which is rows and columns,
nice and formatted data. You could have images of any kind. You could have videos. I mean,
YouTube has lots of data. You could have audio like songs or podcasts. You could have even DNA
these days. Patents and DNA are starting to get discovered by machine learning. And then, of course,
you could have text like what we're writing here. And so what we're going to be focusing on
throughout this entire course is the fact that machine learning is a game of two parts.
So one, get data into a numerical representation to build a model to learn patterns in that
numerical representation. Of course, there's more around it. Yes, yes, yes. I understand you can
get as complex as you like, but these are the main two concepts. And machine learning, when I say
machine learning, saying goes for deep learning, you need some kind of, oh, number a call. Number
a call. I like that word, number a call representation. Then you want to build a model to learn patterns
in that numerical representation. And if you want, I've got a nice pretty picture that describes that
machine learning a game of two parts. Let's refer to our data. Remember, data can be almost
anything. These are our inputs. So the first step that we want to do is create some form
of numerical encoding in the form of tenses to represent these inputs, how this looks will be
dependent on the data, depending on the numerical encoding you choose to use. Then we're going to
build some sort of neural network to learn a representation, which is also referred to as
patterns features or weights within that numerical encoding. It's going to output that
representation. And then we want to do something without representation, such as in the case of
this, we're doing image recognition, image classification, is it a photo of Raman or spaghetti?
Is this tweet spam or not spam? Is this audio file saying what it says here? I'm not going to say
this because my audio assistant that's also named to this word here is close by and I don't want it
to go off. So this is our game of two parts. One here is convert our data into a numerical
representation. And two here is build a model or use a pre trained model to find patterns in
that numerical representation. And so we've got a little stationary picture here, turn data into
numbers, part two, build a model to learn patterns in numbers. So with that being said,
now let's create some data to showcase this. So to showcase this, let's create some known
data using the linear regression formula. Now, if you're not sure what linear regression is,
or the formula is, let's have a look linear regression formula. This is how I'd find it.
Okay, we have some fancy Greek letters here. But essentially, we have y equals a function of x
and b plus epsilon. Okay. Well, there we go. A linear regression line has the equation in the
form of y equals a plus bx. Oh, I like this one better. This is nice and simple. We're going to
start from as simple as possible and work up from there. So y equals a plus bx, where x is the
explanatory variable, and y is the dependent variable. The slope of the line is b. And the
slope is also known as the gradient. And a is the intercept. Okay, the value of when y
when x equals zero. Now, this is just text on a page. This is formula on a page. You know how I
like to learn things? Let's code it out. So let's write it here. We'll use a linear regression formula
to make a straight line with known parameters. I'm going to write this down because parameter
is a common word that you're going to hear in machine learning as well. So a parameter is
something that a model learns. So for our data set, if machine learning is a game of two parts,
we're going to start with this. Number one is going to be done for us, because we're going to
start with a known representation, a known data set. And then we want our model to learn that

Section 46: Main Topics

Key Topics:

Well, it's because typically X in machine learning you'll find is a matrix or a tensor
Of course, we know what the relationship is between X and Y because we've coded this formula here
That is the whole premise of machine learning

📄 Click to view detailed content

representation. This is all just talk, Daniel, let's get into coding. Yes, you're right. You're
right. Let's do it. So create known parameters. So I'm going to use a little bit different
names to what that Google definition did. So weight is going to be 0.7 and bias is going to be 0.3.
Now weight and bias are another common two terms that you're going to hear in neural networks.
So just keep that in mind. But for us, this is going to be the equivalent of our weight will be B
and our bias will be A. But forget about this for the time being. Let's just focus on the code.
So we know these numbers. But we want to build a model that is able to estimate these numbers.
How? By looking at different examples. So let's create some data here. We're going to create a
range of numbers. Start equals zero and equals one. We're going to create some numbers between
zero and one. And they're going to have a gap. So the step the gap is going to be 0.02.
Now we're going to create an X variable. Why is X a capital here?
Well, it's because typically X in machine learning you'll find is a matrix or a tensor.
And if we remember back to the fundamentals, a capital represents a matrix or a tensor
and a lowercase represents a vector. But now case it's going to be a little confusing because
X is a vector. But later on, X will start to be a tensor and a matrix. So for now,
we'll just keep the capital, not capital notation.
We're going to create the formula here, which is remember how I said our weight is in this case,
the B and our bias is the A. So we've got the same formula here. Y equals weight times X plus
bias. Now let's have a look at these different numbers. So we'll view the first 10 of X and we'll
view the first 10 of Y. We'll have a look at the length of X and we'll have a look at the length of
Y. Wonderful. So we've got some values here. We've got 50 numbers of each. This is a little
confusing. Let's just view the first 10 of X and Y first. And then we can have a look at the
length here. So what we're going to be doing is building a model to learn some values,
to look at the X values here and learn what the associated Y value is and the relationship
between those. Of course, we know what the relationship is between X and Y because we've
coded this formula here. But you won't always know that in the wild. That is the whole premise of
machine learning. This is our ideal output and this is our input. The whole premise of machine
learning is to learn a representation of the input and how it maps to the output. So here are our
input numbers and these are our output numbers. And we know that the parameters of the weight and
bias are 0.7 and 0.3. We could have set these to whatever we want, by the way. I just like the
number 7 and 3. You could set these to 0.9, whatever, whatever. The premise would be the same.
So, oh, and what I've just done here, I kind of just coded this without talking.
But I just did torch a range and it starts at 0 and it ends at 1 and the step is 0.02. So there
we go, 000 by 0.02, 04. And I've unsqueezed it. So what does unsqueezed do? Removes the extra
dimensions. Oh, sorry, ads are extra dimension. You're getting confused here. So if we remove that,
we get no extra square bracket. But if we add unsqueeze, you'll see that we need this later on
for when we're doing models. Wonderful. So let's just leave it at that. That's enough for this
video, we've got some data to work with. Don't worry if this is a little bit confusing for now,
we're going to keep coding on and see what we can do to build a model to infer patterns in this
data. But right now, I want you to have a think, this is tensor data, but it's just numbers on a
page. What might be a better way to hint, this is a hint by the way, visualize it. What's our
data explorer's motto? Let's have a look at that in the next video. Welcome back. In the last
video, we created some numbers on a page using the linear regression formula with some known
parameters. Now, there's a lot going on here, but that's all right. We're going to keep building
upon what we've done and learn by doing. So in this video, we're going to cover one of the most
important concepts in machine learning in general. So splitting data into training and test sets.
One of the most important concepts in machine learning in general. Now, I know I've said this
already a few times. One of the most important concepts, but truly, this is possibly, in terms
of data, this is probably the number one thing that you need to be aware of. And if you've come
from a little bit of a machine learning background, you probably well and truly know all about this.
But we're going to recover it anyway. So let's jump in to some pretty pictures. Oh, look at that
one speaking of pretty pictures. But that's not what we're focused on now. We're looking at the
three data sets. And I've written down here possibly the most important concept in machine
learning, because it definitely is from a data perspective. So the course materials,
imagine you're at university. So this is going to be the training set. And then you have the

Section 47: Main Topics

Key Topics:

So say you're trying to learn something at university or through this course, you might have all of the materials, which is your training set
Now, let's just see if you're learning the course materials well
And this is to see if you've gone through the entire course materials, and you've learned some things

📄 Click to view detailed content

practice exam, which is the validation set. Then you have the final exam, which is the test set.
And the goal of all of this is for generalization. So let's step back. So say you're trying to learn
something at university or through this course, you might have all of the materials, which is your
training set. So this is where our model learns patterns from. And then to practice what you've
done, you might have a practice exam. So the mid semester exam or something like that. Now,
let's just see if you're learning the course materials well. So in the case of our model,
we might tune our model on this plastic exam. So we might find that on the validation set,
our model doesn't do too well. And we adjusted a bit, and then we retrain it, and then it does
better. Before finally, at the end of semester, the most important exam is your final exam. And
this is to see if you've gone through the entire course materials, and you've learned some things.
Now you can adapt to unseen material. And that's a big point here. We're going to see this in
practice is that when the model learns something on the course materials, it never sees the validation
set or the test set. So say we started with 100 data points, you might use 70 of those data points
for the training material. You might use 15% of those data points, so 15 for the practice.
And you might use 15 for the final exam. So this final exam is just like if you're at university
learning something is to see if, hey, have you learned any skills from this material at all?
Are you ready to go into the wild into the quote unquote real world? And so this final exam is to
test your model's generalization, because it's never seen this data is, let's define generalization
is the ability for a machine learning model or a deep learning model to perform well on data it
hasn't seen before, because that's our whole goal, right? We want to build a machine learning model
on some training data that we can deploy in our application or production setting. And then
more data comes in that it hasn't seen before. And it can make decisions based on that new data
because of the patterns it's learned in the training set. So just keep this in mind,
three data sets training validation test. And if we jump in to the learn pytorch book,
we've got split data. So we're going to create three sets. Or in our case, we're only going to
create two or training in a test. Why is that? Because you don't always need a validation set.
There is often a use case for a validation set. But the main two that are always used is the training
set and the testing set. And how much should you split? Well, usually for the training set,
you'll have 60 to 80% of your data. If you do create a validation set, you'll have somewhere
between 10 and 20. And if you do create a testing set, it's a similar split to the validation set,
you'll have between 10 and 20%. So training, always testing always validation often, but
not always. So with that being said, I'll let you refer to those materials if you want. But now
let's create a training and test set with our data. So we saw before that we have 50 points,
we have X and Y, we have one to one ratio. So one value of X relates to one value of Y.
And we know that the split now for the training set is 60 to 80%. And the test set is 10 to 20%.
So let's go with the upper bounds of each of these, 80% and 20%, which is a very common split,
actually 80, 20. So let's go create a train test split. And we're going to go train split.
We'll create a number here so we can see how much. So we want an integer of 0.8, which is 80%
of the length of X. What does that give us? Train split should be about 40 samples. Wonderful.
So we're going to create 40 samples of X and 40 samples of Y. Our model will train on those 40
samples to predict what? The other 10 samples. So let's see this in practice. So X train,
Y train equals X. And we're going to use indexing to get all of the samples up until the train
split. That's what this colon does here. So hey, X up until the train split, Y up until the train
split, and then for the testing. Oh, thanks for that. Auto correct cola, but didn't actually need that
one. X test. Y test equals X. And then we're going to get everything from the train split onwards.
So the index onwards, that's what this notation means here. And Y from the train split onwards as
well. Now, there are many different ways to create a train and test split. Ours is quite simple here,
but that's because we're working with quite a simple data set. One of the most popular methods
that I like is scikit learns train test split. We're going to see this one later on. It adds a
little bit of randomness into splitting your data. But that's for another video, just to make you
aware of it. So let's go length X train. We should have 40 training samples to
how many testing samples length X test and length Y test. Wonderful 40 40 10 10 because we have
training features, training labels, testing features, testing labels. So essentially what we've
created here is now a training set. We've split our data. Training set could also be referred to
as training split yet another example of where machine learning has different names for different

Section 48: Main Topics

Key Topics:

One of your biggest, biggest, biggest hurdles in machine learning will be creating proper training and test sets

📄 Click to view detailed content

things. So set split same thing training split test split. This is what we've created. Remember,
the validation set is used often, but not always because our data set is quite simple. We're just
sticking with the necessities training and test. But keep this in mind. One of your biggest,
biggest, biggest hurdles in machine learning will be creating proper training and test sets. So
it's a very important concept. With that being said, I did issue the challenge in the last video
to visualize these numbers on a page. We haven't done that in this video. So let's move towards
that next. I'd like you to think of how could you make these more visual? Right. These are just
numbers on a page right now. Maybe that plot lib can help. Let's find out. Hey, hey, hey, welcome
back. In the last video, we split our data into training and test sets. And now later on,
we're going to be building a model to learn patterns in the training data to relate to the
testing data. But as I said, right now, our data is just numbers on a page. It's kind of
hard to understand. You might be able to understand this, but I prefer to get visual. So let's write
this down. How might we better visualize our data? And I'm put a capital here. So we're grammatically
correct. And this is where the data Explorers motto comes in. Visualize, visualize, visualize.
Ha ha. Right. So if ever you don't understand a concept, one of the best ways to start
understanding it more for me is to visualize it. So let's write a function to do just that.
We're going to call this plot predictions. We'll see why we call it this later on. That's the
benefit of making these videos is that I've got a plan for the future. Although it might seem
like I'm winging it, there is a little bit of behind the scenes happening here. So we'll have
the train data, which is our X train. And then we'll have the train labels, which is our Y train.
And we'll also have the test data. Yeah, that's a good idea. X test. And we'll also have the test
labels, equals Y test. Excuse me. I was looking at too many X's there. And then the predictions.
And we'll set this to none, because we don't have any predictions yet. But as you might have guessed,
we might have some later on. So we'll put a little doc string here, so that we're being nice and
Pythonic. So plots training data, test data, and compares predictions. Nice and simple.
Nothing too outlandish. And then we're going to create a figure. This is where map plot lib comes
in. Plot figure. And we'll go fig size equals 10, seven, which is my favorite hand in poker.
And we'll plot the training data in blue also happens to be a good dimension for a map plot.
Plot dot scatter. Train data. Creating a scatter plot here. We'll see what it does in a second.
Color. We're going to give this a color of B for blue. That's what C stands for in map plot lib
scatter. We'll go size equals four and label equals training data. Now, where could you find
information about this scatter function here? We've got command shift space. Is that going to
give us a little bit of a doc string? Or sometimes if command not space is not working,
you can also hover over this bracket. I think you can even hover over this.
There we go. But this is a little hard for me to read. Like it's there, but it's got a lot going
on. X, Y, S, C, C map. I just like to go map plot lib scatter. There we go. We've got a whole
bunch of information there. A little bit easier to read for me here. And then you can see some
examples. Beautiful. So now let's jump back into here. So in our function plot predictions,
we've taken some training data, test data. We've got the training data plotting in blue. What
color should we use for the testing data? How about green? I like that idea. Plot.scatter.
Test data. Green's my favorite color. What's your favorite color? C equals G. You might be
able to just plot it in your favorite color here. Just remember though, it'll be a little bit
different from the videos. And then we're going to call this testing data. So just the exact same
line is above, but with a different set of data. Now, let's check if there are predictions. So
are there predictions? So if predictions is not none, let's plot the predictions, plot the
predictions, if they exist. So plot scatter test data. And why are we plotting the test data?
Remember, what is our scatter function? Let's go back up to here. It takes in x and y. So
our predictions are going to be compared to the testing data labels. So that's the whole
game that we're playing here. We're going to train our model on the training data.
And then to evaluate it, we're going to get our model to predict the y values
as with the input of x test. And then to evaluate our model, we compare how good our models
predictions are. In other words, predictions versus the actual values of the test data set.
But we're going to see this in practice. Rather than just talk about it. So let's do our predictions
in red. And label equals predictions. Wonderful. So let's also show the legend, because, I mean,
we're legends. So we could just put in a mirror here. Now I'm kidding. Legend is going to show
our labels on the map plot. So prop equals size and prop stands for properties. Well,

Section 49: Main Topics

Key Topics:

So the whole idea of what we're going to be doing with our machine learning model is we don't actually really need to build a machine learning model for this
We could do other things, but machine learning is fun
But now let's get into building our first PyTorch model

📄 Click to view detailed content

it may or may not. I just like to think it does. That's how I remember it. So we have a beautiful
function here to plot our data. Should we try it out? Remember, we've got hard coded inputs here,
so we don't actually need to input anything to our function. We've got our train and test data
ready to go. If in doubt, run the code, let's check it out. Did we make a mistake in our plot
predictions function? You might have caught it. Hey, there we go. Beautiful. So because we don't
have any predictions, we get no red dots. But this is what we're trying to do. We've got a simple
straight line. You can't get a much more simple data set than that. So we've got our training data
in blue, and we've got our testing data in green. So the whole idea of what we're going to be doing
with our machine learning model is we don't actually really need to build a machine learning
model for this. We could do other things, but machine learning is fun. So we're going to take
in the blue dots. There's quite a pattern here, right? This is the relationship we have an x value
here, and we have a y value. So we're going to build a model to try and learn the pattern
of these blue dots, so that if we fed our model, our model, the x values of the green dots,
could it predict the appropriate y values for that? Because remember, these are the test data set.
So pass our model x test to predict y test. So blue dots as input, green dots as the ideal output.
This is the ideal output, a perfect model would have red dots over the top of the green dots. So
that's what we will try to work towards. Now, we know the relationship between x and y.
How do we know that? Well, we set that up above here. This is our weight and bias.
We created that line y equals weight times x plus bias, which is the simple version of the
linear regression formula. So mx plus c, you might have heard that in high school algebra,
so gradient plus intercept. That's what we've got. With that being said,
let's move on to the next video and build a model. Well, this is exciting. I'll see you there.
Welcome back. In the last video, we saw how to get visual with our data. We followed the data
explorer's motto of visualize, visualize, visualize. And we've got an idea of the training data that
we're working with and the testing data that we're trying to build a model to learn the patterns
in the training data, essentially this upwards trend here, to be able to predict the testing data.
So I just want to give you another heads up. I took a little break after the recording last
video. And so now my colab notebook has disconnected. So I'm going to click reconnect.
And my variables here may not work. So this is what might happen on your end. If you take a break
from using Google Colab and come back, if I try to run this function, they might have been saved,
it looks like they have. But if not, you can go restart and run all. This is typically one of the
most helpful troubleshooting steps of using Google Colab. If a cell, say down here isn't working,
you can always rerun the cells above. And that may help with a lower cell here, such as if this
function wasn't instantiated because this cell wasn't run, and we couldn't run this cell here,
which calls this function here, we just have to rerun this cell above so that we can run this one.
But now let's get into building our first PyTorch model. We're going to jump straight into the code.
So our first PyTorch model. Now this is very exciting.
Let's do it. So we'll turn this into Markdown. Now we're going to create a linear regression model.
So look at linear regression formula again, we're going to create a model that's essentially going
to run this computation. So we need to create a model that has a parameter for A, a parameter for B,
and in our case it's going to be weight and bias, and a way to do this forward computation.
What I mean by that, we're going to see with code. So let's do it. We'll do it with pure PyTorch.
So create a linear regression model class. Now if you're not experienced with using Python classes,
I'm going to be using them throughout the course, and I'm going to call this one linear regression
model. If you haven't dealt with Python classes before, that's okay. I'm going to be explaining
what we're doing as we're doing it. But if you'd like a deeper dive, I'd recommend you to real Python
classes. OOP in Python three. That's a good rhyming. So I'm just going to link this here.
Because we're going to be building classes throughout the course,
I'd recommend getting familiar with OOP, which is object oriented programming, a little bit of a
mouthful, hence the OOP in Python. To do so, you can use the following resource from real Python.
But when I'm going to go through that now, I'd rather just code it out and talk it out while we
do it. So we've got a class here. Now the first thing you might notice is that the class inherits
from nn.module. And you might be wondering, well, what's nn.module? Well, let's write down here,
almost everything in PyTorch inherits from nn.module. So you can imagine nn.module as the
Lego building bricks of PyTorch model. And so nn.module has a lot of helpful inbuilt things that's

Section 50: Main Topics

Key Topics:

going to help us build our PyTorch models
And of course, how could you learn more about it
module, PyTorch

📄 Click to view detailed content

going to help us build our PyTorch models. And of course, how could you learn more about it?
Well, you could go nn.module, PyTorch. Module. Here we go. Base class for all neural network
modules. Wonderful. Your models should also subclass this class. So that's what we're building. We're
building our own PyTorch model. And so the documentation here says that your models should
also subclass this class. And another thing with PyTorch, this is what makes it, it might seem very
confusing when you first begin. But modules can contain other modules. So what I mean by being a
Lego brick is that you can stack these modules on top of each other and make progressively more
complex neural networks as you go. But we'll leave that for later on. For now, we're going to start
with something nice and simple. And let's clean up our web browser. So we're going to create a
constructor here, which is with the init function. It's going to take self as a parameter. If you're
not sure of what's going on here, just follow along with the code for now. And I'd encourage you
to read this documentation here after the video. So then we have super dot init. I know when I
first started learning this, I was like, why do we have to write a knit twice? And then what's
super and all that jazz. But just for now, just take this as being some required Python syntax.
And then we have self dot weights. So that means we're going to create a weights parameter. We'll
see why we do this in a second. And to create that parameter, we're going to use nn dot parameter.
And just a quick reminder that we imported nn from torch before. And if you remember,
nn is the building block layer for neural networks. And within nn, so nn stands for neural network
is module. So we've got nn dot parameter. Now, we're going to start with random parameters.
So torch dot rand n. One, we're going to talk through each of these in a second. So I'm also
going to put requires, requires grad equals true. We haven't touched any of these, but that's okay.
D type equals torch dot float. So let's see what nn parameter tells us. What do we have here?
A kind of tensor that is to be considered a module parameter. So we've just created a module
using nn module. Parameters are torch tensor subclasses. So this is a tensor in itself
that have a very special property when used with modules. When they're assigned as a module
attribute, they are automatically added to the list of its parameters. And we'll appear e g
in module dot parameters iterator. Oh, we're going to see that later on. Assigning a tensor
doesn't have such effect. So we're creating a parameter here. Now requires grad. What does that
mean? Well, let's just rather than just try to read the doc string collab, let's look it up.
nn dot parameter. What does it say requires grad optional. If the parameter requires gradient.
Hmm. What does requires gradient mean? Well, let's come back to that in a second. And then
for now, I just want you to think about it. D type equals torch dot float. Now,
the data type here torch dot float is, as we've discussed before, is the default
for pytorch to watch dot float. This could also be torch dot float 32. So we're just going to
leave it as torch float 32, because pytorch likes to work with flight 32. Now, do we have
this by default? We do. So we don't necessarily have to set requires grad equals true. So just
keep that in mind. So now we've created a parameter for the weights. We also have to create a parameter
for the bias. Let's finish creating this. And then we'll write the code, then we'll talk about it.
So rand n. Now requires grad equals true. And d type equals torch dot float. There we go.
And now we're going to write a forward method. So forward method to define the computation
in the model. So let's go def forward, which self takes in a parameter x, which is data,
which X is expected to be of type torch tensor. And it returns a torch dot tensor. And then we go
here. And so we say X, we don't necessarily need this comment. I'm just going to write it anyway.
X is the input data. So in our case, it might be the training data. And then from here, we want
it to return self dot weights times X plus self dot bias. Now, where have we seen this before?
Well, this is the linear regression formula. Now, let's take a step back into how we created our data.
And then we'll go back through and talk a little bit more about what's going on here.
So if we go back up to our data, where did we create that? We created it here. So you see how
we've created known parameters, weight and bias. And then we created our y variable, our target,
using the linear regression formula, wait times X plus bias, and X were a range of numbers.
So what we've done with our linear regression model that we've created from scratch,
if we go down here, we've created a parameter, weights. This could just be weight, if we wanted to.
We've created a parameter here. So when we created our data, we knew what the parameters weight and
bias were. The whole goal of our model is to start with random numbers. So these are going to be
random parameters. And to look at the data, which in our case will be the training samples,
and update those random numbers to represent the pattern here. So ideally, our model, if it's

Section 51: Main Topics

Key Topics:

learning correctly, will take our weight, which is going to be a random value, and our bias, which is going to be a random value
So that's the premise of machine learning
And so what this is going to do is when we run computations using this model here, pytorch is going to keep track of the gradients of our weights parameter and our bias parameter

📄 Click to view detailed content

learning correctly, will take our weight, which is going to be a random value, and our bias,
which is going to be a random value. And it will run it through this forward calculation,
which is the same formula that we use to create our data. And it will adjust the weight and bias
to represent as close as possible, if not perfect, the known parameters. So that's the premise of
machine learning. And how does it do this? Through an algorithm called gradient descent. So I'm just
going to write this down because we've talked a lot about this, but I'd like to just tie it together
here. So what our model does, so start with random values, weight and bias, look at training data,
and adjust the random values to better represent the, or get closer to the ideal values. So the
weight and bias values we use to create the data. So that's what it's going to do. It's going to
start with random values, and then continually look at our training data to see if it can adjust
those random values to be what would represent this straight line here. Now, how does it do so?
How does it do so? Through two main algorithms. So one is gradient descent, and two is back
propagation. So I'm going to leave it here for the time being, but we're going to continue talking
about this gradient descent is why we have requires grad equals true. And so what this is going to
do is when we run computations using this model here, pytorch is going to keep track of the gradients
of our weights parameter and our bias parameter. And then it's going to update them through a
combination of gradient descent and back propagation. Now, I'm going to leave this as extracurricular
for you to look through and gradient descent and back propagation. I'm going to add some
resources here. There will also be plenty of resources in the pytorch workflow fundamentals
book chapter on how these algorithms work behind the scenes. We're going to be focused on the code,
the pytorch code, to trigger these algorithms behind the scenes. So pytorch, lucky for us,
has implemented gradient descent and back propagation for us. So we're writing the higher level code
here to trigger these two algorithms. So in the next video, we're going to step through this a
little bit more, and then further discuss some of the most useful and required modules of pytorch,
particularly an N and a couple of others. So let's leave it there, and I'll see you in the next video.
Welcome back. In the last video, we covered a whole bunch in creating our first pytorch model
that inherits from nn.module. We talked about object oriented programming and how a lot of
pytorch uses object oriented programming. I can't say that. I might just say OOP for now.
What I've done since last video, though, is I've added two resources here for gradient descent
and back propagation. These are two of my favorite videos on YouTube by the channel three blue
one brown. So this is on gradient descent. I would highly recommend watching this entire series,
by the way. So that's your extra curriculum for this video, in particular, and for this course overall
is to go through these two videos. Even if you're not sure entirely what's happening,
you will gain an intuition for the code that we're going to be writing with pytorch.
So just keep that in mind as we go forward, a lot of what pytorch is doing behind the scenes for us
is taking care of these two algorithms for us. And we also created two parameters here in our model
where we've instantiated them as random values. So one parameter for each of the ones that we use,
the weight and bias for our data set. And now I want you to keep in mind that we're working
with a simple data set here. So we've created our known parameters. But in a data set that you
haven't created by yourself, you've maybe gathered that from the internet, such as images,
you won't be necessarily defining these parameters. Instead, another module from nn will define the
parameters for you. And we'll work out what those parameters should end up being. But since we're
working with a simple data set, we can define our two parameters that we're trying to estimate.
This is a key point here is that our model is going to start with random values. That's the
annotation I've added here. Start with a random weight value using torch random. And then we've
told it that it can update via gradient descent. So pytorch is going to track the gradients of
this parameter for us. And then we've told it that the D type we want is float 32. We don't
necessarily need these two set explicitly, because a lot of the time the default in pytorch is to
set these two requires grad equals true and d type equals torch dot float. Does that for us
behind the scenes? But just to keep things as fundamental and as straightforward as possible,
we've set all of this explicitly. So let's jump into the keynote. I'd just like to explain
what's going on one more time in a visual sense. So here's the exact code that we've
just written. I've just copied it from here. And I've just made it a little bit more colorful.
But here's what's going on. So when you build a model in pytorch, it subclasses the nn.modgable
class. This contains all the building blocks for neural networks. So our class of model, subclasses
nn.modgable. Now, inside the constructor, we initialize the model parameters. Now, as we'll see,
later on with bigger models, we won't necessarily always explicitly create the weights and biases.

Section 52: Main Topics

Key Topics:

So this, in turn, means that pytorch behind the scenes will track all of the gradients for these parameters here for use with torch
auto grad module of pytorch is what implements gradient descent
Now, a lot of this will happen behind the scenes for when we write our pytorch training code

📄 Click to view detailed content

We might initialize whole layers. Now, this is a concept we haven't touched on yet, but
we might initialize a list of layers or whatever we need. So basically, what happens in here is that
we create whatever variables that we need for our model to use. And so these could be different
layers from torch.nn, single parameters, which is what we've done in our case, hard coded values,
or even functions. Now, we've explicitly set requires grad equals true for our model parameters.
So this, in turn, means that pytorch behind the scenes will track all of the gradients
for these parameters here for use with torch.auto grad. So torch.auto grad module of pytorch is what
implements gradient descent. Now, a lot of this will happen behind the scenes for when we write
our pytorch training code. So if you'd like to know what's happening behind the scenes,
I'd highly recommend you checking out these two videos, hence is why I've linked them here.
Oh, and for many pytorch.nn modules requires grad is true is set by default.
Finally, we've got a forward method. Now, any subclass of nn.modgable, which is what we've done,
requires a forward method. Now, we can see this in the documentation. If we go torch
dot nn.modgable.
Click on module. Do we have forward?
Yeah, there we go. So forward, we've got a lot of things built into an nn.modgable.
So you see here, this is a subclass of an nn.modgable. And then we have forward.
So forward is what defines the computation performed at every call. So if we were
to call linear regression model and put some data through it, the forward method is the
operation that this module does that this model does. And in our case, our forward method is
the linear regression function. So keep this in mind, any subclass of nn.modgable needs to
override the forward method. So you need to define a forward method if you're going to subclass
nn.modgable. We'll see this very hands on. But for now, I believe that's enough coverage of what
we've done. If you have any questions, remember, you can ask it in the discussions. We've got a
fair bit going on here. But I think we've broken it down a fair bit. The next step is for us to,
I know I mentioned this in a previous video is to cover some PyTorch model building essentials.
But we're going to cover a few more of them. We've seen some already. But the next way to really
start to understand what's going on is to check the contents of our model, train one, and make
some predictions with it. So let's get hands on with that in the next few videos. I'll see you there.
Welcome back. In the last couple of videos, we stepped through creating our first PyTorch model.
And it looks like there's a fair bit going on here. But some of the main takeaways is that almost
every model in PyTorch inherits from nn.modgable. And if you are going to inherit from nn.modgable,
you should override the forward method to define what computation is happening in your model.
And for later on, when our model is learning things, in other words, updating its weights and
bias values from random values to values that better fit the data, it's going to do so via
gradient descent and back propagation. And so these two videos are some extra curriculum
for what's happening behind the scenes. But we haven't actually written any code yet to trigger
these two. So I'll refer back to these when we actually do write code to do that. For now,
we've just got a model that defines some forward computation. But speaking of models, let's have
a look at a couple of PyTorch model building essentials. So we're not going to write too much
code for this video, and it's going to be relatively short. But I just want to introduce you to some
of the main classes that you're going to be interacting with in PyTorch. And we've seen
some of these already. So one of the first is torch.nn. So contains all of the building blocks
for computational graphs. Computational graphs is another word for neural networks.
Well, actually computational graphs is quite general. I'll just write here, a neural network
can be considered a computational graph. So then we have torch.nn.parameter. We've seen this.
So what parameters should our model try and learn? And then we can write here often a PyTorch
layer from torch.nn will set these for us. And then we've got torch.nn.module, which is
what we've seen here. And so torch.nn.module is the base class for all neural network modules.
If you subclass it, you should overwrite forward, which is what we've done here. We've created our
own forward method. So what else should we cover here? We're going to see these later
on, but I'm going to put it here, torch.optim. This is where the optimizers in PyTorch live.
They will help with gradient descent. So optimizer, an optimizer is, as we've said before,
that our model starts with random values. And it looks at training data and adjusts the random
values to better represent the ideal values. The optimizer contains algorithm that's going to
optimize these values, instead of being random, to being values that better represent our data.
So those algorithms live in torch.optim. And then one more for now, I'll link to extra resources.
And we're going to cover them as we go. That's how I like to do things, cover them as we need them.
So all nn.module. So this is the forward method. I'm just going to explicitly say here that all
nn.module subclasses require you to overwrite forward. This method defines what happens
in the forward computation. So in our case, if we were to pass some data to our linear regression

Section 53: Main Topics

Key Topics:

PyTorch is central neural network building modules
And if you'd like more, one of my favorite resources is the PyTorch cheat sheet
As I said, this course is not a replacement for the documentation

📄 Click to view detailed content

model, the forward method would take that data and perform this computation here.
And as your models get bigger and bigger, ours is quite straightforward here.
This forward computation can be as simple or as complex as you like, depending on what you'd
like your model to do. And so I've got a nice and fancy slide here, which basically reiterates
what we've just discussed. PyTorch is central neural network building modules.
So the module torch.nn, torch.nn.module, torch.optim, torch.utils.dataset. We haven't actually talked
about this yet. And I believe there's one more data loader. We're going to see these two later on.
But these are very helpful when you've got a bit more of a complicated data set. In our case,
we've got just 50 integers for our data set. We've got a simple straight line. But when we need
to create more complex data sets, we're going to use these. So this will help us build models.
This will help us optimize our models parameters. And this will help us load data. And if you'd
like more, one of my favorite resources is the PyTorch cheat sheet. Again, we're referring
back to the documentation. See, all of this documentation, right? As I said, this course is
not a replacement for the documentation. It's just my interpretation of how one should best
become familiar with PyTorch. So we've got imports, the general import torch from torch.utils.dataset
data loader. Oh, did you look at that? We've got that mentioned here, data, data set data loader.
And torch, script and jit, neural network API. I want an X. I'll let you go through here.
We're covering some of the most fundamental ones here. But there's, of course, PyTorch is
quite a big library. So some extra curricula for this video would be to go through this for
five to 10 minutes and just read. You don't have to understand them all. We're going to start to
get more familiar with all of these. We're not all of them because, I mean, that would require
making videos for the whole documentation. But a lot of these through writing them via code.
So that's enough for this video. I'll link this PyTorch cheat sheet in the video here.
And in the next video, how about we, we haven't actually checked out what happens if we do
create an instance of our linear regression model. I think we should do that. I'll see you there.
Welcome back. In the last video, we covered some of the PyTorch model building essentials. And look,
I linked a cheat sheet here. There's a lot going on. There's a lot of text going on in the page.
Of course, the reference material for here is in the Learn PyTorch book. PyTorch model building
essentials under 0.1, which is the notebook we're working on here. But I couldn't help myself.
I wanted to add some color to this. So before we inspect our model, let's just add a little bit
of color to our text on the page. We go to whoa. Here's our workflow. This is what we're covering
in this video, right? Or in this module, 0.1. But to get data ready, here are some of the most
important PyTorch modules. Torchvision.transforms. We'll see that when we cover computer vision later
on. Torch.utils.data.data set. So that's if we want to create a data set that's a little bit
more complicated than because our data set is so simple, we haven't used either of these
data set creator or data loader. And if we go build a picker model, well, we can use torch.nn.
We've seen that one. We've seen torch.nn.module. So in our case, we're building a model. But if we
wanted a pre-trained model, well, there's some computer vision models that have already been
built for us in torchvision.models. Now torchvision stands for PyTorch's computer vision
module. So we haven't covered that either. But this is just a spoiler for what's coming on
later on. Then if the optimizer, if we wanted to optimize our model's parameters to better
represent a data set, we can go to torch.optim. Then if we wanted to evaluate the model,
well, we've got torch metrics for that. We haven't seen that, but we're going to be
hands-on with all of these later on. Then if we wanted to improve through experimentation,
we've got torch.utils.tensorboard. Hmm. What's this? But again, if you want more,
there's some at the PyTorch cheat sheet. But now this is just adding a little bit of color
and a little bit of code to our PyTorch workflow. And with that being said, let's get a little bit
deeper into what we've built, which is our first PyTorch model. So checking the contents of our
PyTorch model. So now we've created a model. Let's see what's inside. You might already be able
to guess this by the fact of what we've created in the constructor here in the init function.
So what do you think we have inside our model? And how do you think we'd look in that? Now,
of course, these are questions you might not have the answer to because you've just, you're like,
Daniel, I'm just starting to learn PyTorch. I don't know these, but I'm asking you just to start
thinking about these different things, you know? So we can check out our model parameters or what's
inside our model using, wait for it, dot parameters. Oh, don't you love it when things are nice and
simple? Well, let's check it out. Hey, well, first things we're going to do is let's create a random
seed. Now, why are we creating a random seed? Well, because recall, we're creating these parameters
with random values. And if we were to create them with outer random seed, we would get different
values every time. So for the sake of the educational sense, for the sake of this video,

Section 54: Main Topics

Key Topics:

Model zero, because it's going to be the zeroth model, the first model that we've ever created in this whole course, how amazing linear regression model, which is what our class is called
So this is the essence of what our machine learning models and deep learning models are going to do
And so of course, we're not going to do them all by hand

📄 Click to view detailed content

we're going to create a manual seed here, torch dot manual seed. I'm going to use 42 or maybe 43,
I could use 43 now 42 because I love 42. It's the answer to the universe. And we're going to create
an instance of the model that we created. So this is a subclass of an end up module.
So let's do it. Model zero, because it's going to be the zeroth model, the first model that
we've ever created in this whole course, how amazing linear regression model, which is what
our class is called. So we can just call it like that. That's all I'm doing, just calling this class.
And so let's just see what happens there. And then if we go model zero, what does it give us? Oh,
linear regression. Okay, it doesn't give us much. But we want to find out what's going on in here.
So check out the parameters. So model zero dot parameters. What do we get from this? Oh, a generator.
Well, let's turn this into a list that'll be better to look at. There we go. Oh, how exciting is that?
So parameter containing. Look at the values tensor requires grad equals true parameter containing
wonderful. So these are our model parameters. So why are they the values that they are? Well,
it's because we've used torch rand n. Let's see what happens if we go, let's just create torch dot
rand n one, what happens? We get a value like that. And now if we run this again,
we get the same values. But if we run this again, so keep this in one two, three, four,
five, actually, that's, wow, that's pretty cool that we got a random value that was all in order,
four in a row. Can we do it twice in a row? Probably not. Oh, we get it the same one. Now,
why is that? Oh, we get a different one. Did we just get the same one twice? Oh, my gosh,
we got the same value twice in a row. You saw that. You saw that. That's incredible. Now,
the reason why we get this is because this one is different every time because there's no random
seed. Watch if we put the random seed here, torch dot manual seed, 42, 3, 3, 6, 7, what happens?
3, 3, 6, 7, what happens? 3, 3, 6, 7. Okay. And what if we commented out the random seed
here, initialized our model, different values, two, three, five, two, three, four, five, it must
like that value. Oh, my goodness. Let me know if you get that value, right? So if we keep going,
we get different values every single time. Why is this? Why are we getting different values
every single time? You might be, Daniel, you sound like a broken record, but I'm trying to
really drive home the fact that we initialize our models with random parameters. So this is the
essence of what our machine learning models and deep learning models are going to do. Start with
random values, weights and bias. Maybe we've only got two parameters here, but the future models
that we build might have thousands. And so of course, we're not going to do them all by hand.
We'll see how we do that later on. But for now, we start with random values. And our ideal model
will look at the training data and adjust these random values. But just so that we can get
reproducible results, I'll get rid of this cell. I've set the random seed here. So you should be
getting similar values to this. If you're not, because there's maybe some sort of pytorch update
and how the random seeds calculated, you might get slightly different values. But for now,
we'll use torch.manualc.42. And I want you to just be aware of this can be a little bit confusing.
If you just do the list of parameters, for me, I understand it better if I list the name parameters.
So the way we do that is with model zero, and we call state dict on it. This is going to give us
our dictionary of the parameters of our model. So as you can see here, we've got weights,
and we've got bias, and they are random values. So where did weights and bias come from? Well,
of course, they came from here, weights, bias. But of course, as well up here,
we've got known parameters. So now our whole goal is what? Our whole goal is to build code,
or write code, that is going to allow our model to look at these blue dots here,
and adjust this weight and bias value to be weights as close as possible to weight and bias.
Now, how do we go from here and here to here and here? Well, we're going to see that in future
videos, but the closer we get these values to these two, the better we're going to be able to
predict and model our data. Now, this principle, I cannot stress enough, is the fundamental
entire foundation, the fundamental foundation. Well, good description, Daniel. The entire
foundation of deep learning, we start with some random values, and we use gradient descent and
back propagation, plus whatever data that we're working with to move these random values as close
as possible to the ideal values. And in most cases, you won't know what the ideal values are.
But in our simple case, we already know what the ideal values are.
So just keep that in mind going forward. The premise of deep learning is to start with random
values and make them more representative closer to the ideal values. With that being said,
let's try and make some predictions with our model as it is. I mean, it's got random values.

Section 55: Main Topics

Key Topics:

In the last video, we checked out the internals of our first PyTorch model
And we also discussed the entire premise of deep learning is to start with random numbers and slowly progress those towards more ideal numbers, slightly less random numbers based on the data
Because remember again, another premise of a machine learning model is to take some features as input and make some predictions close to some sort of labels

📄 Click to view detailed content

How do you think the predictions will go? So I think in the next video, we'll make some predictions
on this test data and see what they look like. I'll see you there. Welcome back. In the last
video, we checked out the internals of our first PyTorch model. And we found out that because we're
creating our model with torch dot or the parameters of our model, with torch dot rand, they begin as
random variables. And we also discussed the entire premise of deep learning is to start with random
numbers and slowly progress those towards more ideal numbers, slightly less random numbers based
on the data. So let's see, before we start to improve these numbers, let's see what their predictive
power is like right now. Now you might be able to guess how well these random numbers will be
able to predict on our data. You're not sure what that predicting means? Let's have a look. So making
predictions using torch dot inference mode, something we haven't seen. But as always, we're going to
discuss it while we use it. So to check our models predictive power, let's see how well
it predicts Y test based on X test. Because remember again, another premise of a machine
learning model is to take some features as input and make some predictions close to some sort of
labels. So when we pass data through our model, it's going to run it through the forward method.
So here's where it's a little bit confusing. We defined a forward method and it takes X as input.
Now I've done a little X, but we're going to pass it in a large X as its input. But the reason why I've
done a little X is because oftentimes in pytorch code, you're going to find all over the internet
is that X is quite common, commonly used in the forward method here, like this as the input data.
So I've just left it there because that's what you're going to find quite often.
So let's test it out. We haven't discussed what inference mode does yet, but we will make predictions
with model. So with torch dot inference mode, let's use it. And then we will discuss what's going
on. Y threads equals a model zero X test. So that's all we're doing. We're passing the X test data
through our model. Now, when we pass this X test in here, let's remind ourselves of what X test is.
X test 10 variables here. And we're trying to our ideal model will predict the exact values of Y test.
So this is what our model will do if it's a perfect model. It will take these X test values as input,
and it will return these Y test values as output. That's an ideal model. So the predictions are the
exact same as the test data set. How do you think our model will go considering it's starting with
random values as its parameters? Well, let's find out, hey. So what can that Y threads?
Oh, what's happened here? Not implemented error. Ah, this is an error I get quite often in Google
Colab when I'm creating a high-torch model. Now, it usually happens. I'm glad we've stumbled upon
this. And I think I know the fix. But if not, we might see a little bit of troubleshooting in this
video is that when we create this, if you see this not implemented error, right, it's saying that
the Ford method. Here we go. Ford implemented. There we go. It's a little bit of a rabbit hole
this not implemented area. I've come across it a fair few times and it took me a while to figure
out that for some reason the spacing. So in Python, you know how you have space space and that defines
a function space space. There's another thing there and another line there. For some reason,
if you look at this line in my notebook, and by the way, if you don't have these lines or if you
don't have these numbers, you can go into tools, settings, editor, and then you can define them here.
So show line numbers, show notation guides, all that sort of jazz there. You can customize what's
going on. But I just have these two on because I've run into this error a fair few times. And so
it's because that this Ford method is not in line with this bracket here. So we need to highlight
this and click shift tab, move it over. So now you see that it's in line here. And then if we run
this, won't change any output there. See, that's the hidden gotcha. Is that when we ran this before,
it found no error. But then when we run it down here, it works. So just keep that in mind. I'm
really glad we stumbled upon that because indentation errors, not implemented errors,
one of the most common errors you'll find in PyTorch, or in, well, when you're writing PyTorch
code in Google CoLab, I'm not sure why, but it just happens. So these are our models predictions
so far by running the test data through our models Ford method that we defined. And so if
we look at Y test, are these close? Oh my gosh, they are shocking. So why don't we visualize them?
Plot predictions. And we're going to put in predictions equals Y threads.
Let's have a look. Oh my goodness. All the way over here. Remember how we discussed before
that an ideal model will have, what, red dots on top of the green dots because our ideal model
will be perfectly predicting the test data. So right now, because our model is initialized with
random parameters, it's basically making random predictions. So they're extremely far from where
our ideal predictions are, is that we'll have some training data. And our model predictions,

Section 56: Main Topics

Key Topics:

That's something called transfer learning

📄 Click to view detailed content

when we first create our model will be quite bad. But we want to write some code that will
hopefully move these red dots closer to these green dots. I'm going to see how we can do that in
later videos. But we did one thing up here, which we haven't discussed, which is with torch dot
inference mode. Now this is a context manager, which is what happens when we're making predictions.
So making predictions, another word for predictions is inference torch uses inference. So I'll try
to use that a bit more, but I like to use predictions as well. We could also just go
Y preds equals model zero dot X test. And we're going to get quite a similar output.
Right. But I've put on inference mode because I want to start making that a habit for later on,
when we make predictions, put on inference mode. Now why do this? You might notice something different.
What's the difference here between the outputs? Y preds equals model. There's no inference mode
here, no context manager. Do you notice that there's a grad function here? And we don't need to go
into discussing what exactly this is doing here. But do you notice that this one is lacking that
grad function? So do you remember how behind the scenes I said that pie torch does a few things
with requires grad equals true, it keeps track of the gradients of different parameters so that
they can be used in gradient descent and back propagation. Now what inference mode does is it
turns off that gradient tracking. So it essentially removes all of the, because when we're doing
inference, we're not doing training. So we don't need to keep track of the gradient. So we don't
need to do to keep track of how we should update our models. So inference mode disables all of the
useful things that are available during training. What's the benefit of this? Well, it means that
pie torch behind the scenes is keeping track of less data. So in turn, it will, with our small
data set, it probably won't be too dramatic. But with a larger data set, it means that your
predictions will potentially be a lot faster because a whole bunch of numbers aren't being
kept track of or a whole bunch of things that you don't need during prediction mode or inference
mode. That's why it's called inference mode. I'm not being saved to memory. If you'd like to
learn more about this, you go pie torch inference mode Twitter. I just remember to search for Twitter
because they did a big tweet storm about it. Here we go. So oh, this is another thing that we can
cover. I'm going to copy this in here. But there's also a blog post about what's going on behind
the scenes. Long story short, it makes your code faster. Want to make your inference code and pie
torch run faster? Here's a quick thread on doing exactly that. And that's what we're doing. So
I'm going to write down here. See more on inference mode here.
And I just want to highlight something as well is that they referenced torch no grad with the
torch inference mode context manager. Inference mode is fairly new in pie torch. So you might
see a lot of code existing pie torch code with torch dot no grad. You can use this as well.
Why? Preds equals model zero. And this will do much of the same as what inference mode is doing.
But inference mode has a few things that are advantages over no grad, which are discussed in
this thread here. But if we do this, we get very similar output to what we got before.
Grad function. But as you'll read in here and in the pie torch documentation, inference mode is
the favored way of doing inference for now. I just wanted to highlight this. So you can also do
something similar with torch dot no grad. However, inference mode is preferred. Alrighty. So I'm
just going to comment this out. So we just have one thing going on there. The main takeaway
from this video is that when we're making predictions, we use the context manager torch
dot inference mode. And right now, because our models variables or internal parameters are
randomly initialized, our models predictions are as good as random. So they're actually not too far
off where our values are. At least the red dots aren't like scattered all over here. But in the
upcoming videos, we're going to be writing some pie torch training code to move these values
closer to the green dots by looking at the training data here. So with that being said,
I'll see you in the next video. Friends, welcome back. In the last video, we saw that our model
performs pretty poorly. Like, ideally, these red dots should be in line with these green dots.
And we know that because why? Well, it's because our model is initialized with random parameters.
And I just want to put a little note here. You don't necessarily have to initialize your model
with random parameters. You could initialize it with this could be zero. Yeah, these two values,
weights can bias could be zero and you could go from there. Or you could also use the parameters
from another model. But we're going to see that later on. That's something called transfer learning.
That's just a little spoiler for what's to come. And so we've also discussed that an ideal model
will replicate these known parameters. So in other words, start with random unknown parameters,
these two values here. And then we want to write some code for our model to move towards estimating
the ideal parameters here. Now, I just want to be explicit here and write down some intuition

Section 57: Main Topics

Key Topics:

We're about to get into training our very first machine learning model
And so if we go pytorch loss functions, we're going to see that pytorch has a fair few loss functions built in
We're learning something fun

📄 Click to view detailed content

before we jump into the training code. But this is very exciting. We're about to get into
training our very first machine learning model. So what's right here, the whole idea of training
is for a model to move from some unknown parameters, these may be random to some known parameters.
Or in other words, from a poor representation, representation of the data to a better representation
of the data. And so in our case, would you say that our models representation of the green dots
here with this red dots, is that a good representation? Or is that a poor representation?
I mean, I don't know about you, but I would say that to me, this is a fairly poor representation.
And one way to measure the representation between your models outputs, in our case, the red dots,
the predictions, and the testing data, is to use a loss function. So I'm going to write
this down here. This is what we're moving towards. We're moving towards training, but we need a
way to measure how poorly our models predictions are doing. So one way to measure how poor or how
wrong your models predictions are, is to use a loss function. And so if we go pytorch loss
functions, we're going to see that pytorch has a fair few loss functions built in. But the essence
of all of them is quite similar. So just wait for this to load my internet's going a little bit
slow today, but that's okay. We're not in a rush here. We're learning something fun.
If I search here for loss, loss functions, here we go. So yeah, this is torch in N. These are the
basic building blocks for graphs, whole bunch of good stuff in here, including loss functions.
Beautiful. And this is another thing to note as well, another one of those scenarios where
there's more words for the same thing. You might also see a loss function referred to as a criterion.
There's another word called cost function. So I might just write this down so you're aware of it.
Yeah, cost function versus loss function. And maybe some formal definitions about what all of these
are. Maybe they're used in different fields. But in the case of we're focused on machine learning,
right? So I'm just going to go note, loss function may also be called cost function or criterion in
different areas. For our case, we're going to refer to it as a loss function. And let's
just formally define a loss function here, because we're going to go through a fair few steps in
the upcoming videos. So this is a warning, nothing we can't handle. But I want to put some formal
definitions on things. We're going to see them in practice. That's what I prefer to do,
rather than just sit here defining stuff. This lecture has already had enough text on the page.
So hurry up and get into coding Daniel. A loss function is a function to measure how wrong your
models predictions are to the ideal outputs. So lower is better. So ideally, think of a measurement,
how could we measure the difference between the red dots and the green dots? One of the
simplest ways to do so would be just measure the distance here, right? So if we go, let's just
estimate this is 035 to 0.8. They're abouts. So what's the difference there? About 0.45.
Then we could do the same again for all of these other dots, and then maybe take the average of that.
Now, if you've worked with loss functions before, you might have realized that I've just
reproduced mean absolute error. But we're going to get to that in a minute. So we need a loss
function. I'm going to write down another little dot point here. This is just setting up intuition.
Things we need to train. We need a loss function. This is PyTorch. And this is machine learning
in general, actually. But we're focused on PyTorch. We need an optimizer. What does the optimizer do?
Takes into account the loss of a model and adjusts the model's parameters. So the parameters recall
our weight and bias values. Weight and biases. We can check those or bias. We can check those by
going model dot parameter or parameters. But I also like, oh, that's going to give us a generator,
isn't it? Why do we not define the model yet? What do we call our model? Oh, model zero. Excuse me.
I forgot where. I'm going to build a lot of models in this course. So we're giving them numbers.
Modeled up parameters. Yeah, we've got a generator. So we'll turn that into a list.
But model zero, if we want to get them labeled, we want state dict here.
There we go. So our weight is this value. That's a random value we've set. And there's the bias.
And now we've only got two parameters for our model. So it's quite simple. However, the principles
that we're learning here are going to be the same principles, taking a loss function,
trying to minimize it, so getting it to lower. So the ideal model will predict exactly what our
test data is. And an optimizer will take into account the loss and will adjust a model's parameter.
And our case weights and bias to be, let's finish this definition takes into account the
loss of a model and adjust the model's parameters, e.g. weight and bias, in our case, to improve the
loss function. And specifically, for PyTorch, we need a training loop and a testing loop.
Now, this is what we're going to work towards building throughout the next couple of videos.
We're going to focus on these two first, the loss function and optimizer. There's the formal
definition of those. You're going to find many different definitions. That's how I'm going to

Section 58: Main Topics

Key Topics:

I mean, the whole course is fun
But this is really exciting because training your first machine learning model seems a little bit like magic, but it's even more fun when you're writing the code yourself what's going on behind the scenes
That is the whole idea of a training loop in PyTorch, or an optimization loop in PyTorch

📄 Click to view detailed content

find them. Loss function measures how wrong your model's predictions are, lower is better,
optimizer takes into account the loss of your model. So how wrong it is, and starts to move
these two values into a way that improves where these red dots end up. But these, again, these
principles of a loss function and an optimizer can be for models with two parameters or models
with millions of parameters, can be for computer vision models, or could be for simple models like
ours that predict the dots on a straight line. So with that being said, let's jump into the next
video. We'll start to look a little deeper into loss function, row problem, and an optimizer.
I'll see you there. Welcome back. We're in the exciting streak of videos coming up here. I mean,
the whole course is fun. Trust me. But this is really exciting because training your first machine
learning model seems a little bit like magic, but it's even more fun when you're writing the code
yourself what's going on behind the scenes. So we discussed that the whole concept of training
is from going unknown parameters, random parameters, such as what we've got so far
to parameters that better represent the data. And we spoke of the concept of a loss function.
We want to minimize the loss function. That is the whole idea of a training loop in PyTorch,
or an optimization loop in PyTorch. And an optimizer is one of those ways that can
nudge the parameters of our model. In our case, weights or bias towards values rather than just
being random values like they are now towards values that lower the loss function. And if we
lower the loss function, what does a loss function do? It measures how wrong our models
predictions are compared to the ideal outputs. So if we lower that, well, hopefully we move
these red dots towards the green dots. And so as you might have guessed, PyTorch has some built
in functionality for implementing loss functions and optimizers. And by the way, what we're covering
so far is in the train model section of the PyTorch workflow fundamentals, I've got a little
nice table here, which describes a loss function. What does it do? Where does it live in PyTorch?
Common values, we're going to see some of these hands on. If you'd like to read about it,
of course, you have the book version of the course here. So loss functions in PyTorch,
I'm just in docstorch.nn. Look at this. Look at all these loss functions. There's far too many
for us to go through all in one hit. So we're just going to focus on some of the most common ones.
Look at that. We've got about what's our 15 loss functions, something like that? Well, truth be
told is that which one should use? You're not really going to know unless you start to work hands
on with different problems. And so in our case, we're going to be looking at L1 loss. And this is
an again, once more another instance where different machine learning libraries have different names
for the same thing, this is mean absolute error, which we kind of discussed in the last video,
which is if we took the distance from this red dot to this green dot and say at 0.4, they're about
0.4, 0.4, and then took the mean, well, we've got the mean absolute error. But in PyTorch,
they call it L1 loss, which is a little bit confusing because then we go to MSE loss,
which is mean squared error, which is L2. So naming conventions just takes a little bit of getting
used to this is a warning for you. So let's have a look at the L1 loss function. Again,
I'm just making you aware of where the other loss functions are. We'll do with some binary
cross entropy loss later in the course. And maybe even is that categorical cross entropy?
We'll see that later on. But all the others will be problem specific. For now, a couple of loss
functions like this, L1 loss, MSE loss, we use for regression problems. So that's predicting a number.
Cross entropy loss is a loss that you use with classification problems. But we'll see those hands
on later on. Let's have a look at L1 loss. So L1 loss creates a criterion. As I said, you might
hear the word criterion used in PyTorch for a loss function. I typically call them loss functions.
The literature typically calls it loss functions. That measures the mean absolute error. There we
go. L1 loss is the mean absolute error between each element in the input X and target Y. Now,
your extracurricular measure might have guessed is to read through the documentation for the
different loss functions, especially L1 loss. But for the sake of this video, let's just implement
it for ourselves. Oh, and if you want a little bit of a graphic, I've got one here. This is where
we're up to, by the way, picking a loss function optimizer for step two. This is a fun part, right?
We're getting into training a model. So we've got mean absolute error. Here's that graph we've
seen before. Oh, look at this. Okay. So we've got the difference here. I've actually measured
this before in the past. So I kind of knew what it was. Mean absolute error is if we repeat for
all samples in our set that we're working with. And if we take the absolute difference between
these two dots, well, then we take the mean, we've got mean absolute error. So MAE loss equals
torch mean we could write it out. That's the beauty of pine torch, right? We could write this out.
Or we could use the torch and N version, which is recommended. So let's jump in. There's a colorful

Section 59: Main Topics

Key Topics:

Our objective for training a machine learning model will be two
Often picking a loss function and optimizer and pytorch come as part of the same package because they work together
So again, pytorch has torch

📄 Click to view detailed content

slide describing what we're about to do. So let's go set up a loss function. And then we're also
going to put in here, set up an optimizer. So let's call it loss FN equals NN dot L1 loss.
Simple as that. And then if we have a look at what's our loss function, what does this say?
Oh my goodness. My internet is going quite slow today.
It's raining outside. So there might be some delays somewhere. But that's right. Gives us a
chance to sit here and be mindful about what we're doing. Look at that. Okay. Loss function.
L1 loss. Beautiful. So we've got a loss function. Our objective for training a machine learning
model will be two. Let's go back. Look at the colorful graphic will be to minimize these
distances here. And in turn, minimize the overall value of MAE. That is our goal.
If our red dots line up with our green dots, we will have a loss value of zero, the ideal point
for a model to be. And so let's go here. We now need an optimizer. As we discussed before,
the optimizer takes into account the loss of a model. So these two work in tandem.
That's why I've put them as similar steps if we go back a few slides.
So this is why I put these as 2.1. Often picking a loss function and optimizer and pytorch
come as part of the same package because they work together. The optimizer's objective is to
give the model values. So parameters like a weight and a bias that minimize the loss function.
They work in tandem. And so let's see what an optimizer optimizes. Where might that be?
What if we search here? I typically don't use this search because I prefer just using Google
search. But does this give us optimizer? Hey, there we go. So again, pytorch has torch.optim
which is where the optimizers are. Torch.optim. Let me put this link in here.
This is another bit of your extracurricular. If you want to read more about different optimizers
in pytorch, as you might have guessed, they have a few. Torch.optim is a package implementing
various optimization algorithms. Most commonly used methods are already supported and the interface
is general enough so that more sophisticated ones can also be easily integrated into the future.
So if we have a look at what algorithms exist here, again, we're going to throw a lot of names
at you. But in the literature, a lot of them that have made it into here are already good working
algorithms. So it's a matter of picking whichever one's best for your problem. How do you find that
out? Well, SGD, stochastic gradient descent, is possibly the most popular. However, there are
some iterations on SGD, such as Adam, which is another one that's really popular. So again,
this is one of those other machine learning is part art, part science is trial and error of
figuring out what works best for your problem for us. We're going to start with SGD because
it's the most popular. And if you were paying attention to a previous video, you might have
seen that I said, look up gradient descent, wherever we got this gradient descent. There we go.
So this is one of the main algorithms that improves our models. So gradient descent and back
propagation. So if we have a look at this stochastic gradient descent, bit of a tongue twister,
is random gradient descent. So that's what stochastic means. So basically, our model
improves by taking random numbers, let's go down here, here, and randomly adjusting them
so that they minimize the loss. And once how optimizer, that's right here, once how optimizer
torch dot opt in, let's implement SGD, SGD stochastic gradient descent. We're going to write this here,
stochastic gradient descent. It starts by randomly adjusting these values. And once it's found
some random values or random steps that have minimized the loss value, we're going to see
this in action later on, it's going to continue adjusting them in that direction. So say it says,
oh, weights, if I increase the weights, it reduces the loss. So it's going to keep increasing the
weights until the weights no longer reduce the loss. Maybe it gets to a point at say 0.65.
If you increase the weights anymore, the loss is going to go up. So the optimizer is like,
well, I'm going to stop there. And then for the bias, the same thing happens. If it decreases the
bias and finds that the loss increases, well, it's going to go, well, I'm going to try increasing
the bias instead. So again, one last summary of what's going on here, a loss function measures
how wrong our model is. And the optimizer adjust our model parameters, no matter whether there's
two parameters or millions of them to reduce the loss. There are a couple of things that
an optimizer needs to take in. It needs to take in as an argument, params. So this is if we go to
SGD, I'm just going to link this as well. SGD, there's the formula of what SGD does. I look at this
and I go, hmm, there's a lot going on here. And take me a while to understand that. So I like to
see it in code. So we need params. This is short for what parameters should I optimize as an optimizer.
And then we also need an LR, which stands for, I'm going to write this in a comment, LR equals
learning rate, possibly the most, oh, I didn't even type rate, did I possibly the most important
hyper parameter you can set? So let me just remind you, I'm throwing lots of words out here, but I'm
kind of like trying to write notes about what we're doing. Again, we're going to see these in action

Section 60: Main Topics

Key Topics:

So learning rate equals possibly the most important learning hyper parameter
I don't need learning there, do I
And a hyper parameter is a value that us as a data scientist or a machine learning engineer set ourselves, you can set

📄 Click to view detailed content

in a second. So check out our models and parameters. So a parameter is a value that the model sets
itself. So learning rate equals possibly the most important learning hyper parameter. I don't
need learning there, do I? Hyper parameter. And a hyper parameter is a value that us as a data scientist
or a machine learning engineer set ourselves, you can set. So the learning rate is, in our case,
let's go 0.01. You're like, Daniel, where did I get this value from? Well, again, these type of
values come with experience. I think it actually says it in here, LR, LR 0.1. Yeah, okay, so the
default is 0.1. But then if we go back to Optim, I think I saw it somewhere. Did I see it somewhere?
0.0? Yeah, there we go. Yeah, so a lot of the default settings are pretty good in torch optimizers.
However, the learning rate, what does it actually do? We could go 0.01. These are all common values
here. Triple zero one. I'm not sure exactly why. Oh, model, it's model zero. The learning rate says
to our optimizer, yes, it's going to optimize our parameters here. But the higher the learning
rate, the more it adjusts each of these parameters in one hit. So let's say it's 0.01. And it's going
to optimize this value here. So it's going to take that big of a step. If we changed it to here,
it's going to take a big step on this three. And if we changed it to all the way to the end 0.01,
it's only going to change this value. So the smaller the learning rate, the smaller the change
in the parameter, the larger the learning rate, the larger the change in the parameter.
So we've set up a loss function. We've set up an optimizer. Let's now move on to the next step
in our training workflow. And that's by building a training loop. Far out. This is exciting. I'll
see you in the next video. Welcome back. In the last video, we set up a loss function. And we set
up an optimizer. And we discussed the roles of each. So loss function measures how wrong our model
is. The optimizer talks to the loss function and goes, well, if I change these parameters a certain
way, does that reduce the loss function at all? And if it does, yes, let's keep adjusting them in
that direction. If it doesn't, let's adjust them in the opposite direction. And I just want to show
you I added a little bit of text here just to concretely put down what we were discussing.
Inside the optimizer, you'll often have to set two parameters, params and lr, where params is
the model parameters you'd like to optimize for an example, in our case, params equals our model
zero parameters, which were, of course, a weight and a bias. And the learning rate, which is lr
in optimizer, lr stands for learning rate. And the learning rate is a hyper parameter. Remember,
a hyper parameter is a value that we the data scientist or machine learning engineer sets,
whereas a parameter is what the model sets itself defines how big or smaller optimizer changes
the model parameters. So a small learning rate, so the smaller this value results in small
changes, a large learning rate results in large changes. So another question might be,
well, very valid question. Hey, I put this here already, is which loss function and which optimizer
should I use? So this is another tough one, because it's problem specific. But with experience
and machine learning, I'm showing you one example here, you'll get an idea of what works for your
particular problem for a regression problem, like ours, a loss function of l1 loss, which is mai
and pytorch. And an optimizer like torch dot opt in slash s gd like sarcastic gradient descent
will suffice. But for a classification problem, we're going to see this later on.
Not this one specifically, whether a photo is a cat of a dog, that's just an example of a binary
classification problem, you might want to use a binary classification loss. But with that being
said, we now are moving on to, well, here's our whole goal is to reduce the MAE of our model.
Let's get the workflow. We've done these two steps. Now we want to build a training loop. So
let's get back into here. There's going to be a fair few steps going on. We've already covered
a few, but hey, nothing we can't handle together. So building a training loop in pytorch.
So I thought about just talking about what's going on in the training loop, but we can talk
about the steps after we've coded them. How about we do that? So we want to build a training loop
and a testing loop. How about we do that? So a couple of things we need in a training loop.
So there's going to be a fair few steps here if you've never written a training loop before,
but that is completely fine because you'll find that the first couple of times that you write this,
you'll be like, oh my gosh, there's too much going on here. But then when you have practice,
you'll go, okay, I see what's going on here. And then eventually you'll write them with your
eyes closed. I've got a fun song for you to help you out remembering things. It's called the
unofficial pytorch optimization loop song. We'll see that later on, or actually, I'll probably leave
that as an extension, but you'll see that you can also functionize these things, which we will do
later in the course so that you can just write them once and then forget about them. But we're
going to write it all from scratch to begin with so we know what's happening. So we want to,

Section 61: Main Topics

Key Topics:

Many people, one of the main things I get asked from machine learning is how do I learn machine learning if I didn't do math
Well, the beautiful thing about PyTorch is that it implements a lot of the math of back propagation
So these two algorithms drive the majority of our learning

📄 Click to view detailed content

or actually step zero, is loop through the data. So we want to look at the data multiple times
because our model is going to, at first, start with random predictions on the data, make some
predictions. We're trying to improve those. We're trying to minimize the loss to make those
predictions. We do a forward pass. So forward pass. Why is it called a forward pass? So this
involves data moving through our model's forward functions. Now that I say functions because there
might be plural, there might be more than one. And the forward method recall, we wrote in our model
up here. Ford. A forward pass is our data going through this function here. And if you want to
look at it visually, let's look up a neural network graphic. Images, a forward pass is just
data moving from the inputs to the output layer. So starting here input layer moving through the
model. So that's a forward pass, also called forward propagation. Another time we'll have
more than one name is used for the same thing. So we'll go back down here, forward pass. And
I'll just write here also called forward propagation, propagation. Wonderful. And then we need to
calculate the loss. So forward pass. Let me write this. To calculate or to make predictions,
make predictions on data. So calculate the loss, compare forward pass predictions. Oh, there's
an undergoing in the background here of my place. We might be in for a storm. Perfect time to write
code, compare forward pass predictions to ground truth labels. We're going to see all this in code
in a second, calculate the loss. And then we're going to go optimise a zero grad. We haven't
spoken about what this is, but that's okay. We're going to see that in a second. I'm not going to
put too much there. Loss backward. We haven't discussed this one either. There's probably three
steps that we haven't really discussed. We've discussed the idea behind them, but not too much
in depth. Optimise our step. So this one is loss backwards is move backwards. If the forward pass
is forwards, like through the network, the forward pass is data goes into out. The backward pass
data goes, our calculations happen backwards. So we'll see what that is in a second. Where were
we over here? We've got too much going on. I'm getting rid of these moves backwards through
the network to calculate the gradients. Oh, oh, the gradients of each of the parameters
of our model with respect to the loss. Oh my gosh, that is an absolute mouthful,
but that'll do for now. Optimise a step. This is going to use the optimiser to adjust our
model's parameters to try and improve the loss. So remember how I said in a previous video
that I'd love you to watch the two videos I linked above. One on gradient descent and one
on back propagation. If you did, you might have seen like there's a fair bit of math going on in
there. Well, that's essentially how our model goes from random parameters to better parameters,
using math. Many people, one of the main things I get asked from machine learning is how do I
learn machine learning if I didn't do math? Well, the beautiful thing about PyTorch is that it
implements a lot of the math of back propagation. So this is back propagation. I'm going to write
this down here. This is an algorithm called back, back propagation, hence the loss backward. We're
going to see this in code in a second, don't you worry? And this is gradient descent. So these
two algorithms drive the majority of our learning. So back propagation, calculate the gradients of
the parameters of our model with respect to the loss function and optimise our step,
we'll trigger code to run gradient descent, which is to minimise the gradients because what is a
gradient? Let's look this up. What is a gradient? I know we haven't written a code yet, but we're
going to do that. Images. Gradient, there we go. Changing y, changing x. Gradient is from high
school math. Gradient is a slope. So if you were on a hill, let's find a picture of a hill.
Picture of a hill. There we go. This is a great big hill. So if you were on the top of this hill,
and you wanted to get to the bottom, how would you get to the bottom? Well, of course, you just
walked down the hill. But if you're a machine learning model, what are you trying to do? Let's
imagine your loss is the height of this hill. You start off with your losses really high, and you
want to take your loss down to zero, which is the bottom, right? Well, if you measure the gradient
of the hill, the bottom of the hill is in the opposite direction to where the gradient is steep.
Does that make sense? So the gradient here is an incline. We want our model to move towards the
gradient being nothing, which is down here. And you could argue, yeah, the gradient's probably
nothing up the top here, but let's just for argument's sake say that we want to get to the
bottom of the hill. So we're measuring the gradient, and one of the ways an optimisation algorithm
works is it moves our model parameters so that the gradient equals zero, and then if the gradient
of the loss equals zero, while the loss equals zero two. So now let's write some code. So we're
going to set up a parameter called or a variable called epochs. And we're going to start with one,
even though this could be any value, let me define these as we go. So we're going to write code to

Section 62: Main Topics

Key Topics:

And I know you could argue that, hey, our machine learning parameters of model zero, or our model parameters, model zero aren't actually parameters, because we've set them
So pytorch models have a couple of different modes
So what does train mode do in a pytorch model

📄 Click to view detailed content

do all of this. So epochs, an epoch is one loop through the data dot dot dot. So epochs, we're
going to start with one. So one time through all of the data, we don't have much data. And so
for epoch, let's go this, this is step zero, zero, loop through the data. By the way, when I say
loop through the data, I want you to do all of these steps within the loop. And do dot dot dot
loop through the data. So for epoch in range epochs, even though it's only going to be one,
we can adjust this later. And because epochs, we've set this ourselves, it is a,
this is a hyper parameter, because we've set it ourselves. And I know you could argue that,
hey, our machine learning parameters of model zero, or our model parameters, model zero aren't
actually parameters, because we've set them. But in the models that you build in the future,
they will likely be set automatically rather than you setting them explicitly like we've done when
we created model zero. And oh my gosh, this is taking quite a while to run. That's all right.
We don't need it to run fast. We just, we need to write some more code, then you'll come on.
There's a step here I haven't discussed either. Set the model to training mode. So pytorch models
have a couple of different modes. The default is training mode. So we can set it to training
mode by going like this. Train. So what does train mode do in a pytorch model? My goodness.
Is there a reason my engineer is going this slide? That's all right. I'm just going to
discuss this with talking again list. Train mode. Train mode in pytorch sets. Oh, there we go.
Requires grad equals true. Now I wonder if we do with torch dot no grad member no grad is similar
to inference mode. Will this adjust? See, I just wanted to take note of requires grad equals
true. Actually, what I might do is we do this in a different cell. Watch this. This is just going
to be rather than me just spit words at you. I reckon we might be able to get it work in doing
this. Oh, that didn't list the model parameters. Why did that not come out? Model zero dot eval.
So there's two modes of our mode and train mode model dot eval parameters. Hey, we're experimenting
together on the fly here. And actually, this is what I want you to do is I want you to experiment
with different things. It's not going to say requires grad equals false. Hmm. With torch dot no
grad. Model zero dot parameters. I don't know if this will work, but it definitely works behind
the scenes. And what I mean by works behind the scenes are not here. It works behind the scenes
when calculations have been made, but not if we're trying to explicitly print things out.
Well, that's an experiment that I thought was going to work and it didn't work. So train
mode in pytorch sets all parameters that require gradients to require gradients.
So do you remember with the picture of the hill? I spoke about how we're trying to minimize the
gradient. So the gradient is the steepness of the hill. If the height of the hill is a loss function
and we want to take that down to zero, we want to take the gradient down to zero. So same thing
with the gradients of our model parameters, which are here with respect to the loss function,
we want to try and minimize that gradient. So that's gradient descent is take that gradient down to
zero. So model dot train. And then there's also model zero dot a vowel. So turns off gradient
tracking. So we're going to see that later on. But for now, I feel like this video is getting far
too long. Let's finish the training loop in the next video. I'll see you there.
Friends, welcome back. In the last video, I promised a lot of code, but we didn't get there. We
discussed some important steps. I forgot how much behind the scenes there is to apply towards training
loop. And I think it's important to spend the time that we did discussing what's going on,
because there's a fair few steps. But once you know what's going on, I mean, later on, we don't
have to write all the code that we're going to write in this video, you can functionize it. We're
going to see that later on in the course, and it's going to run behind the scenes for us. But we're
spending a fair bit of time here, because this is literally the crux of how our model learns. So
let's get into it. So now we're going to implement the forward pass, which involves our model's
forward function, which we defined up here. When we built our model, the forward pass runs through
this code here. So let's just write that. So in our case, because we're training, I'm just
going to write here. This is training. We're going to see dot of our later on. We'll talk
about that when it comes. Let's do the forward pass. So the forward pass, we want to pass data
through our model's forward method. We can do this quite simply by going y pred. So y predictions,
because remember, we're trying to use our ideal model is using x test to predict y test
on our test data set. We make predictions on our test data set. We learn on our training data set.
So we're passing, which is going to get rid of that because we don't need that. So we're
passing our model x train and model zero is going to be our current model. There we go. So we learn

Section 63: Main Topics

Key Topics:

And in our case, we're going to set loss equal to our loss function, which is L one loss in PyTorch, but it is MAE
But in the case of staying true to the documentation, let's do inputs first and then targets next for the rest of the course
And now the reason why it accumulates, that's pretty deep in the pytorch documentation

📄 Click to view detailed content

patterns on the training data to evaluate our model on the test data. Number two, where we are.
So we have to calculate the loss. Now, in a previous video, we set up a loss function.
So this is going to help us calculate the what what kind of loss are we using? We want to calculate
the MAE. So the difference or the distance between our red dot and a green dot. And the formula would
be the same if we had 10,000 red dots and 10,000 green dots, we're calculating how far they are
apart. And then we're taking the mean of that value. So let's go back here. So calculate the loss.
And in our case, we're going to set loss equal to our loss function, which is L one loss in
PyTorch, but it is MAE. Y-pred and Y-train. So we're calculating the difference between our models
predictions on the training data set and the ideal training values. And if you want to go into
torch dot NN loss functions, that's going to show you the order because sometimes this confuses me
to what order the values go in here, but it goes prediction first, then labels and I may be wrong
there because I get confused here. My dyslexia kicks in, but I'm pretty sure it's predictions first,
then actual labels. Do we have an example of where it's used? Yeah, import first, target next.
So there we go. And truth be told, because it's mean absolute error, it shouldn't actually matter
too much. But in the case of staying true to the documentation, let's do inputs first and then
targets next for the rest of the course. Then we're going to go optimizer zero grad. Hmm,
haven't discussed this one, but that's okay. I'm going to write the code and then I'm going to
discuss what it does. So what does this do? Actually, before we discuss this, I'm going to write
these two steps because they kind of all work together. And it's a lot easier to discuss what
optimizer zero grad does in the context of having everything else perform back propagation
on the loss with respect to the parameters of the model. Back propagation is going to take
the loss value. So lost backward, I always say backwards, but it's just backward. That's the code
there. And then number five is step the optimizer. So perform gradient descent. So optimizer dot
step. Oh, look at us. We just wrote the five major steps of a training loop. Now let's discuss
how all of these work together. So it's kind of strange, like the ordering of these, you might
think, Oh, what should I do the order? Typically the forward pass and the loss come straight up.
Then there's a little bit of ambiguity around what order these have to come in. But the optimizer
step should come after the back propagation. So I just like to keep this order how it is because
this works. Let's just keep it that way. But what happens here? Well, it also is a little bit
confusing in the first iteration of the loop because we've got zero grad. But what happens here is
that the optimizer makes some calculations in how it should adjust model parameters with regards to
the back propagation of the loss. And so by default, these will by default, how the optimizer
changes will accumulate through the loop. So we have to zero them above in step three
for the next iteration of the loop. So a big long comment there. But what this is saying is,
let's say we go through the loop and the optimizer chooses a value of one, change it by one. And
then it goes through a loop again, if we didn't zero it, if we didn't take it to zero, because
that's what it is doing, it's going one to zero, it would go, okay, next one, two, three, four,
five, six, seven, eight, all through the loop, right? Because we're looping here. If this was
10, it would accumulate the value that it's supposed to change 10 times. But we want it to start
again, start fresh each iteration of the loop. And now the reason why it accumulates, that's
pretty deep in the pytorch documentation. From my understanding, there's something to do with
like efficiency of computing. If you find out what the exact reason is, I'd love to know.
So we have to zero it, then we perform back propagation. If you recall, back propagation is
discussed in here. And then with optimizer step, we form gradient descent. So the beauty of pytorch,
this is the beauty of pytorch, is that it will perform back propagation, we're going to have a
look at this in second, and gradient descent for us. So to prevent this video from getting too long,
I know we've just written code, but I would like you to practice writing a training loop
yourself, just write this code, and then run it and see what happens. Actually, you can comment
this out, we're going to write the testing loop in a second. So your extra curriculum for this
video is to, one, rewrite this training loop, is to, two, sing the pytorch optimization loop
song, let's go into here. If you want to remember the steps, well, I've got a song for you. This is
the training loop song, we haven't discussed the test step, but maybe you could try this yourself.
So this is an old version of the song, actually, I've got a new one for you. But let's sing this
together. It's training time. So we do the forward pass, calculate the loss, optimise a zero grad,
loss backwards, optimise a step, step, step. Now you only have to call optimise a step once,
this is just for jingle purposes. But for test time, let's test with torch no grad, do the forward

Section 64: Main Topics

Key Topics:

And if you want the video version of it, well, you're just going to have to search unofficial pytorch optimisation loop song
In the last few videos, we've been discussing the steps in a training loop in pytorch
Essentially, it sets up a whole bunch of settings behind the scenes in our model parameters so that they can track the gradients and do a whole bunch of learning behind the scenes with these functions down here

📄 Click to view detailed content

pass, calculate the loss, watch it go down, down, down. That's from my Twitter, but this is a way
that I help myself remember the steps that are going on in the code here. And if you want the
video version of it, well, you're just going to have to search unofficial pytorch optimisation loop
song. Oh, look at that, who's that guy? Well, he looks pretty cool. So I'll let you check that
out in your own time. But for now, go back through the training loop steps. I've got a colorful
graphic coming up in the next video, we're going to write the testing steps. And then we're going
to go back one more time and talk about what's happening in each of them. And again, if you'd
like some even more extra curriculum, don't forget the videos I've shown you on back propagation
and gradient descent. But for now, let's leave this video here. I'll see you in the next one.
Friends, welcome back. In the last few videos, we've been discussing the steps in a training
loop in pytorch. And there's a fair bit going on. So in this video, we're going to step back
through what we've done just to recap. And then we're going to get into testing. And it's nice
and early where I am right now. The sun's about to come up. It's a very, very beautiful morning
to be writing code. So let's jump in. We've got a little song here for what we're doing in the
training steps. For an epoch in a range, comodel.train, do the forward pass, calculate the loss of the
measure zero grad, last backward of the measure step step step. That's the little jingle I use to
remember the steps in here, because the first time you write it, there's a fair bit going on.
But subsequent steps and subsequent times that you do write it, you'll start to memorize this.
And even better later on, we're going to put it into a function so that we can just call it
over and over and over and over again. With that being said, let's jump in to a colorful slide,
because that's a lot of code on the page. Let's add some color to it, understand what's happening.
That way you can refer to this and go, Hmm, I see what's going on now. So for the loop, this is why
it's called a training loop. We step through a number of epochs. One epoch is a single forward
pass through the data. So pass the data through the model for a number of epochs. Epox is a
hyper parameter, which means you could set it to 100, you could set it to 1000, you could set it
to one as we're going to see later on in this video. We skip this step with the colors, but
we put the model in we call model.train. This is the default mode that the model is in.
Essentially, it sets up a whole bunch of settings behind the scenes in our model parameters so that
they can track the gradients and do a whole bunch of learning behind the scenes with these
functions down here. PyTorch does a lot of this for us. So the next step is the forward pass.
We perform a forward pass on the training data in the training loop. This is an important note.
In the training loop is where the model learns patterns on the training data. Whereas in the
testing loop, we haven't got to that yet is where we evaluate the patterns that our model has learned
or the parameters that our model has learned on unseen data. So we pass the data through the model,
this will perform the forward method located within the model object. So because we created
a model object, you can actually call your models whatever you want, but it's good practice to
you'll often see it just called model. And if you remember, we'll go back to the code.
We created a forward method in our model up here, which is this, because our linear regression model,
class, subclasses, nn.module, we need to create our own custom forward method. So that's why it's
called a forward pass is because not only does it, well, the technical term is forward propagation.
So if we have a look at a neural network picture, forward propagation just means going through
the network from the input to the output, there's a thing called back propagation, which we're going
to discuss in a second, which happens when we call loss.backward, which is going backward through
the model. But let's return to our colorful slide. We've done the forward pass, call a forward method,
which performs some calculation on the data we pass it. Next is we calculate the loss value,
how wrong the model's predictions are. And this will depend on what loss function you use,
what kind of predictions your model is outputting, and what kind of true values you have.
But that's what we're doing here. We're comparing our model's predictions on the training data
to what they should ideally be. And these will be the training labels. The next step, we zero
the optimizer gradients. So why do we do this? Well, it's a little confusing for the first epoch in
the loop. But as we get down to optimizer dot step here, the gradients that the optimizer
calculates accumulate over time so that for each epoch for each loop step, we want them to go back
to zero. And now the exact reason behind why the optimizer accumulates gradients is buried somewhere
within the pie torch documentation. I'm not sure of the exact reason from memory. It's because of
compute optimization. It just adds them up in case you wanted to know what they were. But if

Section 65: Main Topics

Key Topics:

Now we want it in machine learning
What's it going to give us in machine learning, a gradient is a derivative of a function that has more than one input variable
And then we take a learning step

📄 Click to view detailed content

you find out exactly, I'd love to know. Next step is to perform back propagation on the loss function.
That's what we're calling loss. backward. Now back propagation is we compute the gradient of
every parameter with requires grad equals true. And if you recall, we go back to our code.
We've set requires grad equals true for our parameters. Now the reason we've set requires
grad equals true is not only so back propagation can be performed on it. But let me show you what
the gradients look like. So let's go loss function curve. That's a good idea. So we're looking for
so we're looking for some sort of convex curve here. There we go. L two loss. We're using L one loss
at the moment. Is there a better one here? All we need is just a nice looking curve. Here we go.
So this is why we keep track of the gradients behind the scenes. Pie torch is going to create
some sort of curve for all of our parameters that looks like this. Now this is just a 2d plot.
So the reason why we're just using an example from Google images is one, because you're going to
spend a lot of your time Googling different things. And two, in practice, when you have your own
custom neural networks, right now we only have two parameters. So it's quite easy to visualize a
loss function curve like this. But when you have say 10 million parameters, you basically can't
visualize what's going on. And so pie torch again will take care of these things behind the scenes.
But what it's doing is when we say requires grad pie torch is going to track the gradients
of each of our parameters. And so what we're trying to do here with back propagation and
subsequently gradient descent is calculate where the lowest point is. Because this is a loss function,
this is MSC loss, we could trade this out to be MAE loss in our case or L1 loss for our specific
problem. But this is some sort of parameter. And we calculate the gradients because what is the
gradient? Let's have a look. What is a gradient? A gradient is an inclined part of a road or railway.
Now we want it in machine learning. What's it going to give us in machine learning, a gradient
is a derivative of a function that has more than one input variable. Okay, let's dive in a little
deeper. See, here's some beautiful loss landscapes. We're trying to get to the bottom of here. This
is what gradient descent is all about. So oh, there we go. So this is a cost function, which is also a
loss function. We start with a random initial variable. What have we done? We started with a
random initial variable. Right? Okay. And then we take a learning step. Beautiful. This is W. So
this could be our weight parameter. Okay, we're connecting the dots here. This is exciting.
We've got a lot of tabs here, but that's all right. We'll bring this all together in a second.
And what we're trying to do is come to the minimum. Now, why do we need to calculate the gradients?
Well, the gradient is what? Oh, value of weight. Here we go. This is even better.
I love Google images. So this is our loss. And this is a value of a weight. So we calculate the
gradients. Why? Because the gradient is the slope of a line or the steepness. And so if we
calculate the gradient here, and we find that it's really steep right up the top of this,
this incline, we might head in the opposite direction to that gradient. That's what gradient
descent is. And so if we go down here, now, what are these step points? There's a little thing that
I wrote down in the last video at the end of the last video I haven't told you about yet,
but I was waiting for a moment like this. And if you recall, I said kind of all of these three steps
optimizes zero grad loss backward, optimizes step are all together. So we calculate the
gradients because we want to head in the opposite direction of that gradient to get to a gradient
value of zero. And if we get to a gradient value of zero with a loss function, well, then the loss
is also zero. So that's why we keep track of a gradient with requires grad equals true.
And again, PyTorch does a lot of this behind the scenes. And if you want to dig more into
what's going on here, I'm going to show you some extra resources for back propagation,
which is calculating this gradient curve here, and gradient descent, which is finding the bottom
of it towards the end of this video. And again, if we started over this side, we would just go
in the opposite direction of this. So maybe this is a positive gradient here, and we just go in the
opposite direction here. We want to get to the bottom. That is the main point of gradient descent.
And so if we come back, I said, just keep this step size in mind here. If we come back to where
we created our loss function and optimizer, I put a little tidbit here for the optimizer.
Because we've written a lot of code, and we haven't really discussed what's going on, but
I like to do things on the fly as we need them. So inside our optimizer, we'll have main two
parameters, which is params. So the model parameters you'd like to optimize,
params equals model zero dot parameters in our case. And then PyTorch is going to create
something similar to this curve, not visually, but just mathematically behind the scenes for

Section 66: Main Topics

Key Topics:

That's the beauty of PyTorch
Then within the optimizer, once we've told it what parameters to optimize, we have the learning rate
So the learning rate is another hyper parameter that defines how big or small the optimizer changes the parameters with each step

📄 Click to view detailed content

every parameter. Now, this is a value of weight. So this would just be potentially the weight
parameter of our network. But again, if you have 10 million parameters, there's no way you could
just create all of these curves yourself. That's the beauty of PyTorch. It's doing this behind the
scenes through a mechanism called torch autograd, which is auto gradient calculation. And there's
beautiful documentation on this. If you'd like to read more on how it works, please go through
that. But essentially behind the scenes, it's doing a lot of this for us for each parameter.
That's the optimizer. Then within the optimizer, once we've told it what parameters to optimize,
we have the learning rate. So the learning rate is another hyper parameter that defines how big or
small the optimizer changes the parameters with each step. So a small learning rate results in
small changes, whereas a large learning rate is in large changes. And so if we look at this
curve here, we might at the beginning start with large steps, so we can get closer and closer to
the bottom. But then as we get closer and closer to the bottom, to prevent stepping over to this
side of the curve, we might do smaller and smaller steps. And the optimizer in PyTorch,
there are optimizers that do that for us. But there is also another concept called learning
rate scheduling, which is, again, something if you would like to look up and do more. But
learning rate scheduling essentially says, hey, maybe start with some big steps. And then as we
get closer and closer to the bottom, reduce how big the steps are that we take. Because if you've
ever seen a coin, coin at the back of couch. This is my favorite analogy for this. If you've ever
tried to reach a coin at the back of a couch, like this excited young chap, if you're reaching
towards the back of a couch, you take quite big steps as you say your arm was over here,
you would take quite big steps until you get to about here. And in the closer you get to the coin,
the smaller and smaller your steps are. Otherwise, what's going to happen? The coin is going to be
lost. Or if you took two small steps, you'd never get to the coin. It would take forever to get there.
So that's the concept of learning rate. If you take two big steps, you're going to just end up
over here. If you take two small steps, it's going to take you forever to get to the bottom here.
And this bottom point is called convergence. That's another term you're going to come across. I
know I'm throwing a lot of different terms at you, but that's the whole concept of the learning
rate. How big is your step down here? In gradient descent. Gradient descent is this. Back propagation
is calculating these derivative curves or the gradient curves for each of the parameters in our
model. So let's get out of here. We'll go back to our training steps. Where were we? I think we're
up to back propagation. Have we done backward? Yes. So the back propagation is where we do the
backward steps. So the forward pass, forward propagation, go from input to output. Back propagation,
we take the gradients of the loss function with respect to each parameter in our model
by going backwards. That's what happens when we call loss.backward. PyTorch does that for us
behind the scenes. And then finally, step number five is step the optimizer. We've kind of discussed
that. As I said, if we take a step, let's get our loss curve back up. Loss function curve.
Doesn't really matter what curve we use. The optimizer step is taking a step this way to try
and optimize the parameters so that we can get down to the bottom here. And I also just noted
here that you can turn all of this into a function so we don't necessarily have to remember to
write these every single time. The ordering of this, you'll want to do the forward pass first.
And then calculate the loss because you can't calculate the loss unless you do the forward pass.
I like this ordering here of these three as well. But you also want to do the optimizer step
after the loss backward. So this is my favorite ordering. It works. If you like this ordering,
you can take that as well. With that being said, I think this video has gotten long enough.
In the next video, I'd like to step through this training loop one epoch at a time so that we can
see, I know I've just thrown a lot of words at you that this optimizer is going to try and
optimize our parameters each step. But let's see that in action how our parameters of our model
actually change every time we go through each one of these steps. So I'll see you in the next video.
Let's step through our model. Welcome back. And we've spent a fair bit of time on the training loop
and the testing loop. Well, we haven't even got to that yet, but there's a reason behind this,
because this is possibly one of the most important things aside from getting your data ready,
which we're going to see later on in PyTorch deep learning is writing the training loop,
because this is literally like how your model learns patterns and data. So that's why we're
spending a fair bit of time on here. And we'll get to the testing loop, because that's how you
evaluate the patterns that your model has learned from data, which is just as important as learning
the patterns themselves. And following on from the last couple of videos, I've just linked some

Section 67: Main Topics

Key Topics:

Remember, this course focuses on writing PyTorch code
But if you'd like to dive into what math PyTorch is triggering behind the scenes, I'd highly recommend these two videos
But with experience over time, you work with machine learning problems, you write a lot of code, you get an idea of what works and what doesn't with your particular problem set

📄 Click to view detailed content

YouTube videos that I would recommend for extra curriculum for back propagation,
which is what happens when we call loss stop backward down here. And for the optimizer step,
gradient descent is what's happening there. So I've linked some extra resources for what's going
on behind the scenes there from a mathematical point of view. Remember, this course focuses on
writing PyTorch code. But if you'd like to dive into what math PyTorch is triggering behind the
scenes, I'd highly recommend these two videos. And I've also added a note here as to which
loss function and optimizer should I use, which is a very valid question. And again,
it's another one of those things that's going to be problem specific. But with experience over time,
you work with machine learning problems, you write a lot of code, you get an idea of what works
and what doesn't with your particular problem set. For example, like a regression problem,
like ours, regression is again predicting a number. We use MAE loss, which PyTorch causes
L1 loss. You could also use MSE loss and an optimizer like torch opt-in stochastic gradient
descent will suffice. But for classification, you might want to look into a binary classification,
a binary cross entropy loss, but we'll look at a classification problem later on in the course.
For now, I'd like to demonstrate what's going on in the steps here. So let's go model zero.
Let's look up the state dict and see what the parameters are for now.
Now they aren't the original ones I don't think. Let's re-instantiate our model so we get
re new parameters. Yeah, we recreated it here. I might just get rid of that. So we'll rerun our
model code, rerun model state dict. And we will create an instance of our model and just make
sure your parameters should be something similar to this. If it's not exactly like that, it doesn't
matter. But yeah, I'm just going to showcase you'll see on my screen what's going on anyway.
State dict 3367 for the weight and 012888 for the bias. And again, I can't stress enough. We've
only got two parameters for our model and we've set them ourselves future models that you build
and later ones in the course will have much, much more. And we won't actually explicitly set any
of them ourselves. We'll check out some predictions. They're going to be terrible because we're using
random parameters to begin with. But we'll set up a new loss function and an optimizer. Optimizer
is going to optimize our model zero parameters, the weight and bias. The learning rate is 0.01,
which is relatively large step. That would be a bit smaller. Remember, the larger the learning rate,
the bigger the step, the more the optimizer will try to change these parameters every step.
But let's stop talking about it. Let's see it in action. I've set a manual seed here too, by the way,
because the optimizer steps are going to be quite random as well, depending on how the models
predictions go. But this is just to try and make it as reproduces possible. So keep this in mind,
if you get different values to what we're going to output here from my screen to your screen,
don't worry too much. What's more important is the direction they're going. So ideally,
we're moving these values here. This is from we did one epoch before. We're moving these values
closer to the true values. And in practice, you won't necessarily know what the true values are.
But that's where evaluation of your model comes in. We're going to cover that when we write a
testing loop. So let's run one epoch. Now I'm going to keep that down there. Watch what happens.
We've done one epoch, just a single epoch. We've done the forward pass. We've calculated the loss.
We've done optimizer zero grad. We've performed back propagation. And we've stepped the optimizer.
What is stepping the optimizer do? It updates our model parameters to try and get them further
closer towards the weight and bias. If it does that, the loss will be closer to zero. That's what
it's trying to do. How about we print out the loss at the same time. Print loss and the loss.
Let's take another step. So the loss is 0301. Now we check the weights and the bias. We've changed
again three, three, four, four, five, one, four, eight, eight. We go again. The loss is going down.
Check it. Hey, look at that. The values are getting closer to where they should be if over so slightly.
Loss went down again. Oh my goodness, this is so amazing. Look, we're training our,
let's print this out in the same cell. Print our model. State dict. We're training our first
machine learning model here, people. This is very exciting, even if it's only step by step and it's
only a small model. This is very important. Loss is going down again. Values are getting closer to
where they should be. Again, we won't really know where they should be in real problems, but for
now we do. So let's just get excited. The real way to sort of measure your model's progress and
practice is a lower loss value. Remember, lower is better. A loss value measures how wrong your
model is. We're going down. We're going in the right direction. So that's what I meant by,
as long as your values are going in the similar direction. So down, we're writing similar code
here, but if your values are slightly different in terms of the exact numbers, don't worry too
much because that's inherent to the randomness of machine learning, because the steps that the
optimizer are taking are inherently random, but they're sort of pushed in a direction.
So we're doing gradient descent here. This is beautiful. How low can we get the loss? How about

Section 68: Main Topics

Key Topics:

We've measured the gradient pytorch has done that behind the scenes for us
Thank you pytorch
We're training our first machine learning model

📄 Click to view detailed content

we try to get to 0.1? Look at that. We're getting close to 0.1. And then, I mean, we don't have to
do this hand by hand. The bias is getting close to where it exactly should be. We're below 0.1.
Beautiful. So that was only about, say, 10 passes through the data, but now you're seeing it in
practice. You're seeing it happen. You're seeing gradient descent. Let's go gradient descent work
in action. We've got images. This is what's happening. We've got our cost function. J is
another term for cost function, which is also our loss function. We start with an initial weight.
What have we done? We started with an initial weight, this value here. And what are we doing?
We've measured the gradient pytorch has done that behind the scenes for us. Thank you pytorch.
And we're taking steps towards the minimum. That's what we're trying to do. If we minimize the
gradient of our weight, we minimize the cost function, which is also a loss function. We could
keep going here for hours and get as long as we want. But my challenge for you, or actually,
how about we make some predictions with our model we've got right now? Let's make some predictions.
So with torch dot inference mode, we'll make some predictions together. And then I'm going
to set you a challenge. How about you run this code here for 100 epochs after this video,
and then you make some predictions and see how that goes. So why preds? Remember how
poor our predictions are? Why preds new equals, we just do the forward pass here. Model zero
on the test data. Let's just remind ourselves quickly of how poor our previous predictions were.
Plot predictions, predictions equals y. Do we still have this saved? Why preds?
Hopefully, this is still saved. There we go. Shocking predictions, but we've just done 10 or so
epochs. So 10 or so training steps have our predictions. Do they look any better? Let's run
this. We'll copy this code. You know my rule. I don't really like to copy code, but in this case,
I just want to exemplify a point. I like to write all the code myself. What do we got? Why preds
new? Look at that. We are moving our predictions close at the red dots closer to the green dots.
This is what's happening. We're reducing the loss. In other words, we're reducing the difference
between our models predictions and our ideal outcomes through the power of back propagation
and gradient descent. So this is super exciting. We're training our first machine learning model.
My challenge to you is to run this code here. Change epochs to 100. See how low you can get this
loss value and run some predictions, plot them. And I think it's time to start testing. So give
that a go yourself, and then we'll write some testing code in the next video. I'll see you there.
Welcome back. In the last video, we did something super excited. We saw our loss go down. So the
loss is remember how different our models predictions are to what we'd ideally like them. And we saw
our model update its parameters through the power of back propagation and gradient descent, all
taken care of behind the scenes for us by PyTorch. So thank you, PyTorch. And again, if you'd like
some extra resources on what's actually happening from a math perspective for back propagation and
gradient descent, I would refer to you to these. Otherwise, this is also how I learn about things.
Gradient descent. There we go. How does gradient descent work? And then we've got back propagation.
And just to reiterate, I am doing this and just Googling these things because that's what you're
going to do in practice. You're going to come across a lot of different things that aren't
covered in this course. And this is seriously what I do day to day as a machine learning engineer
if I don't know what's going on. Just go to Google, read, watch a video, write some code,
and then I build my own intuition for it. But with that being said, I also issued you the challenge
of trying to run this training code for 100 epochs. Did you give that a go? I hope you did. And
how low did your loss value? Did the weights and bias get anywhere close to where they should have
been? How do the predictions look? Now, I'm going to save that for later on, running this code for
100 epochs. For now, let's write some testing code. And just a note, you don't necessarily have to
write the training and testing loop together. You can functionize them, which we will be doing later
on. But for the sake of this intuition, building and code practicing and first time where we're
writing this code together, I'm going to write them together. So testing code, we call model.ofour,
what does this do? So this turns off different settings in the model not needed for evaluation
slash testing. This can be a little confusing to remember when you're writing testing code. But
we're going to do it a few times until it's habit. So just make it a habit. If you're training your
model, call model dot train to make sure it's in training mode. If you're testing or evaluating
your model. So that's what a vowel stands for evaluate, call model dot a vowel. So it turns off
different settings in the model not needed for evaluation. So testing, this is things like drop
out. We haven't seen what drop out is slash batch norm layers. But if we go into torch dot
and end, I'm sure you'll come across these things in your future machine learning endeavors. So drop

Section 69: Main Topics

Key Topics:

So we discussed that if parameters in our model have requires grad equals true, which is the default for many different parameters in pytorch, pytorch will behind the scenes keep track of the gradients of our model and use them in lost up backward and optimizer step for back propagation and gradient descent
However, we only need those two back propagation and gradient descent during training because that is when our model is learning
So we don't need to do any learning when we're testing

📄 Click to view detailed content

out drop out layers. There we go. And batch norm. Do we have batch batch norm? There we go. If you'd
like to work out what they are, feel free to check out the documentation. Just take it from me for
now that model of our turns off different settings not needed for evaluation and testing. Then we
set up with torch dot inference mode, inference mode. So what does this do? Let's write down here.
So this turns off gradient tracking. So as we discussed, if we have parameters in our model,
and it turns off actually a few more things and a couple more things behind the scenes,
these are things again, not needed for testing. So we discussed that if parameters in our model
have requires grad equals true, which is the default for many different parameters in pytorch,
pytorch will behind the scenes keep track of the gradients of our model and use them in
lost up backward and optimizer step for back propagation and gradient descent. However,
we only need those two back propagation and gradient descent during training because that
is when our model is learning. When we are testing, we are just evaluating the parameters the patterns
that our model has learned on the training data set. So we don't need to do any learning
when we're testing. So we turn off the things that we don't need. And is this going to have
the correct spacing for me? I'm not sure we'll find out. So we still do the forward pass
in testing mode, do the forward pass. And if you want to look up torch inference mode,
just go torch inference mode. There's a great tweet about it that pytorch did, which explains
what's happening. I think we've covered this before, but yeah, want to make your inference
code and pytorch run faster. Here's a quick thread on doing exactly that. So inference
mode is torch no grad. Again, you might see torch no grad. I think I'll write that down just to
let you know. But here's what's happening behind the scenes. A lot of optimization code,
which is beautiful. This is why we're using pytorch so that our code runs nice and far.
Let me go there. You may also see with torch dot no grad in older pytorch code. It does
similar things, but inference mode is the faster way of doing things according to the thread.
And according to there's a blog post attached to there as well, I believe.
So you may also see torch dot no grad in older pytorch code, which would be valid. But again,
inference mode is the better way of doing things. So do forward pass. So let's get our model. We
want to create test predictions here. So we're going to go model zero. There's a lot of code
going on here, but I'm going to just step by step it in a second. We'll go back through it all.
And then number two is calculate the loss. Now we're doing the test predictions here,
calculate the loss test predictions with model zero. So now we want to calculate the what we want
to calculate the test loss. So this will be our loss function, the difference between the test
pred and the test labels. That's important. So for testing, we're working with test data,
for training, we're working with training data. Model learns patterns on the training data,
and it evaluates those patterns that it's learned, the different parameters on the testing data. It
has never seen before, just like in a university course, you'd study the course materials, which
is the training data, and you'd evaluate your knowledge on materials you'd hopefully never
seen before, unless you sort of were friends with your professor, and they gave you the exam before
the actual exam that would be cheating right. So that's a very important point for the test data
set. Don't let your model see the test data set before you evaluate it. Otherwise, you'll get
poor results. And that's putting it out what's happening. Epoch, we're going to go Epoch,
and then I will introduce you to my little jingle to remember all of these steps because
there's a lot going on. Don't you worry. I know there's a lot going on, but again, with practice,
we're going to know what's happening here. Like it's the back of our hand. All right.
So do we need this? Oh, yeah, we could say that. Oh, no, we don't need test here. Loss. This is
loss, not test. Print out what's happening. Okay. And we don't actually need to do this
every epoch. We could just go say if epoch divided by 10 equals zero, print out what's happening.
Let's do that rather than clutter everything up, print it out, and we'll print out this.
So let's just step through what's happening. We've got 100 epochs. That's what we're about to run,
100 epochs. Our model is trained for about 10 so far. So it's got a good base. Maybe we'll just
get rid of that base. Start a new instance of our model. So we'll come right back down.
So our model is back to randomly initialized parameters, but of course, randomly initialized
flavored with a random seed of 42. Lovely, lovely. And so we've got our training code here. We've
discussed what's happening there. Now, we've got our testing code. We call model dot eval,
which turns off different settings in the model, not needed for evaluation slash testing. We call
with torch inference mode context manager, which turns off gradient tracking and a couple more
things behind the scenes to make our code faster. We do the forward pass. We do the test predictions.
We pass our model, the test data, the test features to calculate the test predictions.

Section 70: Main Topics

Key Topics:

But the beauty of PyTorch is you can use basic Python printing statements to see what's happening with your model
This is, I can't stress enough, like what we are doing here is going to be very similar throughout the entire rest of the course for training more and more models
So this step that we've done here for training our model and evaluating it is seriously like the fundamental steps of deep learning with PyTorch is training and evaluating a model

📄 Click to view detailed content

Then we calculate the loss using our loss function. We can use the same loss function that we used
for the training data. And it's called the test loss, because it's on the test data set.
And then we print out what's happening, because we want to know what's happening while our
model's training, we don't necessarily have to do this. But the beauty of PyTorch is you can
use basic Python printing statements to see what's happening with your model. And so,
because we're doing 100 epochs, we don't want to clutter up everything here. So we'll just
print out what's happening every 10th epoch. Again, you can customize this as much as you like
what's printing out here. This is just one example. If you had other metrics here, such as calculating
model accuracy, we might see that later on, hint hint. We might print out our model accuracy.
So this is very exciting. Are you ready to run 100 epochs? How low do you think our loss can go?
This loss was after about 10. So let's just save this here. Let's give it a go. Ready?
Three, two, one. Let's run. Oh my goodness. Look at that. Waits. Here we go. Every 10 epochs
were printing out what's happening. So the zero epoch, we started with losses 312. Look at it go
down. Yes, that's what we want. And our weights and bias, are they moving towards our ideal weight
and bias values of 0.7 and 0.3? Yes, they're moving in the right direction here. The loss is
going down. Epoch 20, wonderful. Epoch 30, even better. 40, 50, going down, down, down. Yes,
this is what we want. This is what we want. Now, we're predicting a straight line here. Look how
low the loss gets. After 100 epochs, we've got about three times less than what we had before.
And then we've got these values are quite close to where they should be, 0.5629, 0.3573. We'll make
some predictions. What do they look like? Why preds new? This is the original predictions
with random values. And if we make why preds new, look how close it is after 100 epochs.
Now, what's our, do we print out the test loss? Oh no, we're printing out loss as well.
Let's get rid of that. I think this is this. Yeah, that's this statement here. Our code would have
been a much cleaner if we didn't have that, but that's all right. Life goes on. So our test loss,
because this is the test predictions that we're making, is not as low as our training loss.
I wonder how we could get that lower. What do you think we could do? We just trained it for
longer. And what happened? How do you think you could get these red dots to line up with these
green dots? Do you think you could? So that's my challenge to you for the next video.
Think of something that you could do to get these red dots to match up with these green dots,
maybe train for longer. How do you think you could do that? So give that a shot. And I'll see
in the next video, we'll review what our testing code is doing. I'll see you there.
Welcome back. In the last video, we did something super exciting. We trained our model for 100 epochs
and look how good the predictions got. But I finished it off with challenging you to see if you could
align the red dots with the green dots. And it's okay if you're not sure how the best way to do
that. That's what we're here for. We're here to learn what are the best way to do these things
together. But you might have had the idea of potentially training the model for a little bit
longer. So how could we do that? Well, we could just rerun this code. So the model is going to
remember the parameters that it has from what we've done here. And if we rerun it, well, it's going
to start from where it finished off, which is already pretty good for our data set. And then
it's going to try and improve them even more. This is, I can't stress enough, like what we are
doing here is going to be very similar throughout the entire rest of the course for training more
and more models. So this step that we've done here for training our model and evaluating it
is seriously like the fundamental steps of deep learning with PyTorch is training and evaluating
a model. And we've just done it. Although I'll be it to predict some red dots and green dots.
That's all right. So let's try to line them up, hey, red dots onto green dots. I reckon if we
train it for another 100 epochs, we should get pretty darn close. Ready? Three, two, one. I'm
going to run this cell again. Runs really quick because our data's nice and simple. But
look at this, lastly, we started 0244. Where do we get down to? 008. Oh my goodness. So we've
improved it by another three X or so. And now this is where our model has got really good.
On the test loss, we've gone from 00564. We've gone down to 005. So almost 10X improvement there.
And so we make some more predictions. What are our model parameters? Remember the ideal ones here.
We won't necessarily know them in practice, but because we're working with a simple data set,
we know what the ideal parameters are. Model zero state dig weights. These are what they
previously were. What are they going to change to? Oh, would you look at that? Oh,
06990. Now, again, if yours are very slightly different to mine, don't worry too much. That is
the inherent randomness of machine learning and deep learning. Even though we set a manual seed,

Section 71: Main Topics

Key Topics:

Of course, we could just create a model and set the parameters ourselves manually
We just wrote some machine learning code to do it for us with the power of back propagation and gradient descent
We might use a different learning rate

📄 Click to view detailed content

it may be slightly different. The direction is more important. So if your number here is not
exactly what mine is, it should still be quite close to 0.7. And the same thing with this one.
If it's not exactly what mine is, don't worry too much. The same with all of these loss values
as well. The direction is more important. So we're pretty darn close. How do these predictions
look? Remember, these are the original ones. We started with random. And now we've trained a model.
So close. So close to being exactly that. So a little bit off. But that's all right. We could
tweak a few things to improve this. But I think that's well and truly enough for this example
purpose. You see what's happened. Of course, we could just create a model and set the parameters
ourselves manually. But where would be the fun in that? We just wrote some machine learning code
to do it for us with the power of back propagation and gradient descent. Now in the last video,
we wrote the testing loop. We discussed a few other steps here. But now let's go over it with
a colorful slide. Hey, because I mean, code on a page is nice, but colors are even nicer. Oh,
we haven't done this. We might set up this in this video too. But let's just discuss what's going on.
Create an empty list for storing useful value. So this is helpful for tracking model progress.
How can we just do this right now? Hey, we'll go here and we'll go.
So what did we have? Epoch count equals that. And then we'll go
lost values. So why do we keep track of these? It's because
if we want to monitor our models progress, this is called tracking experiments. So track
different values. If we wanted to try and improve upon our current model with a future model. So
our current results, such as this, if we wanted to try and improve upon it, we might build an
entire other model. And we might train it in a different setup. We might use a different learning
rate. We might use a whole bunch of different settings, but we track the values so that we
can compare future experiments to past experiments, like the brilliant scientists that we are.
And so where could we use these lists? Well, we're calculating the loss here. And we're calculating
the test loss here. So maybe we each time append what's going on here as we do a status update.
So epoch count dot append, and we're going to go a current epoch. And then we'll go loss values
dot append, a current loss value. And then we'll do test loss values dot append, the current test
loss values. Wonderful. And now let's re-instantiate our model so that it starts from fresh. So this
is just create another instance. So we're just going to re-initialize our model parameters to
start from zero. If we wanted to, we could functionize all of this so we don't have to
go right back up to the top of the code. But just for demo purposes, we're doing it how we're doing
it. And I'm going to run this for let's say 200 epochs, because that's what we ended up doing,
right? We ran it for 200 epochs, because we did 100 epochs twice. And I want to show you something
beautiful, one of the most beautiful sites in machine learning. So there we go, we run it for
200 epochs, we start with a fairly high training loss value and a fairly high test loss value. So
remember, what is our loss value? It's ma e. So if we go back, yeah, this is what we're measuring
for loss. So this means for the test loss on average, each of our dot points here, the red
predictions are 0.481. That's the average distance between each dot point. And then ideally, what
are we doing? We're trying to minimize this distance. That's the ma e. So the mean absolute error.
And we get it right down to 0.05. And if we make predictions, what do we have here, we get very
close to the ideal weight and bias, make our predictions, have a look at the new predictions.
Yeah, very small distance here. Beautiful. That's a low loss value.
Ideally, they'd line up, but we've got as close as we can for now. So this is one of the most
beautiful sites in machine learning. So plot the loss curves. So let's make a plot, because what
we're doing, we were tracking the value of epoch count, loss values and test loss values.
Let's have a look at what these all look like. So epoch count goes up, loss values ideally go down.
So we'll get rid of that. We're going to create a plot p l t dot plot. We're going to step back
through the test loop in a second with some colorful slides, label equals train loss.
And then we're going to go plot. You might be able to tell what's going on here. Test loss
values. We're going to visualize it, because that's the data explorer's motto, right, is visualize,
visualize, visualize. This is equals. See, collab does this auto correct. That doesn't really work
very well. And I don't know when it does it and why it doesn't. And we got, I know, we didn't,
we didn't say loss value. So that's a good auto correct. Thank you, collab.
So training and loss and test loss curves. So this is another term you're going to come across
often is a loss curve. Now you might be able to think about a loss curve. If we're doing a loss
curve, and it's starting at the start of training, what do we want that curve to do?

Section 72: Main Topics

Key Topics:

So what we're doing here is our loss values are still on PyTorch, and they can't be because mapplotlib works with NumPy
One of the most beautiful sides in machine learning is a declining loss curve
And of course, lower is better

📄 Click to view detailed content

What do we want our loss value to do? We want it to go down. So what should an ideal loss
curve look like? Well, we're about to see a couple. Let's have a look. Oh, what do we got wrong?
Well, we need to, I'll turn it into NumPy. Is this what we're getting wrong? So why is this wrong?
Loss values. Why are we getting an issue? Test loss values.
Ah, it's because they're all tens of values. So I think we should, let's,
I might change this to NumPy. Oh, can I just do that? If I just call this as a NumPy array,
we're going to try and fix this on the fly. People, NumPy array, we'll just turn this into a NumPy
array. Let's see if we get NumPy. I'm figuring these things out together. NumPy as NumPy,
because mapplotlib works with NumPy. Yeah, there we go. So can we do loss values? Maybe
I'm going to try one thing, torch dot tensor, loss values, and then call
CPU dot NumPy. See what happens here.
There we go. Okay, so let's just copy this. So what we're doing here is
our loss values are still on PyTorch, and they can't be because mapplotlib works with
NumPy. And so what we're doing here is we're converting our loss values of the training loss
to NumPy. And if you call from the fundamental section, we call CPU and NumPy, I wonder if we
can just do straight up NumPy, because we're not working on there. Yeah, okay, we don't need
CPU because we're not working on the GPU yet, but we might need that later on. Well, this work.
Beautiful. There we go. One of the most beautiful sides in machine learning is a declining loss
curve. So this is how we keep track of our experiments, or one way, quite rudimentary. We'd like to
automate this later on. But I'm just showing you one way to keep track of what's happening.
So the training loss curve is going down here. The training loss starts at 0.3, and then it goes
right down. The beautiful thing is they match up. If there was a two bigger distance behind the
train loss and the test loss, or sorry, between, then we're running into some problems. But if they
match up closely at some point, that means our model is converging and the loss is getting as
close to zero as it possibly can. If we trained for longer, maybe the loss will go almost basically
to zero. But that's an experiment I'll leave you to try to train that model for longer.
Let's just step back through our testing loop to finish off this video. So we did that. We created
empty lists for strong useful values, storing useful values, strong useful values. Told the
model what we want to evaluate or that we want to evaluate. So we put it in an evaluation mode.
It turns off functionality used for training, but not evaluations, such as drop out and batch
normalization layers. If you want to learn more about them, you can look them up in the documentation.
Turn on torch inference mode. So this is for faster performance. So we don't necessarily need this,
but it's good practice. So I'm going to say that yes, turn on torch inference mode. So this
disables functionality such as gradient tracking for inference. Gradient tracking is not needed
for inference only for training. Now we pass the test data through the model. So this will call
the models implemented forward method. The forward pass is the exact same as what we did in the
training loop, except we're doing it on the test data. So big notion there, training loop,
training data, testing loop, testing data. Then we calculate the test loss value,
how wrong the models predictions are on the test data set. And of course, lower is better.
And finally, we print out what's happening. So we can keep track of what's going on during
training. We don't necessarily have to do this. You can customize this print value to print out
almost whatever you want, because it's pie torches, basically very beautifully interactive with pure
Python. And then we keep track of the values of what's going on on epochs and train loss and test
loss. We could keep track of other values here. But for now, we're just going, okay, what's the loss
value at a particular epoch for the training set? And for the test set. And of course, all of this
could be put into a function. And that way we won't have to remember these steps off by heart.
But the reason why we've spent so much time on this is because we're going to be using this
training and test functionality for all of the models that we build throughout this course.
So give yourself a pat in the back for getting through all of these videos. We've written a lot
of code. We've discussed a lot of steps. But if you'd like a song to remember what's happening,
let's finish this video off with my unofficial PyTorch optimization loop song.
So for an epoch in a range, go model dot train, do the forward pass, calculate the loss, optimize
a zero grad, loss backward, optimize a step, step, step. No, you only have to call this once.
But now let's test, go model dot eval with torch inference mode, do the forward pass,
calculate the loss. And then the real song goes for another epoch because you keep going back
through. But we finish off with print out what's happening. And then of course, we evaluate what's
going on. With that being said, it's time to move on to another thing. But if you'd like to review
what's happening, please, please, please try to run this code for yourself again and check out the

Section 73: Main Topics

Key Topics:

Oh, by the way, if you want to link to all of the extra curriculum, just go to the book version of the course
So what we might cover in this video is saving a model in PyTorch
So now let's see how we can save our models in PyTorch

📄 Click to view detailed content

slides and also check out the extra curriculum. Oh, by the way, if you want to link to all
of the extra curriculum, just go to the book version of the course. And it's all going to be in here.
So that's there ready to go. Everything I link is extra curriculum will be in the extra curriculum
of each chapter. I'll see you in the next video. Welcome back. In the last video, we saw how to
train our model and evaluate it by not only looking at the loss metrics and the loss curves,
but we also plotted our predictions and we compared them. Hey, have a go at these random
predictions. Quite terrible. But then we trained a model using the power of back propagation and
gradient descent. And now look at our predictions. They're almost exactly where we want them to be.
And so you might be thinking, well, we've trained this model and it took us a while to
write all this code to get some good predictions. How might we run that model again? So I've took
in a little break after the last video, but now I've come back and you might notice that my Google
Colab notebook has disconnected. So what does this mean if I was to run this? Is it going to work?
I'm going to connect to a new Google Colab instance. But will we have all of the code that we've run
above? You might have already experienced this if you took a break before and came back to the
videos. Ah, so plot predictions is no longer defined. And do you know what that means? That
means that our model is also no longer defined. So we would have lost our model. We would have
lost all of that effort of training. Now, luckily, we didn't train the model for too long. So we can
just go run time, run all. And it's going to rerun all of the previous cells and be quite quick.
Because we're working with a small data set and using a small model. But we've been through all
of this code. Oh, what have we got wrong here? Model zero state dict. Well, that's all right.
This is good. We're finding errors. So if you want to as well, you can just go run after. It's going
to run all of the cells after. Beautiful. And we come back down. There's our model training.
We're getting very similar values to what we got before. There's the lost curves. Beautiful.
Still going. Okay. Now our predictions are back because we've rerun all the cells and we've got
our model here. So what we might cover in this video is saving a model in PyTorch. Because if
we're training a model and you get to a certain point, especially when you have a larger model,
you probably want to save it and then reuse it in this particular notebook itself. Or you might
want to save it somewhere and send it to your friend so that your friend can try it out. Or you
might want to use it in a week's time. And if Google Colab is disconnected, you might want to
be able to load it back in somehow. So now let's see how we can save our models in PyTorch. So
I'm going to write down here. There are three main methods you should know about
for saving and loading models in PyTorch because of course with saving comes loading. So we're
going to over the next two videos discuss saving and loading. So one is torch.save. And as you might
guess, this allows you to save a PyTorch object in Python's pickle format. So you may or may not
be aware of Python pickle. There we go. Python object serialization. There we go. So we've got
the pickle module implements a binary protocols or implements binary protocols for serializing
and deserializing a Python object. So serializing means I understand it is saving and deserializing
means that it's loading. So this is what PyTorch uses behind the scenes, which is from pure Python.
So if we go back here in Python's pickle format, number two is torch.load, which you might be able
to guess what that does as well, allows you to load a saved PyTorch object. And number three is
also very important is torch.nn.module.loadStatedict. Now what does this allow you to do? Well,
this allows you to load a model's saved dictionary or save state dictionary. Yeah, that's what we'll
call it. Save state dictionary. Beautiful. And what's the model state dict? Well, let's have a look,
model zero dot state dict. The beauty of PyTorch is that it stores a lot of your model's important
parameters in just a simple Python dictionary. Now it might not be that simple because our model,
again, only has two parameters. In the future, you may be working with models with millions of
parameters. So looking directly at the state deck may not be as simple as what we've got here.
But the principle is still the same. It's still a dictionary that holds the state of your model.
And so I've got these three methods I want to show you where from because this is going to be
your extra curriculum, save and load models, your extra curriculum for this video.
If we go into here, this is a very, very, very important piece of PyTorch documentation,
or maybe even a tutorial. So your extra curriculum for this video is to go through it.
Here we go. We've got torch, save, torch, load, torch, module, state deck. That's where, or load
state deck, that's where I've got the three things that we've just written down. And there's a fair
few different pieces of information. So what is a state deck? So in PyTorch, the learnable
parameters, i.e. the weights and biases of a torch and end module, which is our model.

Section 74: Main Topics

Key Topics:

So the recommended way of saving and loading a PyTorch model is by saving its state deck
So PyTorch save and load code
So if we go saving our PyTorch model

📄 Click to view detailed content

Remember, our model subclasses and end module are contained in the model's parameters. Access
with model.parameters, a state deck is simply a Python dictionary object that maps each layer
to its parameter tensor. That's what we've seen. And so then if we define a model,
we can initialize the model. And if we wanted to print the state decked, we can use that.
The optimizer also has a state deck. So that's something to be aware of. You can go optimizer.state
deck. And then you get an output here. And this is our saving and loading model for inference. So
inference, again, is making a prediction. That's probably what we want to do in the future at some
point. For now, we've made predictions right within our notebook. But if we wanted to use our model
outside of our notebook, say in an application, or in another notebook that's not this one,
you'll want to know how to save and load it. So the recommended way of saving and loading a
PyTorch model is by saving its state deck. Now, there is another method down here,
which is saving and loading the entire model. So your extracurricular for this lesson,
we're going to go through the code to do this. But your extracurricular is to read all of the
sections in here, and then figure out what the pros and cons are of saving and loading the entire
model versus saving and loading just the state deck. So that's a challenge for you for this video.
I'm going to link this in here. And now let's write some code to save our model.
So PyTorch save and load code. Code tutorial plus extracurricular. So if we go
saving our PyTorch model. So what might we want? What do you think the save parameter takes?
If we have torch.save, what do you think it takes inside it? Well, let's find out together.
Hey, so let's import part lib. We're going to see why in a second. This is Python's
module for dealing with writing file paths. So if we wanted to save something to this is Google
Colab's file section over here. But just remember, if we do save this from within Google Colab,
the model will disappear if our Google Colab notebook instance disconnects. So I'll show you
how to download it from Google Colab if you want. Google Colab also has a way save from Google Colab
Google Colab to Google Drive to save it to your Google Drive if you wanted to. But I'll leave you
to look at that on your own if you like. So we're first going to create a model directory.
So create models directory. So this is going to help us create a folder over here called models.
And of course, we could create this by hand by adding a new folder here somewhere. But I like
to do it with code. So model path, we're going to set this to path, which is using the path library
here to create us a path called models. Simple. We're just going to save all of our models to
models to the models file. And then we're going to create model path, we're going to make that
directory model path dot mkdir for make directory. We're going to set parents to equals true.
And we're also going to set exist okay equals to true. That means if it already existed,
it won't throw us an error. It will try to create it. But if it already exists, it'll just recreate
the parents directory or it'll leave it there. It won't error out on us. We're also going to
create a model save path. This way, we can give our model a name. Right now, it's just model zero.
We want to save it under some name to the models directory. So let's create the model name.
Model name equals 01. I'm going to call it 01 for the section. That way, if we have more models
later on the course, we know which ones come from where you might create your own naming
convention, model workflow, pytorch workflow, model zero dot pth. And now this is another
important point. Pytorch objects usually have the extension dot pth for pytorch or dot pth.
So if we go in here, and if we look up dot pth, yeah, a common convention is to save models
using either a dot pth or dot pth file extension. I'll let you choose which one you like. I like
dot pth. So if we go down here dot pth, they both result in the same thing. You just have to remember
to make sure you write the right loading path and right saving path. So now we're going to create
our model save path, which is going to be our model path. And because we're using the path lib,
we can use this syntax that we've got here, model path slash model name. And then if we just print out
model save path, what does this look like? There we go. So it creates a supposic path
using the path lib library of models slash 01 pytorch workflow model zero dot pth. We haven't
saved our model there yet. It's just got the path that we want to save our model ready. So if we
refresh this, we've got models over here. Do we have anything in there? No, we don't yet. So now
is our step to save the model. So three is save the model state dict. Why are we saving the state
dict? Because that's the recommended way of doing things. If we come up here, saving and loading the
model for inference, save and load the state dict, which is recommended. We could also save the entire
model. But that's part of your extra curriculum to look into that. So let's use some syntax. It's

Section 75: Main Topics

Key Topics:

Now, of course, we've saved a model
So we'll come back here and we'll go loading a pytorch model

📄 Click to view detailed content

quite like this torch dot save. And then we pass it an object. And we pass it a path of where to
save it. We already have a path. And good thing is we already have a model. So we just have to call
this. Let's try it out. So let's go print f saving model to and we'll put in the path here.
Model save path. I like to print out some things here and there that way. We know what's going on.
And I don't need that capital. Why do I? Getting a little bit trigger happy here with the typing.
So torch dot save. And we're going to pass in the object parameter here. And if we looked up torch
save, we can go. What does this code take? So torch save object f. What is f? A file like object.
Okay. Or a string or OS path like object. Beautiful. That's what we've got. A path like
object containing a file name. So let's jump back into here. The object is what? It's our model zero
dot state dict. That's what we're saving. And then the file path is model save path. You ready?
Let's run this and see what happens. Beautiful. Saving model to models. So it's our model path.
And there's our model there. So if we refresh this, what do we have over here?
Wonderful. We've saved our trained model. So that means we could potentially if we wanted to,
you could download this file here. That's going to download it from Google CoLab to your local
machine. That's one way to do it. But there's also a guide here to save from Google Collaboratory
to Google Drive. That way you could use it later on. So there's many different ways.
The beauty of pie torches is flexibility. So now we've got a saved model. But let's just check
using our LS command. We're going to check models. Yeah, let's just check models. This is going to
check here. So this is list. Wonderful. There's our 01 pie torch workflow model zero dot pth. Now,
of course, we've saved a model. How about we try loading it back in and seeing how it works. So if
you want to challenge, read ahead on the documentation and try to use torch dot load to bring our model
back in. See what happens. I'll see in the next video. Welcome back. In the last video, we wrote
some code here to save our pie torch model. I'm just going to exit out of this couple of things
that we don't need just to clear up the screen. And now we've got our dot pth file, because remember
dot pth or dot pth is a common convention for saving a pie torch model. We've got it saved there,
and we didn't necessarily have to write all of this path style code. But this is just handy for
later on if we wanted to functionize this and create it in say a save dot pie file over here,
so that we could just call our save function and pass it in a file path where we wanted to save
like a directory and a name, and then it'll save it exactly how we want it for later on.
But now we've got a saved model. I issued a challenge of trying to load that model in.
So do we have torch dot load in here? Did you try that out? We've got, oh, we've got a few options
here. Wonderful. But we're using one of the first ones. So let's go back up here. If we wanted to
check the documentation for torch dot load, we've got this option here, load. What happens? Loads
and objects saved with torch dot save from a file. Torch dot load uses Python's unpickling
facilities, but treat storages which underlie tenses specially. They are firstly serialized
on the CPU, and then I moved the device they were saved from. Wonderful. So this is moved to the
device. If later on when we're using a GPU, this is just something to keep in mind. We'll see that
when we start to use a CPU and a GPU. But for now, let's practice using the torch dot load method
and see how we can do it. So we'll come back here and we'll go loading a pytorch model.
And since we, she's going to start writing here, since we saved our models state debt,
so just the dictionary of parameters from a model, rather than
the entire model, we'll create a new instance of our model class and load the state deck,
load the saved state deck. That's better state deck into that.
Now, this is just words on a page. Let's see this in action. So to load in a state deck,
which is what we say, we didn't save the entire model itself, which is one option.
That's extra curriculum, but we saved just the model state deck. So if we remind ourselves what
model zero dot state deck looks like, we saved just this. So to load this in, we have to
instantiate a new class or a new instance of our linear regression model class. So to load in a
saved state deck, we have to instantiate a new instance of our model class. So let's call this
loaded model zero. I like that. That way we can differentiate because it's still going to be the
same parameters as model zero, but this way we know that this instance is the loaded version,
not just the version we've been training before. So we'll create a new version of it here,
linear regression model. This is just the code that we wrote above, linear regression model.
And then we're going to load the saved state deck of model zero. And so this will update the new

Section 76: Main Topics

Key Topics:

So that's saving and loading a model in pytorch
Over the past few videos, we've covered a whole bunch of ground in a pytorch workflow, starting with data, then building a model
We then learned how to save and load a model in pytorch

📄 Click to view detailed content

instance with updated parameters. So let's just check before we load it, we haven't written any
code to actually load anything. What does loaded model zero? What does the state deck look like here?
It won't have anything. It'll be initialized with what?
Oh, loaded. That's what I called it loaded. See how it's initialized with random parameters.
So essentially all we're doing when we load a state dictionary into our new instance of our
model is that we're going, hey, take the saved state deck from this model and plug it into this.
So let's see what happens when we do that. So loaded model zero. Remember how I said there's
a method to also be aware of up here, which is torch nn module dot load state deck. And because
our model is a what, it's a subclass of torch dot nn dot module. So we can call load state deck
on our model directly or on our instance. So recall linear regression model is a subclass
of nn dot module. So let's call in load state deck. And this is where we call the torch dot load
method. And then we pass it the model save path. Is that what we call it? Because torch dot load,
it takes in F. So what's F a file like object or a string or a OS path like object. So that's
why we created this path like object up here. Model save path. So all we're doing here,
we're creating a new instance, linear regression model, which is a subclass of nn dot module.
And then on that instance, we're calling in load state deck of torch dot load model save path.
Because what's saved at the model save path, our previous models state deck, which is here.
So if we run this, let's see what happens. All keys match successfully. That is beautiful.
And so see the values here, loaded state deck of model zero. Well, let's check the loaded version
of that. We now have wonderful, we have the exact same values as above. But there's a little
way that we can test this. So how about we go make some predictions. So make some predictions.
Just to make sure with our loaded model. So let's put it in a valve mode. Because when you make
predictions, you want it in evaluation mode. So it goes a little bit faster. And we want to
also use inference mode. So with torch dot inference mode for making predictions. We want to write
this loaded model preds, we're going to make some predictions on the test data as well. So loaded
model zero, we're going to forward pass on the X test data. And then we can have a look at the
loaded model preds. Wonderful. And then to see if the two models are the same, we can compare
loaded model preds with original model preds. So why preds? These should be equivalent equals
equals loaded model preds. Do we have the same thing? False, false, false, what's going on here?
Why preds? How much different are they? Oh, where's that happened? Have we made some
model preds with this yet? So how about we make some model preds? This is troubleshooting on
the fly team. So let's go model zero dot eval. And then with torch dot inference mode,
this is how we can check to see that our two models are actually equivalent. Why preds equals,
I have a feeling why preds actually save somewhere else equals model zero. And then we pass it the
X test data. And then we might move this above here. And then have a look at what why preds equals.
Do we get the same output? Yes, we should. Wonderful. Okay, beautiful. So now we've covered
saving and loading models or specifically saving the models state deck. So we saved it here with
this code. And then we loaded it back in with load state deck plus torch load. And then we
checked to see by testing equivalents of the predictions of each of our models. So the original
one that we trained here, model zero, and the loaded version of it here. So that's saving and
loading a model in pytorch. There are a few more things that we could cover. But I'm going to leave
that for extra curriculum. We've covered the two main things or three main things. One, two, three.
If you'd like to read more, I'd highly encourage you to go through and read this tutorial here.
But with that being said, we've covered a fair bit of ground over the last few videos. How about
we do a few videos where we put everything together just to reiterate what we've done.
I think that'll be good practice. I'll see you in the next video.
Welcome back. Over the past few videos, we've covered a whole bunch of ground in a pytorch
workflow, starting with data, then building a model. Well, we split the data, then we built a
model. We looked at the model building essentials. We checked the contents of our model. We made
some predictions with a very poor model because it's based off random numbers. We spent a whole
bunch of time figuring out how we could train a model. We figured out what the loss function is.
We saw an optimizer. We wrote a training and test loop. We then learned how to save and load a
model in pytorch. So now I'd like to spend the next few videos putting all this together. We're
not going to spend as much time on each step, but we're just going to have some practice together
so that we can reiterate all the things that we've done. So putting it all together,
let's go back through the steps above and see it all in one place. Wonderful.

Section 77: Main Topics

Key Topics:

We're going to cover those throughout the course
So let's start by importing pytorch
And we're going to check out pytorch version

📄 Click to view detailed content

So we're going to start off with 6.1 and we'll go have a look at our workflow. So 6.1 is data,
but we're going to do one step before that. And I'm just going to get rid of this so we have a bit
more space. So we've got our data ready. We've turned it into tenses way back at the start.
Then we built a model and then we picked a loss function and an optimizer. We built a training
loop. We trained our model. We made some predictions. We saw that they were better. We evaluated our
model. We didn't use torch metrics, but we got visual. We saw our red dots starting to line up
with the green dots. We haven't really improved through experimentation. We did a little bit of
it though, as in we saw that if we trained our model for more epochs, we got better results.
So you could argue that we have done a little bit of this, but there are other ways to experiment.
We're going to cover those throughout the course. And then we saw how to save and reload a trained
model. So we've been through this entire workflow, which is quite exciting, actually.
So now let's go back through it, but we're going to do it a bit quicker than what we've done before,
because I believe you've got the skills to do so now. So let's start by importing pytorch.
So you could start the code from here if you wanted to. And that plot live. And actually,
if you want, you can pause this video and try to recode all of the steps that we've done
by putting some headers here, like data, and then build a model and then train the model,
save and load a model, whatever, and try to code it out yourself. If not, feel free to follow along
with me and we'll do it together. So import torch from torch import. Oh, would help if I could spell
torch import and n because we've seen that we use an n quite a bit. And we're going to also
import map plot live because we like to make some plots because we like to get visual.
Visualize visualize visualize as PLT. And we're going to check out pytorch version.
That way we know if you're on an older version, some of the code might not work here. But if you're
on a newer version, it should work. If it doesn't, let me know. There we go. 1.10. I'm using 1.10
for this. By the time you watch this video, there may be a later version out. And we're also going
to let's create some device agnostic code. So create device agnostic code, because I think we're
up to this step now. This means if we've got access to a GPU, our code will use it for potentially
faster computing. If no GPU is available, the code will default to using CPU. We don't necessarily
need to use a GPU for our particular problem that we're working on right now because it's a small
model and it's a small data set, but it's good practice to write device agnostic code. So that
means our code will use a GPU if it's available, or a CPU by default, if a GPU is not available.
So set up device agnostic code. We're going to be using a similar setup to this throughout the
entire course from now on. So that's why we're bringing it back. CUDA is available. So remember
CUDA is NVIDIA's programming framework for their GPUs, else use CPU. And we're going to print
what device are we using? Device. So what we might do is if we ran this, it should be just a CPU
for now, right? Yours might be different to this if you've enabled a GPU, but let's change this
over to use CUDA. And we can do that if you're using Google Colab, we can change the runtime type
by selecting GPU here. And then I'm going to save this, but what's going to happen is it's
going to restart the runtime. So we're going to lose all of the code that we've written above.
How can we get it all back? Well, we can go. Run all. This is going to run all of the cells
above here. They should all work and it should be quite quick because our model and data aren't
too big. And if it all worked, we should have CUDA as our device that we can use here. Wonderful.
So the beauty of Google Colab is that they've given us access to on a video GPU. So thank you,
Google Colab. Just once again, I'm paying for the paid version of Google Colab. You don't have to.
The free version should give you access to a GPU, or be it it might not be as a later version as
GPU as the pro versions give access to. But this will be more than enough for what we're about to
recreate. So I feel like that's enough for this video. We've got some device agnostic code ready
to go. And for the next few videos, we're going to be rebuilding this except using device agnostic
code. So give it a shot yourself. There's nothing in here that we haven't covered before. So I'll
see you in the next video. Let's create some data. Welcome back. In the last video, we set up some
device agnostic code and we got ready to start putting everything we've learned together.
So now let's continue with that. We're going to recreate some data. Now we could just copy this
code, but we're going to write it out together so we can have some practice creating a dummy data
set. And we want to get to about this stage in this video. So we want to have some data that we can

Section 78: Main Topics

Key Topics:

There's a lot of things in machine learning that are quite flexible
It's tenses because we use PyTorch to create it
So building a PyTorch linear model

📄 Click to view detailed content

plot so that we can build a model to once again, learn on the blue dots to predict the green dots.
So we'll come down here data. I'm going to get out of this as well so that we have a bit more room.
Let's now create some data using the linear regression formula of y equals weight times
features plus bias. And you may have heard this as y equals mx plus c or mx plus b or something like
that, or you can substitute these for different names. Images when I learned this in high school,
it was y equals mx plus c. Yours might be slightly different. Yeah, bx plus a. That's what they use
here. A whole bunch of different ways to name things, but they're all describing the same thing.
So let's see this in code rather than formulaic examples. So we're going to create our weight,
which is 0.7 and a bias, which is 0.3. These are the values we previously used for a challenge you
could change these to 0.1 maybe and 0.2. These could be whatever values you'd like to set them as.
So weight and bias, the principle is going to be the same thing. We're going to try and build a
model to estimate these values. So we're going to start at 0 and we're going to end at 1.
So we can just create a straight line and we're going to fill in those between 0 and 1 with a
step of 0.02. And now we'll create the x and y features x and y, which is features and labels
actually. So x is our features and y are our labels. x equals torch dot a range and x is a
capital Y is that because typically x is a feature matrix. Even though ours is just a vector now,
we're going to unsqueeze this so we don't run into dimensionality issues later on.
You can check this for yourself without unsqueeze, errors will pop up and y equals weight times
x plus bias. You see how we're going a little bit faster now? This is sort of the pace that we're
going to start going for things that we've already covered. If we haven't covered something, we'll
slow down, but if we have covered something, I'm going to step it through. We're going to start
speeding things up a little. So if we get some values here, wonderful. We've got some x values
and they correlate to some y values. We're going to try and use the training values of x to predict
the training values of y and subsequently for the test values. Oh, and speaking of training and test
values, how about we split the data? So let's split the data. Split data. So we'll create the
train split equals int 0.8. We're going to use 80%, which is where 0.8 comes from,
for the length of x. So we use 80% of our samples for the training, which is a typical
training and test split, 80, 20. They're abouts. You could use like 70, 30. You could use 90, 10.
It all depends on how much data you have. There's a lot of things in machine learning that are
quite flexible. Train split, we're going to index on our data here so that we can create our splits.
Google Colab auto corrected my code in a non-helpful way just then. And we're going to do the
opposite split for the testing data. Now let's have a look at the lengths of these. If my calculations
are correct, we should have about 40 training samples and 10 testing samples. And again, this
may change in the future. When you work with larger data sets, you might have 100,000 training
samples and 20,000 testing samples. The ratio will often be quite similar. And then let's plot
what's going on here. So plot the data and note, if you don't have the plot predictions
function loaded, this will error. So we can just run plot predictions here if we wanted to. And
we'll pass it in X train, Y train, X test, Y test. And this should come up with our
plot. Wonderful. So we've just recreated the data that we've been previously using. We've got
blue dots to predict green dots. But if this function errors out because you've started the notebook
from here, right from this cell, and you've gone down from there, just remember, you'll just have
to go up here and copy this function. We don't have to do it because we've run all the cells,
but if you haven't run that cell previously, you could put it here and then run it, run it,
and we'll get the same outcome here. Wonderful. So what's next? Well, if we go back to our workflow,
we've just created some data. And have we turned it into tenses yet? I think it's just still, oh,
yeah, it is. It's tenses because we use PyTorch to create it. But now we're up to building or
picking a model. So we've built a model previously. We did that back in build model. So you could
refer to that code and try to build a model to fit the data that's going on here. So that's
your challenge for the next video. So building a PyTorch linear model. And why do we call it linear?
Because linear refers to a straight line. What's nonlinear? Non-straight. So I'll see you in the
next video. Give it a shot before we get there. But we're going to build a PyTorch linear model.
Welcome back. We're going through some steps to recreate everything that we've done. In the last
video, we created some dummy data. And we've got a straight line here. So now by the workflow,
we're up to building a model or picking a model. In our case, we're going to build one

Section 79: Main Topics

Key Topics:

And I've put building a PyTorch linear model here
module because why a lot of PyTorch models, subclass, and then module
Remember, nn in PyTorch stands for neural network

📄 Click to view detailed content

to suit our problem. So we've got some linear data. And I've put building a PyTorch linear model
here. I issued you the challenge of giving it a go. You could do exactly the same steps that
we've done in build model. But I'm going to be a little bit cheeky and introduce something
new here. And that is the power of torch.nn. So let's see it. What we're going to do is we're
going to create a linear model by subclassingnn.module because why a lot of PyTorch models,
subclass, and then module. So class linear regression, what should we call this one?
Linear regression model v2. How about that? And we'll subclassnn.module. So much similar code to
what we've been writing so far. Or when we first created our linear regression model.
And then we're going to put the standard constructor code here, def init underscore underscore.
And it's going to take as an argument self. And then we're going to call super dot another
underscore init underscore underscore brackets. But we're going to instead of if you recall above
back in the build model section, we initialized these parameters ourselves. And I've been hinting
at in the past in videos we've seen before that oftentimes you won't necessarily initialize the
parameters yourself. You'll instead initialize layers that have the parameters in built in those
layers. We still have to create a forward method. But what we're going to see is how we can use our
torch linear layer to do these steps for us. So let's write the code and then we'll step through it.
So we'll go usenn.linear because why we're building linear regression model and our data is linear.
And in the past, our previous model has implemented linear regression formula. So for creating the
model parameters. So we can go self dot linear layer equals. So this is constructing a variable
that this class can use self linear layer equals nn dot linear. Remember, nn in PyTorch stands for
neural network. And we have in features as one of the parameters and out features as another
parameter. This means we want to take as input of size one and output of size one. Where does that
come from? Well, if we have a look at x train and y train, we have one value of x. Maybe there's
too many here. x five will be the first five five and five. So recall, we have one value of x
equates to one value of y. So that means within this linear layer, we want to take as one feature
x to output one feature y. And we're using just one layer here. So the input and the output shapes
of your model in features, out features, what data goes in and what data comes out. These values
will be highly dependent on the data that you're working with. And we're going to see different
data or different examples of input features and output features all throughout this course. So
but that is what's happening. We have one in feature to one out feature. Now what's happening
inside nn.linear. Let's have a look torch and then linear. We go the documentation
applies a linear transformation to the incoming data. Where have we seen this before?
y equals x a t plus b. Now they're using different letters, but we've got the same formula as
what's happening up here. Look at the same formula as our data. Wait times x plus bias. And then if
we look up linear regression formula once again, linear regression formula. We've got this formula
here. Now again, these letters can be replaced by whatever letters you like. But this linear layer
is implementing the linear regression formula that we created in our model before. So it's
essentially doing this part for us. And behind the scenes, the layer creates these parameters for us.
So that's a big piece of the puzzle of pie torch is that as I've said, you won't always be
initializing the parameters your model yourself. You'll generally initialize layers. And then you'll
use those layers in some Ford computation. So let's see how we could do that. So we've got a linear
layer which takes us in features one and out features one. What should we do now? Well, because
we've subclassed nn.module we need to override the Ford method. So we need to tell our model
what should it do as the Ford computation. And in here it's going to take itself as input,
as well as x, which is conventional for the input data. And then we're just going to return
here, self dot linear layer x. Right. And actually, we might use some typing here to say that this
should be a torch tensor. And it's also going to return a torch dot tensor. That's using Python's
type ins. So this is just saying, hey, X should be a torch tensor. And I'm going to return you a
torch tensor, because I'm going to pass x through the linear layer, which is expecting one in feature
and one out feature. And it's going to this linear transform. That's another word for it. Again,
pytorch and machine learning in general has many different names of the same thing. I would call
this linear layer. I'm going to write here, also called linear transform, probing layer,
fully connected layer, dense layer, intensive flow. So a whole bunch of different names for
the same thing, but they're all implementing a linear transform. They're all implementing a
version of linear regression y equals x, a ranspose plus b, in features, out features,
wonderful. So let's see this in action. So we're going to go set the manual seed so we can
get reproducibility as well, torch dot manual seed. And we're going to set model one equals
linear regression. This is model one, because we've already got model zero, linear regression

Section 80: Main Topics

Key Topics:

This goes behind the scenes of how PyTorch creates its different layers
So this style of what's going on here is how you're going to see the majority of your PyTorch deep learning models created using pre-existing layers from the torch
So for all of the common layers in deep learning, because that's what neural networks are, they're layers of different mathematical transformations, PyTorch has a lot of pre-built implementations

📄 Click to view detailed content

V two, and we're going to check model one, and we're going to check its state dictionary,
state dict. There we go. What do we have inside this ordered dict? Has that not created anything
for us? Model one, dot state dinked, ordered dink. We haven't got anything here in the regression
model V two. Ideally, this should be outputting a weight and a bias. Yeah, variables, weight,
and bias. Let's dig through our code line by line and see what we've got wrong. Ah, did you notice
this? The init function so the constructor had the wrong amount of underscores. So it was never
actually constructing this linear layer troubleshooting on the fly team. There we go. Beautiful. So we
have a linear layer, and we have it is created for us inside a weight and a bias. So effectively,
we've replaced the code we wrote above for build model, initializing a weight and bias parameter
with the linear layer. And you might be wondering why the values are slightly different, even though
we've used the manual seed. This goes behind the scenes of how PyTorch creates its different
layers. It's probably using a different form of randomness to create different types of
variables. So just keep that in mind. And to see this in action, we have a conversion here.
So this is what's going on. We've converted, this is our original model class, linear regression.
We initialize our model parameters here. We've got a weight and a bias. But instead, we've
swapped this in our linear regression model V2. This should be V2 to use linear layer. And then
in the forward method, we had to write the formula manually here when we initialize the parameters
manually. But because of the power of torch.nn, we have just passed it through the linear layer,
which is going to perform some predefined forward computation in this layer. So this
style of what's going on here is how you're going to see the majority of your PyTorch
deep learning models created using pre-existing layers from the torch.nn module. So if we go back
into torch.nn, torch.nn, we have a lot of different layers here. So we have convolutional layers,
pooling layers, padding layers, normalization, recurrent, transformer, linear, we're using a
linear layer, dropout, et cetera, et cetera. So for all of the common layers in deep learning,
because that's what neural networks are, they're layers of different mathematical transformations,
PyTorch has a lot of pre-built implementations. So that's a little bit of a sneaky trick that
I've done to alter our model. But we've still got basically the exact same model as we had before.
So what's next? Well, it's to train this model. So let's do that in the next video.
Welcome back. So in the last video, we built a PyTorch linear model, nice and simple using a
single nn.linear layer with one in feature, one out feature. And we over read the forward method
of nn.module using the linear layer that we created up here. So what's going to happen is when we do
the forward parser on our model, we're going to put some data in and it's going to go through
this linear layer, which behind the scenes, as we saw with torch and n linear,
behind the scenes, it's going to perform the linear regression formula here. So y equals x,
a t plus b. But now case, we've got weight and bias. So let's go back. It's now time to write
some training code. But before we do, let's set the model to use the target device. And so in
our case, we've got a device of CUDA. But because we've written device agnostic code, if we didn't
have access to a CUDA device, a GPU, our default device would be a CPU. So let's check the model
device. We can do that first up here, check the model current device, because we're going to use
the GPU here, or we're going to write device agnostic code. That's better to say device agnostic code.
That's the proper terminology device. What device are we currently using? This is the CPU, right?
So by default, the model will end up on the CPU. But if we set it to model one call dot two device,
what do you think it's going to do now? If our current target device is CUDA, we've seen what
two does in the fundamental section, two is going to send the model to the GPU memory. So now let's
check whether parameters of our model live dot device. If we send them to the device previously,
it was the CPU, it's going to take a little bit longer while the GPU gets fired up and goes,
PyTorch goes, Hey, I'm about to send you this model. You ready for it? Boom, there we go. Wonderful.
So now our model is on the device or the target device, which is CUDA. And if CUDA wasn't available,
the target device would be CPU. So this would just come out just exactly how we've got it here.
But with that being said, now let's get on to some training code. And this is the fun part.
What do we have to do? We've already seen this for training. I'm just going to clear up our
workspace a little bit here. For training, we need, this is part of the PyTorch workflow,
we need a loss function. What does a loss function do? Measures how wrong our model is,
we need an optimizer, we need a training loop and a testing loop. And the optimizer, what does that
do? Well, it optimizes the parameters of our model. So in our case, model one dot state dig,
what do we have? So we have some parameters here within the linear layer, we have a weight,
and we have a bias. The optimizer is going to optimize these random parameters so that they

Section 81: Main Topics

Key Topics:

Well, pytorch offers a lot of optimizers in torch dot opt in SGD
LR, which stands for learning rate
In other words, how big of a step will our optimizer change our parameters with every iteration, a smaller learning rate

📄 Click to view detailed content

hopefully reduce the loss function, which remember the loss function measures how wrong our model
is. So in our case, because we're working with the regression problem, let's set up the loss
function. And by the way, all of these steps are part of the workflow. We've got data ready,
we've built or picked a model, we're using a linear model. Now we're up to here 2.1 pick a loss
function and an optimizer, we're going to do build a training loop in the same session,
because you know what, we're getting pretty darn good at this, loss function equals what?
Well, we're going to use l one loss. So let's set that up and then dot l one loss, which is the
same as ma and if we wanted to set up our optimizer, what optimizer could we use? Well, pytorch offers
a lot of optimizers in torch dot opt in SGD. That's stochastic gradient descent, because remember
gradient descent is the algorithm that optimizes our model parameters. Adam is another popular option.
For now, we're going to stick with SGD. LR, which stands for learning rate. In other words,
how big of a step will our optimizer change our parameters with every iteration, a smaller
learning rate. So such as 0001 will be a small step. And then a large learning rate, such as 0.1
will be a larger step. Too big of a step. Our model learns too much. And it explodes too small of a
step. Our model never learns anything. But oh, we actually have to pass params first. I forgot
about that. I got ahead of myself with a learning rate. Params is the parameters we'd like our
optimizer to optimize. So in our case, it's model one dot parameters, because model one is our current
target model. Beautiful. So we've got a loss function and an optimizer. Now, let's write a training
loop. So I'm going to set torch manual seeds so we can try and get as reproducible as results as
possible. Remember, if you get different numbers to what I'm getting, don't worry too much if they're
not exactly the same, the direction is more important. So that means if my loss function is
getting smaller, yours should be getting smaller too. Don't worry too much if your fourth decimal
place isn't the same as what my values are. So we have a training loop ready to be written here.
Epox, how many should we do? Well, we did 200 last time and that worked pretty well. So let's do 200
again. Did you go through the extra curriculum yet? Did you watch the video for the unofficial
PyTorch optimization loop song yet? This one here, listen to the unofficial PyTorch optimization
loop song. If not, it's okay. Let's sing it together. So for an epoch in range, epochs, we're going to
go through the song in a second. We're going to set the model to train. In our case, it's model one,
model to train. Now, step number one is what? Do the forward pass. This is where we calculate
the predictions. So we calculate the predictions by passing the training data through our model.
And in our case, because the forward method in model one implements the linear layer,
this data is going to go through the linear layer, which is torch.nn.linear and go through
the linear regression formula. And then we calculate the loss, which is how wrong our models predictions
are. So the loss value equals loss fn. And here we're going to pass in y-pred and y-train.
Then what do we do? We zero the optimizer, optimizer zero grad, which because by default,
the optimizer is going to accumulate gradients behind the scenes. So every epoch, we want to
reduce those back to zero. So it starts from fresh. We're going to perform back propagation here,
back propagation, by calling loss, stop backwards. If the forward pass goes forward through the
network, the backward pass goes backwards through the network, calculating the gradients for the
loss function with respect to each parameter in the model. So optimizer step, this next part,
is going to look at those gradients and go, you know what? Which way should I optimize the parameters?
So because the optimizer is optimizing the model parameters, it's going to look at the
loss and go, you know what? I'm going to adjust the weight to be increased. And I'm going to lower
the bias and see if that reduces the loss. And then we can do testing. We can do both of these in
the same hit. Now we are moving quite fast through this because we spent a whole bunch of time
discussing what's going on here. So for testing, what do we do? We set the model into evaluation
mode. That's going to turn off things like dropout and batch normalization layers. We don't have any
of that in our model for now, but just it's good practice to always call a vowel whenever you're
doing testing. And same with inference mode. We don't need to track gradients and a whole bunch of
other things PyTorch does behind the scenes when we're testing or making predictions. So we use
the inference mode context manager. This is where we're going to create test pred, which is going
to be our test predictions, because here we're going to pass the test data features, forward
pass through our model. And then we can calculate the test loss, which is our loss function. And we're
going to compare the test pred to Y test. Wonderful. And then we can print out what's happening.
So what should we print out? How about if epoch divided by 10 equals zero. So every 10 epochs,
let's print something out, print. We'll do an F string here, epoch is epoch. And then we'll go

Section 82: Main Topics

Key Topics:

Oh, of course
Yes, of course, that's what's happened
So remember, one of the biggest issues with pytorch aside from shape errors is that you should have your data or all of the things that you're computing with on the same device

📄 Click to view detailed content

loss, which is the training loss, and just be equal to the loss. And then we'll go test loss is
equal to test loss. So do you think this will work? It's okay if you're not sure. But let's find
out together, hey, oh, we've got a, we need a bracket there. Oh my goodness, what's going on?
Run time error. Expected all tenses to be on the same device. Oh, of course. Do you know what's
happening here? But we found at least two devices, CUDA and CPU. Yes, of course, that's what's happened.
So what have we done? Up here, we put our model on the GPU. But what's going on here? Our data?
Has our data on the GPU? No, it's not. By default, it's on the CPU. So we haven't written device
agnostic code for our data. So let's write it here, put data on the target device.
Device agnostic code for data. So remember, one of the biggest issues with pytorch aside from
shape errors is that you should have your data or all of the things that you're computing with
on the same device. So that's why if we set up device agnostic code for our model,
we have to do the same for our data. So now let's put X train to device. Y train equals Y train
to device. This is going to create device agnostic code. In our case, it's going to use CUDA because
we have access to a CUDA device. But if we don't, this code will still work. It will still default
to CPU. So this is good. I like that we got that error because that's the sum of the things you're
going to come across in practice, right? So now let's run this. What's happening here?
Hey, look at that. Wonderful. So our loss starts up here nice and high. And then it starts to go
right down here for the training data. And then the same for the testing data. Beautiful.
Right up here. And then all the way down. Okay. So this looks pretty good on the test data set. So
how can we check this? How can we evaluate our model? Well, one way is to check its state
deck. So state decked. What do we got here? What are our weight and bias? Oh my gosh, so close.
So we just set weight and bias before to be 0.7 and 0.3. So this is what our model has estimated
our parameters to be based on the training data. 0.6968. That's pretty close to 0.7,
nearly perfect. And the same thing with the bias 0.3025 versus the perfect value is 0.93. But remember,
in practice, you won't necessarily know what the ideal parameters are. This is just to exemplify
what our model is doing behind the scenes. It's moving towards some ideal representative
parameters of whatever data we're working with. So in the next video, I'd like you to give it a go
of before we get to the next video, make some predictions with our model and plot them on the
original data. How close to the green dots match up with the red dots? And you can use this plot
predictions formula or function that we've been using in the past. So give that a go and I'll
see you in the next video. But congratulations. Look how quickly we just trained a model using
the steps that we've covered in a bunch of videos so far and device agnostic code. So good.
I'll see you soon. In the last video, we did something very, very exciting. We worked through
training an entire neural network. Some of these steps took us an hour or so worth of videos to
go back through before. But we coded that in one video. So you're ready listening the song just to
remind ourselves of what's going on. For an epoch in a range, call model dot train, do the forward
pass, calculate the loss, optimizer zero grad, loss backward, optimizer step, step, step, let's
test, come on a dot eval with torch inference mode, do the forward pass, calculate the loss,
print out what's happening. And then we do it again, again, again, for another epoch in a range.
Now I'm kidding. We'll just leave it there. We'll just leave it there. But that's the
unofficial pytorch optimization loop song. We created some device agnostic code so that we could
make the calculations on the same device as what our model is because the models also using device
agnostic code. And so now we've got to evaluate our models. We've looked at the loss and the test
lost here. And we know that our models loss is going down. But what does this actually equate to
when it makes predictions? That's what we're most interested in, right? And we've looked at the
parameters. They're pretty close to the ideal parameters. So at the end of last video, I issued
you the challenge to making and evaluating predictions to make some predictions and plot them. I hope
you gave it a shot. Let's see what it looks like together. Hey, so turn the model into evaluation
mode. Why? Because every time we're making predictions or inference, we want our model to be in a
vowel mode. And every time we're training, we want our model to be in training mode. And then we're
going to make predictions on the test data, because we train on the train data, and we evaluate our
model on the test data data that our model has never actually seen, except for when it makes
predictions. With torch inference mode, we turn on inference mode whenever we make inference or
predictions. So we're going to set Y threads equal to model one, and the test data goes in here.
Let's have a look at what the Y threads look like. Wonderful. So we've got a tensor here. It shows

Section 83: Main Topics

Key Topics:

Oh, of course
It uses matplotlib, of course, and matplotlib works with NumPy, not pytorch
So of course, we're running into another error down here, because we just said that our predictions are on the CUDA device

📄 Click to view detailed content

us that they're still on the device CUDA. Why is that? Well, that's because previously we set the
model one to the device, the target device, the same with the test data. So subsequently,
our predictions are also on the CUDA device. Now, let's bring in the plot predictions function here.
So check out our model predictions visually. We're going to adhere to the data explorer's motto
of visualize visualize visualize plot predictions. And predictions are going to be set to
equals Y threads. And let's have a look. How good do these look? Oh, no.
Oh, we've got another error type error. Can't convert CUDA device type tensor to NumPy.
Oh, of course. Look what we've done. So our plot predictions function, if we go back up,
where did we define that? What does our plot predictions function use? It uses matplotlib,
of course, and matplotlib works with NumPy, not pytorch. And NumPy is CPU based. So of course,
we're running into another error down here, because we just said that our predictions are on the CUDA
device. They're not on the CPU. They're on a GPU. So it's giving us this helpful information here.
Use tensor dot CPU to copy the tensor to host memory first. So this is our tensor. Let's call
dot CPU and see what happens then. Is that going to go to CPU? Oh, my goodness. Look at that.
Look at that. Go the linear layer. The red dots, the predictions are basically on top of the testing
data. That is very exciting. Now again, you may not get the exact same numbers here, and that is
perfectly fine. But the direction should be quite similar. So your red dots should be basically on
top of the green dots, if not very slightly off. But that's okay. That's okay. We just want to focus
on the direction here. So thanks to the power of back propagation here and gradient descent,
our models random parameters have updated themselves to be as close as possible to the ideal parameters.
And now the predictions are looking pretty darn good for what we're trying to predict.
But we're not finished there. We've just finished training this model. What would happen if our
notebook disconnected right now? Well, that wouldn't be ideal, would it? So in the next part,
we're going to move on to 6.5, saving, and loading a trained model. So I'm going to give you a
challenge here as well, is to go ahead and go back and refer to this code here, saving model
in PyTorch, loading a PyTorch model, and see if you can save model one, the state dictionary of
model one, and load it back in and get something similar to this. Give that a shot, and I'll see you
in the next video. Welcome back. In the last video, we saw the power of the torch.nn.linear layer,
and back propagation and gradient descent. And we've got some pretty darn good predictions
out of our model. So that's very exciting. Congratulations. You've now trained two machine
learning models. But it's not over yet. We've got to save and load our trained model. So I
issued you the challenge in the last video to try and save and load the model yourself. I hope
you gave that a go. But we're going to do that together in this video. So we're going to start
by importing path because we would like a file path to save our model to. And the first step we're
going to do is create models directory. We don't have to recreate this because I believe we already
have one. But I'm going to put the code here just for completeness. And this is just so if you
didn't have a models directory, this would create one. So model path is going to go to path
models. And then we'd like to model path dot maker, we're going to call maker for make directory.
We'll set parents equal to true. And if it exists, okay, that'll also be true. So we won't get an error.
Oh my gosh, Google collab. I didn't want that. We won't get an error if it already exists.
And two, we're going to create a model save path. So if you recall that pytorch objects in general
have the extension of what? There's a little pop quiz before we get to the end of this sentence.
So this is going to be pytorch workflow for this module that we're going through. This one here,
chapter 01 pytorch workflow model one. And they usually have the extension dot PT for pytorch or
PT H for pytorch as well. I like PT H. But just remember, sometimes you might come across slightly
different versions of that PT or PT H. And we're going to create the model save name or the save
path. It's probably a better way to do it is going to be model path. And then we can use because we're
using the path lib module from Python, we can save it under model name. And so if we look at this,
what do we get model save path? We should get Oh, path is not defined. Oh, too many capitals here,
Daniel. The reason why I'm doing these in capitals is because oftentimes hyper parameters such as epochs
in machine learning are set as hyper parameters LR could be learning rate. And then you could have
as well model name equals Yeah, yeah, yeah. But that's just a little bit of nomenclature trivia for
later on. And model save path, we've done that. Now we're going to save the model state dictionary
rather than the whole model, save the model state deck, which you will find the pros and cons of
in where in the pytorch documentation for saving and loading model, which was a little bit of extra

Section 84: Main Topics

Key Topics:

And the file path we're going to use is, of course, the model save path, which we've seen here is a POSIX path
This one for us from the workflow we did before up here, saving a model in PyTorch, loading a PyTorch model
And now the one we've got, of course, model one is the one that we've just saved

📄 Click to view detailed content

curriculum for a previous video. But let's have a look at our model save path will print it out.
And we'll go torch save, we'll set the object that we're trying to save to equal model one dot state
deck, which is going to contain our trained model parameters. We can inspect what's going on in here,
state deck. They'll show us our model parameters. Remember, because we're only using a single linear
layer, we only have two parameters. But in practice, when you use a model with maybe hundreds of layers
or tens of millions of parameters, viewing the state deck explicitly, like we are now,
might not be too viable of an option. But the principle still remains a state deck contains
all of the models trained or associated parameters, and what state they're in. And the file path we're
going to use is, of course, the model save path, which we've seen here is a POSIX path. Let's save
our model. Wonderful saving model to this file path here. And if we have a look at our folder,
we should have two saved models now, beautiful to save models. This one for us from the workflow
we did before up here, saving a model in PyTorch, loading a PyTorch model. And now the one we've got,
of course, model one is the one that we've just saved. Beautiful. So now let's load a model. We're
going to do both of these in one video. Load a PyTorch model. You know what, because we've had a
little bit of practice so far, and we're going to pick up the pace. So let's go loaded, let's call
it, we'll create a new instance of loaded model one, which is, of course, our linear regression model
V2, which is the version two of our linear regression model class, which subclasses, what?
Subclasses and n.module. So if we go back here up here to where we created it. So linear regression
model V2 uses a linear layer rather than the previous iteration of linear regression model,
which we created right up here. If we go up to here, which explicitly defined the parameters,
and then implemented a linear regression formula in the forward method, the difference between
what we've got now is we use PyTorch's pre-built linear layer, and then we call that linear layer
in the forward method, which is probably the far more popular way of building PyTorch models,
is stacking together pre-built NN layers, and then calling them in some way in the forward method.
So let's load it in. So we'll create a new instance of linear regression model V2, and now what do
we do? We've created a new instance, I'm just going to get out of this, make some space for us.
We want to load the model state deck, the saved model one state deck, which is the state deck that
we just saved beforehand. So we can do this by going loaded model one, calling the load state
decked method, and then passing it torch dot load, and then the file path of where we saved that
PyTorch object before. But the reason why we use the path lib is so that we can just call model
save path in here. Wonderful. And then let's check out what's going on. Or actually, we need to
put the target model or the loaded model to the device. The reason being is because we're doing all
our computing with device agnostic code. So let's send it to the device. And I think that'll be about
it. Let's see if this works. Oh, there we go. Linear regression model V2 in features one,
out features one, bias equals true. Wonderful. Let's check those parameters. Hey, next loaded model
one dot parameters. Are they on the right device? Let's have a look. Beautiful. And let's just check
the loaded state dictionary of loaded model one. Do we have the same values as we had previously?
Yes, we do. Okay. So to conclusively make sure what's going on, let's evaluate the loaded model.
Evaluate loaded model, loaded model one. What do we do for making predictions? Or what do we do to
evaluate? We call dot a vowel. And then if we're going to make some predictions, we use torch
inference mode with torch inference mode. And then let's create loaded model one, threads
equals loaded model one. And we'll pass it the test data. And now let's check for a quality
between Y threads, which is our previous model one preds that we made up here, Y threads.
And we're going to compare them to the fresh loaded model one preds. And should they be the same?
Yes, they are beautiful. And we can see that they're both on the device CUDA. How amazing is that? So
I want to give you a big congratulations, because you've come such a long way. We've gone through
the entire PyTorch workflow from making data, preparing and loading it to building a model.
All of the steps that come in building a model, there's a whole bunch there, making predictions,
training a model, we spent a lot of time going through the training steps. But trust me, it's
worth it, because we're going to be using these exact steps all throughout the course. And in fact,
you're going to be using these exact steps when you build PyTorch models after this course. And
then we looked at how to save a model so we don't lose all our work, we looked at loading a model,
and then we put it all together using the exact same problem, but in far less time. And as you'll
see later on, we can actually make this even quicker by functionalizing some of the code we've already
written. But I'm going to save that for later. I'll see you in the next video, where I'm just

Section 85: Main Topics

Key Topics:

going to show you where you can find some exercises and all of the extra curriculum I've been talking about throughout this section 01 PyTorch workflow
In the last video, we finished up putting things together by saving and loading our trained model, which is super exciting, because let's come to the end of the PyTorch workflow section
So within the book version of the course materials, which is at learnpytorch

📄 Click to view detailed content

going to show you where you can find some exercises and all of the extra curriculum I've been talking
about throughout this section 01 PyTorch workflow. I'll see you there. Welcome back. In the last
video, we finished up putting things together by saving and loading our trained model, which is
super exciting, because let's come to the end of the PyTorch workflow section. So now, this section
is going to be exercises and extra curriculum, or better yet, where you can find them. So I'm
going to turn this into markdown. And I'm going to write here for exercises and extra curriculum.
Refer to. So within the book version of the course materials, which is at learnpytorch.io,
we're in the 01 section PyTorch workflow fundamentals. There'll be more here by the time you watch
this video likely. And then if we go down here, at the end of each of these sections, we've got
the table of contents over here. We've got exercises and extra curriculum. I listed a bunch of things
throughout this series of 01 videos, like what's gradient descent and what's back propagation. So
I've got plenty of resources to learn more on that. There's the loading and saving PyTorch
documentation. There's the PyTorch cheat sheet. There's a great article by Jeremy Howard for a
deeper understanding of what's going on in torch.nn. And there's, of course, the unofficial PyTorch
optimization loop song by yours truly, which is a bit of fun. And here's some exercises. So
the exercises here are all based on the code that we wrote throughout section 01. So there's
nothing in the exercises that we haven't exactly covered. And if so, I'll be sure to put a note
in the exercise itself. But we've got create a straight line data set using the linear regression
formula. And then build a model by subclassing and end up module. So for these exercises, there's an
exercise notebook template, which is, of course, linked here. And in the PyTorch deep learning
GitHub, if we go into here, and then if we go into extras, and if we go into exercises, you'll
find all of these templates here. They're numbered by the same section that we're in. This is PyTorch
workflow exercises. So if you wanted to complete these exercises, you could click this notebook
here, open in Google CoLab. I'll just wait for this to load. There we go. And you can start to
write some code here. You could save a copy of this in your own Google Drive and go through this.
It's got some notes here on what you should be doing. You can, of course, refer to the text-based
version of them. They're all here. And then if you want an example of what some solutions look
like, now, please, I can't stress enough that I would highly, highly recommend trying the exercises
yourself. You can use the book that we've got here. This is just all the code from the videos.
You can use this. You can use, I've got so many notebooks here now, you can use all of the code
that we've written here to try and complete the exercises. But please give them a go yourself.
And then if you go back into the extras folder, you'll also find solutions. And this is just one
example solutions for section 01. But I'm going to get out of that so you can't cheat and look
at the solutions first. But there's a whole bunch of extra resources all contained within
the PyTorch deep loaning repo, extras, exercises, solutions, and they're also in the book version
of the course. So I'm just going to link this in here. I'm going to put this right at the bottom
here. Wonderful. But that is it. That is the end of the section 01 PyTorch workflow.
So exciting. We went through basically all of the steps in a PyTorch workflow,
getting data ready, turning into tenses, build or pick a model, picking a loss function on an
optimizer. We built a training loop. We fit the model to the data. We made a prediction.
We evaluated our model. We improved through experimentation by training for more epochs.
We'll do more of this later on. And we saved and reload our trained model.
But that's going to finish 01. I will see you in the next section. Friends, welcome back.
We've got another very exciting module. You ready? Neural network classification with
PyTorch. Now combining this module once we get to the end with the last one, which was
regression. So remember classification is predicting a thing, but we're going to see this in a second.
And regression is predicting a number. Once we've covered this, we've covered two of the
the biggest problems in machine learning, predicting a number or predicting a thing.
So let's start off with before we get into any ideas or code, where can you get help?
First things first is follow along with the code. If you can, if in doubt, run the code.
Try it for yourself. Write the code. I can't stress how important this is.
If you're still stuck, press shift, command, and space to read the doc string of any of the
functions that we're running. If you are on Windows, it might be control. I'm on a Mac, so I put command
here. If you're still stuck, search for your problem. If an error comes up, just copy and paste
that into Google. That's what I do. You might come across resources like Stack Overflow or,
of course, the PyTorch documentation. We'll be referring to this a lot again throughout this
section. And then finally, oh wait, if you're still stuck, try again. If in doubt, run the code.
And then finally, if you're still stuck, don't forget, you can ask a question. The best place to

Section 86: Main Topics

Key Topics:

do so will be on the course GitHub, which will be at the discussions page, which is linked here
If we load this up, there's nothing here yet, because as I record these videos, the course hasn't launched yet, but press new discussion
Finally, don't forget that this notebook that we're about to go through is based on chapter two of the Zero to Mastery Learn PyTorch for deep learning, which is neural network classification with PyTorch

📄 Click to view detailed content

do so will be on the course GitHub, which will be at the discussions page, which is linked here.
If we load this up, there's nothing here yet, because as I record these videos, the course
hasn't launched yet, but press new discussion. Talk about what you've got. Problem with XYZ.
Let's go ahead. Leave a video number here and a timestamp, and that way, we'll be able to help
you out as best as possible. So video number, timestamp, and then your question here, and you
can select Q&A. Finally, don't forget that this notebook that we're about to go through is based
on chapter two of the Zero to Mastery Learn PyTorch for deep learning, which is neural network
classification with PyTorch. All of the text-based code that we're about to write is here. That
was a little spoiler. And don't forget, this is the home page. So my GitHub repo slash PyTorch
deep learning for all of the course materials, everything you need will be here. So that's very
important. How can you get help? But this is the number one. Follow along with the code and try
to write it yourself. Well, with that being said, when we're talking about classification,
what is a classification problem? Now, as I said, classification is one of the main problems of
machine learning. So you probably already deal with classification problems or machine learning
powered classification problems every day. So let's have a look at some examples. Is this email
spam or not spam? Did you check your emails this morning or last night or whenever? So chances are
that there was some sort of machine learning model behind the scenes. It may have been a neural
network. It may have not that decided that some of your emails won't spam. So to Daniel,
at mrdberg.com, hey, Daniel, this steep learning course is incredible. I can't wait to use what
I've learned. Oh, that's such a nice message. If you want to send that email directly to me,
you can. That's my actual email address. But if you want to send me this email, well, hopefully
my email, which is hosted by some email service detects this as spam because although that is a
lot of money and it would be very nice, I think if someone can't spell too well, are they really
going to pay me this much money? So thank you email provider for classifying this as spam. And now
because this is one thing or another, not spam or spam, this is binary classification. So in this
case, it might be one here and this is a zero or zero or one. So one thing or another, that's binary
classification. If you can split it into one thing or another, binary classification. And then we
have an example of say we had the question, we asked our photos app on our smartphone or whatever
device you're using. Is this photo of sushi steak or pizza? We wanted to search our photos for every
time we've eaten sushi or every time we've eaten steak or every time we've eaten pizza far out
and this looks delicious. But this is multi class classification. Now, why is this? Because we've
got more than two things. We've got 123. And now this could be 10 different foods. It could be 100
different foods. It could be 1000 different categories. So the image net data set, which is a popular
data set for computer vision, image net, we go to here, does it say 1000 anywhere, 1k or 1000?
No, it doesn't. But if we go image net 1k, download image net data, maybe it's here.
It won't say it, but you just, oh, there we go, 1000 object classes. So this is multi class
classification because it has 1000 classes, that's a lot, right? So that's multi class classification,
more than one thing or another. And finally, we might have multi label classification,
which is what tags should this article have when I first got into machine learning, I got these
two mixed up a whole bunch of times. Multi class classification has multiple classes such as sushi
steak pizza, but assigns one label to each. So this photo would be sushi in an ideal world. This is
steak and this is pizza. So one label to each. Whereas multi label classification means you could
have multiple different classes. But each of your target samples such as this Wikipedia article,
what tags should this article have? It may have more than one label. It might have three labels,
it might have 10 labels. In fact, what if we went to the Wikipedia page for deep learning Wikipedia
and does it have any labels? Oh, there we go. Where was that? I mean, you can try this yourself.
This is just the Wikipedia page for deep learning. There is a lot, there we go categories deep
learning, artificial neural networks, artificial intelligence and emerging technologies. So that
is an example. If we wanted to build a machine learning model to say, read all of the text in
here and then go tell me what are the most relevant categories to this article? It might come up
with something like these. In this case, because it has one, two, three, four, it has multiple labels
rather than just one label of deep learning, it could be multi label classification. So we'll go
back. But there's a few more. These will get you quite far in the world of classification.
So let's dig a little deeper on binary versus multi class classification. You may have already
experienced this. So in my case, if I search on my phone in the photos app for photos of a dog,
it might come here. If I search for photos of a cat, it might come up with this. But if I wanted

Section 87: Main Topics

Key Topics:

In other words, because remember, machine learning models, neural networks love to have numerical inputs
Well, of course, we're going to be part cook, part chemist, part artist, part science
Well, let's break it down to inputs, some kind of machine learning algorithm, and then outputs

📄 Click to view detailed content

to train an algorithm to detect the difference between photos of these are my two dogs.
Aren't they cute? They're nice and tired and they're sleeping like a person. This is seven.
Number seven, that's her name. And this is Bella. This is a cat that me and my partner rescued.
And so I'm not sure what this cat's name is actually. So I'd love to give it a name, but I can't.
So binary classification, if we wanted to build an algorithm, we wanted to feed it, say, 10,000
photos of dogs and 10,000 photos of cats. And then we wanted to find a random image on the
internet and pass it through to our model and say, hey, is this a dog or is this a cat? It would
be binary classification because the options are one thing or another dog or cat. But then for
multi-class classification, let's say we've been working on a farm and we've been taking some photos
of chickens because they groovy, right? Well, we updated our model and added some chicken photos
in there. We would now be working with a multi-class classification problem because we've got more
than one thing or another. So let's jump in to what we're going to cover. This is broadly,
by the way, because this is just text on a page. You know, I like to just write code of what we're
actually doing. So we're going to look at the architecture of a neural network classification
model. We're going to check what the input shapes and output shapes of a clas

License

Ali-hey-0/deep-learning-with-pytorch

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Packages