-
Notifications
You must be signed in to change notification settings - Fork 2
/
Copy pathmodule5.html
210 lines (150 loc) · 5.88 KB
/
module5.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
<!DOCTYPE html>
<html>
<head>
<title>Writing a PyTorch module[Marc Lelarge]</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<link rel="stylesheet" href="./assets/katex.min.css">
<link rel="stylesheet" type="text/css" href="./assets/slides.css">
<link rel="stylesheet" type="text/css" href="./assets/grid.css">
</head>
<body>
<textarea id="source">
class: center, middle, title-slide
count: false
# Module 5
## Stacking layers
<br/><br/>
.bold[Marc Lelarge]
---
# Deep Learning pipeline
## Dataset and Dataloader + .red[Model] + .grey[Loss and Optimizer] = .grey[Training]
.center[
<img src="images/lesson1/ml4.png" style="width: 1000px;" />
]
---
# Overview of the course:
1- .grey[Course overview: machine learning pipeline]
2- .grey[PyTorch tensors and automatic differentiation]
3- .grey[Classification with deep learning]
4- Convolutional neural networks
* .red[Building block for a model] - in PyTorch: [Convolutional layrers](https://pytorch.org/docs/stable/nn.html#convolution-layers) and [Pooling layers](https://pytorch.org/docs/stable/nn.html#pooling-layers) in `torch.nn`
---
# Writing a PyTorch module
We only built once a Neural Network from scratch in [Module 2](https://github.com/dataflowr/notebooks/blob/master/Module2/02b_linear_reg.ipynb)
```
model = torch.nn.Sequential(torch.nn.Linear(2, 1))
```
--
count: false
A Multilayer perceptron (MLP) with a 2 dimension input, 1 dimension output, ReLU activation and one hidden layers of dimensions 50 and a final sigmoid layer can be written as:
```
import torch.nn as nn
model = torch.nn.Sequential(nn.Linear(2, 50), nn.ReLU(), nn.Linear(50,1), nn.Sigmoid())
```
--
count: false
But for any model of reasonable complexity, the best is to write a sub-class of `torch.nn.Module`.
---
# Writing a PyTorch module
To create a module, one has to inherit from the base class `torch.nn.Module` and implement the constructor `__init__(self, ...)` and the forward pass `forward(self, x)`.
Here is the code for our last model:
```
import torch
import torch.nn as nn
*import torch.nn.functional as F
class MLP(nn.Module):
def __init__(self):
super(MLP, self).__init__()
self.fc1 = nn.Linear(in_features=2, out_features=50)
self.fc2 = nn.Linear(in_features=50, out_features=1)
def forward(self,x):
x = self.fc1(x)
* x = F.relu(x)
x = self.fc2(x)
* return F.sigmoid(x)
```
[difference between `nn.ReLU()` and `F.relu`](https://discuss.pytorch.org/t/whats-the-difference-between-nn-relu-vs-f-relu/27599/2)
---
# Writing a PyTorch module
Inheriting from `torch.nn.Module` provides many mechanisms implemented in the superclass.
First, the `(...)` operator is redefined to call the `forward(...)` method and run additional operations, see [ref](https://discuss.pytorch.org/t/any-different-between-model-input-and-model-forward-input/3690).
The forward pass should be executed through this operator and not by calling forward explicitly.
```
model = MLP()
input = torch.empty(12, 2).normal_()
output = model(input)
```
--
count: false
All `Parameters` added as class attributes are seen by `Module.parameters()`.
As long as you use autograd-compliant operations, the backward pass is implemented automatically.
This is crucial to allow the update of the `Parameters` with your optimizer.
---
# Writing a PyTorch module
Define a loss and an optimizer as before:
```
loss_fn = nn.BCELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)
```
--
count: false
One step of gradient descent is done as before:
```
output = model(x)
loss = loss_fn(output, label)
optimizer.zero_grad()
loss.backward()
optimizer.step()
```
So now is time to construct more complex architectures!
---
class: end-slide, center
count: false
The end.
</textarea>
<script src="./assets/remark-latest.min.js"></script>
<script src="./assets/auto-render.min.js"></script>
<script src="./assets/katex.min.js"></script>
<script type="text/javascript">
function getParameterByName(name, url) {
if (!url) url = window.location.href;
name = name.replace(/[\[\]]/g, "\\$&");
var regex = new RegExp("[?&]" + name + "(=([^&#]*)|&|#|$)"),
results = regex.exec(url);
if (!results) return null;
if (!results[2]) return '';
return decodeURIComponent(results[2].replace(/\+/g, " "));
}
//var options = {sourceUrl: getParameterByName("p"),
// highlightLanguage: "python",
// // highlightStyle: "tomorrow",
// // highlightStyle: "default",
// highlightStyle: "github",
// // highlightStyle: "googlecode",
// // highlightStyle: "zenburn",
// highlightSpans: true,
// highlightLines: true,
// ratio: "16:9"};
var options = {sourceUrl: getParameterByName("p"),
highlightLanguage: "python",
highlightStyle: "github",
highlightSpans: true,
highlightLines: true,
ratio: "16:9",
slideNumberFormat: (current, total) => `
<div class="progress-bar-container">${current}/${total} <br/><br/>
<div class="progress-bar" style="width: ${current/total*100}%"></div>
</div>
`};
var renderMath = function() {
renderMathInElement(document.body, {delimiters: [ // mind the order of delimiters(!?)
{left: "$$", right: "$$", display: true},
{left: "$", right: "$", display: false},
{left: "\\[", right: "\\]", display: true},
{left: "\\(", right: "\\)", display: false},
]});
}
var slideshow = remark.create(options, renderMath);
</script>
</body>
</html>