Visualization
As we can see in this plot, our original data didn't come from a normal distribution. Standardization transforms the data such that $\mu = 0$ and $\sigma=1$, but doesn't make it normally distributed.
You can also implement more variants or simply use those implemented in Optimisers.jl.
Click plot_trajectory below to reveal the code that produced the plot. The most relevant part is the inner loop where the optimizer is called:
state = Optimisers.setup(opt, params)
for i in 1:steps
grad = ... # Compute gradient. We will learn how this works next week.
state, params = Optimisers.update(state, params, grad)
end
Let's plot the trajectory of our optimizers on the "loss surface".
Add as many optimizers and sliders as you want, change the initial parameters and the number of steps taken.
Task
Please add your student information to the cell below.
Task (1 point)
Define an Optimisers.init function for GradientDescent that returns nothing.
We return nothing since gradient descent doesn't require a state.
Keep working on it!
Make sure that you define a struct called GradientDescent.
Optional task
Implement the Adam optimizer, which was introduced in this paper by Kingma and Ba. Add it to the visualization above and compare it to Optimizer.jl's Adam implementation.
Task (1 point)
Define an Optimisers.apply! function for GradientDescent that
mimetext/htmlrootassigneelast_run_timestampA( ݫpersist_js_state·has_pluto_hook_features§cell_id$c2adc38f-84e3-486f-93d8-b88eea63a09bdepends_on_disabled_cells§runtime v_published_object_keysdepends_on_skipped_cells§errored$5370c50b-a9b8-41dc-8cfe-1f0389df97c3queued¤logsrunning¦outputbodylOptional Task – Adam optimizer
mimetext/htmlrootassigneelast_run_timestampA( 袱persist_js_state·has_pluto_hook_features§cell_id$5370c50b-a9b8-41dc-8cfe-1f0389df97c3depends_on_disabled_cells§runtime Spublished_object_keysdepends_on_skipped_cells§errored$32e8ee04-4378-4377-ae47-b65e5ce70667queued¤logsrunning¦outputbodycExercise 2.2 – init
mimetext/htmlrootassigneelast_run_timestampA( persist_js_state·has_pluto_hook_features§cell_id$32e8ee04-4378-4377-ae47-b65e5ce70667depends_on_disabled_cells§runtime y~published_object_keysdepends_on_skipped_cells§errored$befba318-8595-4fc4-b42e-9c9125e7abfbqueued¤logsrunning¦outputbodyTip
Statefulness could also have been implemented by making the state a field in the struct of the optimizer and/or making the struct mutable.
However, mutable structs are usually less performant. The dimensionality of the state also often depends on the dimensionality of the input parameters we optimize over.
Tip
Reminder: By convention, functions that modify their inputs in-place are given an exclamation point ! at the end of their name.
Tip
Gradient descent is typically used in deep learning in combination with mini-batches, which introduce stochasticity.
This example had no stochasticity and other optimization algorithms would have been better suited.
For the purpose of plotting, we define a range of parameters:
Initializing the optimizer
Some optimizers require a state. This state is used to save additional information about previous optimization steps, for example in optimizers that use "momentum" terms like ADAM.
When setting up an Optimizer in Optimisers.jl, init is called on it, which initializes a state. init takes two arguments: an optimizer object and an array of initial parameters.
function Optimisers.init(alg::MyOptimiser, params_init::AbstractArray)
state = ... # some computation
return state
end
The state is then updated in the apply! function.
Exercise 2.1 – Constructor
mimetext/htmlrootassigneelast_run_timestampA( Opersist_js_state·has_pluto_hook_features§cell_id$d16b1a65-5da9-46ac-b0b8-59e55ced37f3depends_on_disabled_cells§runtime ^published_object_keysdepends_on_skipped_cells§errored$7de85c6f-996a-4d5b-8236-271922aeb908queued¤logsrunning¦outputbodymsg(UndefVarError: `Standardize` not definedstacktracecall_shorttop-level scopeinlined£urlpathu/home/runner/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#7de85c6f-996a-4d5b-8236-271922aeb908source_packagecalltop-level scopelinfo_typeCore.CodeInfolinefile:H3_Custom_Types.jl#==#7de85c6f-996a-4d5b-8236-271922aeb908functop-level scopeparent_modulefrom_c«plain_errorٻUndefVarError: `Standardize` not defined
Stacktrace:
[1] top-level scope
@ ~/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#7de85c6f-996a-4d5b-8236-271922aeb908:1mime'application/vnd.pluto.stacktrace+objectrootassigneelast_run_timestampA(|b2persist_js_state·has_pluto_hook_features§cell_id$7de85c6f-996a-4d5b-8236-271922aeb908depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$edb7814a-eddf-4c87-8857-19bb0a0c0241queued¤logsrunning¦outputbodyFeedback
Please help us improve the course by giving us feedback!
You can write whatever you want in the following string. Feel free to add or delete whatever you want.
Task (4 points)
Create a parametric composite type Standardize with two fields mean and std (for $\mu$ and $\sigma$) using the type parameter T<:Real.
These two fields will hold the mean and the standard deviation of the standardization transformation.
Standardize should have two constructors:
An inner constructor that takes two numbers $μ$ and $σ$. Throw an error if $σ ≤ 0$.
An outer constructor that takes an array data, computes its mean and standard deviation and calls the inner constructor.
You are allowed to use the mean and std functions from the standard library package Statistics.jl.
Julia for Machine Learning
Homework 3: Custom types
TU Berlin, Summer Semester 2026
mimetext/htmlrootassigneelast_run_timestampA( epersist_js_state·has_pluto_hook_features§cell_id$f7347c06-c1b7-11ed-3b8e-fbf167ce9cbadepends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$c322fd53-78db-40ae-b7ba-612200ab6617queued¤logsrunning¦outputbodyGInterlude: Gradient-based optimization in Deep Learning
Gradient-based optimization methods are used in deep learning, where we want to optimize a neural network $f$ that is parametrized by parameters $\theta$ (e.g. the weights and biases of $f$).
This neural network could e.g. be a classifier that predicts the output class $\hat{y}$ for some input image $x$:
$$\hat{y} = f_\theta(x)$$
In supervised learning, we are also given true labels $y$. Based on the true and predicted labels, we define a loss
$$\mathcal{L}_\theta(\hat{y}, y)$$
which depends on $\theta$, since the parameters of our neural network determine the prediction $\hat{y}$.
To optimize $\theta$, we typically compute the gradient of our loss with respect to our parameters and update the parameters using gradient-based optimization methods.
Interlude: Understanding Optimisers.jl
All optimization algorithms in Optimisers.jl are sub-types of the abstract type AbstractRule.
To extend Optimisers.jl, we simply need to define a custom sub-type and overload two functions init and apply! using multiple dispatch.
Due date: Monday, May 11th 2026 at 23:59
Tip
The purpose of this exercise is to teach you how to extend functionality from external Julia packages.
Missing Response
Please replace "Mara Mustermann" with your name. Use Shift+Enter to run your edits.
Applying the update step
The actual update step is handled automatically by Optimisers.jl. After implementing the "internal" functions init and apply! to define a custom optimizer, we can use the "public" API:
Initializing packages
Note: Running this notebook for the first time can take several minutes.
mimetext/htmlrootassigneelast_run_timestampA( persist_js_state·has_pluto_hook_features§cell_id$06c0ad65-22d4-4c8e-ae19-4f05ba125e79depends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$dcbf8963-4a5c-4e7d-81a9-3dce27fb3117queued¤logsrunning¦outputbodyjLet's check whether the transformed data has $\mu'=0$ and $\sigma'=1$.
Note that due to rounding errors arising from floating point arithmetic, the results will not be exactly zero and one:
Keep working on it!
Make sure that you define a struct called GradientDescent.
Keep working on it!
Make sure that you define a struct called Standardize.
Updating the optimizer
The function apply! can do two things before the final update step is applied:
it can modify the gradient
it can update the optimizer state
function Optimisers.apply!(alg::MyOptimiser, state, params, grad)
grad_new = ... # some computation
state_new = ... # some computation
return state_new, grad_new
end
Julia for Machine Learning
Homework 3: Custom types
TU Berlin, Summer Semester 2026
"""metadatashow_logsèdisabled®skip_as_script«code_folded$c322fd53-78db-40ae-b7ba-612200ab6617cell_id$c322fd53-78db-40ae-b7ba-612200ab6617codepmd"""### Interlude: Gradient-based optimization in Deep Learning
Gradient-based optimization methods are used in deep learning, where we want to optimize a neural network $f$ that is parametrized by parameters $\theta$ (e.g. the weights and biases of $f$).
This neural network could e.g. be a classifier that predicts the output class $\hat{y}$ for some input image $x$:
$\hat{y} = f_\theta(x)$
In [supervised learning](https://en.wikipedia.org/wiki/Supervised_learning),
we are also given true labels $y$.
Based on the true and predicted labels, we define a loss
$\mathcal{L}_\theta(\hat{y}, y)$
which depends on $\theta$, since the parameters of our neural network determine the prediction $\hat{y}$.
To optimize $\theta$, we typically compute the gradient of our loss with respect to our parameters and update the parameters using gradient-based optimization methods.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$dc4f7f55-dde5-4ee6-b43d-7f64269b55d1cell_id$dc4f7f55-dde5-4ee6-b43d-7f64269b55d1code# Write your code heremetadatashow_logsèdisabled®skip_as_script«code_folded$cd7c244d-8a4c-4aaf-ad4b-891744e004becell_id$cd7c244d-8a4c-4aaf-ad4b-891744e004becodeplot_loss_surface(; fill=true)metadatashow_logsèdisabled®skip_as_script«code_folded$ed873ea2-d4bd-46e7-b89b-4e5494b606e6cell_id$ed873ea2-d4bd-46e7-b89b-4e5494b606e6codeoptimizers = Dict(
"GradientDescent" => GradientDescent(lr1), # our implementation
"Momentum" => Momentum(lr2), # from Optimisers.jl
"ADAM" => ADAM(0.1), # from Optimisers.jl
)metadatashow_logsèdisabled®skip_as_script«code_folded$b3832fb6-dd19-4b38-8117-b0a11d1bc34fcell_id$b3832fb6-dd19-4b38-8117-b0a11d1bc34fcode md"""### Interlude: Understanding Optimisers.jl
All optimization algorithms in Optimisers.jl are sub-types of the abstract type `AbstractRule`.
To extend Optimisers.jl, we simply need to define a custom sub-type and overload two functions `init` and `apply!` using multiple dispatch.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$bdcb27c5-0603-49ac-b831-d78c558b31f0cell_id$bdcb27c5-0603-49ac-b831-d78c558b31f0code0md"Due date: **Monday, May 11th 2026 at 23:59**"metadatashow_logsèdisabled®skip_as_script«code_folded$33866006-11d7-46cc-870b-fec519a2b541cell_id$33866006-11d7-46cc-870b-fec519a2b541codevtip(
md"The purpose of this exercise is to teach you how to extend functionality from external Julia packages.",
)metadatashow_logsèdisabled®skip_as_script«code_folded$44ec9e94-f6af-431e-8841-bae44431dfa3cell_id$44ec9e94-f6af-431e-8841-bae44431dfa3codeif student.name == "Mara Mustermann"
still_missing(md"""Please replace `"Mara Mustermann"` with your name.
Use `Shift+Enter` to run your edits.""")
elseif student.email == "m.mustermann@campus.tu-berlin.de"
still_missing(md"Please enter your TU Berlin E-Mail address.")
elseif student.id == 456123
still_missing(md"Please enter your TU Berlin student ID (Matrikelnummer).")
elseif !isa(student.id, Number)
still_missing(md"Please enter your TU Berlin student ID as number, not a string.")
else
correct()
endmetadatashow_logsèdisabled®skip_as_script«code_folded$57a8108f-9ad3-4cc2-9372-bb291fc3bf86cell_id$57a8108f-9ad3-4cc2-9372-bb291fc3bf86codeTableOfContents()metadatashow_logsèdisabled®skip_as_script«code_folded$50b34968-a45f-4c5b-a83d-7a5b47740007cell_id$50b34968-a45f-4c5b-a83d-7a5b47740007codemd"""#### Applying the update step
The actual update step is handled automatically by Optimisers.jl.
After implementing the "internal" functions `init` and `apply!` to define a custom optimizer, we can use the "public" API:
- `Optimisers.setup`: calls `init` to set up the optimizer
- `Optimisers.update`: calls `apply!` to compute `new_grad` and `next_state` and performs the update step `params_new = params - new_grad`
"""metadatashow_logsèdisabled®skip_as_script«code_folded$06c0ad65-22d4-4c8e-ae19-4f05ba125e79cell_id$06c0ad65-22d4-4c8e-ae19-4f05ba125e79codekmd"### Initializing packages
**Note:** Running this notebook for the first time can take several minutes.
"metadatashow_logsèdisabled®skip_as_script«code_folded$dcbf8963-4a5c-4e7d-81a9-3dce27fb3117cell_id$dcbf8963-4a5c-4e7d-81a9-3dce27fb3117codemd"Let's check whether the transformed data has $\mu'=0$ and $\sigma'=1$.
Note that due to [rounding errors](https://en.wikipedia.org/wiki/Machine_epsilon) arising from floating point arithmetic, the results will not be exactly zero and one:
"metadatashow_logsèdisabled®skip_as_script«code_folded$5762be30-3084-4eda-9642-19dc9d4dcb10cell_id$5762be30-3084-4eda-9642-19dc9d4dcb10codeif !@isdefined(GradientDescent)
keep_working(md"Make sure that you define a struct called `GradientDescent`.")
elseif !hasmethod(Optimisers.init, Tuple{GradientDescent,Vector{Float64}})
keep_working()
else
let
gd = GradientDescent()
state = Optimisers.init(gd, [1.0])
if !isnothing(state)
keep_working(md"Initializing `GradientDescent` should return `nothing`.")
else
correct()
end
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$39063d08-d687-45a2-8367-734b12dcefeacell_id$39063d08-d687-45a2-8367-734b12dcefeacode(if !@isdefined(Standardize)
keep_working(md"Make sure that you define a struct called `Standardize`.")
elseif !@isdefined(transform)
not_defined(:transform)
elseif !@isdefined(inverse_transform)
not_defined(:inverse_transform)
else
let
train = [1, 4, 5]
test = [2, 1, 3]
dtt = Standardize(train)
res_tf = transform(test, dtt)
sol_tf = [-0.6405126152203485, -1.12089707663561, -0.1601281538050872]
res_itf = inverse_transform(res_tf, dtt)
if ismissing(res_tf)
still_missing(md"Replace `missing` in `transform` with your answer.")
elseif isnothing(res_tf)
keep_working(md"Did you forget to write `return` in `transform`?")
elseif !(res_tf ≈ sol_tf)
keep_working(md"The answer is not quite right. Keep working on `transform`.")
elseif ismissing(res_itf)
still_missing(md"Replace `missing` in `inverse_transform` with your answer.")
elseif isnothing(res_itf)
keep_working(md"Did you forget to write `return` in `inverse_transform`?")
elseif !(res_itf ≈ test)
keep_working(
md"The answer is not quite right. Keep working on `inverse_transform`."
)
else
correct()
end
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$582cf667-b85b-4a7d-bb17-88d4776fc240cell_id$582cf667-b85b-4a7d-bb17-88d4776fc240codeusing Optimisersmetadatashow_logsèdisabled®skip_as_script«code_folded$d03e4e95-faab-4ab3-ab27-81189cbd8231cell_id$d03e4e95-faab-4ab3-ab27-81189cbd8231codeّstudent = (
name="Mara Mustermann",
email="m.mustermann@campus.tu-berlin.de", # TU Berlin email address
id=456123, # Matrikelnummer
)metadatashow_logsèdisabled®skip_as_script«code_folded$e0705a25-a01e-43a8-8c1d-ccf0da22fca7cell_id$e0705a25-a01e-43a8-8c1d-ccf0da22fca7codermd"""#### Updating the optimizer
The function `apply!` can do two things before the final update step is applied:
1. it can modify the gradient
1. it can update the optimizer state
```julia
function Optimisers.apply!(alg::MyOptimiser, state, params, grad)
grad_new = ... # some computation
state_new = ... # some computation
return state_new, grad_new
end
```
"""metadatashow_logsèdisabled®skip_as_script«code_foldedënotebook_id$5c9c9f8c-4a20-11f1-0bf5-5997a3917a21in_temp_dir¨metadata