From c7b02eb229a2aa1b867481dd79462627062da704 Mon Sep 17 00:00:00 2001 From: "Documenter.jl" Date: Fri, 15 Dec 2023 09:22:56 +0000 Subject: [PATCH] build based on 2a8c838 --- dev/.documenter-siteinfo.json | 2 +- dev/how_to_submit_hw/index.html | 2 +- dev/index.html | 2 +- dev/installation/index.html | 2 +- dev/lecture_01/basics/index.html | 2 +- dev/lecture_01/demo/index.html | 2 +- dev/lecture_01/hw/986c9d56.svg | 60 ------ dev/lecture_01/hw/b38604ce.svg | 60 ++++++ dev/lecture_01/hw/index.html | 2 +- dev/lecture_01/lab/index.html | 2 +- dev/lecture_01/motivation/index.html | 2 +- dev/lecture_01/outline/index.html | 2 +- dev/lecture_02/hw/index.html | 2 +- dev/lecture_02/lab/index.html | 14 +- dev/lecture_02/lecture/index.html | 2 +- dev/lecture_03/hw/index.html | 8 +- dev/lecture_03/lab/index.html | 16 +- dev/lecture_03/lecture/index.html | 2 +- dev/lecture_04/hw/index.html | 2 +- .../lab/{f48b3c02.svg => 2c8fa7d3.svg} | 68 +++--- dev/lecture_04/lab/index.html | 19 +- dev/lecture_04/lecture/index.html | 2 +- dev/lecture_05/hw/index.html | 2 +- dev/lecture_05/lab/index.html | 12 +- dev/lecture_05/lecture/index.html | 2 +- dev/lecture_06/hw/index.html | 2 +- .../lab/{2693a6cc.svg => 28287c05.svg} | 90 ++++---- dev/lecture_06/lab/index.html | 32 +-- dev/lecture_06/lecture/index.html | 2 +- dev/lecture_07/hw/index.html | 2 +- dev/lecture_07/lab/index.html | 2 +- dev/lecture_07/lecture/index.html | 2 +- dev/lecture_07/macros/index.html | 2 +- dev/lecture_08/hw/index.html | 2 +- .../lab/{adbd310d.svg => a2df3780.svg} | 202 +++++++++--------- dev/lecture_08/lab/index.html | 42 ++-- .../lecture/{5454f9cc.svg => 8d3f576c.svg} | 68 +++--- dev/lecture_08/lecture/index.html | 4 +- dev/lecture_09/ircode/index.html | 2 +- dev/lecture_09/lab/index.html | 42 ++-- dev/lecture_09/lecture/index.html | 4 +- dev/lecture_10/hw/index.html | 2 +- dev/lecture_10/lab/index.html | 2 +- dev/lecture_10/lecture/index.html | 2 +- .../hw/{ac32a377.svg => 681d107c.svg} | 84 ++++---- dev/lecture_12/hw/index.html | 2 +- dev/lecture_12/lab/1bab4067.svg | 48 +++++ .../lab/{703c4e58.svg => 330135ea.svg} | 80 +++---- .../lab/{893cdf9c.svg => cb3fec90.svg} | 76 +++---- dev/lecture_12/lab/index.html | 18 +- dev/lecture_12/lecture/index.html | 2 +- dev/projects/index.html | 2 +- dev/search_index.js | 2 +- 53 files changed, 578 insertions(+), 533 deletions(-) delete mode 100644 dev/lecture_01/hw/986c9d56.svg create mode 100644 dev/lecture_01/hw/b38604ce.svg rename dev/lecture_04/lab/{f48b3c02.svg => 2c8fa7d3.svg} (52%) rename dev/lecture_06/lab/{2693a6cc.svg => 28287c05.svg} (81%) rename dev/lecture_08/lab/{adbd310d.svg => a2df3780.svg} (96%) rename dev/lecture_08/lecture/{5454f9cc.svg => 8d3f576c.svg} (86%) rename dev/lecture_12/hw/{ac32a377.svg => 681d107c.svg} (92%) create mode 100644 dev/lecture_12/lab/1bab4067.svg rename dev/lecture_12/lab/{703c4e58.svg => 330135ea.svg} (97%) rename dev/lecture_12/lab/{893cdf9c.svg => cb3fec90.svg} (91%) diff --git a/dev/.documenter-siteinfo.json b/dev/.documenter-siteinfo.json index 2471dc8a..df4496ba 100644 --- a/dev/.documenter-siteinfo.json +++ b/dev/.documenter-siteinfo.json @@ -1 +1 @@ -{"documenter":{"julia_version":"1.9.4","generation_timestamp":"2023-12-14T13:31:46","documenter_version":"1.2.1"}} \ No newline at end of file +{"documenter":{"julia_version":"1.9.4","generation_timestamp":"2023-12-15T09:22:49","documenter_version":"1.2.1"}} \ No newline at end of file diff --git a/dev/how_to_submit_hw/index.html b/dev/how_to_submit_hw/index.html index 995e3ef4..7acd77f8 100644 --- a/dev/how_to_submit_hw/index.html +++ b/dev/how_to_submit_hw/index.html @@ -1,2 +1,2 @@ -Homework submission · Scientific Programming in Julia
+Homework submission · Scientific Programming in Julia
diff --git a/dev/index.html b/dev/index.html index f6cfb7b9..4fe47f99 100644 --- a/dev/index.html +++ b/dev/index.html @@ -6,4 +6,4 @@ Learn the power of abstraction. Example: The essence of forward mode automatic differentiation. -

Before joining the course, consider reading the following two blog posts to figure out if Julia is a language in which you want to invest your time.

What will you learn?

First and foremost you will learn how to think julia - meaning how write fast, extensible, reusable, and easy-to-read code using things like optional typing, multiple dispatch, and functional programming concepts. The later part of the course will teach you how to use more advanced concepts like language introspection, metaprogramming, and symbolic computing. Amonst others you will implement your own automatic differetiation (the backbone of modern machine learning) package based on these advanced techniques that can transform intermediate representations of Julia code.

Organization

This course webpage contains all information about the course that you need, including lecture notes, lab instructions, and homeworks. The official format of the course is 2+2 (2h lectures/2h labs per week) for 4 credits.

The official course code is: B0M36SPJ and the timetable for the winter semester 2022 can be found here.

The course will be graded based on points from your homework (max. 20 points) and points from a final project (max. 30 points).

Below is a table that shows which lectures have homeworks (and their points).

Homework12345678910111213
Points22222222-2-2-

Hint: The first few homeworks are easier. Use them to fill up your points.

Final project

The final project will be individually agreed on for each student. Ideally you can use this project to solve a problem you have e.g. in your thesis, but don't worry - if you cannot come up with an own project idea, we will suggest one to you. More info and project suggestion can be found here.

Grading

Your points from the homeworks and the final project are summed and graded by the standard grading scale below.

GradeABCDEF
Points45-5040-4435-3930-3425-290-25

Teachers

–E-mailRoomRole
Tomáš Pevnýpevnak@protonmail.chKN:E-406Lecturer
Vašek Šmídlsmidlva1@fjfi.cvut.czKN:E-333Lecturer
Matěj Zorekzorekmat@fel.cvut.czKN:E-333Lab Instructor
Niklas Heimheimnikl@fel.cvut.czKN:E-333Lab Instructor

Prerequisites

There are no hard requirements to take the course, but if you are not at all familiar with Julia we recommend you to take Julia for Optimization and Learning before enrolling in this course. The Functional Programming course also contains some helpful concepts for this course. And knowledge about computer hardware, namely basics of how CPU works, how it interacts with memory through caches, and basics of multi-threadding certainly helps.

References

+

Before joining the course, consider reading the following two blog posts to figure out if Julia is a language in which you want to invest your time.

What will you learn?

First and foremost you will learn how to think julia - meaning how write fast, extensible, reusable, and easy-to-read code using things like optional typing, multiple dispatch, and functional programming concepts. The later part of the course will teach you how to use more advanced concepts like language introspection, metaprogramming, and symbolic computing. Amonst others you will implement your own automatic differetiation (the backbone of modern machine learning) package based on these advanced techniques that can transform intermediate representations of Julia code.

Organization

This course webpage contains all information about the course that you need, including lecture notes, lab instructions, and homeworks. The official format of the course is 2+2 (2h lectures/2h labs per week) for 4 credits.

The official course code is: B0M36SPJ and the timetable for the winter semester 2022 can be found here.

The course will be graded based on points from your homework (max. 20 points) and points from a final project (max. 30 points).

Below is a table that shows which lectures have homeworks (and their points).

Homework12345678910111213
Points22222222-2-2-

Hint: The first few homeworks are easier. Use them to fill up your points.

Final project

The final project will be individually agreed on for each student. Ideally you can use this project to solve a problem you have e.g. in your thesis, but don't worry - if you cannot come up with an own project idea, we will suggest one to you. More info and project suggestion can be found here.

Grading

Your points from the homeworks and the final project are summed and graded by the standard grading scale below.

GradeABCDEF
Points45-5040-4435-3930-3425-290-25

Teachers

–E-mailRoomRole
Tomáš Pevnýpevnak@protonmail.chKN:E-406Lecturer
Vašek Šmídlsmidlva1@fjfi.cvut.czKN:E-333Lecturer
Matěj Zorekzorekmat@fel.cvut.czKN:E-333Lab Instructor
Niklas Heimheimnikl@fel.cvut.czKN:E-333Lab Instructor

Prerequisites

There are no hard requirements to take the course, but if you are not at all familiar with Julia we recommend you to take Julia for Optimization and Learning before enrolling in this course. The Functional Programming course also contains some helpful concepts for this course. And knowledge about computer hardware, namely basics of how CPU works, how it interacts with memory through caches, and basics of multi-threadding certainly helps.

References

diff --git a/dev/installation/index.html b/dev/installation/index.html index bd18d25f..6b55647f 100644 --- a/dev/installation/index.html +++ b/dev/installation/index.html @@ -14,4 +14,4 @@ _/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release |__/ | -julia>

Julia IDE

There is no one way to install/develop and run Julia, which may be strange users coming from MATLAB, but for users of general purpose languages such as Python, C++ this is quite common. Most of the Julia programmers to date are using

This setup is described in a comprehensive step-by-step guide in our bachelor course Julia for Optimization & Learning.

Note that this setup is not a strict requirement for the lectures/labs and any other text editor with the option to send code to the terminal such as Vim (+Tmux), Emacs, or Sublime Text will suffice.

GitHub registration & Git setup

As one of the goals of the course is writing code that can be distributed to others, we recommend a GitHub account, which you can create here (unless you already have one). In order to interact with GitHub repositories, we will be using git. For installation instruction (Windows only) see the section in the bachelor course.

+julia>

Julia IDE

There is no one way to install/develop and run Julia, which may be strange users coming from MATLAB, but for users of general purpose languages such as Python, C++ this is quite common. Most of the Julia programmers to date are using

This setup is described in a comprehensive step-by-step guide in our bachelor course Julia for Optimization & Learning.

Note that this setup is not a strict requirement for the lectures/labs and any other text editor with the option to send code to the terminal such as Vim (+Tmux), Emacs, or Sublime Text will suffice.

GitHub registration & Git setup

As one of the goals of the course is writing code that can be distributed to others, we recommend a GitHub account, which you can create here (unless you already have one). In order to interact with GitHub repositories, we will be using git. For installation instruction (Windows only) see the section in the bachelor course.

diff --git a/dev/lecture_01/basics/index.html b/dev/lecture_01/basics/index.html index a70c811a..24538df6 100644 --- a/dev/lecture_01/basics/index.html +++ b/dev/lecture_01/basics/index.html @@ -16,4 +16,4 @@ out = fsum(varargin{1},varargin{2:end}) end

The need to build intuition for function composition.

Dispatch is easier to optimize by the compiler.

Operators are functions

operatorfunction name
[A B C ...]hcat
[A; B; C; ...]vcat
[A B; C D; ...]hvcat
A'adjoint
A[i]getindex
A[i] = xsetindex!
A.ngetproperty
A.n = xsetproperty!
struct Foo end
 
-Base.getproperty(a::Foo, x::Symbol) = x == :a ? 5 : error("does not have property $(x)")

Can be redefined and overloaded for different input types. The getproperty method can define access to the memory structure.

Broadcasting revisited

The a.+b syntax is a syntactic sugar for broadcast(+,a,b).

The special meaning of the dot is that they will be fused into a single call:

The same logic works for lists, tuples, etc.

+Base.getproperty(a::Foo, x::Symbol) = x == :a ? 5 : error("does not have property $(x)")

Can be redefined and overloaded for different input types. The getproperty method can define access to the memory structure.

Broadcasting revisited

The a.+b syntax is a syntactic sugar for broadcast(+,a,b).

The special meaning of the dot is that they will be fused into a single call:

The same logic works for lists, tuples, etc.

diff --git a/dev/lecture_01/demo/index.html b/dev/lecture_01/demo/index.html index c9151357..542cb8a6 100644 --- a/dev/lecture_01/demo/index.html +++ b/dev/lecture_01/demo/index.html @@ -24,4 +24,4 @@ prob = ODEProblem(lotka_volterra,u0,tspan,p) sol = solve(prob) -plot(sol,denseplot=false)

Integration with other toolkits

Flux: toolkit for modelling Neural Networks. Neural network is a function.

Turing: Probabilistic modelling toolkit

+plot(sol,denseplot=false)

Integration with other toolkits

Flux: toolkit for modelling Neural Networks. Neural network is a function.

Turing: Probabilistic modelling toolkit

diff --git a/dev/lecture_01/hw/986c9d56.svg b/dev/lecture_01/hw/986c9d56.svg deleted file mode 100644 index 14cd6c9f..00000000 --- a/dev/lecture_01/hw/986c9d56.svg +++ /dev/null @@ -1,60 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/dev/lecture_01/hw/b38604ce.svg b/dev/lecture_01/hw/b38604ce.svg new file mode 100644 index 00000000..3e2c6dc4 --- /dev/null +++ b/dev/lecture_01/hw/b38604ce.svg @@ -0,0 +1,60 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/lecture_01/hw/index.html b/dev/lecture_01/hw/index.html index 533a9b71..4bc56e12 100644 --- a/dev/lecture_01/hw/index.html +++ b/dev/lecture_01/hw/index.html @@ -44,4 +44,4 @@ Series{9}: num_edges_nodes: (10, 10) Series{10}: - num_edges_nodes: (10, 10)Example block output

+ num_edges_nodes: (10, 10)Example block output

diff --git a/dev/lecture_01/lab/index.html b/dev/lecture_01/lab/index.html index 90ef4d67..ae39131e 100644 --- a/dev/lecture_01/lab/index.html +++ b/dev/lecture_01/lab/index.html @@ -260,4 +260,4 @@ :b => [1, 23]
julia> d[:c]ERROR: KeyError: key :c not found

TypeError

Type assertion failure, or calling an intrinsic function (inside LLVM, where code is strictly typed) with incorrect argument type. In practice this error comes up most often when comparing value of a type against the Bool type as seen in the example bellow.

julia> if 1 end                # calls internally typeassert(1, Bool)ERROR: TypeError: non-boolean (Int64) used in boolean context
julia> typeassert(1, Bool)ERROR: TypeError: non-boolean (Int64) used in boolean context

In order to compare inside conditional statements such as if-elseif-else or the ternary operator x ? a : b the condition has to be always of Bool type, thus the example above can be fixed by the comparison operator: if 1 == 1 end (in reality either the left or the right side of the expression contains an expression or a variable to compare against).

UndefVarError

While this error is quite self-explanatory, the exact causes are often quite puzzling for the user. The reason behind the confusion is to do with code scoping, which comes into play for example when trying to access a local variable from outside of a given function or just updating a global variable from within a simple loop.

In the first example we show the former case, where variable is declared from within a function and accessed from outside afterwards.

julia> function plusone(x)
            uno = 1
            return x + uno
-       endplusone (generic function with 1 method)
julia> uno # defined only within plusoneERROR: UndefVarError: `uno` not defined

Unless there is variable I_am_not_defined in the global scope, the following should throw an error.

julia> I_am_not_definedERROR: UndefVarError: `I_am_not_defined` not defined

Often these kind of errors arise as a result of bad code practices, such as long running sessions of Julia having long forgotten global variables, that do not exist upon new execution (this one in particular has been addressed by the authors of the reactive Julia notebooks Pluto.jl).

For more details on code scoping we recommend particular places in the bachelor course lectures here and there.

ErrorException & error function

ErrorException is the most generic error, which can be thrown/raised just by calling the error function with a chosen string message. As a result developers may be inclined to misuse this for any kind of unexpected behavior a user can run into, often providing out-of-context/uninformative messages.

+ endplusone (generic function with 1 method)
julia> uno # defined only within plusoneERROR: UndefVarError: `uno` not defined

Unless there is variable I_am_not_defined in the global scope, the following should throw an error.

julia> I_am_not_definedERROR: UndefVarError: `I_am_not_defined` not defined

Often these kind of errors arise as a result of bad code practices, such as long running sessions of Julia having long forgotten global variables, that do not exist upon new execution (this one in particular has been addressed by the authors of the reactive Julia notebooks Pluto.jl).

For more details on code scoping we recommend particular places in the bachelor course lectures here and there.

ErrorException & error function

ErrorException is the most generic error, which can be thrown/raised just by calling the error function with a chosen string message. As a result developers may be inclined to misuse this for any kind of unexpected behavior a user can run into, often providing out-of-context/uninformative messages.

diff --git a/dev/lecture_01/motivation/index.html b/dev/lecture_01/motivation/index.html index 53e221f1..84944e02 100644 --- a/dev/lecture_01/motivation/index.html +++ b/dev/lecture_01/motivation/index.html @@ -35,4 +35,4 @@ m ~ Normal(0, sqrt(s²)) x ~ Normal(m, sqrt(s²)) y ~ Normal(m, sqrt(s²)) -end

Such tools allow building a very convenient user experience on abstract level, and reaching very efficient code.

Reproducibile research

Think about a code that was written some time ago. To run it, you often need to be able to have the same version of the language it was written for.

Environment

Is an independent set of packages that can be local to an individual project or shared and selected by name.

Package

A package is a source tree with a standard layout providing functionality that can be reused by other Julia projects.

This allows Julia to be a rapidly evolving ecosystem with frequent changes due to:

Package manager

Julia from user's point of view

  1. compilation of everything to as specialized as possible

    • very fast code
    • slow interaction (caching...)
    • generating libraries is harder
      • think of fsum,
      • everything is ".h" (Eigen library)
    • debugging is different to matlab/python

  2. extensibility, Multiple dispatch = multi-functions

    • allows great extensibility and code composition
    • not (yet) mainstream thinking
    • Julia is not Object-oriented
    • Julia is (not pure) functional language
+end

Such tools allow building a very convenient user experience on abstract level, and reaching very efficient code.

Reproducibile research

Think about a code that was written some time ago. To run it, you often need to be able to have the same version of the language it was written for.

Environment

Is an independent set of packages that can be local to an individual project or shared and selected by name.

Package

A package is a source tree with a standard layout providing functionality that can be reused by other Julia projects.

This allows Julia to be a rapidly evolving ecosystem with frequent changes due to:

Package manager

Julia from user's point of view

  1. compilation of everything to as specialized as possible

    • very fast code
    • slow interaction (caching...)
    • generating libraries is harder
      • think of fsum,
      • everything is ".h" (Eigen library)
    • debugging is different to matlab/python

  2. extensibility, Multiple dispatch = multi-functions

    • allows great extensibility and code composition
    • not (yet) mainstream thinking
    • Julia is not Object-oriented
    • Julia is (not pure) functional language
diff --git a/dev/lecture_01/outline/index.html b/dev/lecture_01/outline/index.html index 24255c17..fa63e0e3 100644 --- a/dev/lecture_01/outline/index.html +++ b/dev/lecture_01/outline/index.html @@ -1,2 +1,2 @@ -Outline · Scientific Programming in Julia

Course outline

  1. Introduction

  2. Type system

    • user: tool for abstraction
    • compiler: tool for memory layout

  3. Design patterns (mental setup)

    • Julia is a type-based language
    • multiple-dispatch generalizes OOP and FP

  4. Packages

    • way how to organize code
    • code reuse (alternative to libraries)
    • experiment reproducibility

  5. Benchmarking

    • how to measure code efficiency

  6. Introspection

    • understand how the compiler process the data

  7. Macros

    • automate writing of boring the boilerplate code
    • good macro create cleaner code

  8. Automatic Differentiation

    • Theory: difference between the forward and backward mode
    • Implementation techniques

  9. Intermediate representation

    • how to use internal the representation of the code
    • example in automatic differentiation

  10. Parallel computing

    • threads, processes

  11. Graphics card coding

    • types for GPU
    • specifics of architectures

  12. Ordinary Differential Equations

    • simple solvers
    • error propagation

  13. Data driven ODE

    • combine ODE with optimization
    • automatic differentiation (adjoints)
+Outline · Scientific Programming in Julia

Course outline

  1. Introduction

  2. Type system

    • user: tool for abstraction
    • compiler: tool for memory layout

  3. Design patterns (mental setup)

    • Julia is a type-based language
    • multiple-dispatch generalizes OOP and FP

  4. Packages

    • way how to organize code
    • code reuse (alternative to libraries)
    • experiment reproducibility

  5. Benchmarking

    • how to measure code efficiency

  6. Introspection

    • understand how the compiler process the data

  7. Macros

    • automate writing of boring the boilerplate code
    • good macro create cleaner code

  8. Automatic Differentiation

    • Theory: difference between the forward and backward mode
    • Implementation techniques

  9. Intermediate representation

    • how to use internal the representation of the code
    • example in automatic differentiation

  10. Parallel computing

    • threads, processes

  11. Graphics card coding

    • types for GPU
    • specifics of architectures

  12. Ordinary Differential Equations

    • simple solvers
    • error propagation

  13. Data driven ODE

    • combine ODE with optimization
    • automatic differentiation (adjoints)
diff --git a/dev/lecture_02/hw/index.html b/dev/lecture_02/hw/index.html index df0df361..f86a25d2 100644 --- a/dev/lecture_02/hw/index.html +++ b/dev/lecture_02/hw/index.html @@ -4,4 +4,4 @@

  1. Implement a function agent_count that can be called on a single Agent and returns a number between $(0,1)$ (i.e. always 1 for animals; and size(plant)/max_size(plant) for plants).

  2. Add a method for a vector of agents Vector{<:Agent} which sums all agent counts.

  3. Add a method for a World which returns a dictionary that contains pairs of Symbols and the agent count like below:

julia> grass1 = Grass(1,5,5);
julia> agent_count(grass1)1.0
julia> grass2 = Grass(2,1,5);
julia> agent_count([grass1,grass2]) # one grass is fully grown; the other only 20% => 1.21.2
julia> sheep = Sheep(3,10.0,5.0,1.0,1.0);
julia> wolf  = Wolf(4,20.0,10.0,1.0,1.0);
julia> world = World([grass1, grass2, sheep, wolf]);
julia> agent_count(world)Dict{Symbol, Real} with 3 entries:
   :Wolf  => 1
   :Grass => 1.2
-  :Sheep => 1

Hint: You can get the name of a type by using the nameof function:

julia> nameof(Grass):Grass

Use as much dispatch as you can! ;)

+ :Sheep => 1

Hint: You can get the name of a type by using the nameof function:

julia> nameof(Grass):Grass

Use as much dispatch as you can! ;)

diff --git a/dev/lecture_02/lab/index.html b/dev/lecture_02/lab/index.html index e6a1b8f3..042cd80c 100644 --- a/dev/lecture_02/lab/index.html +++ b/dev/lecture_02/lab/index.html @@ -30,7 +30,7 @@ # hint: to type the leaf in the julia REPL you can do: # \:herb:<tab> print(io,"🌿 #$(g.id) $(round(Int,x))% grown") -end

Creating a few Grass agents can then look like this:

julia> Grass(1,5)🌿 #1 40% grown
julia> g = Grass(2)🌿 #2 50% grown
julia> g.id = 5ERROR: setfield!: const field .id of type Grass cannot be changed

Sheep and Wolf Agents

Animals are slightly different from plants. They will have an energy $E$, which will be increase (or decrease) if the agent eats (or reproduces) by a certain amount $\Delta E$. Later we will also need a probability to find food $p_f$ and a probability to reproduce $p_r$.c

+end

Creating a few Grass agents can then look like this:

julia> Grass(1,5)🌿 #1 40% grown
julia> g = Grass(2)🌿 #2 100% grown
julia> g.id = 5ERROR: setfield!: const field .id of type Grass cannot be changed

Sheep and Wolf Agents

Animals are slightly different from plants. They will have an energy $E$, which will be increase (or decrease) if the agent eats (or reproduces) by a certain amount $\Delta E$. Later we will also need a probability to find food $p_f$ and a probability to reproduce $p_r$.c

Exercise:
  1. Define two mutable structs Sheep and Wolf that are subtypes of Animal and have the fields id, energy, Δenergy, reprprob, and foodprob.
  2. Define constructors with the following default values:
    • For 🐑: $E=4$, $\Delta E=0.2$, $p_r=0.8$, and $p_f=0.6$.
    • For 🐺: $E=10$, $\Delta E=8$, $p_r=0.1$, and $p_f=0.2$.
  3. Overload Base.show to get pretty printing for your two new animals.
@@ -91,11 +91,11 @@ Solution:

function eat!(sheep::Sheep, grass::Grass, w::World)
     sheep.energy += grass.size * sheep.Δenergy
     grass.size = 0
-end

Below you can see how a fully grown grass is eaten by a sheep. The sheep's energy changes size of the grass is set to zero.

julia> grass = Grass(1)🌿 #1 50% grown
julia> sheep = Sheep(2)🐑 #2 E=4.0 ΔE=0.2 pr=0.8 pf=0.6
julia> world = World([grass, sheep])Main.World{Main.Agent}
+end

Below you can see how a fully grown grass is eaten by a sheep. The sheep's energy changes size of the grass is set to zero.

julia> grass = Grass(1)🌿 #1 20% grown
julia> sheep = Sheep(2)🐑 #2 E=4.0 ΔE=0.2 pr=0.8 pf=0.6
julia> world = World([grass, sheep])Main.World{Main.Agent}
   🐑 #2 E=4.0 ΔE=0.2 pr=0.8 pf=0.6
-  🌿 #1 50% grown
julia> eat!(sheep,grass,world);ERROR: setfield!: const field .energy of type Sheep cannot be changed
julia> worldMain.World{Main.Agent}
+  🌿 #1 20% grown
julia> eat!(sheep,grass,world);ERROR: setfield!: const field .energy of type Sheep cannot be changed
julia> worldMain.World{Main.Agent}
   🐑 #2 E=4.0 ΔE=0.2 pr=0.8 pf=0.6
-  🌿 #1 50% grown

Note that the order of the arguments has a meaning here. Calling eat!(grass,sheep,world) results in a MethodError which is great, because Grass cannot eat Sheep.

julia> eat!(grass,sheep,world);ERROR: MethodError: no method matching eat!(::Main.Grass, ::Main.Sheep, ::Main.World{Main.Agent})
+  🌿 #1 20% grown

Note that the order of the arguments has a meaning here. Calling eat!(grass,sheep,world) results in a MethodError which is great, because Grass cannot eat Sheep.

julia> eat!(grass,sheep,world);ERROR: MethodError: no method matching eat!(::Main.Grass, ::Main.Sheep, ::Main.World{Main.Agent})
 
 Closest candidates are:
   eat!(::Main.Sheep, ::Main.Grass, ::Main.World)
@@ -111,9 +111,9 @@
 kill_agent!(a::Agent, w::World) = delete!(w.agents, a.id)

With a correct eat! method you should get results like this:

julia> grass = Grass(1);
julia> sheep = Sheep(2);
julia> wolf  = Wolf(3);
julia> world = World([grass, sheep, wolf])Main.World{Main.Agent}
   🐑 #2 E=4.0 ΔE=0.2 pr=0.8 pf=0.6
   🐺 #3 E=10.0 ΔE=8.0 pr=0.1 pf=0.2
-  🌿 #1 40% grown
julia> eat!(wolf,sheep,world);
julia> worldMain.World{Main.Agent}
+  🌿 #1 70% grown
julia> eat!(wolf,sheep,world);
julia> worldMain.World{Main.Agent}
   🐺 #3 E=42.0 ΔE=8.0 pr=0.1 pf=0.2
-  🌿 #1 40% grown

The sheep is removed from the world and the wolf's energy increased by $\Delta E$.

Reproduction

Currently our animals can only eat. In our simulation we also want them to reproduce. We will do this by adding a reproduce! method to Animal.

+ 🌿 #1 70% grown

The sheep is removed from the world and the wolf's energy increased by $\Delta E$.

Reproduction

Currently our animals can only eat. In our simulation we also want them to reproduce. We will do this by adding a reproduce! method to Animal.

Exercise

Write a function reproduce! that takes an Animal and a World. Reproducing will cost an animal half of its energy and then add an almost identical copy of the given animal to the world. The only thing that is different from parent to child is the ID. You can simply increase the max_id of the world by one and use that as the new ID for the child.

@@ -135,4 +135,4 @@ 🐑 #2 E=4.0 ΔE=0.2 pr=0.8 pf=0.6 🐑 #1 E=4.0 ΔE=0.2 pr=0.8 pf=0.6
julia> reproduce!(s1, w);ERROR: setfield!: const field .energy of type Sheep cannot be changed
julia> wMain.World{Main.Sheep} 🐑 #2 E=4.0 ΔE=0.2 pr=0.8 pf=0.6 - 🐑 #1 E=4.0 ΔE=0.2 pr=0.8 pf=0.6
+ 🐑 #1 E=4.0 ΔE=0.2 pr=0.8 pf=0.6 diff --git a/dev/lecture_02/lecture/index.html b/dev/lecture_02/lecture/index.html index 77fef909..65cc59af 100644 --- a/dev/lecture_02/lecture/index.html +++ b/dev/lecture_02/lecture/index.html @@ -171,4 +171,4 @@

typeof(Position(1.0,2.0)) typeof(Position(1,2)) Position(1,2) isa Position{Float64} Position(1,2) isa Position{Real} Position(1,2) isa Position{<:Real} typeof(Position(1,2)) <: Position{<:Float64} typeof(Position(1,2)) <: Position{<:Real}


 ### A bizzare definition which you can encounter
 The following definition of a one-hot matrix is taken from [Flux.jl](https://github.com/FluxML/Flux.jl/blob/1a0b51938b9a3d679c6950eece214cd18108395f/src/onehot.jl#L10-L12)
-

julia struct OneHotArray{T<:Integer, L, N, var"N+1", I<:Union{T,AbstractArray{T, N}}} <: AbstractArray{Bool, var"N+1"} indices::I end ```

The parameters of the type carry information about the type used to encode the position of one in each column in T, the dimension of one-hot vectors in L, the dimension of the storage of indices in N (which is zero for OneHotVector and one for OneHotMatrix), number of dimensions of the OneHotArray in var"N+1" and the type of underlying storage of indicies I.

+

julia struct OneHotArray{T<:Integer, L, N, var"N+1", I<:Union{T,AbstractArray{T, N}}} <: AbstractArray{Bool, var"N+1"} indices::I end ```

The parameters of the type carry information about the type used to encode the position of one in each column in T, the dimension of one-hot vectors in L, the dimension of the storage of indices in N (which is zero for OneHotVector and one for OneHotMatrix), number of dimensions of the OneHotArray in var"N+1" and the type of underlying storage of indicies I.

diff --git a/dev/lecture_03/hw/index.html b/dev/lecture_03/hw/index.html index 623b2ee1..0246b0f6 100644 --- a/dev/lecture_03/hw/index.html +++ b/dev/lecture_03/hw/index.html @@ -2,11 +2,11 @@ Homework · Scientific Programming in Julia

Homework 3

In this homework we will implement a function find_food and practice the use of closures. The solution of lab 3 can be found here. You can use this file and add the code that you write for the homework to it.

How to submit?

Put all your code (including your or the provided solution of lab 2) in a script named hw.jl. Zip only this file (not its parent folder) and upload it to BRUTE.

Agents looking for food

Homework:

Implement a method find_food(a::Animal, w::World) returns one randomly chosen agent from all w.agents that can be eaten by a or nothing if no food could be found. This means that if e.g. the animal is a Wolf you have to return one random Sheep, etc.

Hint: You can write a general find_food method for all animals and move the parts that are specific to the concrete animal types to a separate function. E.g. you could define a function eats(::Animal{Wolf}, ::Animal{Sheep}) = true, etc.

You can check your solution with the public test:

julia> sheep = Sheep(1,pf=1.0)🐑♀ #1 E=4.0 ΔE=0.2 pr=0.8 pf=1.0
julia> world = World([Grass(2), sheep])Main.World{Main.Agent}
-  🌿  #2 30% grown
+  🌿  #2 60% grown
   🐑♀ #1 E=4.0 ΔE=0.2 pr=0.8 pf=1.0
julia> find_food(sheep, world) isa Plant{Grass}true

Callbacks & Closures

Homework:

Implement a function every_nth(f::Function,n::Int) that takes an inner function f and uses a closure to construct an outer function g that only calls f every nth call to g. For example, if n=3 the inner function f be called at the 3rd, 6th, 9th ... call to g (not at the 1st, 2nd, 4th, 5th, 7th... call).

Hint: You can use splatting via ... to pass on an unknown number of arguments from the outer to the inner function.

You can use every_nth to log (or save) the agent count only every couple of steps of your simulation. Using every_nth will look like this:

julia> w = World([Sheep(1), Grass(2), Wolf(3)])Main.World{Main.Agent}
-  🌿  #2 40% grown
+  🌿  #2 100% grown
   🐺♂ #3 E=10.0 ΔE=8.0 pr=0.1 pf=0.2
-  🐑♂ #1 E=4.0 ΔE=0.2 pr=0.8 pf=0.6
julia> # `@info agent_count(w)` is executed only every 3rd call to logcb(w)
-       logcb = every_nth(w->(@info agent_count(w)), 3);
julia> logcb(w);  # x->(@info agent_count(w)) is not called
julia> logcb(w);  # x->(@info agent_count(w)) is not called
julia> logcb(w);  # x->(@info agent_count(w)) *is* called[ Info: Dict(:Wolf => 1.0, :Grass => 0.4, :Sheep => 1.0)
+ 🐑♀ #1 E=4.0 ΔE=0.2 pr=0.8 pf=0.6
julia> # `@info agent_count(w)` is executed only every 3rd call to logcb(w) + logcb = every_nth(w->(@info agent_count(w)), 3);
julia> logcb(w); # x->(@info agent_count(w)) is not called
julia> logcb(w); # x->(@info agent_count(w)) is not called
julia> logcb(w); # x->(@info agent_count(w)) *is* called[ Info: Dict(:Wolf => 1.0, :Grass => 1.0, :Sheep => 1.0) diff --git a/dev/lecture_03/lab/index.html b/dev/lecture_03/lab/index.html index 6c172b8f..1671b028 100644 --- a/dev/lecture_03/lab/index.html +++ b/dev/lecture_03/lab/index.html @@ -23,9 +23,9 @@ else setfield!(s,name,x) end -end

You should be able to do the following with your overloads now

julia> sheep = ⚥Sheep(1)Main.⚥Sheep(🐑 #1 E=4.0 ΔE=0.2 pr=0.8 pf=0.6, :male)
julia> sheep.id1
julia> sheep.sex:male
julia> sheep.energy += 15.0
julia> sheepMain.⚥Sheep(🐑 #1 E=5.0 ΔE=0.2 pr=0.8 pf=0.6, :male)

In order to make the ⚥Sheep work with the rest of the code we only have to forward the eat! method

julia> eat!(s::⚥Sheep, food, world) = eat!(s.sheep, food, world);
julia> sheep = ⚥Sheep(1);
julia> grass = Grass(2);
julia> world = World([sheep,grass])Main.World{Main.Agent}
-  🌿 #2 70% grown
-  Main.⚥Sheep(🐑 #1 E=4.0 ΔE=0.2 pr=0.8 pf=0.6, :male)
julia> eat!(sheep, grass, world)0

and implement a custom reproduce! method with the behaviour that we want.

However, the extension of Sheep to ⚥Sheep is a very object-oriented approach. With a little bit of rethinking, we can build a much more elegant solution that makes use of Julia's powerful parametric types.

Part II: A new, parametric type hierarchy

First, let us note that there are two fundamentally different types of agents in our world: animals and plants. All species such as grass, sheep, wolves, etc. can be categorized as one of those two. We can use Julia's powerful, parametric type system to define one large abstract type for all agents Agent{S}. The Agent will either be an Animal or a Plant with a type parameter S which will represent the specific animal/plant species we are dealing with.

This new type hiearchy can then look like this:

abstract type Species end
+end

You should be able to do the following with your overloads now

julia> sheep = ⚥Sheep(1)Main.⚥Sheep(🐑 #1 E=4.0 ΔE=0.2 pr=0.8 pf=0.6, :female)
julia> sheep.id1
julia> sheep.sex:female
julia> sheep.energy += 15.0
julia> sheepMain.⚥Sheep(🐑 #1 E=5.0 ΔE=0.2 pr=0.8 pf=0.6, :female)

In order to make the ⚥Sheep work with the rest of the code we only have to forward the eat! method

julia> eat!(s::⚥Sheep, food, world) = eat!(s.sheep, food, world);
julia> sheep = ⚥Sheep(1);
julia> grass = Grass(2);
julia> world = World([sheep,grass])Main.World{Main.Agent}
+  🌿 #2 10% grown
+  Main.⚥Sheep(🐑 #1 E=4.0 ΔE=0.2 pr=0.8 pf=0.6, :female)
julia> eat!(sheep, grass, world)0

and implement a custom reproduce! method with the behaviour that we want.

However, the extension of Sheep to ⚥Sheep is a very object-oriented approach. With a little bit of rethinking, we can build a much more elegant solution that makes use of Julia's powerful parametric types.

Part II: A new, parametric type hierarchy

First, let us note that there are two fundamentally different types of agents in our world: animals and plants. All species such as grass, sheep, wolves, etc. can be categorized as one of those two. We can use Julia's powerful, parametric type system to define one large abstract type for all agents Agent{S}. The Agent will either be an Animal or a Plant with a type parameter S which will represent the specific animal/plant species we are dealing with.

This new type hiearchy can then look like this:

abstract type Species end
 
 abstract type PlantSpecies <: Species end
 abstract type Grass <: PlantSpecies end
@@ -74,7 +74,7 @@
 # get the per species defaults back
 randsex() = rand(instances(Sex))
 Sheep(id; E=4.0, ΔE=0.2, pr=0.8, pf=0.6, s=randsex()) = Sheep(id, E, ΔE, pr, pf, s)
-Wolf(id; E=10.0, ΔE=8.0, pr=0.1, pf=0.2, s=randsex()) = Wolf(id, E, ΔE, pr, pf, s)

We have our convenient, high-level behaviour back!

julia> Sheep(1)🐑♀ #1 E=4.0 ΔE=0.2 pr=0.8 pf=0.6
julia> Wolf(2)🐺♀ #2 E=10.0 ΔE=8.0 pr=0.1 pf=0.2
+Wolf(id; E=10.0, ΔE=8.0, pr=0.1, pf=0.2, s=randsex()) = Wolf(id, E, ΔE, pr, pf, s)

We have our convenient, high-level behaviour back!

julia> Sheep(1)🐑♀ #1 E=4.0 ΔE=0.2 pr=0.8 pf=0.6
julia> Wolf(2)🐺♂ #2 E=10.0 ΔE=8.0 pr=0.1 pf=0.2
Exercise:

Check the methods for eat! and kill_agent! which involve Animals and update their type signatures such that they work for the new type hiearchy.

@@ -147,8 +147,8 @@ sheep.energy += grass.size * sheep.Δenergy grass.size = 0 end -eats(::Animal{Sheep},g::Plant{Grass}) = g.size > 0

julia> g = Grass(2)🌿  #2 60% grown
julia> s = Sheep(3)🐑♀ #3 E=4.0 ΔE=0.2 pr=0.8 pf=0.6
julia> w = World([g,s])Main.World{Main.Agent}
-  🌿  #2 60% grown
-  🐑♀ #3 E=4.0 ΔE=0.2 pr=0.8 pf=0.6
julia> eat!(s,g,w); wMain.World{Main.Agent}
+eats(::Animal{Sheep},g::Plant{Grass}) = g.size > 0

julia> g = Grass(2)🌿  #2 30% grown
julia> s = Sheep(3)🐑♂ #3 E=4.0 ΔE=0.2 pr=0.8 pf=0.6
julia> w = World([g,s])Main.World{Main.Agent}
+  🌿  #2 30% grown
+  🐑♂ #3 E=4.0 ΔE=0.2 pr=0.8 pf=0.6
julia> eat!(s,g,w); wMain.World{Main.Agent}
   🌿  #2 0% grown
-  🐑♀ #3 E=5.2 ΔE=0.2 pr=0.8 pf=0.6
+ 🐑♂ #3 E=4.6 ΔE=0.2 pr=0.8 pf=0.6 diff --git a/dev/lecture_03/lecture/index.html b/dev/lecture_03/lecture/index.html index 2c865b07..37f7ae78 100644 --- a/dev/lecture_03/lecture/index.html +++ b/dev/lecture_03/lecture/index.html @@ -121,4 +121,4 @@ end end

Is this confusing? What can cb() do and what it can not?

Note that function train! does not have many local variables. The important ones are arguments, i.e. exist in the scope from which the function was invoked.

loss(x,y)=mse(model(x),y)
 cb() = @info "training" loss(x,y)
-train!(loss, ps, data, opt; cb=cb)

Usage

Usage of closures:

Beware: Performance of captured variables

Inference of types may be difficult in closures: https://github.com/JuliaLang/julia/issues/15276

Aditional materials

+train!(loss, ps, data, opt; cb=cb)

Usage

Usage of closures:

Beware: Performance of captured variables

Inference of types may be difficult in closures: https://github.com/JuliaLang/julia/issues/15276

Aditional materials

diff --git a/dev/lecture_04/hw/index.html b/dev/lecture_04/hw/index.html index 9e352848..1bec5ed9 100644 --- a/dev/lecture_04/hw/index.html +++ b/dev/lecture_04/hw/index.html @@ -13,4 +13,4 @@
Homework:
  1. Create a Sheep with food probability $p_f=1$
  2. Create fully grown Grass and a World with the two agents.
  3. Execute eat!(::Animal{Sheep}, ::Plant{Grass}, ::World)
  4. @test that the size of the Grass now has size == 0

Test Wolf

Homework:
-
  1. Create a Wolf with food probability $p_f=1$
  2. Create a Sheep and a World with the two agents.
  3. Execute eat!(::Animal{Wolf}, ::Animal{Sheep}, ::World)
  4. @test that the World only has one agent left in the agents dictionary
+
  1. Create a Wolf with food probability $p_f=1$
  2. Create a Sheep and a World with the two agents.
  3. Execute eat!(::Animal{Wolf}, ::Animal{Sheep}, ::World)
  4. @test that the World only has one agent left in the agents dictionary
diff --git a/dev/lecture_04/lab/f48b3c02.svg b/dev/lecture_04/lab/2c8fa7d3.svg similarity index 52% rename from dev/lecture_04/lab/f48b3c02.svg rename to dev/lecture_04/lab/2c8fa7d3.svg index c3db1a69..018c42ad 100644 --- a/dev/lecture_04/lab/f48b3c02.svg +++ b/dev/lecture_04/lab/2c8fa7d3.svg @@ -1,48 +1,48 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/lecture_04/lab/index.html b/dev/lecture_04/lab/index.html index 9a6c6535..a72c36f7 100644 --- a/dev/lecture_04/lab/index.html +++ b/dev/lecture_04/lab/index.html @@ -56,17 +56,12 @@ end end
world_step! (generic function with 1 method)

julia> w = World([Sheep(1), Sheep(2), Wolf(3)])Main.World{Main.Animal}
   🐑♀ #2 E=4.0 ΔE=0.2 pr=0.8 pf=0.6
-  🐺♂ #3 E=10.0 ΔE=8.0 pr=0.1 pf=0.2
+  🐺♀ #3 E=10.0 ΔE=8.0 pr=0.1 pf=0.2
   🐑♀ #1 E=4.0 ΔE=0.2 pr=0.8 pf=0.6
julia> world_step!(w); wMain.World{Main.Animal}
   🐑♀ #2 E=3.0 ΔE=0.2 pr=0.8 pf=0.6
-  🐺♂ #3 E=9.0 ΔE=8.0 pr=0.1 pf=0.2
-  🐑♀ #1 E=3.0 ΔE=0.2 pr=0.8 pf=0.6
julia> world_step!(w); wMain.World{Main.Animal}
-  🐑♀ #2 E=2.0 ΔE=0.2 pr=0.8 pf=0.6
-  🐺♂ #3 E=8.0 ΔE=8.0 pr=0.1 pf=0.2
-  🐑♀ #1 E=2.0 ΔE=0.2 pr=0.8 pf=0.6
julia> world_step!(w); wMain.World{Main.Animal}
-  🐑♀ #2 E=1.0 ΔE=0.2 pr=0.8 pf=0.6
-  🐺♂ #3 E=7.0 ΔE=8.0 pr=0.1 pf=0.2
-  🐑♀ #1 E=1.0 ΔE=0.2 pr=0.8 pf=0.6

Finally, lets run a few simulation steps and plot the solution

n_grass  = 1_000
+  🐺♀ #3 E=41.0 ΔE=8.0 pr=0.1 pf=0.2
julia> world_step!(w); wMain.World{Main.Animal}
+  🐺♀ #3 E=56.0 ΔE=8.0 pr=0.1 pf=0.2
julia> world_step!(w); wMain.World{Main.Animal}
+  🐺♀ #3 E=55.0 ΔE=8.0 pr=0.1 pf=0.2

Finally, lets run a few simulation steps and plot the solution

n_grass  = 1_000
 n_sheep  = 40
 n_wolves = 4
 
@@ -88,7 +83,7 @@
 for (n,c) in counts
     plot!(plt, c, label=string(n), lw=2)
 end
-plt
Example block output

Package: Ecosystem.jl

In the main section of this lab you will create your own Ecosystem.jl package to organize and test (!) the code that we have written so far.

PkgTemplates.jl

+pltExample block output

Package: Ecosystem.jl

In the main section of this lab you will create your own Ecosystem.jl package to organize and test (!) the code that we have written so far.

PkgTemplates.jl

Exercise:

The simplest way to create a new package in Julia is to use PkgTemplates.jl. ]add PkgTemplates to your global julia env and create a new package by running:

using PkgTemplates
 Template(interactive=true)("Ecosystem")

to interactively specify various options for your new package or use the following snippet to generate it programmatically:

using PkgTemplates
@@ -192,6 +187,6 @@
            @test repr(w) == "🐺♀ #3 E=1.0 ΔE=1.0 pr=1.0 pf=1.0"
        endTest Summary: | Pass  Total  Time
 Base.show     |    3      3  0.3s
-Test.DefaultTestSet("Base.show", Any[], 3, false, false, true, 1.702560641387629e9, 1.702560641658057e9, false)

Github CI

+Test.DefaultTestSet("Base.show", Any[], 3, false, false, true, 1.702632109363292e9, 1.702632109620566e9, false)

Github CI

Exercise:
-

If you want you can upload you package to Github and add the julia-runtest Github Action to automatically test your code for every new push you make to the repository.

+

If you want you can upload you package to Github and add the julia-runtest Github Action to automatically test your code for every new push you make to the repository.

diff --git a/dev/lecture_04/lecture/index.html b/dev/lecture_04/lecture/index.html index 8bae9245..32e558bf 100644 --- a/dev/lecture_04/lecture/index.html +++ b/dev/lecture_04/lecture/index.html @@ -103,4 +103,4 @@ precompile(fsum,(Float64,Float64,Float64)) end

Can be investigated using MethodAnalysis.

using MethodAnalysis
-mi =methodinstances(fsum)

Useful packages:

  • PackageCompiler.jl has three main purposes:

    • Creating custom sysimages for reduced latency when working locally with packages that has a high startup time.
    • Creating "apps" which are a bundle of files including an executable that can be sent and run on other machines without Julia being installed on that machine.
    • Creating a relocatable C library bundle form of Julia code.
  • AutoSysimages.jl allows easy generation of precompiles images - reduces package loading

Additional material

+mi =methodinstances(fsum)

Useful packages:

  • PackageCompiler.jl has three main purposes:

    • Creating custom sysimages for reduced latency when working locally with packages that has a high startup time.
    • Creating "apps" which are a bundle of files including an executable that can be sent and run on other machines without Julia being installed on that machine.
    • Creating a relocatable C library bundle form of Julia code.
  • AutoSysimages.jl allows easy generation of precompiles images - reduces package loading

Additional material

diff --git a/dev/lecture_05/hw/index.html b/dev/lecture_05/hw/index.html index 6b49630d..366e1cb3 100644 --- a/dev/lecture_05/hw/index.html +++ b/dev/lecture_05/hw/index.html @@ -11,4 +11,4 @@

Voluntary exercise

Voluntary exercise
-

Use Plots.jl to plot the polynomial $p$ on the interval $[-5, 5]$ and visualize the progress/convergence of each method, with a dotted vertical line and a dot on the x-axis for each subsequent root approximation x̃.

HINTS:

  • plotting scalar function f - plot(r, f), where r is a range of x values at which we evaluate f
  • updating an existing plot - either plot!(plt, ...) or plot!(...), in the former case the plot lives in variable plt whereas in the latter we modify some implicit global variable
  • plotting dots - for example with scatter/scatter!
  • plot([(1.0,2.0), (1.0,3.0)], ls=:dot) will create a dotted line from position (x=1.0,y=2.0) to (x=1.0,y=3.0)
+

Use Plots.jl to plot the polynomial $p$ on the interval $[-5, 5]$ and visualize the progress/convergence of each method, with a dotted vertical line and a dot on the x-axis for each subsequent root approximation x̃.

HINTS:

  • plotting scalar function f - plot(r, f), where r is a range of x values at which we evaluate f
  • updating an existing plot - either plot!(plt, ...) or plot!(...), in the former case the plot lives in variable plt whereas in the latter we modify some implicit global variable
  • plotting dots - for example with scatter/scatter!
  • plot([(1.0,2.0), (1.0,3.0)], ls=:dot) will create a dotted line from position (x=1.0,y=2.0) to (x=1.0,y=3.0)
diff --git a/dev/lecture_05/lab/index.html b/dev/lecture_05/lab/index.html index 73cd2d4b..dacc69f1 100644 --- a/dev/lecture_05/lab/index.html +++ b/dev/lecture_05/lab/index.html @@ -146,8 +146,8 @@ │ %19 = Base.not_int(%18)::Bool └── goto #4 if not %19 3 ─ goto #2 -4 ┄ return accumulator

julia> @time polynomial(a, xf)  0.000008 seconds (1 allocation: 16 bytes)
-128.0
julia> @time polynomial_stable(a, xf)  0.000005 seconds (1 allocation: 16 bytes)
+4 ┄       return accumulator

julia> @time polynomial(a, xf)  0.000002 seconds (1 allocation: 16 bytes)
+128.0
julia> @time polynomial_stable(a, xf)  0.000002 seconds (1 allocation: 16 bytes)
 128.0

Only really visible when evaluating multiple times.

julia> using BenchmarkTools
 
 julia> @btime polynomial($a, $xf)
@@ -417,12 +417,12 @@
 end
 world = create_world();
w = Wolf(4000)
 find_food(w, world)
-@code_warntype find_food(w, world)
MethodInstance for Main.find_food(::Main.Animal{🐺, ♀}, ::Main.World{NamedTuple{(:Grass, :SheepFemale, :SheepMale, :WolfFemale, :WolfMale), Tuple{Dict{Int64, Main.Plant{🌿}}, Dict{Int64, Main.Animal{🐑, ♀}}, Dict{Int64, Main.Animal{🐑, ♂}}, Dict{Int64, Main.Animal{🐺, ♀}}, Dict{Int64, Main.Animal{🐺, ♂}}}}})
+@code_warntype find_food(w, world)
MethodInstance for Main.find_food(::Main.Animal{🐺, ♂}, ::Main.World{NamedTuple{(:Grass, :SheepMale, :SheepFemale, :WolfFemale, :WolfMale), Tuple{Dict{Int64, Main.Plant{🌿}}, Dict{Int64, Main.Animal{🐑, ♂}}, Dict{Int64, Main.Animal{🐑, ♀}}, Dict{Int64, Main.Animal{🐺, ♀}}, Dict{Int64, Main.Animal{🐺, ♂}}}}})
   from find_food(::Main.Animal{🐺}, w::Main.World) @ Main ~/work/Scientific-Programming-in-Julia/Scientific-Programming-in-Julia/docs/build/lecture_05/ecosystems/animal_ST_world_NamedTupleDict/Ecosystem.jl:158
 Arguments
   #self#::Core.Const(Main.find_food)
-  _::Main.Animal{🐺, ♀}
-  w::Main.World{NamedTuple{(:Grass, :SheepFemale, :SheepMale, :WolfFemale, :WolfMale), Tuple{Dict{Int64, Main.Plant{🌿}}, Dict{Int64, Main.Animal{🐑, ♀}}, Dict{Int64, Main.Animal{🐑, ♂}}, Dict{Int64, Main.Animal{🐺, ♀}}, Dict{Int64, Main.Animal{🐺, ♂}}}}}
+  _::Main.Animal{🐺, ♂}
+  w::Main.World{NamedTuple{(:Grass, :SheepMale, :SheepFemale, :WolfFemale, :WolfMale), Tuple{Dict{Int64, Main.Plant{🌿}}, Dict{Int64, Main.Animal{🐑, ♂}}, Dict{Int64, Main.Animal{🐑, ♀}}, Dict{Int64, Main.Animal{🐺, ♀}}, Dict{Int64, Main.Animal{🐺, ♂}}}}}
 Body::Union{Nothing, Main.Animal{🐑, ♀}, Main.Animal{🐑, ♂}}
 1 ─ %1 = Main.find_agent(Main.Sheep, w)::Union{Nothing, Main.Animal{🐑, ♀}, Main.Animal{🐑, ♂}}
-└──      return %1

Useful resources

+└── return %1

Useful resources

diff --git a/dev/lecture_05/lecture/index.html b/dev/lecture_05/lecture/index.html index c68439cc..1d52eb71 100644 --- a/dev/lecture_05/lecture/index.html +++ b/dev/lecture_05/lecture/index.html @@ -561,4 +561,4 @@ end x end -@btime find_min!($f, $x₀, $params_tuple; verbose=true)
+@btime find_min!($f, $x₀, $params_tuple; verbose=true)
diff --git a/dev/lecture_06/hw/index.html b/dev/lecture_06/hw/index.html index acea923c..f866ef5f 100644 --- a/dev/lecture_06/hw/index.html +++ b/dev/lecture_06/hw/index.html @@ -7,4 +7,4 @@
Voluntary exercise

Create a function that replaces each of +, -, * and / with the respective checked operation, which checks for overflow. E.g. + should be replaced by Base.checked_add.

+Solution:

Not yet published.

diff --git a/dev/lecture_06/lab/2693a6cc.svg b/dev/lecture_06/lab/28287c05.svg similarity index 81% rename from dev/lecture_06/lab/2693a6cc.svg rename to dev/lecture_06/lab/28287c05.svg index 054f4978..79603d59 100644 --- a/dev/lecture_06/lab/2693a6cc.svg +++ b/dev/lecture_06/lab/28287c05.svg @@ -1,59 +1,59 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/lecture_06/lab/index.html b/dev/lecture_06/lab/index.html index e73906b4..023e56d0 100644 --- a/dev/lecture_06/lab/index.html +++ b/dev/lecture_06/lab/index.html @@ -51,13 +51,13 @@ 16 ─ goto #17 17 ─ return %3 ) => Vector{Int64}
  • LLVM code generation
julia> @code_llvm f();  @ REPL[1]:1 within `f`
-define nonnull {}* @julia_f_6252() #0 {
+define nonnull {}* @julia_f_6273() #0 {
 top:
 ; ┌ @ array.jl:787 within `collect`
 ; │┌ @ array.jl:671 within `_array_for`
 ; ││┌ @ abstractarray.jl:883 within `similar` @ abstractarray.jl:884
 ; │││┌ @ boot.jl:486 within `Array` @ boot.jl:477
-      %0 = call nonnull {}* inttoptr (i64 140113439253952 to {}* ({}*, i64)*)({}* inttoptr (i64 140113098683488 to {}*), i64 10)
+      %0 = call nonnull {}* inttoptr (i64 140189603620288 to {}* ({}*, i64)*)({}* inttoptr (i64 140189263049824 to {}*), i64 10)
 ; │└└└
 ; │ @ array.jl:792 within `collect`
 ; │┌ @ array.jl:817 within `collect_to_with_first!`
@@ -94,7 +94,7 @@
 }
  • native code generation
julia> @code_native f()	.text
 	.file	"f"
 	.section	.rodata.cst32,"aM",@progbits,32
-	.p2align	5                               # -- Begin function julia_f_6289
+	.p2align	5                               # -- Begin function julia_f_6310
 .LCPI0_0:
 	.quad	1                               # 0x1
 	.quad	2                               # 0x2
@@ -111,10 +111,10 @@
 	.quad	9                               # 0x9
 	.quad	10                              # 0xa
 	.text
-	.globl	julia_f_6289
+	.globl	julia_f_6310
 	.p2align	4, 0x90
-	.type	julia_f_6289,@function
-julia_f_6289:                           # @julia_f_6289
+	.type	julia_f_6310,@function
+julia_f_6310:                           # @julia_f_6310
 ; ┌ @ REPL[1]:1 within `f`
 	.cfi_startproc
 # %bb.0:                                # %top
@@ -123,8 +123,8 @@
 	.cfi_offset %rbp, -16
 	movq	%rsp, %rbp
 	.cfi_def_cfa_register %rbp
-	movabsq	$140113098683488, %rdi          # imm = 0x7F6E9F79B460
-	movabsq	$140113439253952, %rax          # imm = 0x7F6EB3C665C0
+	movabsq	$140189263049824, %rdi          # imm = 0x7F805B39B460
+	movabsq	$140189603620288, %rax          # imm = 0x7F806F8665C0
 ; │┌ @ array.jl:787 within `collect`
 ; ││┌ @ array.jl:671 within `_array_for`
 ; │││┌ @ abstractarray.jl:883 within `similar` @ abstractarray.jl:884
@@ -189,7 +189,7 @@
 	movq	$1, (%rsi)
 	callq	*%rcx
 .Lfunc_end0:
-	.size	julia_f_6289, .Lfunc_end0-julia_f_6289
+	.size	julia_f_6310, .Lfunc_end0-julia_f_6310
 	.cfi_endproc
 ; └└└└
                                         # -- End function
@@ -243,7 +243,7 @@
 
 @code_lowered polynomial(a,x)       # cannot be seen here as optimizations are not applied
 @code_typed polynomial(a,x)         # loop unrolling is not part of type inference optimization
julia> @code_llvm polynomial(a,x);  @ lab.md:113 within `polynomial`
-define double @julia_polynomial_6341([20 x double]* nocapture noundef nonnull readonly align 8 dereferenceable(160) %0, double %1) #0 {
+define double @julia_polynomial_6362([20 x double]* nocapture noundef nonnull readonly align 8 dereferenceable(160) %0, double %1) #0 {
 pass.18:
 ;  @ lab.md:114 within `polynomial`
 ; ┌ @ tuple.jl:29 within `getindex`
@@ -446,7 +446,7 @@
 ;  @ lab.md:118 within `polynomial`
   ret double %79
 }
julia> @code_llvm polynomial(ac,x);  @ lab.md:113 within `polynomial`
-define double @julia_polynomial_6343({}* noundef nonnull align 16 dereferenceable(40) %0, double %1) #0 {
+define double @julia_polynomial_6364({}* noundef nonnull align 16 dereferenceable(40) %0, double %1) #0 {
 top:
 ;  @ lab.md:114 within `polynomial`
 ; ┌ @ abstractarray.jl:419 within `lastindex`
@@ -497,7 +497,7 @@
 ; ┌ @ range.jl:22 within `Colon`
 ; │┌ @ range.jl:24 within `_colon`
 ; ││┌ @ range.jl:373 within `StepRange` @ range.jl:320
-     %13 = call i64 @j_steprange_last_6345(i64 signext %5, i64 signext -1, i64 signext 1) #0
+     %13 = call i64 @j_steprange_last_6366(i64 signext %5, i64 signext -1, i64 signext 1) #0
 ; └└└
 ; ┌ @ range.jl:887 within `iterate`
 ; │┌ @ range.jl:659 within `isempty`
@@ -544,8 +544,8 @@
     %.not15 = icmp eq i64 %value_phi3, %13
 ; └└
   br i1 %.not15, label %L40, label %L23
-}

More than 2x speedup

julia> @btime polynomial($a,$x)  8.966 ns (0 allocations: 0 bytes)
-1.048575e6
julia> @btime polynomial($ac,$x)  23.447 ns (0 allocations: 0 bytes)
+}

More than 2x speedup

julia> @btime polynomial($a,$x)  8.976 ns (0 allocations: 0 bytes)
+1.048575e6
julia> @btime polynomial($ac,$x)  23.598 ns (0 allocations: 0 bytes)
 1.048575e6

Recursion inlining depth

Inlining[2] is another compiler optimization that allows us to speed up the code by avoiding function calls. Where applicable compiler can replace f(args) directly with the function body of f, thus removing the need to modify stack to transfer the control flow to a different place. This is yet another optimization that may improve speed at the expense of binary size.

Exercise

Rewrite the polynomial function from the last lab using recursion and find the length of the coefficients, at which inlining of the recursive calls stops occurring.

function polynomial(a, x)
@@ -699,7 +699,7 @@
       args: Array{Any}((3,))
         1: Symbol +
         2: Symbol x
-        3: Symbol y

The type of both multiline and single line expression is Expr with fields head and args. Notice that Expr type is recursive in the args, which can store other expressions resulting in a tree structure - abstract syntax tree (AST) - that can be visualized for example with the combination of GraphRecipes and Plots packages. 

plot(code_expr_block, fontsize=12, shorten=0.01, axis_buffer=0.15, nodeshape=:rect)
Example block output

This recursive structure has some major performance drawbacks, because the args field is of type Any and therefore modifications of this expression level AST won't be type stable. Building blocks of expressions are Symbols and literal values (numbers).

A possible nuisance of working with multiline expressions is the presence of LineNumber nodes, which can be removed with Base.remove_linenums! function.

julia> Base.remove_linenums!(code_parse_block)quote
+        3: Symbol y

The type of both multiline and single line expression is Expr with fields head and args. Notice that Expr type is recursive in the args, which can store other expressions resulting in a tree structure - abstract syntax tree (AST) - that can be visualized for example with the combination of GraphRecipes and Plots packages. 

plot(code_expr_block, fontsize=12, shorten=0.01, axis_buffer=0.15, nodeshape=:rect)
Example block output

This recursive structure has some major performance drawbacks, because the args field is of type Any and therefore modifications of this expression level AST won't be type stable. Building blocks of expressions are Symbols and literal values (numbers).

A possible nuisance of working with multiline expressions is the presence of LineNumber nodes, which can be removed with Base.remove_linenums! function.

julia> Base.remove_linenums!(code_parse_block)quote
     x = 2
     y = 3
     x + y
@@ -731,4 +731,4 @@
            end
        
            return ex
-       endwrap! (generic function with 1 method)
julia> wrap!(ex::Number) = Expr(:call, :f, ex)wrap! (generic function with 2 methods)
julia> wrap!(ex) = exwrap! (generic function with 3 methods)
julia> ext, x, y = copy(ex), 2, 3(:(x * x + 2 * y * x + y * y), 2, 3)
julia> @test wrap!(ex) == :(x*x + f(2)*y*x + y*y)Test Passed
julia> eval(ext)25
julia> eval(ex)25.0

This kind of manipulation is at the core of some pkgs, such as aforementioned IntervalArithmetics.jl where every number is replaced with a narrow interval in order to find some bounds on the result of a computation.

Resources

+ endwrap! (generic function with 1 method)
julia> wrap!(ex::Number) = Expr(:call, :f, ex)wrap! (generic function with 2 methods)
julia> wrap!(ex) = exwrap! (generic function with 3 methods)
julia> ext, x, y = copy(ex), 2, 3(:(x * x + 2 * y * x + y * y), 2, 3)
julia> @test wrap!(ex) == :(x*x + f(2)*y*x + y*y)Test Passed
julia> eval(ext)25
julia> eval(ex)25.0

This kind of manipulation is at the core of some pkgs, such as aforementioned IntervalArithmetics.jl where every number is replaced with a narrow interval in order to find some bounds on the result of a computation.

Resources

diff --git a/dev/lecture_06/lecture/index.html b/dev/lecture_06/lecture/index.html index 22961474..687c0835 100644 --- a/dev/lecture_06/lecture/index.html +++ b/dev/lecture_06/lecture/index.html @@ -369,4 +369,4 @@ s = "Base.$(f)(A::MyMatrix, args...) = $(f)(A.x, args...)" println(s) eval(Meta.parse(s)) -end

for f in [:setindex!, :getindex, :size, :length] @eval f(A::MyMatrix, args...) = f(A.x, args...) end

Notice that we have just hand-implemented parts of @forward macro from MacroTools, which does exactly this.

Resources

+end

for f in [:setindex!, :getindex, :size, :length] @eval f(A::MyMatrix, args...) = f(A.x, args...) end

Notice that we have just hand-implemented parts of @forward macro from MacroTools, which does exactly this.

Resources

diff --git a/dev/lecture_07/hw/index.html b/dev/lecture_07/hw/index.html index f3bdc4dd..0e9cf896 100644 --- a/dev/lecture_07/hw/index.html +++ b/dev/lecture_07/hw/index.html @@ -15,4 +15,4 @@ genex = _ecosystem(ex) world = eval(genex) +Solution:

Nothing to see here

diff --git a/dev/lecture_07/lab/index.html b/dev/lecture_07/lab/index.html index d83f2811..31cd45f0 100644 --- a/dev/lecture_07/lab/index.html +++ b/dev/lecture_07/lab/index.html @@ -264,4 +264,4 @@ species = :Rabbit foodlist = :([Grass => 0.5, Broccoli => 1.0]) -_eats(species, foodlist)

Resources

Type{T} type selectors

We have used ::Type{T} signature[2] at few places in the Ecosystem family of packages (and it will be helpful in the HW as well), such as in the show methods

Base.show(io::IO,::Type{World}) = print(io,"🌍")

This particular example defines a method where the second argument is the World type itself and not an instance of a World type. As a result we are able to dispatch on specific types as values.

Furthermore we can use subtyping operator to match all types in a hierarchy, e.g. ::Type{<:AnimalSpecies} matches all animal species

+_eats(species, foodlist)

Resources

Type{T} type selectors

We have used ::Type{T} signature[2] at few places in the Ecosystem family of packages (and it will be helpful in the HW as well), such as in the show methods

Base.show(io::IO,::Type{World}) = print(io,"🌍")

This particular example defines a method where the second argument is the World type itself and not an instance of a World type. As a result we are able to dispatch on specific types as values.

Furthermore we can use subtyping operator to match all types in a hierarchy, e.g. ::Type{<:AnimalSpecies} matches all animal species

diff --git a/dev/lecture_07/lecture/index.html b/dev/lecture_07/lecture/index.html index 91bc7f0b..458c2120 100644 --- a/dev/lecture_07/lecture/index.html +++ b/dev/lecture_07/lecture/index.html @@ -420,4 +420,4 @@ print(io, lb,r.left,",",r.right,rb) end

We can check it does the job by trying Interval("[1,2)"). Finally, we define a string macro as 

macro int_str(s)
 	Interval(s)
-end

which allows us to define interval as int"[1,2)".

Sources

+end

which allows us to define interval as int"[1,2)".

Sources

diff --git a/dev/lecture_07/macros/index.html b/dev/lecture_07/macros/index.html index 70592d08..c128a719 100644 --- a/dev/lecture_07/macros/index.html +++ b/dev/lecture_07/macros/index.html @@ -1,2 +1,2 @@ -Macros · Scientific Programming in Julia
+Macros · Scientific Programming in Julia
diff --git a/dev/lecture_08/hw/index.html b/dev/lecture_08/hw/index.html index 25f20d6e..c6fa77d6 100644 --- a/dev/lecture_08/hw/index.html +++ b/dev/lecture_08/hw/index.html @@ -48,4 +48,4 @@ accum!.(ts) end
gradient (generic function with 1 method)

We will use it to compute the derivative of the Babylonian square root.

babysqrt(x, t=(1+x)/2, n=10) = n==0 ? t : babysqrt(x, (t+x/t)/2, n-1)

In order to differentiate through babysqrt you will need a reverse rule for / for Base.:/(TrackedReal,TrackedReal) as well as the cases where you divide with constants in volved (e.g. Base.:/(TrackedReal,Real)).

Homework (2 points)
-

Write the reverse rules for / and the missing rules for + such that you can differentiate through division and addition with and without constants.

You can verify your solution with the gradient function.

julia> gradient(babysqrt, 2.0)(0.35355339059327373,)
julia> 1/(2babysqrt(2.0))0.3535533905932738
+

Write the reverse rules for / and the missing rules for + such that you can differentiate through division and addition with and without constants.

You can verify your solution with the gradient function.

julia> gradient(babysqrt, 2.0)(0.35355339059327373,)
julia> 1/(2babysqrt(2.0))0.3535533905932738
diff --git a/dev/lecture_08/lab/adbd310d.svg b/dev/lecture_08/lab/a2df3780.svg similarity index 96% rename from dev/lecture_08/lab/adbd310d.svg rename to dev/lecture_08/lab/a2df3780.svg index 3f22a977..699e310e 100644 --- a/dev/lecture_08/lab/adbd310d.svg +++ b/dev/lecture_08/lab/a2df3780.svg @@ -1,120 +1,120 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + diff --git a/dev/lecture_08/lab/index.html b/dev/lecture_08/lab/index.html index 0c48d49c..b5cd0a01 100644 --- a/dev/lecture_08/lab/index.html +++ b/dev/lecture_08/lab/index.html @@ -55,15 +55,15 @@ z end

Forward & Backward Pass

To visualize that with reverse-mode AD we really do save computation we can visualize the computation graph at different stages. We start with the forward pass and keep observing x

julia> x = track(2.0,"x");
julia> y = track(3.0,"y");
julia> a = x*y;
julia> print_tree(x)x data: 2.0 grad: nothing
 └─ * data: 6.0 grad: nothing

We can see that we x now has one child a which has the value 2.0*3.0==6.0. All the gradients are still nothing. Computing another value that depends on x will add another child.

julia> b = sin(x)0.9092974268256817 (tracked sin)
julia> print_tree(x)x data: 2.0 grad: nothing
-├─ sin data: 0.91 grad: nothing
-└─ * data: 6.0 grad: nothing

In the final step we compute z which does not mutate the children of x because it does not depend directly on it. The result z is added as a child to both a and b.

julia> z = a + b6.909297426825682 (tracked +)
julia> print_tree(x)x data: 2.0 grad: nothing
-├─ sin data: 0.91 grad: nothing
+├─ * data: 6.0 grad: nothing
+└─ sin data: 0.91 grad: nothing

In the final step we compute z which does not mutate the children of x because it does not depend directly on it. The result z is added as a child to both a and b.

julia> z = a + b6.909297426825682 (tracked +)
julia> print_tree(x)x data: 2.0 grad: nothing
+├─ * data: 6.0 grad: nothing
 │  └─ + data: 6.91 grad: nothing
-└─ * data: 6.0 grad: nothing
+└─ sin data: 0.91 grad: nothing
    └─ + data: 6.91 grad: nothing

For the backward pass we have to seed the initial gradient value of z and call accum! on the variable that we are interested in.

julia> z.grad = 1.01.0
julia> dx = accum!(x)2.5838531634528574
julia> dx ≈ y.data + cos(x.data)true

By accumulating the gradients for x, the gradients in the sub-tree connected to x will be evaluated. The parts of the tree that are only connected to y stay untouched.

julia> print_tree(x)x data: 2.0 grad: 2.5838531634528574
-├─ sin data: 0.91 grad: 1.0
+├─ * data: 6.0 grad: 1.0
 │  └─ + data: 6.91 grad: 1.0
-└─ * data: 6.0 grad: 1.0
+└─ sin data: 0.91 grad: 1.0
    └─ + data: 6.91 grad: 1.0
julia> print_tree(y)y data: 3.0 grad: nothing
 └─ * data: 6.0 grad: 1.0
    └─ + data: 6.91 grad: 1.0

If we now accumulate the gradients over y we re-use the gradients that are already computed. In larger computations this will save us a lot of effort!

Info

This also means that we have to re-build the graph for every new set of inputs!

Optimizing 2D Functions

@@ -79,7 +79,7 @@ using Plots color_scheme = cgrad(:RdYlBu_5, rev=true) -contour(-4:0.1:4, -2:0.1:2, g, fill=true, c=color_scheme, xlabel="x", ylabel="y")Example block output

We can find a local minimum of $g$ by starting at an initial point $(x_0,y_0)$ and taking small steps in the opposite direction of the gradient

\[\begin{align} +contour(-4:0.1:4, -2:0.1:2, g, fill=true, c=color_scheme, xlabel="x", ylabel="y")Example block output

We can find a local minimum of $g$ by starting at an initial point $(x_0,y_0)$ and taking small steps in the opposite direction of the gradient

\[\begin{align} x_{i+1} &= x_i - \lambda \frac{\partial f}{\partial x_i} \\ y_{i+1} &= y_i - \lambda \frac{\partial f}{\partial y_i}, \end{align}\]

where $\lambda$ is the learning rate that has to be tuned manually.

@@ -187,8 +187,8 @@ # pretty print TrackedArray Base.show(io::IO, x::TrackedArray) = print(io, "Tracked $(x.data)") Base.print_array(io::IO, x::TrackedArray) = Base.print_array(io, x.data)

Creating a TrackedArray should work like this:

julia> track(rand(2,2))2×2 Main.TrackedArray{Float64, 2, Matrix{Float64}}:
- 0.573951  0.103986
- 0.339707  0.196479
function accum!(x::Union{TrackedReal,TrackedArray})
+ 0.616642  0.51326
+ 0.479984  0.23724
function accum!(x::Union{TrackedReal,TrackedArray})
     if isnothing(x.grad)
         x.grad = sum(λ(accum!(Δ)) for (Δ,λ) in x.children)
     end
@@ -204,19 +204,19 @@
     Y.children[Z] = Δ -> X.data' * Δ
     Z
 end

julia> X = rand(2,3) |> track2×3 Main.TrackedArray{Float64, 2, Matrix{Float64}}:
- 0.783772  0.424827  0.11654
- 0.996745  0.902329  0.248027
julia> Y = rand(3,2) |> track3×2 Main.TrackedArray{Float64, 2, Matrix{Float64}}:
- 0.163143  0.361368
- 0.734127  0.633411
- 0.586153  0.145302
julia> Z = X*Y2×2 Main.TrackedArray{Float64, 2, Matrix{Float64}}:
- 0.508055  0.569254
- 0.970418  0.967776
julia> f = X.children[Z]#3 (generic function with 1 method)
julia> Ω̄ = ones(size(Z)...)2×2 Matrix{Float64}:
+ 0.0682162  0.519568  0.0426353
+ 0.422919   0.923472  0.693343
julia> Y = rand(3,2) |> track3×2 Main.TrackedArray{Float64, 2, Matrix{Float64}}:
+ 0.904966   0.903146
+ 0.669698   0.618338
+ 0.0864509  0.944416
julia> Z = X*Y2×2 Main.TrackedArray{Float64, 2, Matrix{Float64}}:
+ 0.413373  0.423143
+ 1.06111   1.60778
julia> f = X.children[Z]#3 (generic function with 1 method)
julia> Ω̄ = ones(size(Z)...)2×2 Matrix{Float64}:
  1.0  1.0
  1.0  1.0
julia> f(Ω̄)2×3 Matrix{Float64}:
- 0.524511  1.36754  0.731455
- 0.524511  1.36754  0.731455
julia> Ω̄*Y.data'2×3 Matrix{Float64}:
- 0.524511  1.36754  0.731455
- 0.524511  1.36754  0.731455
+ 1.80811 1.28804 1.03087 + 1.80811 1.28804 1.03087
julia> Ω̄*Y.data'2×3 Matrix{Float64}: + 1.80811 1.28804 1.03087 + 1.80811 1.28804 1.03087
Exercise

Implement rules for sum, +, -, and abs2.

@@ -288,4 +288,4 @@ end end y -end

You can see a full implementation of our tracing based AD here and a simple implementation of a Neural Network that can learn an approximation to the function g here. Running the latter script will produce an animation that shows how the network is learning.

anim

This lab is heavily inspired by Rufflewind

+end

You can see a full implementation of our tracing based AD here and a simple implementation of a Neural Network that can learn an approximation to the function g here. Running the latter script will produce an animation that shows how the network is learning.

anim

This lab is heavily inspired by Rufflewind

diff --git a/dev/lecture_08/lecture/5454f9cc.svg b/dev/lecture_08/lecture/8d3f576c.svg similarity index 86% rename from dev/lecture_08/lecture/5454f9cc.svg rename to dev/lecture_08/lecture/8d3f576c.svg index f89e3213..35db3f41 100644 --- a/dev/lecture_08/lecture/5454f9cc.svg +++ b/dev/lecture_08/lecture/8d3f576c.svg @@ -1,48 +1,48 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/lecture_08/lecture/index.html b/dev/lecture_08/lecture/index.html index cf040949..6b011a29 100644 --- a/dev/lecture_08/lecture/index.html +++ b/dev/lecture_08/lecture/index.html @@ -57,7 +57,7 @@ -1.25

We now compare the analytic solution to values computed by the forward_diff and byt he finite differencing

\[f(x) = \sqrt{x} \qquad f'(x) = \frac{1}{2\sqrt{x}}\]

julia> using FiniteDifferencesERROR: ArgumentError: Package FiniteDifferences not found in current path.
 - Run `import Pkg; Pkg.add("FiniteDifferences")` to install the FiniteDifferences package.
julia> forward_dsqrt(x) = forward_diff(babysqrt,x)forward_dsqrt (generic function with 1 method)
julia> analytc_dsqrt(x) = 1/(2babysqrt(x))analytc_dsqrt (generic function with 1 method)
julia> forward_dsqrt(2.0)0.35355339059327373
julia> analytc_dsqrt(2.0)0.3535533905932738
julia> central_fdm(5, 1)(babysqrt, 2.0)ERROR: UndefVarError: `central_fdm` not defined
plot(0.0:0.01:2, babysqrt, label="f(x) = babysqrt(x)", lw=3)
 plot!(0.1:0.01:2, analytc_dsqrt, label="Analytic f'", ls=:dot, lw=3)
-plot!(0.1:0.01:2, forward_dsqrt, label="Dual Forward Mode f'", lw=3, ls=:dash)
Example block output

Takeaways

  1. Forward mode $f'$ is obtained simply by pushing a Dual through babysqrt
  2. To make the forward diff work in Julia, we only need to overload a few operators for forward mode AD to work on any function. Therefore the name of the approach is called operator overloading.
  3. For vector valued function we can use Hyperduals
  4. Forward diff can differentiation through the setindex! (called each time an element is assigned to a place in array, e.g. x = [1,2,3]; x[2] = 1)
  5. ForwardDiff is implemented in ForwardDiff.jl, which might appear to be neglected, but the truth is that it is very stable and general implementation.
  6. ForwardDiff does not have to be implemented through Dual numbers. It can be implemented similarly to ReverseDiff through multiplication of Jacobians, which is what is the community work on now (in Diffractor, Zygote with rules defined in ChainRules).

Reverse mode

In reverse mode, the computation of the gradient follow the opposite order. We initialize the computation by setting $\mathbf{J}_0 = \frac{\partial y}{\partial y_0},$ which is again an identity matrix. Then we compute Jacobians and multiplications in the opposite order. The problem is that to calculate $J_i$ we need to know the value of $y_i^0$, which cannot be calculated in the reverse pass. The backward pass therefore needs to be preceded by the forward pass, where $\{y_i^0\}_{i=1}^n$ are calculated.

The complete reverse mode algorithm therefore proceeds as

  1. Forward pass: iterate i from n down to 1 as
    • calculate the next intermediate output as $y^0_{i-1} = f_i(y^0_i)$
  2. Backward pass: iterate i from 1 down to n as
    • calculate Jacobian $J_i = \left.\frac{f_i}{\partial y_i}\right|_{y_i^0}$ at point $y_i^0$
    • pull back the gradient as $\left.\frac{\partial f(x)}{\partial y_{i}}\right|_{y^0_i} = \left.\frac{\partial y_0}{\partial y_{i-1}}\right|_{y^0_{i-1}} \times J_i$

The need to store intermediate outs has a huge impact on memory requirements, which particularly on GPU is a big deal. Recall few lectures ago we have been discussing how excessive memory allocations can be damaging for performance, here we are given an algorithm where the excessive allocation is by design.

Tricks to decrease memory consumptions

  • Define custom rules over large functional blocks. For example while we can auto-grad (in theory) matrix product, it is much more efficient to define make a matrix multiplication as one large function, for which we define Jacobians (note that by doing so, we can dispatch on Blas). e.g

\[\begin{alignat*}{2} +plot!(0.1:0.01:2, forward_dsqrt, label="Dual Forward Mode f'", lw=3, ls=:dash)Example block output

Takeaways

  1. Forward mode $f'$ is obtained simply by pushing a Dual through babysqrt
  2. To make the forward diff work in Julia, we only need to overload a few operators for forward mode AD to work on any function. Therefore the name of the approach is called operator overloading.
  3. For vector valued function we can use Hyperduals
  4. Forward diff can differentiation through the setindex! (called each time an element is assigned to a place in array, e.g. x = [1,2,3]; x[2] = 1)
  5. ForwardDiff is implemented in ForwardDiff.jl, which might appear to be neglected, but the truth is that it is very stable and general implementation.
  6. ForwardDiff does not have to be implemented through Dual numbers. It can be implemented similarly to ReverseDiff through multiplication of Jacobians, which is what is the community work on now (in Diffractor, Zygote with rules defined in ChainRules).

Reverse mode

In reverse mode, the computation of the gradient follow the opposite order. We initialize the computation by setting $\mathbf{J}_0 = \frac{\partial y}{\partial y_0},$ which is again an identity matrix. Then we compute Jacobians and multiplications in the opposite order. The problem is that to calculate $J_i$ we need to know the value of $y_i^0$, which cannot be calculated in the reverse pass. The backward pass therefore needs to be preceded by the forward pass, where $\{y_i^0\}_{i=1}^n$ are calculated.

The complete reverse mode algorithm therefore proceeds as

  1. Forward pass: iterate i from n down to 1 as
    • calculate the next intermediate output as $y^0_{i-1} = f_i(y^0_i)$
  2. Backward pass: iterate i from 1 down to n as
    • calculate Jacobian $J_i = \left.\frac{f_i}{\partial y_i}\right|_{y_i^0}$ at point $y_i^0$
    • pull back the gradient as $\left.\frac{\partial f(x)}{\partial y_{i}}\right|_{y^0_i} = \left.\frac{\partial y_0}{\partial y_{i-1}}\right|_{y^0_{i-1}} \times J_i$

The need to store intermediate outs has a huge impact on memory requirements, which particularly on GPU is a big deal. Recall few lectures ago we have been discussing how excessive memory allocations can be damaging for performance, here we are given an algorithm where the excessive allocation is by design.

Tricks to decrease memory consumptions

  • Define custom rules over large functional blocks. For example while we can auto-grad (in theory) matrix product, it is much more efficient to define make a matrix multiplication as one large function, for which we define Jacobians (note that by doing so, we can dispatch on Blas). e.g

\[\begin{alignat*}{2} \mathbf{C} &= \mathbf{A} * \mathbf{B} \\ \frac{\partial{\mathbf{C}}}{\partial \mathbf{A}} &= \mathbf{B} \\ \frac{\partial{\mathbf{C}}}{\partial \mathbf{B}} &= \mathbf{A}^{\mathrm{T}} \\ @@ -212,4 +212,4 @@ │ %2 = Main.sin(x) │ %3 = %1 + %2 └── return %3 -)

This form is particularly nice for automatic differentiation, as we have on the left hand side always a single variable, which means the compiler has provided us with a form, on which we know, how to apply AD rules.

What if we somehow be able to talk to the compiler and get this form from him?

Sources for this lecture

  • 1Linnainmaa, S. (1976). Taylor expansion of the accumulated rounding error. BIT Numerical Mathematics, 16(2), 146-160.
  • 2Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536.
+)

This form is particularly nice for automatic differentiation, as we have on the left hand side always a single variable, which means the compiler has provided us with a form, on which we know, how to apply AD rules.

What if we somehow be able to talk to the compiler and get this form from him?

Sources for this lecture

  • 1Linnainmaa, S. (1976). Taylor expansion of the accumulated rounding error. BIT Numerical Mathematics, 16(2), 146-160.
  • 2Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536.
diff --git a/dev/lecture_09/ircode/index.html b/dev/lecture_09/ircode/index.html index cc89577f..182fcca9 100644 --- a/dev/lecture_09/ircode/index.html +++ b/dev/lecture_09/ircode/index.html @@ -61,4 +61,4 @@ match = only(mthds) mi = Core.Compiler.specialize_method(match) -ci = Core.Compiler.retrieve_code_info(mi, world)

CodeInfo

IRTools.jl are great for modifying CodeInfo. I have found two tools for modifying IRCode and I wonder if they have been abandoned because they were both dead ends or because of lack of human labor. I am also aware of Also, this is quite cool play with IRStuff.

Resources

  • https://vchuravy.dev/talks/licm/
  • CompilerPluginTools
  • CodeInfoTools.jl.
  • TKF's CodeInfo.jl is nice for visualization of the IRCode
  • Diffractor is an awesome source of howto. For example function my_insert_node! in src/stage1/hacks.jl
  • https://nbviewer.org/gist/tkf/d4734be24d2694a3afd669f8f50e6b0f/00_notebook.ipynb
  • https://github.com/JuliaCompilerPlugins/Mixtape.jl
+ci = Core.Compiler.retrieve_code_info(mi, world)

CodeInfo

IRTools.jl are great for modifying CodeInfo. I have found two tools for modifying IRCode and I wonder if they have been abandoned because they were both dead ends or because of lack of human labor. I am also aware of Also, this is quite cool play with IRStuff.

Resources

  • https://vchuravy.dev/talks/licm/
  • CompilerPluginTools
  • CodeInfoTools.jl.
  • TKF's CodeInfo.jl is nice for visualization of the IRCode
  • Diffractor is an awesome source of howto. For example function my_insert_node! in src/stage1/hacks.jl
  • https://nbviewer.org/gist/tkf/d4734be24d2694a3afd669f8f50e6b0f/00_notebook.ipynb
  • https://github.com/JuliaCompilerPlugins/Mixtape.jl
diff --git a/dev/lecture_09/lab/index.html b/dev/lecture_09/lab/index.html index d069c592..968febd7 100644 --- a/dev/lecture_09/lab/index.html +++ b/dev/lecture_09/lab/index.html @@ -5,7 +5,7 @@ acc = x*acc + p[i] end acc -end

Julia has its own implementation of this function called evalpoly. If we compare the performance of our polynomial and Julia's evalpoly we can observe a pretty big difference:

julia> x = 2.02.0
julia> p = ntuple(float,20);
julia> @btime polynomial($x,$p)  8.985 ns (0 allocations: 0 bytes)
+end

Julia has its own implementation of this function called evalpoly. If we compare the performance of our polynomial and Julia's evalpoly we can observe a pretty big difference:

julia> x = 2.02.0
julia> p = ntuple(float,20);
julia> @btime polynomial($x,$p)  8.975 ns (0 allocations: 0 bytes)
 1.9922945e7
julia> @btime evalpoly($x,$p)  4.348 ns (0 allocations: 0 bytes)
 1.9922945e7

Julia's implementation uses a generated function which specializes on different tuple lengths (i.e. it unrolls the loop) and eliminates the (small) overhead of looping over the tuple. This is possible, because the length of the tuple is known during compile time. You can check the difference between polynomial and evalpoly yourself via the introspectionwtools you know - e.g. @code_lowered.

Exercise
@@ -17,7 +17,7 @@ ex = :(x*$ex + p[$i]) end ex -end

You should get the same performance as evalpoly (and as @poly from Lab 7 with the added convenience of not having to spell out all the coefficients in your code like: p = @poly 1 2 3 ...).

julia> @btime genpoly($x,$p)  8.975 ns (0 allocations: 0 bytes)
+end

You should get the same performance as evalpoly (and as @poly from Lab 7 with the added convenience of not having to spell out all the coefficients in your code like: p = @poly 1 2 3 ...).

julia> @btime genpoly($x,$p)  8.985 ns (0 allocations: 0 bytes)
 1.9922945e7

Fast, Static Matrices

Another great example that makes heavy use of generated functions are static arrays. A static array is an array of fixed size which can be implemented via an NTuple. This means that it will be allocated on the stack, which can buy us a lot of performance for smaller static arrays. We define a StaticMatrix{T,C,R,L} where the paramteric types represent the matrix element type T (e.g. Float32), the number of rows R, the number of columns C, and the total length of the matrix L=C*R (which we need to set the size of the NTuple).

struct StaticMatrix{T,R,C,L} <: AbstractArray{T,2}
     data::NTuple{L,T}
 end
@@ -33,10 +33,10 @@
 Base.length(x::StaticMatrix{T,R,C,L}) where {T,R,C,L} = L
 Base.getindex(x::StaticMatrix, i::Int) = x.data[i]
 Base.getindex(x::StaticMatrix{T,R,C}, r::Int, c::Int) where {T,R,C} = x.data[R*(c-1) + r]

You can check if everything works correctly by comparing to a normal Matrix:

julia> x = rand(2,3)2×3 Matrix{Float64}:
- 0.140681  0.730943  0.357691
- 0.896104  0.461956  0.363701
julia> x[1,2]0.7309432091736742
julia> a = StaticMatrix(x)2×3 Main.StaticMatrix{Float64, 2, 3, 6}:
- 0.140681  0.730943  0.357691
- 0.896104  0.461956  0.363701
julia> a[1,2]0.7309432091736742
+ 0.166925 0.220522 0.0572556 + 0.514926 0.795528 0.308131
julia> x[1,2]0.22052201440244568
julia> a = StaticMatrix(x)2×3 Main.StaticMatrix{Float64, 2, 3, 6}: + 0.166925 0.220522 0.0572556 + 0.514926 0.795528 0.308131
julia> a[1,2]0.22052201440244568
Exercise

Overload matrix multiplication between two static matrices

Base.:*(x::StaticMatrix{T,K,M},y::StaticMatrix{T,M,N})

with a generated function that creates an expression without loops. Below you can see an example for an expression that would be generated from multiplying two $2\times 2$ matrices.

:(StaticMatrix{T,2,2,4}((
     (x[1,1]*y[1,1] + x[1,2]*y[2,1]),
@@ -52,20 +52,20 @@
     z = Expr(:tuple, zs...)
     :(StaticMatrix{$T,$K,$N,$(K*N)}($z))
 end

You can check that your matrix multiplication works by multiplying two random matrices. Which one is faster?

julia> a = rand(2,3)2×3 Matrix{Float64}:
- 0.361048  0.188428  0.23953
- 0.526889  0.74518   0.995482
julia> b = rand(3,4)3×4 Matrix{Float64}:
- 0.283104  0.861352  0.838542  0.827074
- 0.887657  0.854791  0.676118  0.808743
- 0.975118  0.735933  0.787604  0.472445
julia> c = StaticMatrix(a)2×3 Main.StaticMatrix{Float64, 2, 3, 6}:
- 0.361048  0.188428  0.23953
- 0.526889  0.74518   0.995482
julia> d = StaticMatrix(b)3×4 Main.StaticMatrix{Float64, 3, 4, 12}:
- 0.283104  0.861352  0.838542  0.827074
- 0.887657  0.854791  0.676118  0.808743
- 0.975118  0.735933  0.787604  0.472445
julia> a*b2×4 Matrix{Float64}:
- 0.503044  0.648334  0.618808  0.564168
- 1.78134   1.82342   1.72969   1.50875
julia> c*d2×4 Main.StaticMatrix{Float64, 2, 4, 8}:
- 0.503044  0.648334  0.618808  0.564168
- 1.78134   1.82342   1.72969   1.50875

OptionalArgChecks.jl

The package OptionalArgChecks.jl makes is possible to add checks to a function which can then be removed by calling the function with the @skip macro. For example, we can check if the input to a function f is an even number

function f(x::Number)
+ 0.932736  0.764829  0.533826
+ 0.142397  0.409429  0.695706
julia> b = rand(3,4)3×4 Matrix{Float64}:
+ 0.745627   0.989929  0.926135  0.286951
+ 0.891098   0.709065  0.937527  0.170143
+ 0.0044717  0.102473  0.657533  0.112949
julia> c = StaticMatrix(a)2×3 Main.StaticMatrix{Float64, 2, 3, 6}:
+ 0.932736  0.764829  0.533826
+ 0.142397  0.409429  0.695706
julia> d = StaticMatrix(b)3×4 Main.StaticMatrix{Float64, 3, 4, 12}:
+ 0.745627   0.989929  0.926135  0.286951
+ 0.891098   0.709065  0.937527  0.170143
+ 0.0044717  0.102473  0.657533  0.112949
julia> a*b2×4 Matrix{Float64}:
+ 1.3794    1.52036   1.9319   0.458075
+ 0.474128  0.502566  0.97318  0.189102
julia> c*d2×4 Main.StaticMatrix{Float64, 2, 4, 8}:
+ 1.3794    1.52036   1.9319   0.458075
+ 0.474128  0.502566  0.97318  0.189102

OptionalArgChecks.jl

The package OptionalArgChecks.jl makes is possible to add checks to a function which can then be removed by calling the function with the @skip macro. For example, we can check if the input to a function f is an even number

function f(x::Number)
     iseven(x) || error("Input has to be an even number!")
     x
 end

If you are doing more involved argument checking it can take quite some time to perform all your checks. However, if you want to be fast and are completely sure that you are always passing in the correct inputs to your function, you might want to remove them in some cases. Hence, we would like to transform the IR of the function above

julia> using IRTools
julia> using IRTools: @code_ir
julia> @code_ir f(1)1: (%1, %2)
@@ -318,4 +318,4 @@
   %6 = (Main.skip)(Main.bar, %5)
   return %6
julia> foo(-2)The input is negative.
 The input is even.
--2
julia> skip(foo,-2)-2
julia> @skip foo(-2)-2

References

+-2
julia> skip(foo,-2)-2
julia> @skip foo(-2)-2

References

diff --git a/dev/lecture_09/lecture/index.html b/dev/lecture_09/lecture/index.html index 7ad3581d..d98880c2 100644 --- a/dev/lecture_09/lecture/index.html +++ b/dev/lecture_09/lecture/index.html @@ -28,7 +28,7 @@ foo() :(x + y) endgenplus (generic function with 1 method)
julia> foo() = println("foo")foo (generic function with 1 method)
julia> genplus(1,1)ERROR: MethodError: no method matching foo() -The applicable method may be too new: running in world age 34451, while current world is 34452. +The applicable method may be too new: running in world age 34452, while current world is 34453. Closest candidates are: foo() (method too new to be called from this world context.) @@ -528,4 +528,4 @@ %17 = %12 + %16 %18 = Base.tuple(0, %15, %17) return %18

and it calculates the gradient with respect to the input as

julia> pb(1.0)
-(0, 1.0, 1.5403023058681398)

where the first item is gradient with parameters of the function itself.

Conclusion

The above examples served to demonstrate that @generated functions offers extremely powerful paradigm, especially if coupled with manipulation of intermediate representation. Within few lines of code, we have implemented reasonably powerful profiler and reverse AD engine. Importantly, it has been done without a single-purpose engine or tooling.

+(0, 1.0, 1.5403023058681398)

where the first item is gradient with parameters of the function itself.

Conclusion

The above examples served to demonstrate that @generated functions offers extremely powerful paradigm, especially if coupled with manipulation of intermediate representation. Within few lines of code, we have implemented reasonably powerful profiler and reverse AD engine. Importantly, it has been done without a single-purpose engine or tooling.

diff --git a/dev/lecture_10/hw/index.html b/dev/lecture_10/hw/index.html index 196a2d47..50912ed4 100644 --- a/dev/lecture_10/hw/index.html +++ b/dev/lecture_10/hw/index.html @@ -8,4 +8,4 @@ w = [1.0, 2.0, 4.0, 2.0, 1.0] @btime thread_conv1d($x, $w);

On your local machine you should be able to achieve 0.6x reduction in execution time with two threads, however the automatic eval system is a noisy environment and therefore we require only 0.8x reduction therein. This being said, please reach out to us, if you encounter any issues.

HINTS:

  • start with single threaded implementation
  • don't forget to reverse the kernel
  • @threads macro should be all you need
  • for testing purposes create a simple script, that you can run with julia -t 1 and julia -t 2
+Solution:

Nothing to see here

diff --git a/dev/lecture_10/lab/index.html b/dev/lecture_10/lab/index.html index 52eb2b86..6d299979 100644 --- a/dev/lecture_10/lab/index.html +++ b/dev/lecture_10/lab/index.html @@ -401,4 +401,4 @@ get_cat_facts(n) = map(x -> query_cat_fact(), Base.OneTo(n)) @time get_cat_facts_async(10) # ~0.15s -@time get_cat_facts(10) # ~1.1s

Resources

  • parallel computing course by Julia Computing
+@time get_cat_facts(10) # ~1.1s

Resources

  • parallel computing course by Julia Computing
diff --git a/dev/lecture_10/lecture/index.html b/dev/lecture_10/lecture/index.html index 2b654795..8a5f5487 100644 --- a/dev/lecture_10/lecture/index.html +++ b/dev/lecture_10/lecture/index.html @@ -429,4 +429,4 @@ @elapsed foldxd(mergewith(+), files |> Map(histfile)) 86.44577969 @elapsed foldxt(mergewith(+), files |> Map(histfile)) -105.32969331

is much better.

Locks / lock-free multi-threadding

Avoid locks.

Take away message

When deciding, what kind of paralelism to employ, consider following

  • for tightly coupled computation over shared data, multi-threadding is more suitable due to non-existing sharing of data between processes
  • but if the computation requires frequent allocation and freeing of memery, or IO, separate processes are multi-suitable, since garbage collectors are independent between processes
  • Making all cores busy while achieving an ideally linear speedup is difficult and needs a lot of experience and knowledge. Tooling and profilers supporting debugging of parallel processes is not much developped.
  • Transducers thrives for (almost) the same code to support thread- and process-based paralelism.

Materials

+105.32969331

is much better.

Locks / lock-free multi-threadding

Avoid locks.

Take away message

When deciding, what kind of paralelism to employ, consider following

  • for tightly coupled computation over shared data, multi-threadding is more suitable due to non-existing sharing of data between processes
  • but if the computation requires frequent allocation and freeing of memery, or IO, separate processes are multi-suitable, since garbage collectors are independent between processes
  • Making all cores busy while achieving an ideally linear speedup is difficult and needs a lot of experience and knowledge. Tooling and profilers supporting debugging of parallel processes is not much developped.
  • Transducers thrives for (almost) the same code to support thread- and process-based paralelism.

Materials

diff --git a/dev/lecture_12/hw/ac32a377.svg b/dev/lecture_12/hw/681d107c.svg similarity index 92% rename from dev/lecture_12/hw/ac32a377.svg rename to dev/lecture_12/hw/681d107c.svg index 7ebf306e..d448062e 100644 --- a/dev/lecture_12/hw/ac32a377.svg +++ b/dev/lecture_12/hw/681d107c.svg @@ -1,56 +1,56 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/lecture_12/hw/index.html b/dev/lecture_12/hw/index.html index dbb0fc09..e4911e87 100644 --- a/dev/lecture_12/hw/index.html +++ b/dev/lecture_12/hw/index.html @@ -82,4 +82,4 @@ # RK2 solve (t,X) = solve(prob, RK2(0.2)) plot!(p1,t,X[1,:], color=1, lw=3, alpha=0.8, label="x RK2") -plot!(p1,t,X[2,:], color=2, lw=3, alpha=0.8, label="y RK2")Example block output +plot!(p1,t,X[2,:], color=2, lw=3, alpha=0.8, label="y RK2")Example block output diff --git a/dev/lecture_12/lab/1bab4067.svg b/dev/lecture_12/lab/1bab4067.svg new file mode 100644 index 00000000..4de268b4 --- /dev/null +++ b/dev/lecture_12/lab/1bab4067.svg @@ -0,0 +1,48 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/lecture_12/lab/703c4e58.svg b/dev/lecture_12/lab/330135ea.svg similarity index 97% rename from dev/lecture_12/lab/703c4e58.svg rename to dev/lecture_12/lab/330135ea.svg index ba3cbff8..09210e85 100644 --- a/dev/lecture_12/lab/703c4e58.svg +++ b/dev/lecture_12/lab/330135ea.svg @@ -1,54 +1,54 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/lecture_12/lab/893cdf9c.svg b/dev/lecture_12/lab/cb3fec90.svg similarity index 91% rename from dev/lecture_12/lab/893cdf9c.svg rename to dev/lecture_12/lab/cb3fec90.svg index 6829093b..abf73a12 100644 --- a/dev/lecture_12/lab/893cdf9c.svg +++ b/dev/lecture_12/lab/cb3fec90.svg @@ -1,52 +1,52 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/lecture_12/lab/index.html b/dev/lecture_12/lab/index.html index 71571dd4..0c40a916 100644 --- a/dev/lecture_12/lab/index.html +++ b/dev/lecture_12/lab/index.html @@ -55,7 +55,7 @@ (t,X) = solve(prob, Euler(0.2)) plot!(p1,t,X[1,:], color=1, lw=3, alpha=0.8, label="x Euler") -plot!(p1,t,X[2,:], color=2, lw=3, alpha=0.8, label="y Euler")Example block output

As you can see in the plot above, the Euler method quickly becomes quite inaccurate because we make a step in the direction of the tangent which inevitably leads us away from the perfect solution as shown in the plot below. euler

In the homework you will implement a Runge-Kutta solver to get a much better accuracy with the same step size.

Automating GaussNums

Next you will implement your own uncertainty propagation. In the lecture you have already seen the new number type that we need for this:

struct GaussNum{T<:Real} <: Real
+plot!(p1,t,X[2,:], color=2, lw=3, alpha=0.8, label="y Euler")
Example block output

As you can see in the plot above, the Euler method quickly becomes quite inaccurate because we make a step in the direction of the tangent which inevitably leads us away from the perfect solution as shown in the plot below. euler

In the homework you will implement a Runge-Kutta solver to get a much better accuracy with the same step size.

Automating GaussNums

Next you will implement your own uncertainty propagation. In the lecture you have already seen the new number type that we need for this:

struct GaussNum{T<:Real} <: Real
     μ::T
     σ::T
 end
@@ -68,12 +68,14 @@

Implement a helper function uncertain(f, args::GaussNum...) which takes a function f and its args and returns the resulting GaussNum with an uncertainty defined by the equation above.

Hint: You can compute the gradient of a function with Zygote, for example:

julia> using Zygote;
julia> f(x,y) = x*y;
julia> Zygote.gradient(f, 2., 3.)(3.0, 2.0)
Solution:

function uncertain(f, args::GaussNum...)
-    μs  = [x.μ for x in args]
-    dfs = Zygote.gradient(f,μs...)
-    σ = mapreduce(+, zip(dfs,args)) do (df,x)
+    μs  = (x.μ for x in args)
+    dfs = Zygote.gradient(f, μs...)
+
+    σ² = mapreduce(+, zip(dfs,args)) do (df,x)
         (df * x.σ)^2
-    end |> sqrt
-    GaussNum(f(μs...), σ)
+    end
+
+    GaussNum(f(μs...), sqrt(σ²))
 end

Now you can propagate uncertainties through any function like this:

julia> x1 = 2.0 ± 2.02.0 ± 2.0
julia> x2 = 2.0 ± 2.02.0 ± 2.0
julia> uncertain(*, x1, x2)4.0 ± 5.656854249492381

You can verify the correctness of your implementation by comparing to the manual implementation from the lecture.

Exercise

For convenience, implement the macro @register which will define the uncertainty propagation rule for a given function. E.g. for the function * the macro should generate code like below

Base.:*(args::GaussNum...) = uncertain(*, args...)

Hint: If you run into trouble with module names of functions you can make use of

julia> getmodule(f) = first(methods(f)).modulegetmodule (generic function with 1 method)
julia> getmodule(*)Base
@@ -128,7 +130,7 @@ plot!(p, t, mu.(X[1,:])) return p - enduncertainplot (generic function with 1 method)

julia> uncertainplot(t, X[1,:])Plot{Plots.GRBackend() n=2}

Unfortunately, with this approach, we would have to define things like uncertainplot! by hand. To make plotting GaussNums more pleasant we can make use of the @recipe macro from Plots.jl. It allows to define plot recipes for custom types (without having to depend on Plots.jl). Additionally, it makes it easiert to support all the different ways of creating plots (e.g. via plot or plot!, and with support for all keyword args) without having to overload tons of functions manually. If you want to read more about plot recipies in the docs of RecipesBase.jl. An example of a recipe for vectors of GaussNums could look like this:

@recipe function plot(ts::AbstractVector, xs::AbstractVector{<:GaussNum})
+       enduncertainplot (generic function with 1 method)

uncertainplot(t, X[1,:])
Example block output

Unfortunately, with this approach, we would have to define things like uncertainplot! and kwargs to the function by hand. To make plotting GaussNums more pleasant we can make use of the @recipe macro from Plots.jl. It allows to define plot recipes for custom types (without having to depend on Plots.jl). Additionally, it makes it easiert to support all the different ways of creating plots (e.g. via plot or plot!, and with support for all keyword args) without having to overload tons of functions manually. If you want to read more about plot recipies in the docs of RecipesBase.jl. An example of a recipe for vectors of GaussNums could look like this:

@recipe function plot(ts::AbstractVector, xs::AbstractVector{<:GaussNum})
     # you can set a default value for an attribute with `-->`
     # and force an argument with `:=`
     μs = [x.μ for x in xs]
@@ -151,4 +153,4 @@
 
 # now we can easily plot multiple things on to of each other
 p1 = plot(t, X[1,:], label="x", lw=3)
-plot!(p1, t, X[2,:], label="y", lw=3)
Example block output

References

+plot!(p1, t, X[2,:], label="y", lw=3)Example block output

References

diff --git a/dev/lecture_12/lecture/index.html b/dev/lecture_12/lecture/index.html index d196bd89..f2689bc3 100644 --- a/dev/lecture_12/lecture/index.html +++ b/dev/lecture_12/lecture/index.html @@ -72,4 +72,4 @@ setmean!(gop::GaussODEProblem,x::AbstractVector) = begin gop.mean.x0[gop.unc_in_u]=x[1:length(gop.unc_in_u)] gop.mean.θ[gop.unc_in_θ]=x[length(gop.unc_in_u).+[1:length(gop.unc_in_θ)]] -end

Constructor accepts an ODEProblem with uncertain numbers and converts it to GaussODEProblem:

  • goes through ODEProblem $x0$ and $θ$ fields and checks their types
  • replaces GaussNums in ODEProblem by ordinary numbers
  • remembers indices of GaussNum in $x0$ and $θ$
  • copies standard deviations in GaussNum to $sqΣ0$
+end

Constructor accepts an ODEProblem with uncertain numbers and converts it to GaussODEProblem:

  • goes through ODEProblem $x0$ and $θ$ fields and checks their types
  • replaces GaussNums in ODEProblem by ordinary numbers
  • remembers indices of GaussNum in $x0$ and $θ$
  • copies standard deviations in GaussNum to $sqΣ0$
diff --git a/dev/projects/index.html b/dev/projects/index.html index 456e0c6b..4ccabddd 100644 --- a/dev/projects/index.html +++ b/dev/projects/index.html @@ -14,4 +14,4 @@ │ └── Manifest.toml # usually not committed to git as it is generated on the fly ├── README.md # describes in short what the pkg does and how to install pkg (e.g. some external deps) and run the example ├── Project.toml # lists all the pkg dependencies -└── Manifest.toml # usually not committed to git as the requirements may be to restrictive

The first thing that we will look at is README.md, which should warn us if there are some special installation steps, that cannot be handled with Julia's Pkg system. For example if some 3rd party binary dependency with license is required. Secondly we will try to run tests in the test folder, which should run and not fail and should cover at least some functionality of the pkg. Thirdly and most importantly we will instantiate environment in scripts and test if the example runs correctly. Lastly we will focus on documentation in terms of code readability, docstrings and inline comments.

Only after all this we may look at the extent of the project and it's difficulty, which may help us in deciding between grades.

Nice to have things, which are not strictly required but obviously improves the score.

  • Ideally the project should be hosted on GitHub, which could have the continuous integration/testing set up.
  • Include some benchmark and profiling code in your examples, which can show us how well you have dealt with the question of performance.
  • Some parallelization attempts either by multi-processing, multi-threadding, or CUDA. Do not forget to show the improvement.
  • Documentation with a webpage using Documenter.jl.

Here are some examples of how the project could look like:

+└── Manifest.toml # usually not committed to git as the requirements may be to restrictive

The first thing that we will look at is README.md, which should warn us if there are some special installation steps, that cannot be handled with Julia's Pkg system. For example if some 3rd party binary dependency with license is required. Secondly we will try to run tests in the test folder, which should run and not fail and should cover at least some functionality of the pkg. Thirdly and most importantly we will instantiate environment in scripts and test if the example runs correctly. Lastly we will focus on documentation in terms of code readability, docstrings and inline comments.

Only after all this we may look at the extent of the project and it's difficulty, which may help us in deciding between grades.

Nice to have things, which are not strictly required but obviously improves the score.

  • Ideally the project should be hosted on GitHub, which could have the continuous integration/testing set up.
  • Include some benchmark and profiling code in your examples, which can show us how well you have dealt with the question of performance.
  • Some parallelization attempts either by multi-processing, multi-threadding, or CUDA. Do not forget to show the improvement.
  • Documentation with a webpage using Documenter.jl.

Here are some examples of how the project could look like:

diff --git a/dev/search_index.js b/dev/search_index.js index e94f85ad..1d14d8f5 100644 --- a/dev/search_index.js +++ b/dev/search_index.js @@ -1,3 +1,3 @@ var documenterSearchIndex = {"docs": -[{"location":"lecture_08/hw/#hw08","page":"Homework","title":"Homework 08","text":"","category":"section"},{"location":"lecture_08/hw/","page":"Homework","title":"Homework","text":"In this homework you will write an additional rule for our scalar reverse AD from the lab. For this homework, please write all your code in one file hw.jl which you have to zip and upload to BRUTE as usual. The solution to the lab is below.","category":"page"},{"location":"lecture_08/hw/","page":"Homework","title":"Homework","text":"mutable struct TrackedReal{T<:Real}\n data::T\n grad::Union{Nothing,T}\n children::Dict\n # this field is only need for printing the graph. you can safely remove it.\n name::String\nend\n\ntrack(x::Real,name=\"\") = TrackedReal(x,nothing,Dict(),name)\n\nfunction Base.show(io::IO, x::TrackedReal)\n t = isempty(x.name) ? \"(tracked)\" : \"(tracked $(x.name))\"\n print(io, \"$(x.data) $t\")\nend\n\nfunction accum!(x::TrackedReal)\n if isnothing(x.grad)\n x.grad = sum(w*accum!(v) for (v,w) in x.children)\n end\n x.grad\nend\n\nfunction Base.:*(a::TrackedReal, b::TrackedReal)\n z = track(a.data * b.data, \"*\")\n a.children[z] = b.data # dz/da=b\n b.children[z] = a.data # dz/db=a\n z\nend\n\nfunction Base.:+(a::TrackedReal{T}, b::TrackedReal{T}) where T\n z = track(a.data + b.data, \"+\")\n a.children[z] = one(T)\n b.children[z] = one(T)\n z\nend\n\nfunction Base.sin(x::TrackedReal)\n z = track(sin(x.data), \"sin\")\n x.children[z] = cos(x.data)\n z\nend\n\nfunction gradient(f, args::Real...)\n ts = track.(args)\n y = f(ts...)\n y.grad = 1.0\n accum!.(ts)\nend","category":"page"},{"location":"lecture_08/hw/","page":"Homework","title":"Homework","text":"We will use it to compute the derivative of the Babylonian square root.","category":"page"},{"location":"lecture_08/hw/","page":"Homework","title":"Homework","text":"babysqrt(x, t=(1+x)/2, n=10) = n==0 ? t : babysqrt(x, (t+x/t)/2, n-1)\nnothing # hide","category":"page"},{"location":"lecture_08/hw/","page":"Homework","title":"Homework","text":"In order to differentiate through babysqrt you will need a reverse rule for / for Base.:/(TrackedReal,TrackedReal) as well as the cases where you divide with constants in volved (e.g. Base.:/(TrackedReal,Real)).","category":"page"},{"location":"lecture_08/hw/","page":"Homework","title":"Homework","text":"
\n
Homework (2 points)
\n
","category":"page"},{"location":"lecture_08/hw/","page":"Homework","title":"Homework","text":"Write the reverse rules for / and the missing rules for + such that you can differentiate through division and addition with and without constants.","category":"page"},{"location":"lecture_08/hw/","page":"Homework","title":"Homework","text":"
","category":"page"},{"location":"lecture_08/hw/","page":"Homework","title":"Homework","text":"function Base.:/(a::TrackedReal, b::TrackedReal)\n z = track(a.data / b.data)\n a.children[z] = 1/b.data\n b.children[z] = -a.data / b.data^2\n z\nend\nfunction Base.:/(a::TrackedReal, b::Real)\n z = track(a.data/b)\n a.children[z] = 1/b\n z\nend\n\nfunction Base.:+(a::Real, b::TrackedReal{T}) where T\n z = track(a + b.data, \"+\")\n b.children[z] = one(T)\n z\nend\nBase.:+(a::TrackedReal,b::Real) = b+a","category":"page"},{"location":"lecture_08/hw/","page":"Homework","title":"Homework","text":"You can verify your solution with the gradient function.","category":"page"},{"location":"lecture_08/hw/","page":"Homework","title":"Homework","text":"gradient(babysqrt, 2.0)\n1/(2babysqrt(2.0))","category":"page"},{"location":"lecture_04/lab/#Lab-04:-Packaging","page":"Lab","title":"Lab 04: Packaging","text":"","category":"section"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"projdir = dirname(Base.active_project())\ninclude(joinpath(projdir,\"src\",\"lecture_03\",\"Lab03Ecosystem.jl\"))\n\nfunction find_food(a::Animal, w::World)\n as = filter(x -> eats(a,x), w.agents |> values |> collect)\n isempty(as) ? nothing : sample(as)\nend\neats(::Animal{Sheep},g::Plant{Grass}) = g.size > 0\neats(::Animal{Wolf},::Animal{Sheep}) = true\neats(::Agent,::Agent) = false","category":"page"},{"location":"lecture_04/lab/#Warmup-Stepping-through-time","page":"Lab","title":"Warmup - Stepping through time","text":"","category":"section"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"We now have all necessary functions in place to make agents perform one step of our simulation. At the beginning of each step an animal looses energy. Afterwards it tries to find some food, which it will subsequently eat. If the animal then has less than zero energy it dies and is removed from the world. If it has positive energy it will try to reproduce.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"Plants have a simpler life. They simply grow if they have not reached their maximal size.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"Implement a method agent_step!(::Animal,::World) which performs the following steps:\nDecrement E of agent by 1.0.\nWith p_f, try to find some food and eat it.\nIf E0, the animal dies.\nWith p_r, try to reproduce.\nImplement a method agent_step!(::Plant,::World) which performs the following steps:\nIf the size of the plant is smaller than max_size, increment the plant's size by one.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"function agent_step!(p::Plant, w::World)\n if p.size < p.max_size\n p.size += 1\n end\nend\n\nfunction agent_step!(a::Animal, w::World)\n a.energy -= 1\n if rand() <= a.foodprob\n dinner = find_food(a,w)\n eat!(a, dinner, w)\n end\n if a.energy < 0\n kill_agent!(a,w)\n return\n end\n if rand() <= a.reprprob\n reproduce!(a,w)\n end\nend\n\nnothing # hide","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"An agent_step! of a sheep in a world with a single grass should make it consume the grass, let it reproduce, and eventually die if there is no more food and its energy is at zero:","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"sheep = Sheep(1,2.0,2.0,1.0,1.0,male);\ngrass = Grass(2,2,2);\nworld = World([sheep, grass])\nagent_step!(sheep, world); world\n# NOTE: The second agent step leads to an error.\n# Can you figure out what is the problem here?\nagent_step!(sheep, world); world","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"Finally, lets implement a function world_step! which performs one agent_step! for each agent. Note that simply iterating over all agents could lead to problems because we are mutating the agent dictionary. One solution for this is to iterate over a copy of all agent IDs that are present when starting to iterate over agents. Additionally, it could happen that an agent is killed by another one before we apply agent_step! to it. To solve this you can check if a given ID is currently present in the World.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"# make it possible to eat nothing\neat!(::Animal, ::Nothing, ::World) = nothing\n\nfunction world_step!(world::World)\n # make sure that we only iterate over IDs that already exist in the\n # current timestep this lets us safely add agents\n ids = copy(keys(world.agents))\n\n for id in ids\n # agents can be killed by other agents, so make sure that we are\n # not stepping dead agents forward\n !haskey(world.agents,id) && continue\n\n a = world.agents[id]\n agent_step!(a,world)\n end\nend","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"w = World([Sheep(1), Sheep(2), Wolf(3)])\nworld_step!(w); w\nworld_step!(w); w\nworld_step!(w); w","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"Finally, lets run a few simulation steps and plot the solution","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"n_grass = 1_000\nn_sheep = 40\nn_wolves = 4\n\ngs = [Grass(id) for id in 1:n_grass]\nss = [Sheep(id) for id in (n_grass+1):(n_grass+n_sheep)]\nws = [Wolf(id) for id in (n_grass+n_sheep+1):(n_grass+n_sheep+n_wolves)]\nw = World(vcat(gs,ss,ws))\n\ncounts = Dict(n=>[c] for (n,c) in agent_count(w))\nfor _ in 1:100\n world_step!(w)\n for (n,c) in agent_count(w)\n push!(counts[n],c)\n end\nend\n\nusing Plots\nplt = plot()\nfor (n,c) in counts\n plot!(plt, c, label=string(n), lw=2)\nend\nplt","category":"page"},{"location":"lecture_04/lab/#Package:-Ecosystem.jl","page":"Lab","title":"Package: Ecosystem.jl","text":"","category":"section"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"In the main section of this lab you will create your own Ecosystem.jl package to organize and test (!) the code that we have written so far.","category":"page"},{"location":"lecture_04/lab/#PkgTemplates.jl","page":"Lab","title":"PkgTemplates.jl","text":"","category":"section"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"The simplest way to create a new package in Julia is to use PkgTemplates.jl. ]add PkgTemplates to your global julia env and create a new package by running:","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"using PkgTemplates\nTemplate(interactive=true)(\"Ecosystem\")","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"to interactively specify various options for your new package or use the following snippet to generate it programmatically:","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"using PkgTemplates\n\n# define the package template\ntemplate = Template(;\n user = \"GithubUserName\", # github user name\n authors = [\"Author1\", \"Author2\"], # list of authors\n dir = \"/path/to/folder/\", # dir in which the package will be created\n julia = v\"1.8\", # compat version of Julia\n plugins = [\n !CompatHelper, # disable CompatHelper\n !TagBot, # disable TagBot\n Readme(; inline_badges = true), # added readme file with badges\n Tests(; project = true), # added Project.toml file for unit tests\n Git(; manifest = false), # add manifest.toml to .gitignore\n License(; name = \"MIT\") # addedMIT licence\n ],\n)\n\n# execute the package template (this creates all files/folders)\ntemplate(\"Ecosystem\")","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"This should have created a new folder Ecosystem which looks like below.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":".\n├── LICENSE\n├── Project.toml\n├── README.md\n├── src\n│ └── Ecosystem.jl\n└── test\n ├── Manifest.toml\n ├── Project.toml\n └── runtests.jl","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"If you ]activate /path/to/Ecosystem you should be able to run ]test to run the autogenerated test (which is not doing anything) and get the following output:","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"(Ecosystem) pkg> test\n Testing Ecosystem\n Status `/private/var/folders/6h/l9_skfms2v3dt8z3zfnd2jr00000gn/T/jl_zd5Uai/Project.toml`\n [e77cd98c] Ecosystem v0.1.0 `~/repos/Ecosystem`\n [8dfed614] Test `@stdlib/Test`\n Status `/private/var/folders/6h/l9_skfms2v3dt8z3zfnd2jr00000gn/T/jl_zd5Uai/Manifest.toml`\n [e77cd98c] Ecosystem v0.1.0 `~/repos/Ecosystem`\n [2a0f44e3] Base64 `@stdlib/Base64`\n [b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`\n [56ddb016] Logging `@stdlib/Logging`\n [d6f4376e] Markdown `@stdlib/Markdown`\n [9a3f8284] Random `@stdlib/Random`\n [ea8e919c] SHA v0.7.0 `@stdlib/SHA`\n [9e88b42a] Serialization `@stdlib/Serialization`\n [8dfed614] Test `@stdlib/Test`\n Testing Running tests...\nTest Summary: |Time\nEcosystem.jl | None 0.0s\n Testing Ecosystem tests passed ","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"warning: Warning\nFrom now on make sure that you always have the Ecosystem enviroment enabled. Otherwise you will not end up with the correct dependencies in your packages","category":"page"},{"location":"lecture_04/lab/#Adding-content-to-Ecosystem.jl","page":"Lab","title":"Adding content to Ecosystem.jl","text":"","category":"section"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"Next, let's add the types and functions we have defined so far. You can use include(\"path/to/file.jl\") in the main module file at src/Ecosystem.jl to bring some structure in your code. An exemplary file structure could look like below.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":".\n├── LICENSE\n├── Manifest.toml\n├── Project.toml\n├── README.md\n├── src\n│ ├── Ecosystem.jl\n│ ├── animal.jl\n│ ├── plant.jl\n│ └── world.jl\n└── test\n └── runtests.jl","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"While you are adding functionality to your package you can make great use of Revise.jl. Loading Revise.jl before your Ecosystem.jl will automatically recompile (and invalidate old methods!) while you develop. You can install it in your global environment and and create a $HOME/.config/startup.jl which always loads Revise. It can look like this:","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"# try/catch block to make sure you can start julia if Revise should not be installed\ntry\n using Revise\ncatch e\n @warn(e.msg)\nend","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"warning: Warning\nAt some point along the way you should run into problems with the sample functions or when trying using StatsBase. This is normal, because you have not added the package to the Ecosystem environment yet. Adding it is as easy as ]add StatsBase. Your Ecosystem environment should now look like this:(Ecosystem) pkg> status\nProject Ecosystem v0.1.0\nStatus `~/repos/Ecosystem/Project.toml`\n [2913bbd2] StatsBase v0.33.21","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"In order to use your new types/functions like below","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"using Ecosystem\n\nSheep(2)","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"you have to export them from your module. Add exports for all important types and functions.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"# src/Ecosystem.jl\nmodule Ecosystem\n\nusing StatsBase\n\nexport World\nexport Species, PlantSpecies, AnimalSpecies, Grass, Sheep, Wolf\nexport Agent, Plant, Animal\nexport agent_step!, eat!, eats, find_food, reproduce!, world_step!, agent_count\n\n# ....\n\nend","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_04/lab/#Unit-tests","page":"Lab","title":"Unit tests","text":"","category":"section"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"Every package should have tests which verify the correctness of your implementation, such that you can make changes to your codebase and remain confident that you did not break anything.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"Julia's Test package provides you functionality to easily write unit tests.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"In the file test/runtests.jl, create a new @testset and write three @tests which check that the show methods we defined for Grass, Sheep, and Wolf work as expected.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"The function repr(x) == \"some string\" to check if the string representation we defined in the Base.show overload returns what you expect.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"# using Ecosystem\nusing Test\n\n@testset \"Base.show\" begin\n g = Grass(1,1,1)\n s = Animal{Sheep}(2,1,1,1,1,male)\n w = Animal{Wolf}(3,1,1,1,1,female)\n @test repr(g) == \"🌿 #1 100% grown\"\n @test repr(s) == \"🐑♂ #2 E=1.0 ΔE=1.0 pr=1.0 pf=1.0\"\n @test repr(w) == \"🐺♀ #3 E=1.0 ΔE=1.0 pr=1.0 pf=1.0\"\nend","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_04/lab/#Github-CI","page":"Lab","title":"Github CI","text":"","category":"section"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"If you want you can upload you package to Github and add the julia-runtest Github Action to automatically test your code for every new push you make to the repository.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
","category":"page"},{"location":"lecture_03/lab/#lab03","page":"Lab","title":"Lab 3: Predator-Prey Agents","text":"","category":"section"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"projdir = dirname(Base.active_project())\ninclude(joinpath(projdir,\"src\",\"lecture_02\",\"Lab02Ecosystem.jl\"))","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"In this lab we will look at two different ways of extending our agent simulation to take into account that animals can have two different sexes: female and male.","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"In the first part of the lab you will re-use the code from lab 2 and create a new type of sheep (⚥Sheep) which has an additional field sex. In the second part you will redesign the type hierarchy from scratch using parametric types to make this agent system much more flexible and julian.","category":"page"},{"location":"lecture_03/lab/#Part-I:-Female-and-Male-Sheep","page":"Lab","title":"Part I: Female & Male Sheep","text":"","category":"section"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"The code from lab 2 that you will need in the first part of this lab can be found here.","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"The goal of the first part of the lab is to demonstrate the forwarding method (which is close to how things are done in OOP) by implementing a sheep that can have two different sexes and can only reproduce with another sheep of opposite sex.","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"This new type of sheep needs an additonal field sex::Symbol which can be either :male or :female. In OOP we would simply inherit from Sheep and create a ⚥Sheep with an additional field. In Julia there is no inheritance - only subtyping of abstract types. As you cannot inherit from a concrete type in Julia, we will have to create a wrapper type and forward all necessary methods. This is typically a sign of unfortunate type tree design and should be avoided, but if you want to extend a code base by an unforeseen type this forwarding of methods is a nice work-around. Our ⚥Sheep type will simply contain a classic sheep and a sex field","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"struct ⚥Sheep <: Animal\n sheep::Sheep\n sex::Symbol\nend\n⚥Sheep(id, e=4.0, Δe=0.2, pr=0.8, pf=0.6, sex=rand(Bool) ? :female : :male) = ⚥Sheep(Sheep(id,e,Δe,pr,pf),sex)\nnothing # hide","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"sheep = ⚥Sheep(1)\nsheep.sheep\nsheep.sex","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Instead of littering the whole code with custom getters/setters Julia allows us to overload the sheep.field behaviour by implementing custom getproperty/setproperty! methods.","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Implement custom getproperty/setproperty! methods which allow to access the Sheep inside the ⚥Sheep as if we would not be wrapping it.","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"# NOTE: the @forward macro we will discuss in a later lecture is based on this\n\nfunction Base.getproperty(s::⚥Sheep, name::Symbol)\n if name in fieldnames(Sheep)\n getfield(s.sheep,name)\n else\n getfield(s,name)\n end\nend\n\nfunction Base.setproperty!(s::⚥Sheep, name::Symbol, x)\n if name in fieldnames(Sheep)\n setfield!(s.sheep,name,x)\n else\n setfield!(s,name,x)\n end\nend","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"You should be able to do the following with your overloads now","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"sheep = ⚥Sheep(1)\nsheep.id\nsheep.sex\nsheep.energy += 1\nsheep","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"In order to make the ⚥Sheep work with the rest of the code we only have to forward the eat! method","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"eat!(s::⚥Sheep, food, world) = eat!(s.sheep, food, world);\nsheep = ⚥Sheep(1);\ngrass = Grass(2);\nworld = World([sheep,grass])\neat!(sheep, grass, world)","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"and implement a custom reproduce! method with the behaviour that we want.","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"However, the extension of Sheep to ⚥Sheep is a very object-oriented approach. With a little bit of rethinking, we can build a much more elegant solution that makes use of Julia's powerful parametric types.","category":"page"},{"location":"lecture_03/lab/#Part-II:-A-new,-parametric-type-hierarchy","page":"Lab","title":"Part II: A new, parametric type hierarchy","text":"","category":"section"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"First, let us note that there are two fundamentally different types of agents in our world: animals and plants. All species such as grass, sheep, wolves, etc. can be categorized as one of those two. We can use Julia's powerful, parametric type system to define one large abstract type for all agents Agent{S}. The Agent will either be an Animal or a Plant with a type parameter S which will represent the specific animal/plant species we are dealing with.","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"This new type hiearchy can then look like this:","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"abstract type Species end\n\nabstract type PlantSpecies <: Species end\nabstract type Grass <: PlantSpecies end\n\nabstract type AnimalSpecies <: Species end\nabstract type Sheep <: AnimalSpecies end\nabstract type Wolf <: AnimalSpecies end\n\nabstract type Agent{S<:Species} end\n\n# instead of Symbols we can use an Enum for the sex field\n# using an Enum here makes things easier to extend in case you\n# need more than just binary sexes and is also more explicit than\n# just a boolean\n@enum Sex female male","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"mutable struct World{A<:Agent}\n agents::Dict{Int,A}\n max_id::Int\nend\n\nfunction World(agents::Vector{<:Agent})\n max_id = maximum(a.id for a in agents)\n World(Dict(a.id=>a for a in agents), max_id)\nend\n\n# optional: overload Base.show\nfunction Base.show(io::IO, w::World)\n println(io, typeof(w))\n for (_,a) in w.agents\n println(io,\" $a\")\n end\nend","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Now we can create a concrete type Animal with the two parametric types and the fields that we already know from lab 2.","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"mutable struct Animal{A<:AnimalSpecies} <: Agent{A}\n const id::Int\n energy::Float64\n const Δenergy::Float64\n const reprprob::Float64\n const foodprob::Float64\n const sex::Sex\nend","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"To create an instance of Animal we have to specify the parametric type while constructing it","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Animal{Wolf}(1,5,5,1,1,female)","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Note that we now automatically have animals of any species without additional work. Starting with the overload of the show method we can already see that we can abstract away a lot of repetitive work into the type system. We can implement one single show method for all animal species!","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Implement Base.show(io::IO, a::Animal) with a single method for all Animals. You can get the pretty (unicode) printing of the Species types with another overload like this: Base.show(io::IO, ::Type{Sheep}) = print(io,\"🐑\")","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"function Base.show(io::IO, a::Animal{A}) where {A<:AnimalSpecies}\n e = a.energy\n d = a.Δenergy\n pr = a.reprprob\n pf = a.foodprob\n s = a.sex == female ? \"♀\" : \"♂\"\n print(io, \"$A$s #$(a.id) E=$e ΔE=$d pr=$pr pf=$pf\")\nend\n\n# note that for new species/sexes we will only have to overload `show` on the\n# abstract species types like below!\nBase.show(io::IO, ::Type{Sheep}) = print(io,\"🐑\")\nBase.show(io::IO, ::Type{Wolf}) = print(io,\"🐺\")","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Unfortunately we have lost the convenience of creating plants and animals by simply calling their species constructor. For example, Sheep is just an abstract type that we cannot instantiate. However, we can manually define a new constructor that will give us this convenience back. This is done in exactly the same way as defining a constructor for a concrete type:","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Sheep(id,E,ΔE,pr,pf,s=rand(Sex)) = Animal{Sheep}(id,E,ΔE,pr,pf,s)","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Ok, so we have a constructor for Sheep now. But what about all the other billions of species that you want to define in your huge master thesis project of ecosystem simulations? Do you have to write them all by hand? Do not despair! Julia has you covered.","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Overload all AnimalSpecies types with a constructor. You already know how to write constructors for specific types such as Sheep. Can you manage to sneak in a type variable? Maybe with Type?","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"function (A::Type{<:AnimalSpecies})(id::Int,E::T,ΔE::T,pr::T,pf::T,s::Sex) where T\n Animal{A}(id,E,ΔE,pr,pf,s)\nend\n\n# get the per species defaults back\nrandsex() = rand(instances(Sex))\nSheep(id; E=4.0, ΔE=0.2, pr=0.8, pf=0.6, s=randsex()) = Sheep(id, E, ΔE, pr, pf, s)\nWolf(id; E=10.0, ΔE=8.0, pr=0.1, pf=0.2, s=randsex()) = Wolf(id, E, ΔE, pr, pf, s)\nnothing # hide","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"We have our convenient, high-level behaviour back!","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Sheep(1)\nWolf(2)","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Check the methods for eat! and kill_agent! which involve Animals and update their type signatures such that they work for the new type hiearchy.","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"function eat!(wolf::Animal{Wolf}, sheep::Animal{Sheep}, w::World)\n wolf.energy += sheep.energy * wolf.Δenergy\n kill_agent!(sheep,w)\nend\n\n# no change\n# eat!(::Animal, ::Nothing, ::World) = nothing\n\n# no change\n# kill_agent!(a::Agent, w::World) = delete!(w.agents, a.id)\n\neats(::Animal{Wolf},::Animal{Sheep}) = true\neats(::Agent,::Agent) = false\n# this one needs to wait until we have `Plant`s\n# eats(::Animal{Sheep},g::Plant{Grass}) = g.size > 0\n\nnothing # hide","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Finally, we can implement the new behaviour for reproduce! which we wanted. Build a function which first finds an animal species of opposite sex and then lets the two reproduce (same behaviour as before).","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"mates(a::Animal{A}, b::Animal{A}) where A<:AnimalSpecies = a.sex != b.sex\nmates(::Agent, ::Agent) = false\n\nfunction find_mate(a::Animal, w::World)\n ms = filter(x->mates(x,a), w.agents |> values |> collect)\n isempty(ms) ? nothing : rand(ms)\nend\n\nfunction reproduce!(a::Animal{A}, w::World) where {A}\n m = find_mate(a,w)\n if !isnothing(m)\n a.energy = a.energy / 2\n vals = [getproperty(a,n) for n in fieldnames(Animal) if n ∉ [:id, :sex]]\n new_id = w.max_id + 1\n ŝ = Animal{A}(new_id, vals..., randsex())\n w.agents[ŝ.id] = ŝ\n w.max_id = new_id\n end\nend\nnothing # hide","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"s1 = Sheep(1, s=female)\ns2 = Sheep(2, s=male)\nw = World([s1, s2])\nreproduce!(s1, w); w","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Implement the type hiearchy we designed for Plants as well.","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"mutable struct Plant{P<:PlantSpecies} <: Agent{P}\n id::Int\n size::Int\n max_size::Int\nend\n\n# constructor for all Plant{<:PlantSpecies} callable as PlantSpecies(...)\n(A::Type{<:PlantSpecies})(id, s, m) = Plant{A}(id,s,m)\n(A::Type{<:PlantSpecies})(id, m) = (A::Type{<:PlantSpecies})(id,rand(1:m),m)\n\n# default specific for Grass\nGrass(id; max_size=10) = Grass(id, rand(1:max_size), max_size)\n\nfunction Base.show(io::IO, p::Plant{P}) where P\n x = p.size/p.max_size * 100\n print(io,\"$P #$(p.id) $(round(Int,x))% grown\")\nend\n\nBase.show(io::IO, ::Type{Grass}) = print(io,\"🌿\")\n\nfunction eat!(sheep::Animal{Sheep}, grass::Plant{Grass}, w::World)\n sheep.energy += grass.size * sheep.Δenergy\n grass.size = 0\nend\neats(::Animal{Sheep},g::Plant{Grass}) = g.size > 0\n\nnothing # hide","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"g = Grass(2)\ns = Sheep(3)\nw = World([g,s])\neat!(s,g,w); w","category":"page"},{"location":"lecture_12/lecture/#lec12","page":"Lecture","title":"Uncertainty Propagation in Ordinary Differential Equations","text":"","category":"section"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Differential equations are commonly used in science to describe many aspects of the physical world, ranging from dynamical systems and curves in space to complex multi-physics phenomena. ","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"As an example, consider a simple non-linear ordinary differential equation:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"beginalign\ndotx=alpha x-beta xydoty=-delta y+gamma xy \nendalign","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Which describes behavior of a predator-pray models in continuous times:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"x is the population of prey (sheep),\ny is the population of predator (wolfes)\nderivatives represent instantaneous growth rates of the populations\nt is the time and alpha beta gamma delta are parameters.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Can be written in vector arguments mathbfx=xy:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"fracdmathbfxdt=f(mathbfxtheta)","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"with arbitrary function f with vector of parameters theta.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"The first steps we may want to do with an ODE is to see it's evolution in time. The most simple approach is to discretize the time axis into steps: t = t_1 t_2 t_3 ldots t_T and evaluate solution at these points.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Replacing derivatives by differences:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"dot x leftarrow fracx_t-x_t-1Delta t","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"we can derive a general scheme (Euler solution):","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"mathbfx_t = mathbfx_t-1 + Deltat f(mathbfx_ttheta)","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"which can be written genericaly in julia :","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"\nfunction f(x,θ)\n α,β,γ,δ = θ\n x1,x2=x\n dx1 = α*x1 - β*x1*x2\n dx2 = δ*x1*x2 - γ*x2\n [dx1,dx2]\nend\n\nfunction solve(f,x0::AbstractVector,θ,dt,N)\n X = hcat([zero(x0) for i=1:N]...)\n X[:,1]=x0\n for t=1:N-1\n X[:,t+1]=X[:,t]+dt*f(X[:,t],θ)\n end\n X\nend","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Is simple and working (with sufficienty small dt):","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"(Image: )","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"ODE of this kind is an example of a \"complex\" simulation code that we may want to use, interact with, modify or incorporate into a more complex scheme.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"we will test how to re-define the elementary operations using custom types, automatic differentiation and automatic code generation\nwe will redefine the plotting operation to display the new type correctly\nwe will use composition to incorporate the ODE into a more complex solver","category":"page"},{"location":"lecture_12/lecture/#Uncertainty-propagation","page":"Lecture","title":"Uncertainty propagation","text":"","category":"section"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Prediction of the ODE model is valid only if all parameters and all initial conditions are accurate. This is almost never the case. While the number of sheep can be known, the number of wolfes in a forest is more uncertain. The same model holds for predator-prey in insects where the number of individuals can be only estimated.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Uncertain initial conditions:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"number of predators and prey given by a probability distribution \ninterval 0812 corresponds to uniform distribution U(0812)\ngaussian N(musigma), with mean mu and standard deviation sigma e.g. N(101)\nmore complicated distributions are more realistic (the number of animals is not negative!)","category":"page"},{"location":"lecture_12/lecture/#Ensemble-approach","page":"Lecture","title":"Ensemble approach","text":"","category":"section"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"The most simple approach is to represent distribution by an empirical density = discrete samples.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"p(mathbfx)approx frac1Ksum_k=1^K delta(mathbfx-mathbfx^(k))","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"In the case of a Gaussian, we just sample:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"K = 10\nX0 = [x0 .+ 0.1*randn(2) for _=1:K] # samples of initial conditions\nXens=[X=solve(f,X0[i],θ0,dt,N) for i=1:K] # solve multiple times","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"(can be implemented more elegantly using multiple dispatch on Vector{Vector})","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"(Image: )","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"While it is very simple and universal, it may become hard to interpret. ","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"What is the probability that it will higher than x_max?\nImproving accuracy with higher number of samples (expensive!)","category":"page"},{"location":"lecture_12/lecture/#Propagating-a-Gaussian","page":"Lecture","title":"Propagating a Gaussian","text":"","category":"section"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Propagation of uncertainty has been studied in many areas of science. Relation between accuracy and computational speed is always a tradeoff.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"A common appoach to propagation of uncertainty is linearized Gaussian:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"variable x is represented by gaussian N(musigma)\ntransformation of addition: x+asim N(mu+asigma)\ntransformation of multiplication: a*xsim N(a*mua*sigma)\ngeneral transformation approximated:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"g(x)sim N(g(mu)g(mu)*sigma)","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"This can be efficienty implemented in Julia:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"struct GNum{T} where T<:Real\n μ::T\n σ::T\nend\nimport Base: +, *\n+(x::GaussNum{T},a::T) where T =GaussNum(x.μ+a,x.σ)\n+(a::T,x::GaussNum{T}) where T =GaussNum(x.μ+a,x.σ)\n*(x::GaussNum{T},a::T) where T =GaussNum(x.μ*a,a*x.σ)\n*(a::T,x::GaussNum{T}) where T =GaussNum(x.μ*a,a*x.σ)","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"For the ODE we need multiplication of two Gaussians. Using Taylor expansion and neglecting covariances:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"g(x_1x_2)=Nleft(g(mu_1mu_2) sqrtleft(fracdgdx_1(mu_1mu_2)sigma_1right)^2 + left(fracdgdx_2(mu_1mu_2)sigma_2right)^2right)","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"which trivially applies to sum: x_1+x_2=N(mu_1+mu_2 sqrtsigma_1^2 + sigma_2^2)","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"+(x1::GaussNum{T},x2::GaussNum{T}) where T =GaussNum(x1.μ+x2.μ,sqrt(x1.σ.^2 + x2.σ.^2))\n*(x1::GaussNum{T},x2::GaussNum{T}) where T =GaussNum(x1.μ*x2.μ, sqrt(x2.μ*x1.σ.^2 + x1.μ*x2.σ.^2))\n","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Following the principle of defining the necessary functions on the type, we can make it pass through the ODE:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"it is necessary to define new initialization (functions zero)\ndefine nice-looking constructor ()\n±(a::T,b::T) where T: can be automated (macro, generated functions)","category":"page"},{"location":"lecture_12/lecture/#Flexibility","page":"Lecture","title":"Flexibility","text":"","category":"section"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"The great advantage of the former model was the ability to run an arbitrary code with uncertainty at an arbitrary number.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"For example, we may know the initial conditions, but do not know the parameter value.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"GX=solve(f,[1.0±0.1,1.0±0.1],[0.1±0.1,0.2,0.3,0.2],0.1,1000)","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"(Image: )","category":"page"},{"location":"lecture_12/lecture/#Disadvantage","page":"Lecture","title":"Disadvantage","text":"","category":"section"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"The result does not correspond to the ensemble version above.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"we have ignored the covariances\nextension to version with covariances is possible by keeping track of the correlations (Measurements.jl), where other variables are stored in a dictionary:\ncorrelations found by language manipulations\nvery flexible and easy-to-use\ndiscovering the covariances requires to build the covariance from ids. (Expensive if done too often).","category":"page"},{"location":"lecture_12/lecture/#Vector-uncertainty","page":"Lecture","title":"Vector uncertainty","text":"","category":"section"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"The previous simple approach ignores the covariances between variables. Even if we tract covariances linearly in the same fashion (Measurementsjl), the approach will suffer from a loss of precision under non-linearity. ","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"(Image: )","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"The linearization-based approach propogates through the non-linearity only the mean and models its neighborhood by a plane.\nPropagating all samples is too expensive\nMethods based on quadrature or cubature rules are a compromise","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"The cubature approach is based on moment matching:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"mu_g = int g(x) p(x) dx","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"for which is g(mu) poor approximation, corresponding to:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"mu_g = g(mu) = int g(x) delta(x-mu) dx","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"For Gaussian distribution, we can use a smarter integration rule, called the Gauss-Hermite quadrature:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"mu_g = int g(x) p(x) dx approx sum_j=1^J w_j g(x_j)","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"where x_j are prescribed quadrature points (see e.g. (Image: online tables))","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"In multivariate setting, the same problem is typically solved with the aim to reduce the computational cost to linear complexity with dimension. Most often aimimg at O(2d) complexity where d is the dimension of vector x.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"One of the most popular approaches today is based on cubature rules approximating the Gaussian in radial-spherical coordinates.","category":"page"},{"location":"lecture_12/lecture/#Cubature-rules","page":"Lecture","title":"Cubature rules","text":"","category":"section"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Consider Gaussian distribution with mean mu and covariance matrix Sigma that is positive definite with square root sqrtSigma, such that sqrtSigma sqrtSigma^T=Sigma. The quadrature pints are:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"x_i = mu + sqrtSigma q_i","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"beginalign\nq_1=sqrtdbeginbmatrix1\n0\nvdots\nendbmatrix\n\nq_2=sqrtdbeginbmatrix0\n1\nvdots\nendbmatrix ldots \n\nq_d+1=sqrtdbeginbmatrix-1\n0\nvdots\nendbmatrix\nq_d+2=sqrtdbeginbmatrix0\n-1\nvdots\nendbmatrix ldots\nendalign","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"that can be composed into a matrix Q=q_1ldots q_2d that is constant:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Q = sqrtd I_d -I_d","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"(Image: )","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Those quadrature points are in integration weighted by:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"w_i = frac12d i=1ldots2d","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"where d is dimension of the vectors.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"The quadrature points are propogated through the non-linearity in parallel (x_i=g(x_i)) and the resulting Gaussian distribution is:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"beginalign\nx sim N(muSigma)\nmu = frac12dsum_j=1^2d x_i\nSigma = frac12dsum_j=1^2d (x_i-mu)^T (x_i-mu)\nendalign","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"It is easy to check that if the sigma-points are propagated through an identity, they preserve the mean and variance. ","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"beginalign\nmu = frac12dsum_j=1^2d (mu + sqrtSigmaq_i)\n = frac12d(2dmu + sqrtSigma sum_j=1^2d (q_i)\n = mu\nendalign\n","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"For our example:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"(Image: )","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"only 4 trajectories propagated deterministically\ncan not be implemented using a single number type\nthe number of points to store is proportional to the dimension\nmanipulation requires operations from linear algebra\nmoving to representations in vector form\nsimple for initial conditions,\nhow to extend to operate also on parameters?","category":"page"},{"location":"lecture_12/lecture/#Smarter-implementation","page":"Lecture","title":"Smarter implementation","text":"","category":"section"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Easiest solution is to put the corresponding parts of the problem together:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"ode function f, \nits state x0,\nand parameters θ","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"can be wrapped into an ODEProblem","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"struct ODEProblem{F,T,X<:AbstractVector,P<:AbstractVector}\n f::F\n tspan::T\n x0::X\n θ::P\nend","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"the solver can operate on the ODEProbelm type","category":"page"},{"location":"lecture_12/lecture/#Unceratinty-propagation-in-vectors","page":"Lecture","title":"Unceratinty propagation in vectors","text":"","category":"section"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Example: consider uncertainty in state x_1x_2 and the first parameter theta_1. ","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Quick and dirty: ","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"getuncertainty(o::ODEProblem) = [o.u0[1:2];o.θ[1]]\nsetuncertainty!(o::ODEProblem,x::AbstractVector) = o.u0[1:2]=x[1:2],o.θ[1]=x[3]","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"and write a general Cubature solver using multiple dispatch.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Practical issues:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"how to check bounds? (Asserts)\nwhat if we provide an incompatible ODEProblem\ndefine a type that specifies the type of uncertainty? ","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"struct GaussODEProblem\n mean::ODEProblem\n unc_in_u # any indexing type accepted by to_index()\n unc_in_θ\n sqΣ0\nend","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"We can dispatch the cubature solver on GaussODEProblem and the ordinary solve on GaussODEProblem.OP internally.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"getmean(gop::GaussODEProblem) =[ gop.mean.x0[gop.unc_in_u];gop.mean.θ[gop.unc_in_θ]]\nsetmean!(gop::GaussODEProblem,x::AbstractVector) = begin \n gop.mean.x0[gop.unc_in_u]=x[1:length(gop.unc_in_u)]\n gop.mean.θ[gop.unc_in_θ]=x[length(gop.unc_in_u).+[1:length(gop.unc_in_θ)]] \nend","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Constructor accepts an ODEProblem with uncertain numbers and converts it to GaussODEProblem:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"goes through ODEProblem x0 and θ fields and checks their types\nreplaces GaussNums in ODEProblem by ordinary numbers\nremembers indices of GaussNum in x0 and θ\ncopies standard deviations in GaussNum to sqΣ0","category":"page"},{"location":"lecture_04/hw/#hw4","page":"Homework","title":"Homework 4","text":"","category":"section"},{"location":"lecture_04/hw/","page":"Homework","title":"Homework","text":"In this homework you will have to write two additional @testsets for the Ecosystem. One testset should be contained in a file test/sheep.jl and verify that the function eat!(::Animal{Sheep}, ::Plant{Grass}, ::World) works correctly. Another testset should be in the file test/wolf.jl and veryfiy that the function eat!(::Animal{Wolf}, ::Animal{Sheep}, ::World) works correctly.","category":"page"},{"location":"lecture_04/hw/#How-to-submit?","page":"Homework","title":"How to submit?","text":"","category":"section"},{"location":"lecture_04/hw/","page":"Homework","title":"Homework","text":"Zip the whole package folder Ecosystem.jl and upload it to BRUTE. The package has to include at least the following files:","category":"page"},{"location":"lecture_04/hw/","page":"Homework","title":"Homework","text":"├── src\n│ └── Ecosystem.jl\n└── test\n ├── sheep.jl # contains only a single @testset\n ├── wolf.jl # contains only a single @testset\n └── runtests.jl","category":"page"},{"location":"lecture_04/hw/","page":"Homework","title":"Homework","text":"Thet test/runtests.jl file can look like this:","category":"page"},{"location":"lecture_04/hw/","page":"Homework","title":"Homework","text":"using Test\nusing Ecosystem\n\ninclude(\"sheep.jl\")\ninclude(\"wolf.jl\")\n# ...","category":"page"},{"location":"lecture_04/hw/#Test-Sheep","page":"Homework","title":"Test Sheep","text":"","category":"section"},{"location":"lecture_04/hw/","page":"Homework","title":"Homework","text":"
\n
Homework:
\n
","category":"page"},{"location":"lecture_04/hw/","page":"Homework","title":"Homework","text":"Create a Sheep with food probability p_f=1\nCreate fully grown Grass and a World with the two agents.\nExecute eat!(::Animal{Sheep}, ::Plant{Grass}, ::World)\n@test that the size of the Grass now has size == 0","category":"page"},{"location":"lecture_04/hw/","page":"Homework","title":"Homework","text":"
","category":"page"},{"location":"lecture_04/hw/#Test-Wolf","page":"Homework","title":"Test Wolf","text":"","category":"section"},{"location":"lecture_04/hw/","page":"Homework","title":"Homework","text":"
\n
Homework:
\n
","category":"page"},{"location":"lecture_04/hw/","page":"Homework","title":"Homework","text":"Create a Wolf with food probability p_f=1\nCreate a Sheep and a World with the two agents.\nExecute eat!(::Animal{Wolf}, ::Animal{Sheep}, ::World)\n@test that the World only has one agent left in the agents dictionary","category":"page"},{"location":"lecture_04/hw/","page":"Homework","title":"Homework","text":"
","category":"page"},{"location":"lecture_02/lecture/#type_lecture","page":"Lecture","title":"Motivation","text":"","category":"section"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Before going into the details of Julia's type system, we will spend a few minutes motivating the roles of a type system, which are:","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Structuring code\nCommunicating to the compiler how a type will be used","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"The first aspect is important for the convenience of the programmer and enables abstractions in the language, the latter aspect is important for the speed of the generated code. Writing efficient Julia code is best viewed as a dialogue between the programmer and the compiler. [1] ","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Type systems according to Wikipedia:","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"In computer science and computer programming, a data type or simply type is an attribute of data which tells the compiler or interpreter how the programmer intends to use the data.\nA type system is a logical system comprising a set of rules that assigns a property called a type to the various constructs of a computer program, such as variables, expressions, functions or modules. These types formalize and enforce the otherwise implicit categories the programmer uses for algebraic data types, data structures, or other components.","category":"page"},{"location":"lecture_02/lecture/#Structuring-the-code-/-enforcing-the-categories","page":"Lecture","title":"Structuring the code / enforcing the categories","text":"","category":"section"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"The role of structuring the code and imposing semantic restriction means that the type system allows you to logically divide your program, and to prevent certain types of errors. Consider for example two types, Wolf and Sheep which share the same definition but the types have different names.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"struct Wolf\n name::String\n energy::Int\nend\n\nstruct Sheep\n name::String\n energy::Int\nend","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"This allows us to define functions applicable only to the corresponding type","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"howl(wolf::Wolf) = println(wolf.name, \" has howled.\")\nbaa(sheep::Sheep) = println(sheep.name, \" has baaed.\")\nnothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Therefore the compiler (or interpreter) enforces that a wolf can only howl and never baa and vice versa a sheep can only baa. In this sense, it ensures that howl(sheep) and baa(wolf) never happen.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"baa(Sheep(\"Karl\",3))\nbaa(Wolf(\"Karl\",3))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Notice the type of error of the latter call baa(Wolf(\"Karl\",3)). Julia raises MethodError which states that it has failed to find a function baa for the type Wolf (but there is a function baa for type Sheep).","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"For comparison, consider an alternative definition which does not have specified types","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"bark(animal) = println(animal.name, \" has howled.\")\nbaa(animal) = println(animal.name, \" has baaed.\")\nnothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"in which case the burden of ensuring that a wolf will never baa rests upon the programmer which inevitably leads to errors (note that severely constrained type systems are difficult to use).","category":"page"},{"location":"lecture_02/lecture/#Intention-of-use-and-restrictions-on-compilers","page":"Lecture","title":"Intention of use and restrictions on compilers","text":"","category":"section"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Types play an important role in generating efficient code by a compiler, because they tells the compiler which operations are permitted, prohibited, and can indicate invariants of type (e.g. constant size of an array). If compiler knows that something is invariant (constant), it can expoit such information. As an example, consider the following two alternatives to represent a set of animals:","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"a = [Wolf(\"1\", 1), Wolf(\"2\", 2), Sheep(\"3\", 3)]\nb = (Wolf(\"1\", 1), Wolf(\"2\", 2), Sheep(\"3\", 3))\nnothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"where a is an array which can contain arbitrary types and have arbitrary length whereas b is a Tuple which has fixed length in which the first two items are of type Wolf and the third item is of type Sheep. Moreover, consider a function which calculates the energy of all animals as","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"energy(animals) = mapreduce(x -> x.energy, +, animals)\nnothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"A good compiler makes use of the information provided by the type system to generate efficient code which we can verify by inspecting the compiled code using @code_native macro","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"@code_native debuginfo=:none energy(a)\n@code_native debuginfo=:none energy(b)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"one observes the second version produces more optimal code. Why is that?","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"In the first representation, a, the animals are stored in an Array{Any} which can have arbitrary size and can contain arbitrary animals. This means that the compiler has to compile energy(a) such that it works on such arrays.\nIn the second representation, b, the animals are stored in a Tuple, which specializes for lengths and types of items. This means that the compiler knows the number of animals and the type of each animal on each position within the tuple, which allows it to specialize.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"This difference will indeed have an impact on the time of code execution. On my i5-8279U CPU, the difference (as measured by BenchmarkTools) is","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"using BenchmarkTools\n@btime energy($(a))\n@btime energy($(b))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":" 70.2 ns (0 allocations: 0 bytes)\n 2.62 ns (0 allocations: 0 bytes)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Which nicely demonstrates that the choice of types affects performance. Does it mean that we should always use Tuples instead of Arrays? Surely not, it is just that each is better for different use-cases. Using Tuples means that the compiler will compile a special function for each length of tuple and each combination of types of items it contains, which is clearly wasteful.","category":"page"},{"location":"lecture_02/lecture/#type_system","page":"Lecture","title":"Julia's type system","text":"","category":"section"},{"location":"lecture_02/lecture/#Julia-is-dynamicaly-typed","page":"Lecture","title":"Julia is dynamicaly typed","text":"","category":"section"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Julia's type system is dynamic, which means that all types are resolved during runtime. But, if the compiler can infer types of all variables of the called function, it can specialize the function for that given type of variables which leads to efficient code. Consider a modified example where we represent two wolfpacks:","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"wolfpack_a = [Wolf(\"1\", 1), Wolf(\"2\", 2), Wolf(\"3\", 3)]\nwolfpack_b = Any[Wolf(\"1\", 1), Wolf(\"2\", 2), Wolf(\"3\", 3)]\nnothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"wolfpack_a carries a type Vector{Wolf} while wolfpack_b has the type Vector{Any}. This means that in the first case, the compiler knows that all items are of the type Wolfand it can specialize functions using this information. In case of wolfpack_b, it does not know which animal it will encounter (although all are of the same type), and therefore it needs to dynamically resolve the type of each item upon its use. This ultimately leads to less performant code.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"@benchmark energy($(wolfpack_a))\n@benchmark energy($(wolfpack_b))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":" 3.7 ns (0 allocations: 0 bytes)\n 69.4 ns (0 allocations: 0 bytes)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"To conclude, julia is indeed a dynamically typed language, but if the compiler can infer all types in a called function in advance, it does not have to perform the type resolution during execution, which produces performant code. This means and in hot (performance critical) parts of the code, you should be type stable, in other parts, it is not such big deal.","category":"page"},{"location":"lecture_02/lecture/#Classes-of-types","page":"Lecture","title":"Classes of types","text":"","category":"section"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Julia divides types into three classes: primitive, composite, and abstract.","category":"page"},{"location":"lecture_02/lecture/#Primitive-types","page":"Lecture","title":"Primitive types","text":"","category":"section"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Citing the documentation: A primitive type is a concrete type whose data consists of plain old bits. Classic examples of primitive types are integers and floating-point values. Unlike most languages, Julia lets you declare your own primitive types, rather than providing only a fixed set of built-in ones. In fact, the standard primitive types are all defined in the language itself.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"The definition of primitive types look as follows","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"primitive type Float16 <: AbstractFloat 16 end\nprimitive type Float32 <: AbstractFloat 32 end\nprimitive type Float64 <: AbstractFloat 64 end","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"and they are mainly used to jump-start julia's type system. It is rarely needed to define a special primitive type, as it makes sense only if you define special functions operating on its bits. This is almost excusively used for exposing special operations provided by the underlying CPU / LLVM compiler. For example + for Int32 is different from + for Float32 as they call a different intrinsic operations. You can inspect this jump-starting of the type system yourself by looking at Julia's source.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia> @which +(1,2)\n+(x::T, y::T) where T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8} in Base at int.jl:87","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"At int.jl:87","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"(+)(x::T, y::T) where {T<:BitInteger} = add_int(x, y)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"we see that + of integers is calling the function add_int(x, y), which is defined in the core part of the compiler in Intrinsics.cpp (yes, in C++), exposed in Core.Intrinsics","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"From Julia docs: Core is the module that contains all identifiers considered \"built in\" to the language, i.e. part of the core language and not libraries. Every module implicitly specifies using Core, since you can't do anything without those definitions.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Primitive types are rarely used, and they will not be used in this course. We mention them for the sake of completeness and refer the reader to the official Documentation (and source code of Julia).","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"An example of use of primitive type is a definition of one-hot vector in the library PrimitiveOneHot as ","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"primitive type OneHot{K} <: AbstractOneHotArray{1} 32 end","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"where K is the dimension of the one-hot vector. ","category":"page"},{"location":"lecture_02/lecture/#Abstract-types","page":"Lecture","title":"Abstract types","text":"","category":"section"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"An abstract type can be viewed as a set of concrete types. For example, an AbstractFloat represents the set of concrete types (BigFloat,Float64,Float32,Float16). This is used mainly to define general methods for sets of types for which we expect the same behavior (recall the Julia design motivation: if it quacks like a duck, waddles like a duck and looks like a duck, chances are it's a duck). Abstract types are defined with abstract type TypeName end. For example the following set of abstract types defines part of julia's number system.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"abstract type Number end\nabstract type Real <: Number end\nabstract type Complex <: Number end\nabstract type AbstractFloat <: Real end\nabstract type Integer <: Real end\nabstract type Signed <: Integer end\nabstract type Unsigned <: Integer end","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"where <: means \"is a subtype of\" and it is used in declarations where the right-hand is an immediate sypertype of a given type (Integer has the immediate supertype Real.) If the supertype is not supplied, it is considered to be Any, therefore in the above defition Number has the supertype Any. ","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"We can list childrens of an abstract type using function subtypes ``julia using InteractiveUtils: subtypes # hide subtypes(AbstractFloat)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"and we can also list the immediate `supertype` or climb the ladder all the way to `Any` using `supertypes`\n``julia\nusing InteractiveUtils: supertypes # hide\nsupertypes(AbstractFloat)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"supertype and subtypes print only types defined in Modules that are currently loaded to your workspace. For example with Julia without any Modules, subtypes(Number) returns [Complex, Real], whereas if I load Mods package implementing numbers defined over finite field, the same call returns [Complex, Real, AbstractMod].","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"It is relatively simple to print a complete type hierarchy of ","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"using AbstractTrees\nfunction AbstractTrees.children(t::Type)\n t === Function ? Vector{Type}() : filter!(x -> x !== Any,subtypes(t))\nend\nAbstractTrees.printnode(io::IO,t::Type) = print(io,t)\nprint_tree(Number)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"The main role of abstract types allows is in function definitions. They allow to define functions that can be used on variables with types with a given abstract type as a supertype. For example we can define a sgn function for all real numbers as","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"sgn(x::Real) = x > 0 ? 1 : x < 0 ? -1 : 0\nnothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"and we know it would be correct for all real numbers. This means that if anyone creates a new subtype of Real, the above function can be used. This also means that it is expected that comparison operations are defined for any real number. Also notice that Complex numbers are excluded, since they do not have a total order.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"For unsigned numbers, the sgn can be simplified, as it is sufficient to verify if they are different (greater) than zero, therefore the function can read","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"sgn(x::Unsigned) = x > 0 ? 1 : 0\nnothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"and again, it applies to all numbers derived from Unsigned. Recall that Unsigned <: Integer <: Real, how does Julia decide, which version of the function sgn to use for UInt8(0)? It chooses the most specific version, and thus for sgn(UInt8(0)) it will use sgn(x::Unsinged). If the compiler cannot decide, typically it encounters an ambiguity, it throws an error and recommends which function you should define to resolve it.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"The above behavior allows to define default \"fallback\" implementations and while allowing to specialize for sub-types. A great example is matrix multiplication, which has a generic (and slow) implementation with many specializations, which can take advantage of structure (sparse, banded), or use optimized implementations (e.g. blas implementation for dense matrices with eltype Float32 and Float64).","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Again, Julia does not make a difference between abstract types defined in Base libraries shipped with the language and those defined by you (the user). All are treated the same.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"From Julia documentation: Abstract types cannot be instantiated, which means that we cannot create a variable that would have an abstract type (try typeof(Number(1f0))). Also, abstract types cannot have any fields, therefore there is no composition (there are lengthy discussions of why this is so, one of the most definite arguments of creators is that abstract types with fields frequently lead to children types not using some fields (consider circle vs. ellipse)).","category":"page"},{"location":"lecture_02/lecture/#composite_types","page":"Lecture","title":"Composite types","text":"","category":"section"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Composite types are similar to struct in C (they even have the same memory layout) as they logically join together other types. It is not a great idea to think about them as objects (in OOP sense), because objects tie together data and functions on owned data. Contrary in Julia (as in C), functions operate on data of structures, but are not tied to them and they are defined outside them. Composite types are workhorses of Julia's type system, as user-defined types are mostly composite (or abstract).","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Composite types are defined using struct TypeName [fields] end. To define a position of an animal on the Euclidean plane as a type, we would write","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"struct PositionF64\n x::Float64\n y::Float64\nend","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"which defines a structure with two fields x and y of type Float64. Julia's compiler creates a default constructor, where both (but generally all) arguments are converted using (convert(Float64, x), convert(Float64, y) to the correct type. This means that we can construct a PositionF64 with numbers of different type that are convertable to Float64, e.g. PositionF64(1,1//2) but we cannot construct PositionF64 where the fields would be of different type (e.g. Int, Float32, etc.) or they are not trivially convertable (e.g. String).","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Fields in composite types do not have to have a specified type. We can define a VaguePosition without specifying the type","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"struct VaguePosition\n x\n y\nend","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"This works as the definition above except that the arguments are not converted to Float64 now. One can store different values in x and y, for example String (e.g. VaguePosition(\"Hello\",\"world\")). Although the above definition might be convenient, it limits the compiler's ability to specialize, as the type VaguePosition does not carry information about type of x and y, which has a negative impact on the performance. For example","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"using BenchmarkTools\nmove(a,b) = typeof(a)(a.x+b.x, a.y+b.y)\nx = [PositionF64(rand(), rand()) for _ in 1:100]\ny = [VaguePosition(rand(), rand()) for _ in 1:100]\n@benchmark reduce(move, $(x))\n@benchmark reduce(move, $(y))\nnothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Giving fields of a composite type an abstract type does not really solve the problem of the compiler not knowing the type. In this example, it still does not know, if it should use instructions for Float64 or Int8.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"``julia struct LessVaguePosition x::Real y::Real end","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"z = [LessVaguePosition(rand(), rand()) for _ in 1:100]; @benchmark reduce(move, z) nothing #hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nFrom the perspective of generating optimal code, both definitions are equally uninformative to the compiler as it cannot assume anything about the code. However, the `LessVaguePosition` will ensure that the position will contain only numbers, hence catching trivial errors like instantiating `VaguePosition` with non-numeric types for which arithmetic operators will not be defined (recall the discussion on the beginning of the lecture).\n\nAll structs defined above are immutable (as we have seen above in the case of `Tuple`), which means that one cannot change a field (unless the struct wraps a container, like and array, which allows that). For example this raises an error\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"a = LessVaguePosition(1,2) a.x = 2","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nIf one needs to make a struct mutable, use the keyword `mutable` before the keyword `struct` as\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia mutable struct MutablePosition x::Float64 y::Float64 end","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nIn mutable structures, we can change the values of fields.\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"a = MutablePosition(1e0, 2e0) a.x = 2; a","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nNote, that the memory layout of mutable structures is different, as fields now contain references to memory locations, where the actual values are stored (such structures cannot be allocated on stack, which increases the pressure on Garbage Collector).\n\nThe difference can be seen from ","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"a, b = PositionF64(1,2), PositionF64(1,2) @codenative debuginfo=:none move(a,b) a, b = MutablePosition(1,2), MutablePosition(1,2) @codenative debuginfo=:none move(a,b)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Why there is just one addition?\n\nAlso, the mutability is costly.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia x = [PositionF43(rand(), rand()) for _ in 1:100]; z = [MutablePosition(rand(), rand()) for _ in 1:100]; @benchmark reduce(move, x) @benchmark reduce(move, z)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n### Parametric types\nSo far, we had to trade-off flexibility for generality in type definitions. Can we have both? The answer is affirmative. The way to achieve this **flexibility** in definitions of the type while being able to generate optimal code is to **parametrize** the type definition. This is achieved by replacing types with a parameter (typically a single uppercase character) and decorating in definition by specifying different type in curly brackets. For example\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia struct PositionT{T} x::T y::T end u = [PositionT(rand(), rand()) for _ in 1:100] u = [PositionT(rand(Float32), rand(Float32)) for _ in 1:100]","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"@benchmark reduce(move, u) nothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nNotice that the compiler can take advantage of specializing for different types (which does not have an effect here as in modern processors addition of `Float` and `Int` takes the same time).\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia v = [PositionT(rand(1:100), rand(1:100)) for _ in 1:100] @benchmark reduce(move, v) nothing #hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nThe above definition suffers the same problem as `VaguePosition`, which is that it allows us to instantiate the `PositionT` with non-numeric types, e.g. `String`. We solve this by restricting the types `T` to be children of some supertype, in this case `Real`\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia struct Position{T<:Real} x::T y::T end","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nwhich will throw an error if we try to initialize it with `Position(\"1.0\", \"2.0\")`. Notice the flexibility we have achieved. We can use `Position` to store (and later compute) not only over `Float32` / `Float64` but any real numbers defined by other packages, for example with `Posit`s.\n``julia\nusing SoftPosit\nPosition(Posit8(3), Posit8(1))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"also notice that trying to construct the Position with different type of real numbers will fail, example Position(1f0,1e0)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Naturally, fields in structures can be of different types, as is in the below pointless example.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"``julia struct PositionXY{X<:Real, Y<:Real} x::X y::Y end","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nThe type can be parametrized by a concrete types. This is usefuyl to communicate the compiler some useful informations, for example size of arrays. \n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia struct PositionZ{T<:Real,Z} x::T y::T end","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"PositionZ{Int64,1}(1,2)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n\n### Abstract parametric types\nLike Composite types, Abstract types can also have parameters. These parameters define types that are common for all child types. A very good example is Julia's definition of arrays of arbitrary dimension `N` and type `T` of its items as","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia abstract type AbstractArray{T,N} end","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Different `T` and `N` give rise to different variants of `AbstractArrays`,\ntherefore `AbstractArray{Float32,2}` is different from `AbstractArray{Float64,2}`\nand from `AbstractArray{Float64,1}.` Note that these are still `Abstract` types,\nwhich means you cannot instantiate them. Their purpose is\n* to allow to define operations for broad class of concrete types\n* to inform the compiler about constant values, which can be used\nNotice in the above example that parameters of types do not have to be types, but can also be values of primitive types, as in the above example of `AbstractArray` `N` is the number of dimensions which is an integer value.\n\nFor convenience, it is common to give some important partially instantiated Abstract types an **alias**, for example `AbstractVector` as","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia const AbstractVector{T} = AbstractArray{T,1}","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"is defined in `array.jl:23` (in Julia 1.6.2), which allows us to define for example general prescription for the `dot` product of two abstract vectors as\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia function dot(a::AbstractVector, b::AbstractVector) @assert length(a) == length(b) mapreduce(*, +, a, b) end nothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nYou can verify that the above general function can be compiled to performant code if\nspecialized for particular arguments.\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia; ansicolor=true using InteractiveUtils: @codenative @codenative debuginfo=:none mapreduce(*,+, [1,2,3], [1,2,3])","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n## More on the use of types in function definitions\n### Terminology\nA *function* refers to a set of \"methods\" for a different combination of type parameters (the term function can be therefore considered as refering to a mere **name**). *Methods* define different behavior for different types of arguments for a given function. For example\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia move(a::Position, b::Position) = Position(a.x + b.x, a.y + b.y) move(a::Vector{<:Position}, b::Vector{<:Position}) = move.(a,b) nothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n`move` refers to a function with methods `move(a::Position, b::Position)` and `move(a::Vector{<:Position}, b::Vector{<:Position})`. When different behavior on different types is defined by a programmer, as shown above, it is also called *implementation specialization*. There is another type of specialization, called *compiler specialization*, which occurs when the compiler generates different functions for you from a single method. For example for\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"move(Position(1,1), Position(2,2)) move(Position(1.0,1.0), Position(2.0,2.0))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nthe compiler generates two methods, one for `Position{Int64}` and the other for `Position{Float64}`. Notice that inside generated functions, the compiler needs to use different intrinsic operations, which can be viewed from\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia; ansicolor=true @code_native debuginfo=:none move(Position(1,1), Position(2,2))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nand\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia; ansicolor=true @code_native debuginfo=:none move(Position(1.0,1.0), Position(2.0,2.0))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nNotice that `move` works on `Posits` defined in 3rd party libas well","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"move(Position(Posit8(1),Posit8(1)), Position(Posit8(2),Posit8(2)))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n## Intermezzo: How does the Julia compiler work?\nLet's walk through an example. Consider the following definitions\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia move(a::Position, by::Position) = Position(a.x + by.x, a.y + by.y) move(a::T, by::T) where {T<:Position} = Position(a.x + by.x, a.y + by.y) move(a::Position{Float64}, by::Position{Float64}) = Position(a.x + by.x, a.y + by.y) move(a::Vector{<:Position}, by::Vector{<:Position}) = move.(a, by) move(a::Vector{<:Position}, by::Position) = move.(a, by) nothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nand a function call\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia a = Position(1.0, 1.0) by = Position(2.0, 2.0) move(a, by)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n1. The compiler knows that you call the function `move`.\n2. The compiler infers the type of the arguments. You can view the result with\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"(typeof(a),typeof(by))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n3. The compiler identifies all `move`-methods with arguments of type `(Position{Float64}, Position{Float64})`:\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"wc = Base.getworldcounter() m = Base.method_instances(move, (typeof(a), typeof(by)), wc) m = first(m)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n4a. If the method has been specialized (compiled), then the arguments are prepared and the method is invoked. The compiled specialization can be seen from\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"m.cache","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n4b. If the method has not been specialized (compiled), the method is compiled for the given type of arguments and continues as in step 4a.\nA compiled function is therefore a \"blob\" of **native code** living in a particular memory location. When Julia calls a function, it needs to pick the right block corresponding to a function with particular type of parameters.\n\nIf the compiler cannot narrow the types of arguments to concrete types, it has to perform the above procedure inside the called function, which has negative effects on performance, as the type resulution and identification of the methods can be slow, especially for methods with many arguments (e.g. 30ns for a method with one argument,\n100 ns for method with two arguements). **You always want to avoid run-time resolution inside the performant loop!!!**\nRecall the above example\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia wolfpacka = [Wolf(\"1\", 1), Wolf(\"2\", 2), Wolf(\"3\", 3)] @benchmark energy($(Expr(:incomplete, \"incomplete: premature end of input\"))a))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"BenchmarkTools.Trial: 10000 samples with 991 evaluations. Range (min … max): 40.195 ns … 66.641 ns ┊ GC (min … max): 0.00% … 0.00% Time (median): 40.742 ns ┊ GC (median): 0.00% Time (mean ± σ): 40.824 ns ± 1.025 ns ┊ GC (mean ± σ): 0.00% ± 0.00%","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"▂▃ ▃▅▆▅▆█▅▅▃▂▂ ▂ ▇██████████████▇▇▅▅▁▅▄▁▅▁▄▄▃▄▅▄▅▃▅▃▅▁▃▁▄▄▃▁▁▅▃▃▄▃▄▃▄▆▆▇▇▇▇█ █ 40.2 ns Histogram: log(frequency) by time 43.7 ns <","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Memory estimate: 0 bytes, allocs estimate: 0.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nand\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia wolfpackb = Any[Wolf(\"1\", 1), Wolf(\"2\", 2), Wolf(\"3\", 3)] @benchmark energy($(Expr(:incomplete, \"incomplete: premature end of input\"))b))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"BenchmarkTools.Trial: 10000 samples with 800 evaluations. Range (min … max): 156.406 ns … 212.344 ns ┊ GC (min … max): 0.00% … 0.00% Time (median): 157.136 ns ┊ GC (median): 0.00% Time (mean ± σ): 158.114 ns ± 4.023 ns ┊ GC (mean ± σ): 0.00% ± 0.00%","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"▅█▆▅▄▂ ▃▂▁ ▂ ██████▆▇██████▇▆▇█▇▆▆▅▅▅▅▅▃▄▄▅▄▄▄▄▅▁▃▄▄▃▃▄▃▃▃▄▄▄▅▅▅▅▁▅▄▃▅▄▄▅▅ █ 156 ns Histogram: log(frequency) by time 183 ns <","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Memory estimate: 0 bytes, allocs estimate: 0.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nAn interesting intermediate between fully abstract and fully concrete type happens, when the compiler knows that arguments have abstract type, which is composed of a small number of concrete types. This case called Union-Splitting, which happens when there is just a little bit of uncertainty. Julia will do something like","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia argtypes = typeof(args) push!(executionstack, args) if T == Tuple{Int, Bool} @goto compiledblob1234 else # the only other option is Tuple{Float64, Bool} @goto compiledblob_1236 end","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"For example\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia const WolfOrSheep = Union{Wolf, Sheep} wolfpackc = WolfOrSheep[Wolf(\"1\", 1), Wolf(\"2\", 2), Wolf(\"3\", 3)] @benchmark energy($(Expr(:incomplete, \"incomplete: premature end of input\"))c))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"BenchmarkTools.Trial: 10000 samples with 991 evaluations. Range (min … max): 43.600 ns … 73.494 ns ┊ GC (min … max): 0.00% … 0.00% Time (median): 44.106 ns ┊ GC (median): 0.00% Time (mean ± σ): 44.279 ns ± 0.931 ns ┊ GC (mean ± σ): 0.00% ± 0.00%","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":" █ ▁ ▃","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"▂▂▂▆▃██▅▃▄▄█▅█▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▁▂▂▂▂▂▂▂▁▂▂▂▂▂▂▂▂▂▂▂▂▂ ▃ 43.6 ns Histogram: frequency by time 47.4 ns <","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Memory estimate: 0 bytes, allocs estimate: 0.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nThanks to union splitting, Julia is able to have performant operations on arrays with undefined / missing values for example\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"[1, 2, 3, missing] |> typeof","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n### More on matching methods and arguments\nIn the above process, the step, where Julia looks for a method instance with corresponding parameters can be very confusing. The rest of this lecture will focus on this. For those who want to have a formal background, we recommend [talk of Francesco Zappa Nardelli](https://www.youtube.com/watch?v=Y95fAipREHQ) and / or the one of [Jan Vitek](https://www.youtube.com/watch?v=LT4AP7CUMAw).\n\nWhen Julia needs to specialize a method instance, it needs to find it among multiple definitions. A single function can have many method instances, see for example `methods(+)` which lists all method instances of the `+`-function. How does Julia select the proper one?\n1. It finds all methods where the type of arguments match or are subtypes of restrictions on arguments in the method definition.\n2a. If there are multiple matches, the compiler selects the most specific definition.\n\n2b. If the compiler cannot decide, which method instance to choose, it throws an error.\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"confusedmove(a::Position{Float64}, by) = Position(a.x + by.x, a.y + by.y) confusedmove(a, by::Position{Float64}) = Position(a.x + by.x, a.y + by.y) confused_move(Position(1.0,2.0), Position(1.0,2.0))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n2c. If it cannot find a suitable method, it throws an error.\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"move(Position(1,2), VaguePosition(\"hello\",\"world\"))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nSome examples: Consider following definitions\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia move(a::Position, by::Position) = Position(a.x + by.x, a.y + by.y) move(a::T, by::T) where {T<:Position} = T(a.x + by.x, a.y + by.y) move(a::Position{Float64}, by::Position{Float64}) = Position(a.x + by.x, a.y + by.y) move(a::Vector{<:Position}, by::Vector{<:Position}) = move.(a, by) move(a::Vector{T}, by::Vector{T}) where {T<:Position} = move.(a, by) move(a::Vector{<:Position}, by::Position) = move.(a, by) nothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nWhich method will compiler select for\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"move(Position(1.0,2.0), Position(1.0,2.0))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nThe first three methods match the types of argumens, but the compiler will select the third one, since it is the most specific.\n\nWhich method will compiler select for\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"move(Position(1,2), Position(1,2))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nAgain, the first and second method definitions match the argument, but the second is the most specific.\n\nWhich method will the compiler select for\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"move([Position(1,2)], [Position(1,2)])","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nAgain, the fourth and fifth method definitions match the argument, but the fifth is the most specific.\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"move([Position(1,2), Position(1.0,2.0)], [Position(1,2), Position(1.0,2.0)])","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n### Frequent problems\n1. Why does the following fail?\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"foo(a::Vector{Real}) = println(\"Vector{Real}\") foo([1.0,2,3])","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nJulia's type system is **invariant**, which means that `Vector{Real}` is different from `Vector{Float64}` and from `Vector{Float32}`, even though `Float64` and `Float32` are sub-types of `Real`. Therefore `typeof([1.0,2,3])` isa `Vector{Float64}` which is not subtype of `Vector{Real}.` For **covariant** languages, this would be true. For more information on variance in computer languages, [see here](https://en.wikipedia.org/wiki/Covariance_and_contravariance_(computer_science)). If the above definition of `foo` should be applicable to all vectors which has elements of subtype of `Real` we have define it as\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia foo(a::Vector{T}) where {T<:Real} = println(\"Vector{T} where {T<:Real}\") nothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nor equivalently but more tersely as\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia foo(a::Vector{<:Real}) = println(\"Vector{T} where {T<:Real}\") nothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n2. Diagonal rule says that a repeated type in a method signature has to be a concrete type (this is to avoid ambinguity if the repeated type is used inside function definition to define a new variable to change type of variables). Consider for example the function below\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia move(a::T, b::T) where {T<:Position} = T(a.x + by.x, a.y + by.y) nothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nwe cannot call it with `move(Position(1.0,2.0), Position(1,2))`, since in this case `Position(1.0,2.0)` is of type `Position{Float64}` while `Position(1,2)` is of type `Position{Int64}`.\n3. When debugging why arguments do not match a particular method definition, it is useful to use `typeof`, `isa`, and `<:` commands. For example\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"typeof(Position(1.0,2.0)) typeof(Position(1,2)) Position(1,2) isa Position{Float64} Position(1,2) isa Position{Real} Position(1,2) isa Position{<:Real} typeof(Position(1,2)) <: Position{<:Float64} typeof(Position(1,2)) <: Position{<:Real}","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n### A bizzare definition which you can encounter\nThe following definition of a one-hot matrix is taken from [Flux.jl](https://github.com/FluxML/Flux.jl/blob/1a0b51938b9a3d679c6950eece214cd18108395f/src/onehot.jl#L10-L12)\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia struct OneHotArray{T<:Integer, L, N, var\"N+1\", I<:Union{T,AbstractArray{T, N}}} <: AbstractArray{Bool, var\"N+1\"} indices::I end ```","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"The parameters of the type carry information about the type used to encode the position of one in each column in T, the dimension of one-hot vectors in L, the dimension of the storage of indices in N (which is zero for OneHotVector and one for OneHotMatrix), number of dimensions of the OneHotArray in var\"N+1\" and the type of underlying storage of indicies I.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"[1]: Type Stability in Julia, Pelenitsyn et al., 2021](https://arxiv.org/pdf/2109.01950.pdf)","category":"page"},{"location":"installation/#install","page":"Installation","title":"Installation","text":"","category":"section"},{"location":"installation/","page":"Installation","title":"Installation","text":"In order to participate in the course, everyone should install a recent version of Julia together with some text editor of choice. Furthermore during the course we will introduce some best practices of creating/testing and distributing your own Julia code, for which we will require a GitHub account.","category":"page"},{"location":"installation/","page":"Installation","title":"Installation","text":"We recommend to install Julia via juliaup. We are using the latest, stable version of Julia (which at the time of this writing is v1.9). Once you have installed juliaup you can get any Julia version you want via:","category":"page"},{"location":"installation/","page":"Installation","title":"Installation","text":"juliaup add $JULIA_VERSION\n\n# or more concretely:\njuliaup add 1.9\n\n# but please, just use the latest, stable version","category":"page"},{"location":"installation/","page":"Installation","title":"Installation","text":"Now you should be able to start Julia an be greated with the following:","category":"page"},{"location":"installation/","page":"Installation","title":"Installation","text":"$ julia\n _\n _ _ _(_)_ | Documentation: https://docs.julialang.org\n (_) | (_) (_) |\n _ _ _| |_ __ _ | Type \"?\" for help, \"]?\" for Pkg help.\n | | | | | | |/ _` | |\n | | |_| | | | (_| | | Version 1.9.2 (2023-07-05)\n _/ |\\__'_|_|_|\\__'_| | Official https://julialang.org/ release\n|__/ |\n\njulia>","category":"page"},{"location":"installation/#Julia-IDE","page":"Installation","title":"Julia IDE","text":"","category":"section"},{"location":"installation/","page":"Installation","title":"Installation","text":"There is no one way to install/develop and run Julia, which may be strange users coming from MATLAB, but for users of general purpose languages such as Python, C++ this is quite common. Most of the Julia programmers to date are using","category":"page"},{"location":"installation/","page":"Installation","title":"Installation","text":"Visual Studio Code,\nand the corresponding Julia extension.","category":"page"},{"location":"installation/","page":"Installation","title":"Installation","text":"This setup is described in a comprehensive step-by-step guide in our bachelor course Julia for Optimization & Learning.","category":"page"},{"location":"installation/","page":"Installation","title":"Installation","text":"Note that this setup is not a strict requirement for the lectures/labs and any other text editor with the option to send code to the terminal such as Vim (+Tmux), Emacs, or Sublime Text will suffice.","category":"page"},{"location":"installation/#GitHub-registration-and-Git-setup","page":"Installation","title":"GitHub registration & Git setup","text":"","category":"section"},{"location":"installation/","page":"Installation","title":"Installation","text":"As one of the goals of the course is writing code that can be distributed to others, we recommend a GitHub account, which you can create here (unless you already have one). In order to interact with GitHub repositories, we will be using git. For installation instruction (Windows only) see the section in the bachelor course.","category":"page"},{"location":"lecture_01/motivation/#Introduction-to-Scientific-Programming","page":"Motivation","title":"Introduction to Scientific Programming","text":"","category":"section"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"note: Loose definition of Scientific Programming\nScientific programming languages are designed and optimized for implementing mathematical formulas and for computing with matrices.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Examples of Scientific programming languages include ALGOL, APL, Fortran, J, Julia, Maple, MATLAB and R.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Key requirements for a Scientific programming language:","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Fast execution of the code (complex algorithms).\nEase of code reuse / code restructuring.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"
\n \n
\n Julia set.\n Stolen from\n Colorschemes.jl.\n
\n
","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"In contrast, to general-purpose language Julia has:","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"less concern with standalone executable/libraby compilation \nless concern with Application binary interface (ABI)\nless concern with business models (library + header files)\nless concern with public/private separation","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"tip: Example of a scientific task\nIn many applications, we encounter the task of optimization a function given by a routine (e.g. engineering, finance, etc.)using Optim\n\nP(x,y) = x^2 - 3x*y + 5y^2 - 7y + 3 # user defined function\n\nz₀ = [ 0.0\n 0.0 ] # starting point \n\noptimize(z -> P(z...), z₀, ConjugateGradient())\noptimize(z -> P(z...), z₀, Newton())\noptimize(z -> P(z...), z₀, Newton();autodiff = :forward)\n","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Very simple for a user, very complicated for a programmer. The program should:","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"pick the right optimization method (easy by config-like approach)\ncompute gradient (Hessian) of a user function","category":"page"},{"location":"lecture_01/motivation/#Classical-approach:-create-a-*fast*-library-and-flexible-calling-enviroment","page":"Motivation","title":"Classical approach: create a fast library and flexible calling enviroment","text":"","category":"section"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Crucial algorithms (sort, least squares...) are relatively small and well defined. Application of these algorithms to real-world problem is typically not well defined and requires more code. Iterative development. ","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Think of a problem of repeated execution of similar jobs with different options. Different level ","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"binary executable with command-line switches\nbinary executable with configuration file\nscripting language/environment (Read-Eval-Print Loop)","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"It is not a strict boundary, increasing expresivity of the configuration file will create a new scripting language.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Ending up in the 2 language problem. ","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Low-level programming = computer centric\nclose to the hardware\nallows excellent optimization for fast execution\nHigh-level programming = user centric\nrunning code with many different modifications as easily as possible\nallowing high level of abstraction","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"In scientific programming, the most well known scripting languages are: Python, Matlab, R","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"If you care about standard \"configurations\" they are just perfect. (PyTorch, BLAS)\nYou hit a problem with more complex experiments, such a modifying the internal algorithms.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"The scripting language typically makes decisions (if) at runtime. Becomes slow.","category":"page"},{"location":"lecture_01/motivation/#Examples","page":"Motivation","title":"Examples","text":"","category":"section"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Basic Linear Algebra Subroutines (BLAS)–MKL, OpenBlas–-with bindings (Matlab, NumPy)\nMatlab and Mex (C with pointer arithmetics)\nPython with transcription to C (Cython)","category":"page"},{"location":"lecture_01/motivation/#Convergence-efforts","page":"Motivation","title":"Convergence efforts","text":"","category":"section"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Just-in-time compilation (understands high level and converts to low-level)\nautomatic typing (auto in C++) (extends low-level with high-level concepts)","category":"page"},{"location":"lecture_01/motivation/#Julia-approach:-fresh-thinking","page":"Motivation","title":"Julia approach: fresh thinking","text":"","category":"section"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"(Image: )","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"A dance between specialization and abstraction. ","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Specialization allows for custom treatment. The right algorithm for the right circumstance is obtained by Multiple dispatch,\nAbstraction recognizes what remains the same after differences are stripped away. Abstractions in mathematics are captured as code through generic programming.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Why a new language?","category":"page"},{"location":"lecture_01/motivation/#Challenge","page":"Motivation","title":"Challenge","text":"","category":"section"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Translate high-level thinking with as much abstraction as possible into specific fast machine code.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Not so easy!","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"theorem: Indexing array x in Matlab:\nx = [1,2,3]\ny=x(4/2)\ny=x(5/2)In the first case it works, in the second throws an error.type instability \nfunction inde(x,n,m)=x(n/m) can never be fast.\nPoor language design choice!","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Simple solution","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Solved by different floating and integer division operation /,÷\nNot so simple with complex objects, e.g. triangular matrices","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Julia was designed as a high-level language that allows very high level abstract concepts but propagates as much information about the specifics as possible to help the compiler to generate as fast code as possible. Taking lessons from the inability to achieve fast code compilation (mostly from python).","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"(Image: )","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"julia is faster than C?","category":"page"},{"location":"lecture_01/motivation/#Julia-way","page":"Motivation","title":"Julia way","text":"","category":"section"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Design principle: abstraction should have zero runtime cost","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"flexible type system with strong typing (abstract types)\nmultiple dispatch\nsingle language from high to low levels (as much as possible) optimize execution as much as you can during compile time\nfunctions as symbolic abstraction layers","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"(Image: )","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"AST = Abstract Syntax Tree\nIR = Intermediate Representation","category":"page"},{"location":"lecture_01/motivation/#Teaser-example","page":"Motivation","title":"Teaser example","text":"","category":"section"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Function recursion with arbitrary number of arguments:","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"fsum(x) = x\nfsum(x,p...) = x+fsum(p...)","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Defines essentially a sum of inputs. Nice generic and abstract concept.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Possible in many languages:","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Matlab via nargin, varargin using construction if nargin==1, out=varargin{1}, else out=fsum(varargin{2:end}), end","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Julia solves this if at compile time. ","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"The generated code can be inspected by macro @code_llvm?","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"fsum(1,2,3)\n@code_llvm fsum(1,2,3)\n@code_llvm fsum(1.0,2.0,3.0)\nfz()=fsum(1,2,3)\n@code_llvm fz()","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Note that each call of fsum generates a new and different function.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Functions can act either as regular functions or like templates in C++. Compiler decides.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"This example is relatively simple, many other JIT languages can optimize such code. Julia allows taking this approach further.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Generality of the code:","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"fsum('c',1)\nfsum([1,2],[3,4],[5,6])","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Relies on multiple dispatch of the + function.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"More involved example:","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"using Zygote\n\nf(x)=3x+1 # user defined function\n@code_llvm f'(10)","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"The simplification was not achieved by the compiler alone.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Julia provides tools for AST and IR code manipulation\nautomatic differentiation via IR manipulation is implemented in Zygote.jl\nin a similar way, debugger is implemented in Debugger.jl\nvery simple to design domain specific language\nusing Turing\nusing StatsPlots\n\n@model function gdemo(x, y)\n s² ~ InverseGamma(2, 3)\n m ~ Normal(0, sqrt(s²))\n x ~ Normal(m, sqrt(s²))\n y ~ Normal(m, sqrt(s²))\nend","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Such tools allow building a very convenient user experience on abstract level, and reaching very efficient code.","category":"page"},{"location":"lecture_01/motivation/#Reproducibile-research","page":"Motivation","title":"Reproducibile research","text":"","category":"section"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Think about a code that was written some time ago. To run it, you often need to be able to have the same version of the language it was written for. ","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Standard way language freezes syntax and guarantees some back-ward compatibility (Matlab), which prevents future improvements\nJulia approach allows easy recreation of the environment in which the code was developed. Every project (e.g. directory) can have its own environment","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"tip: Environment\nIs an independent set of packages that can be local to an individual project or shared and selected by name.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"tip: Package\nA package is a source tree with a standard layout providing functionality that can be reused by other Julia projects.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"This allows Julia to be a rapidly evolving ecosystem with frequent changes due to:","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"built-in package manager\nswitching between multiple versions of packages","category":"page"},{"location":"lecture_01/motivation/#Package-manager","page":"Motivation","title":"Package manager","text":"","category":"section"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"implemented by Pkg.jl\nsource tree have their structure defined by a convention\nhave its own mode in REPL\nallows adding packages for using (add) or development (dev)\nsupporting functions for creation (generate) and activation (activate) and many others","category":"page"},{"location":"lecture_01/motivation/#Julia-from-user's-point-of-view","page":"Motivation","title":"Julia from user's point of view","text":"","category":"section"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"compilation of everything to as specialized as possible\nvery fast code\nslow interaction (caching...)\ngenerating libraries is harder \nthink of fsum, \neverything is \".h\" (Eigen library)\ndebugging is different to matlab/python\nextensibility, Multiple dispatch = multi-functions\nallows great extensibility and code composition\nnot (yet) mainstream thinking\nJulia is not Object-oriented\nJulia is (not pure) functional language","category":"page"},{"location":"lecture_12/hw/#hw12","page":"Homework","title":"Homework 12 - The Runge-Kutta ODE Solver","text":"","category":"section"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"There exist many different ODE solvers. To demonstrate how we can get significantly better results with a simple update to Euler, you will implement the second order Runge-Kutta method RK2:","category":"page"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"beginalign*\ntilde x_n+1 = x_n + hf(x_n t_n)\n x_n+1 = x_n + frach2(f(x_nt_n)+f(tilde x_n+1t_n+1))\nendalign*","category":"page"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"RK2 is a 2nd order method. It uses not only f (the slope at a given point), but also f (the derivative of the slope). With some clever manipulations you can arrive at the equations above with make use of f without needing an explicit expression for it (if you want to know how, see here). Essentially, RK2 computes an initial guess tilde x_n+1 to then average the slopes at the current point x_n and at the guess tilde x_n+1 which is illustarted below. (Image: rk2)","category":"page"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"The code from the lab that you will need for this homework is given below. As always, put all your code in a file called hw.jl, zip it, and upload it to BRUTE.","category":"page"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"struct ODEProblem{F,T<:Tuple{Number,Number},U<:AbstractVector,P<:AbstractVector}\n f::F\n tspan::T\n u0::U\n θ::P\nend\n\n\nabstract type ODESolver end\n\nstruct Euler{T} <: ODESolver\n dt::T\nend\n\nfunction (solver::Euler)(prob::ODEProblem, u, t)\n f, θ, dt = prob.f, prob.θ, solver.dt\n (u + dt*f(u,θ), t+dt)\nend\n\n\nfunction solve(prob::ODEProblem, solver::ODESolver)\n t = prob.tspan[1]; u = prob.u0\n us = [u]; ts = [t]\n while t < prob.tspan[2]\n (u,t) = solver(prob, u, t)\n push!(us,u)\n push!(ts,t)\n end\n ts, reduce(hcat,us)\nend\n\n\n# Define & Solve ODE\n\nfunction lotkavolterra(x,θ)\n α, β, γ, δ = θ\n x₁, x₂ = x\n\n dx₁ = α*x₁ - β*x₁*x₂\n dx₂ = δ*x₁*x₂ - γ*x₂\n\n [dx₁, dx₂]\nend","category":"page"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"
\n
Homework (2 points)
\n
","category":"page"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"Implement the 2nd order Runge-Kutta solver according to the equations given above by overloading the call method of a new type RK2.","category":"page"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"(solver::RK2)(prob::ODEProblem, u, t)","category":"page"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"
","category":"page"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"struct RK2{T} <: ODESolver\n dt::T\nend\nfunction (solver::RK2)(prob::ODEProblem, u, t)\n f, θ, dt = prob.f, prob.θ, solver.dt\n du = f(u,θ)\n uh = u + du*dt\n u + dt/2*(du + f(uh,θ)), t+dt\nend","category":"page"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"You should be able to use it exactly like our Euler solver before:","category":"page"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"using Plots\nusing JLD2\n\n# Define ODE\nfunction lotkavolterra(x,θ)\n α, β, γ, δ = θ\n x₁, x₂ = x\n\n dx₁ = α*x₁ - β*x₁*x₂\n dx₂ = δ*x₁*x₂ - γ*x₂\n\n [dx₁, dx₂]\nend\n\nθ = [0.1,0.2,0.3,0.2]\nu0 = [1.0,1.0]\ntspan = (0.,100.)\nprob = ODEProblem(lotkavolterra,tspan,u0,θ)\n\n# load correct data\ntrue_data = load(\"lotkadata.jld2\")\n\n# create plot\np1 = plot(true_data[\"t\"], true_data[\"u\"][1,:], lw=4, ls=:dash, alpha=0.7,\n color=:gray, label=\"x Truth\")\nplot!(p1, true_data[\"t\"], true_data[\"u\"][2,:], lw=4, ls=:dash, alpha=0.7,\n color=:gray, label=\"y Truth\")\n\n# Euler solve\n(t,X) = solve(prob, Euler(0.2))\nplot!(p1,t,X[1,:], color=3, lw=3, alpha=0.8, label=\"x Euler\", ls=:dot)\nplot!(p1,t,X[2,:], color=4, lw=3, alpha=0.8, label=\"y Euler\", ls=:dot)\n\n# RK2 solve\n(t,X) = solve(prob, RK2(0.2))\nplot!(p1,t,X[1,:], color=1, lw=3, alpha=0.8, label=\"x RK2\")\nplot!(p1,t,X[2,:], color=2, lw=3, alpha=0.8, label=\"y RK2\")","category":"page"},{"location":"lecture_06/lab/#introspection_lab","page":"Lab","title":"Lab 06: Code introspection and metaprogramming","text":"","category":"section"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"In this lab we are first going to inspect some tooling to help you understand what Julia does under the hood such as:","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"looking at the code at different levels\nunderstanding what method is being called\nshowing different levels of code optimization","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Secondly we will start playing with the metaprogramming side of Julia, mainly covering:","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"how to view abstract syntax tree (AST) of Julia code\nhow to manipulate AST","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"These topics will be extended in the next lecture/lab, where we are going use metaprogramming to manipulate code with macros.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"We will be again a little getting ahead of ourselves as we are going to use quite a few macros, which will be properly explained in the next lecture as well, however for now the important thing to know is that a macro is just a special function, that accepts as an argument Julia code, which it can modify.","category":"page"},{"location":"lecture_06/lab/#Quick-reminder-of-introspection-tooling","page":"Lab","title":"Quick reminder of introspection tooling","text":"","category":"section"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Let's start with the topic of code inspection, e.g. we may ask the following: What happens when Julia evaluates [i for i in 1:10]?","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"parsing ","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"using InteractiveUtils #hide\n:([i for i in 1:10]) |> dump","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"lowering","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Meta.@lower [i for i in 1:10]","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"typing","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"f() = [i for i in 1:10]\n@code_typed f()","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"LLVM code generation","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"@code_llvm f()","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"native code generation","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"@code_native f()","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Let's see how these tools can help us understand some of Julia's internals on examples from previous labs and lectures.","category":"page"},{"location":"lecture_06/lab/#Understanding-runtime-dispatch-and-type-instabilities","page":"Lab","title":"Understanding runtime dispatch and type instabilities","text":"","category":"section"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"We will start with a question: Can we spot internally some difference between type stable/unstable code?","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Inspect the following two functions using @code_lowered, @code_typed, @code_llvm and @code_native.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"x = rand(10^5)\nfunction explicit_len(x)\n length(x)\nend\n\nfunction implicit_len()\n length(x)\nend\nnothing #hide","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"For now do not try to understand the details, but focus on the overall differences such as length of the code.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"info: Redirecting `stdout`\nIf the output of the method introspection tools is too long you can use a general way of redirecting standard output stdout to a fileopen(\"./llvm_fun.ll\", \"w\") do file\n original_stdout = stdout\n redirect_stdout(file)\n @code_llvm fun()\n redirect_stdout(original_stdout)\nendIn case of @code_llvm and @code_native there are special options, that allow this out of the box, see help ? for underlying code_llvm and code_native. If you don't mind adding dependencies there is also the @capture_out from Suppressor.jl","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"@code_warntype explicit_sum(x)\n@code_warntype implicit_sum()\n\n@code_typed explicit_sum(x)\n@code_typed implicit_sum()\n\n@code_llvm explicit_sum(x)\n@code_llvm implicit_sum()\n\n@code_native explicit_sum(x)\n@code_native implicit_sum()","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"In this case we see that the generated code for such a simple operation is much longer in the type unstable case resulting in longer run times. However in the next example we will see that having longer code is not always a bad thing.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_06/lab/#Loop-unrolling","page":"Lab","title":"Loop unrolling","text":"","category":"section"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"In some cases the compiler uses loop unrolling[1] optimization to speed up loops at the expense of binary size. The result of such optimization is removal of the loop control instructions and rewriting the loop into a repeated sequence of independent statements.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"[1]: https://en.wikipedia.org/wiki/Loop_unrolling","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Inspect under what conditions does the compiler unroll the for loop in the polynomial function from the last lab.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"function polynomial(a, x)\n accumulator = a[end] * one(x)\n for i in length(a)-1:-1:1\n accumulator = accumulator * x + a[i]\n end\n accumulator \nend\nnothing #hide","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Compare the speed of execution with and without loop unrolling.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"these kind of optimization are lower level than intermediate language\nloop unrolling is possible when compiler knows the length of the input","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"using Test #hide\nusing BenchmarkTools\na = Tuple(ones(20)) # tuple has known size\nac = collect(a)\nx = 2.0\n\n@code_lowered polynomial(a,x) # cannot be seen here as optimizations are not applied\n@code_typed polynomial(a,x) # loop unrolling is not part of type inference optimization\nnothing #hide","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"@code_llvm polynomial(a,x)\n@code_llvm polynomial(ac,x)","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"More than 2x speedup","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"@btime polynomial($a,$x)\n@btime polynomial($ac,$x)","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_06/lab/#Recursion-inlining-depth","page":"Lab","title":"Recursion inlining depth","text":"","category":"section"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Inlining[2] is another compiler optimization that allows us to speed up the code by avoiding function calls. Where applicable compiler can replace f(args) directly with the function body of f, thus removing the need to modify stack to transfer the control flow to a different place. This is yet another optimization that may improve speed at the expense of binary size.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"[2]: https://en.wikipedia.org/wiki/Inline_expansion","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Rewrite the polynomial function from the last lab using recursion and find the length of the coefficients, at which inlining of the recursive calls stops occurring.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"function polynomial(a, x)\n accumulator = a[end] * one(x)\n for i in length(a)-1:-1:1\n accumulator = accumulator * x + a[i]\n end\n accumulator \nend","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"info: Splatting/slurping operator `...`\nThe operator ... serves two purposes inside function calls [3][4]:combines multiple arguments into onefunction printargs(args...)\n println(typeof(args))\n for (i, arg) in enumerate(args)\n println(\"Arg #$i = $arg\")\n end\nend\nprintargs(1, 2, 3)splits one argument into many different argumentsfunction threeargs(a, b, c)\n println(\"a = $a::$(typeof(a))\")\n println(\"b = $b::$(typeof(b))\")\n println(\"c = $c::$(typeof(c))\")\nend\nthreeargs([1,2,3]...) # or with a variable threeargs(x...)[3]: https://docs.julialang.org/en/v1/manual/faq/#What-does-the-...-operator-do?[4]: https://docs.julialang.org/en/v1/manual/functions/#Varargs-Functions","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"define two methods _polynomial!(ac, x, a...) and _polynomial!(ac, x, a) for the case of ≥2 coefficients and the last coefficient\nuse splatting together with range indexing a[1:end-1]...\nthe correctness can be checked using the built-in evalpoly\nrecall that these kind of optimization are possible just around the type inference stage\nuse container of known length to store the coefficients","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"_polynomial!(ac, x, a...) = _polynomial!(x * ac + a[end], x, a[1:end-1]...)\n_polynomial!(ac, x, a) = x * ac + a\npolynomial(a, x) = _polynomial!(a[end] * one(x), x, a[1:end-1]...)\n\n# the coefficients have to be a tuple\na = Tuple(ones(Int, 21)) # everything less than 22 gets inlined\nx = 2\npolynomial(a,x) == evalpoly(x,a) # compare with built-in function\n\n# @code_llvm polynomial(a,x) # seen here too, but code_typed is a better option\n@code_lowered polynomial(a,x) # cannot be seen here as optimizations are not applied\nnothing #hide","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"@code_typed polynomial(a,x)","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_06/lab/#AST-manipulation:-The-first-steps-to-metaprogramming","page":"Lab","title":"AST manipulation: The first steps to metaprogramming","text":"","category":"section"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Julia is so called homoiconic language, as it allows the language to reason about its code. This capability is inspired by years of development in other languages such as Lisp, Clojure or Prolog.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"There are two easy ways to extract/construct the code structure [5]","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"parsing code stored in string with internal Meta.parse","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"code_parse = Meta.parse(\"x = 2\") # for single line expressions (additional spaces are ignored)\ncode_parse_block = Meta.parse(\"\"\"\nbegin\n x = 2\n y = 3\n x + y\nend\n\"\"\") # for multiline expressions","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"constructing an expression using quote ... end or simple :() syntax","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"code_expr = :(x = 2) # for single line expressions (additional spaces are ignored)\ncode_expr_block = quote\n x = 2\n y = 3\n x + y \nend # for multiline expressions","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Results can be stored into some variables, which we can inspect further.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"typeof(code_parse)\ndump(code_parse)","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"typeof(code_parse_block)\ndump(code_parse_block)","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"The type of both multiline and single line expression is Expr with fields head and args. Notice that Expr type is recursive in the args, which can store other expressions resulting in a tree structure - abstract syntax tree (AST) - that can be visualized for example with the combination of GraphRecipes and Plots packages. ","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"using GraphRecipes #hide\nusing Plots #hide\nplot(code_expr_block, fontsize=12, shorten=0.01, axis_buffer=0.15, nodeshape=:rect)","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"This recursive structure has some major performance drawbacks, because the args field is of type Any and therefore modifications of this expression level AST won't be type stable. Building blocks of expressions are Symbols and literal values (numbers).","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"A possible nuisance of working with multiline expressions is the presence of LineNumber nodes, which can be removed with Base.remove_linenums! function.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Base.remove_linenums!(code_parse_block)","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Parsed expressions can be evaluate using eval function. ","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"eval(code_parse) # evaluation of :(x = 2) \nx # should be defined","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Before doing anything more fancy let's start with some simple manipulation of ASTs.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Define a variable code to be as the result of parsing the string \"j = i^2\". \nCopy code into a variable code2. Modify this to replace the power 2 with a power 3. Make sure that the original code variable is not also modified. \nCopy code2 to a variable code3. Replace i with i + 1 in code3.\nDefine a variable i with the value 4. Evaluate the different code expressions using the eval function and check the value of the variable j.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"code = Meta.parse(\"j = i^2\")\ncode2 = copy(code)\ncode2.args[2].args[3] = 3\ncode3 = copy(code2)\ncode3.args[2].args[2] = :(i + 1)\ni = 4\neval(code), eval(code2), eval(code3)","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Following up on the more general substitution of variables in an expression from the lecture, let's see how the situation becomes more complicated, when we are dealing with strings instead of a parsed AST.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"using Test #hide\nreplace_i(s::Symbol) = s == :i ? :k : s\nreplace_i(e::Expr) = Expr(e.head, map(replace_i, e.args)...)\nreplace_i(u) = u\nnothing #hide","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Given a function replace_i, which replaces variables i for k in an expression like the following","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"ex = :(i + i*i + y*i - sin(z))\n@test replace_i(ex) == :(k + k*k + y*k - sin(z))","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"write a different function sreplace_i(s), which does the same thing but instead of a parsed expression (AST) it manipulates a string, such as","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"s = string(ex)","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Use Meta.parse in combination with replace_i ONLY for checking of correctness.\nYou can use the replace function in combination with regular expressions.\nThink of some corner cases, that the method may not handle properly.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"The naive solution","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"sreplace_i(s) = replace(s, 'i' => 'k')\n@test Meta.parse(sreplace_i(s)) == replace_i(Meta.parse(s))","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"does not work in this simple case, because it will replace \"i\" inside the sin(z) expression. We can play with regular expressions to obtain something, that is more robust","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"sreplace_i(s) = replace(s, r\"([^\\w]|\\b)i(?=[^\\w]|\\z)\" => s\"\\1k\")\n@test Meta.parse(sreplace_i(s)) == replace_i(Meta.parse(s))","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"however the code may now be harder to read. Thus it is preferable to use the parsed AST when manipulating Julia's code.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"If the exercises so far did not feel very useful let's focus on one, that is similar to a part of the IntervalArithmetics.jl pkg.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Write function wrap!(ex::Expr) which wraps literal values (numbers) with a call to f(). You can test it on the following example","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"f = x -> convert(Float64, x)\nex = :(x*x + 2*y*x + y*y) # original expression\nrex = :(x*x + f(2)*y*x + y*y) # result expression\nnothing #hide","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"use recursion and multiple dispatch\ndispatch on ::Number to detect numbers in an expression\nfor testing purposes, create a copy of ex before mutating","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"function wrap!(ex::Expr)\n args = ex.args\n \n for i in 1:length(args)\n args[i] = wrap!(args[i])\n end\n\n return ex\nend\n\nwrap!(ex::Number) = Expr(:call, :f, ex)\nwrap!(ex) = ex\n\next, x, y = copy(ex), 2, 3\n@test wrap!(ex) == :(x*x + f(2)*y*x + y*y)\neval(ext)\neval(ex)","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"This kind of manipulation is at the core of some pkgs, such as aforementioned IntervalArithmetics.jl where every number is replaced with a narrow interval in order to find some bounds on the result of a computation.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"[5]: Once you understand the recursive structure of expressions, the AST can be constructed manually like any other type.","category":"page"},{"location":"lecture_06/lab/#Resources","page":"Lab","title":"Resources","text":"","category":"section"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Julia's manual on metaprogramming\nDavid P. Sanders' workshop @ JuliaCon 2021 \nSteven Johnson's keynote talk @ JuliaCon 2019\nAndy Ferris's workshop @ JuliaCon 2018\nFrom Macros to DSL by John Myles White \nNotes on JuliaCompilerPlugin","category":"page"},{"location":"how_to_submit_hw/#homeworks","page":"Homework submission","title":"Homework submission","text":"","category":"section"},{"location":"how_to_submit_hw/","page":"Homework submission","title":"Homework submission","text":"This document should describe the homework submission procedure.","category":"page"},{"location":"lecture_07/macros/#Macros","page":"Macros","title":"Macros","text":"","category":"section"},{"location":"lecture_01/basics/#Syntax","page":"Basics","title":"Syntax","text":"","category":"section"},{"location":"lecture_01/basics/#Elementary-syntax:-Matlab-heritage","page":"Basics","title":"Elementary syntax: Matlab heritage","text":"","category":"section"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Very much like matlab:","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"indexing from 1\narray as first-class A=[1 2 3]","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Cheat sheet: https://cheatsheets.quantecon.org/","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Introduction: https://juliadocs.github.io/Julia-Cheat-Sheet/","category":"page"},{"location":"lecture_01/basics/#Arrays-are-first-class-citizens","page":"Basics","title":"Arrays are first-class citizens","text":"","category":"section"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Many design choices were motivated considering matrix arguments:","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"x *= 2 is implemented as x = x*2 causing new allocation (vectors).","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"The reason is consistency with matrix operations: A *= B works as A = A*B.","category":"page"},{"location":"lecture_01/basics/#Broadcasting-operator","page":"Basics","title":"Broadcasting operator","text":"","category":"section"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Julia generalizes matlabs .+ operation to general use for any function. ","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"a = [1 2 3]\nsin.(a)\nf(x)=x^2+3x+8\nf.(a)","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Solves the problem of inplace multiplication","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"x .*= 2 ","category":"page"},{"location":"lecture_01/basics/#Functional-roots-of-Julia","page":"Basics","title":"Functional roots of Julia","text":"","category":"section"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Function is a first-class citizen.","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Repetition of functional programming:","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"function mymap(f::Function,a::AbstractArray)\n b = similar(a)\n for i in eachindex(a)\n b[i]=f(a[i])\n end\n b\nend","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Allows for anonymous functions:","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"mymap(x->x^2+2,[1.0,2.0])","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Function properties:","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Arguments are passed by reference (change of mutable inputs inside the function is visible outside)\nConvention: function changing inputs have a name ending by \"!\" symbol\nreturn value \nthe last line of the function declaration, \nreturn keyword\nzero cost abstraction","category":"page"},{"location":"lecture_01/basics/#Different-style-of-writing-code","page":"Basics","title":"Different style of writing code","text":"","category":"section"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Definitions of multiple small functions and their composition","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"fsum(x) = x\nfsum(x,p...) = x+fsum(p[1],p[2:end]...)","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"a single methods may not be sufficient to understand the full algorithm. In procedural language, you may write:","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"function out=fsum(x,varargin)\n if nargin==2 # TODO: better treatment\n out=x\n else\n out = fsum(varargin{1},varargin{2:end})\n end","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"The need to build intuition for function composition.","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Dispatch is easier to optimize by the compiler.","category":"page"},{"location":"lecture_01/basics/#Operators-are-functions","page":"Basics","title":"Operators are functions","text":"","category":"section"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"operator function name\n[A B C ...] hcat\n[A; B; C; ...] vcat\n[A B; C D; ...] hvcat\nA' adjoint\nA[i] getindex\nA[i] = x setindex!\nA.n getproperty\nA.n = x setproperty!","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"struct Foo end\n\nBase.getproperty(a::Foo, x::Symbol) = x == :a ? 5 : error(\"does not have property $(x)\")","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Can be redefined and overloaded for different input types. The getproperty method can define access to the memory structure.","category":"page"},{"location":"lecture_01/basics/#Broadcasting-revisited","page":"Basics","title":"Broadcasting revisited","text":"","category":"section"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"The a.+b syntax is a syntactic sugar for broadcast(+,a,b).","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"The special meaning of the dot is that they will be fused into a single call:","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"f.(g.(x .+ 1)) is treated by Julia as broadcast(x -> f(g(x + 1)), x). \nAn assignment y .= f.(g.(x .+ 1)) is treated as in-place operation broadcast!(x -> f(g(x + 1)), y, x).","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"The same logic works for lists, tuples, etc.","category":"page"},{"location":"lecture_03/lecture/#Design-patterns:-good-practices-and-structured-thinking","page":"Lecture","title":"Design patterns: good practices and structured thinking","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Every software developer has a desire to write better code. A desire to improve system performance. A desire to design software that is easy to maintain, easy to understand and explain.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Design patterns are recommendations and good practices accumulating knowledge of experienced programmers.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"The highest level of experience contains the design guiding principles:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"SOLID: Single Responsibility, Open/Closed, Liskov Substitution, Interface\nSegregation, Dependency Inversion\nDRY: Don't Repeat Yourself\nKISS: Keep It Simple, Stupid!\nPOLA: Principle of Least Astonishment\nYAGNI: You Aren't Gonna Need It (overengineering)\nPOLP: Principle of Least Privilege ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"While these high-level concepts are intuitive, they are too general to give specific answers.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"More detailed patterns arise for programming paradigms (declarative, imperative) with specific instances of functional or object-oriented programming.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"The concept of design patterns originates in the OOP paradigm. OOP defines a strict way how to write software. Sometimes it is not clear how to squeeze real world problems into those rules. Cookbook for many practical situations","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Gamma, E., Johnson, R., Helm, R., Johnson, R. E., & Vlissides, J. (1995). Design patterns: elements of reusable object-oriented software. Pearson Deutschland GmbH.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Defining 23 design patterns in three categories. Became extremely popular.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"(Image: ) (C) Scott Wlaschin","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Is julia OOP or FP? It is different from both, based on:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"types system (polymorphic)\nmultiple dispatch (extending single dispatch of OOP)\nfunctions as first class \ndecoupling of data and functions\nmacros","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Any guidelines to solve real-world problems?","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Hands-On Design Patterns and Best Practices with Julia Proven solutions to common problems in software design for Julia 1.x Tom Kwong, CFA","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Fundamental tradeoff: rules vs. freedom","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"freedom: in the C language it is possible to access assembler instructions, use pointer aritmetics:\nit is possible to write extremely efficient code\nit is easy to segfault, leak memory, etc.\nrules: in strict languages (strict OOP, strict functional programing) you lose freedom for certain guarantees:\ne.g. strict functional programing guarantees that the program provably terminates\noperations that are simple e.g. in pointer arithmetics may become clumsy and inefficient in those strict rules.\nthe compiler can validate the rules and complain if the code does not comply with them. ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Julia is again a dance between freedom and strict rules. It is more inclined to freedom. Provides few simple concepts that allow to construct design patterns common in other languages.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"the language does not enforce too many formalisms (via keywords (interface, trait, etc.) but they can be \nthe compiler cannot check for correctness of these \"patterns\"\nthe user has a lot of freedom (and responsibility)\nlots of features can be added by Julia packages (with various level of comfort)\nmacros","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Read: ","category":"page"},{"location":"lecture_03/lecture/#Design-Patterns-of-OOP-from-the-Julia-viewpoint","page":"Lecture","title":"Design Patterns of OOP from the Julia viewpoint","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"OOP is currently very popular concept (C++, Java, Python). It has strenghts and weaknesses. The Julia authors tried to keep the strength and overcome weaknesses. ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Key features of OOP:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Encapsulation \nInheritance \nPolymorphism ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Classical OOP languages define classes that bind processing functions to the data. Virtual methods are defined only for the attached methods of the classes.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"tip: Encapsulation\nRefers to bundling of data with the methods that operate on that data, or the restricting of direct access to some of an object's components. Encapsulation is used to hide the values or state of a structured data object inside a class, preventing direct access to them by clients in a way that could expose hidden implementation details or violate state invariance maintained by the methods. ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"tip: Making Julia to mimic OOP\nThere are many discussions how to make Julia to behave like an OOP. The best implementation to our knowledge is ObjectOriented","category":"page"},{"location":"lecture_03/lecture/#Encapsulation-Advantage:-Consistency-and-Validity","page":"Lecture","title":"Encapsulation Advantage: Consistency and Validity","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"With fields of data structure freely accessible, the information may become inconsistent.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"mutable struct Grass <: Plant\n id::Int\n size::Int\n max_size::Int\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"What if I create Grass with larger size than max_size?","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"grass = Grass(1,50,5)","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Freedom over Rules. Maybe I would prefer to introduce some rules.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Some encapsulation may be handy keeping it consistent. Julia has inner constructor.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"mutable struct Grass2 <: Plant\n id::Int\n size::Int\n max_size::Int\n Grass2(id,sz,msz) = sz > msz ? error(\"size can not be greater that max_size\") : new(id,sz,msz)\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"When defined, Julia does not provide the default outer constructor. ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"But fields are still accessible:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"grass.size = 10000","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Recall that grass.size=1000 is a syntax of setproperty!(grass,:size,1000), which can be redefined:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"function Base.setproperty!(obj::Grass, sym::Symbol, val)\n if sym==:size\n @assert val<=obj.max_size \"size have to be lower than max_size!\"\n end\n setfield!(obj,sym,val)\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Function setfield! can not be overloaded.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Julia has partial encapsulation via a mechanism for consistency checks. ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"warn: Array in unmutable struct can be mutated\nThe mutability applies to the structure and not to encapsulated structures.struct Foo\n x::Float64\n y::Vector{Float64}\n z::Dict{Int,Int}\nendIn the structure Foo, x cannot be mutated, but fields of y and key-value pairs of z can be mutated, because they are mutable containers. But I cannot replace y with a different Vector.","category":"page"},{"location":"lecture_03/lecture/#Encapsulation-Disadvantage:-the-Expression-Problem","page":"Lecture","title":"Encapsulation Disadvantage: the Expression Problem","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Encapsulation limits the operations I can do with an object. Sometimes too much. Consider a matrix of methods/types(data-structures)","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Consider an existing matrix of data and functions:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"data \\ methods find_food eat! grow! \nWolf \nSheep \nGrass ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"You have a good reason not to modify the original source (maintenance).","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Imagine we want to extend the world to use new animals and new methods for all animals.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Object-oriented programming ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"classes are primary objects (hierarchy)\ndefine animals as classes ( inheriting from abstract class)\nadding a new animal is easy\nadding a new method for all animals is hard (without modifying the original code)","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Functional programming ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"functions are primary\ndefine operations find_food, eat!\nadding a new operation is easy\nadding new data structure to existing operations is hard","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Solutions:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"multiple-dispatch = julia\nopen classes (monkey patching) = add methods to classes on the fly\nvisitor pattern = partial fix for OOP [extended visitor pattern using dynamic_cast]","category":"page"},{"location":"lecture_03/lecture/#Morale:","page":"Lecture","title":"Morale:","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Julia does not enforces creation getters/setters by default (setproperty is mapped to setfield)\nit provides tools to enforce access restriction if the user wants it.\ncan be used to imitate objects: ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"https://stackoverflow.com/questions/39133424/how-to-create-a-single-dispatch-object-oriented-class-in-julia-that-behaves-l/39150509#39150509","category":"page"},{"location":"lecture_03/lecture/#Polymorphism:","page":"Lecture","title":"Polymorphism:","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"tip: Polymorphism in OOP\nPolymorphism is the method in an object-oriented programming language that performs different things as per the object’s class, which calls it. With Polymorphism, a message is sent to multiple class objects, and every object responds appropriately according to the properties of the class. ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Example animals of different classes make different sounds. In Python:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"\nclass Sheep:\n def __init__(self, energy, Denergy):\n self.energy = energy\n self.Denergy = Denergy\n\n def make_sound(self):\n print(\"Baa\")\n\nsheep.make_sound()\nwolf.make_sound()","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Will make distinct sounds (baa, Howl). ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Can we achieve this in Julia?","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"make_sound(s::Sheep)=println(\"Baa\")\nmake_sound(w::Wolf)=println(\"Howl\")","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"tip: Implementation of virtual methods\nVirtual methods in OOP are typically implemented using Virtual Method Table, one for each class. (Image: )Julia has a single method table. Dispatch can be either static or dynamic (slow).","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Freedom vs. Rules. ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Duck typing is a type of polymorphism without static types\nmore programming freedom, less formal guarantees\njulia does not check if make_sound exists for all animals. May result in MethodError. Responsibility of a programmer.\ndefine make_sound(A::AbstractAnimal)","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"So far, the polymorphism coincides for OOP and julia becuase the method had only one argument => single argument dispatch.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Multiple dispatch is an extension of the classical first-argument-polymorphism of OOP, to all-argument polymorphism.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"tip: Challenge for OOP\nHow to code polymorphic behavior of interaction between two agents, e.g. an agent eating another agent in OOP?Complicated.... You need a \"design pattern\" for it.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"class Sheep(Animal):\n energy: float = 4.0\n denergy: float = 0.2\n reprprob: float = 0.5\n foodprob: float = 0.9\n\n # hard, if not impossible to add behaviour for a new type of food\n def eat(self, a: Agent, w: World):\n if isinstance(a, Grass)\n self.energy += a.size * self.denergy\n a.size = 0\n else:\n raise ValueError(f\"Sheep cannot eat {type(a).__name__}.\")","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Consider an extension to:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Flower : easy\nPoisonousGrass: harder","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Simple in Julia:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"eat!(w1::Sheep, a::Grass, w::World)=\neat!(w1::Sheep, a::Flower, w::World)=\neat!(w1::Sheep, a::PoisonousGrass, w::World)=","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Boiler-plate code can be automated by macros / meta programming.","category":"page"},{"location":"lecture_03/lecture/#Inheritance","page":"Lecture","title":"Inheritance","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"tip: Inheritance\nIs the mechanism of basing one object or class upon another object (prototype-based inheritance) or class (class-based inheritance), retaining similar implementation. Deriving new classes (sub classes) from existing ones such as super class or base class and then forming them into a hierarchy of classes. In most class-based object-oriented languages, an object created through inheritance, a \"child object\", acquires all the properties and behaviors of the \"parent object\" , with the exception of: constructors, destructor, overloaded operators.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Most commonly, the sub-class inherits methods and the data.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"For example, in python we can design a sheep with additional field. Think of a situation that we want to refine the reproduction procedure for sheeps by considering differences for male and female. We do not have information about gender in the original implementation. ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"In OOP, we can use inheritance.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"class Sheep:\n def __init__(self, energy, Denergy):\n self.energy = energy\n self.Denergy = Denergy\n\n def make_sound(self):\n print(\"Baa\")\n\nclass SheepWithGender(Sheep):\n def __init__(self, energy, Denergy,gender):\n super().__init__(energy, Denergy)\n self.gender = gender\n # make_sound is inherited \n\n# Can you do this in Julia?!","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Simple answer: NO, not exactly","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Sheep has fields, is a concrete type, we cannot extend it.\nwith modification of the original code, we can define AbstractSheep with subtypes Sheep and SheepWithGender.\nBut methods for AbstractAnimal works for sheeps! Is this inheritance?","category":"page"},{"location":"lecture_03/lecture/#Inheritance-vs.-Subtyping","page":"Lecture","title":"Inheritance vs. Subtyping","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Subtle difference:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"subtyping = equality of interface \ninheritance = reuse of implementation ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"In practice, subtyping reuse methods, not data fields.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"We have seen this in Julia, using type hierarchy: ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"agent_step!(a::Animal, w::World)\nall animals subtype of Animal \"inherit\" this method.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"The type hierarchy is only one way of subtyping. Julia allows many variations, e.g. concatenating different parts of hierarchies via the Union{} type:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"fancy_method(O::Union{Sheep,Grass})=println(\"Fancy\")","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Is this a good idea? It can be done completely Ad-hoc! Freedom over Rules.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"There are very good use-cases:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Missing values:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"x::AbstractVector{<:Union{<:Number, Missing}}","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"theorem: SubTyping issues\nWith parametric types, unions and other construction, subtype resolution may become a complicated problem. Julia can even crash. (Jan Vitek's Keynote at JuliaCon 2021)[https://www.youtube.com/watch?v=LT4AP7CUMAw]","category":"page"},{"location":"lecture_03/lecture/#Sharing-of-data-field-via-composition","page":"Lecture","title":"Sharing of data field via composition","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Composition is also recommended in OOP: (Composition over ingeritance)[https://en.wikipedia.org/wiki/Compositionoverinheritance]","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"struct ⚥Sheep <: Animal\n sheep::Sheep\n sex::Symbol\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"If we want our new ⚥Sheep to behave like the original Sheep, we need to forward the corresponding methods.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"eat!(a::⚥Sheep, b::Grass, w::World)=eat!(a.sheep, b, w)","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"and all other methods. Routine work. Boring! The whole process can be automated using macros @forward from Lazy.jl.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Why so complicated? Wasn't the original inheritance tree structure better?","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"multiple inheritance:\nyou just compose two different \"trees\".\ncommon example with ArmoredVehicle = Vehicle + Weapon\nDo you think there is only one sensible inheritance tree?","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"tip: Animal World\nThink of an inheritance tree of a full scope Animal world.Idea #1: Split animals by biological taxonomy (Image: )Hold on. Sharks and dolphins can swim very well!\nBoth bats and birds fly similarly!Idea #2: Split by the way they move!Idea #3: Split by way of ...","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"In fact, we do not have a tree, but more like a matrix/tensor:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":" swims flies walks\nbirds penguin eagle kiwi\nmammal dolphin bat sheep,wolf\ninsect backswimmer fly beetle","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Single type hierarchy will not work. Other approaches:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"interfaces\nparametric types","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Analyze what features of animals are common and compose the animal:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"abstract type HeatType end\nabstract type MovementType end\nabstract type ChildCare end\n\n\nmutable struct Animal{H<:HeatType,M<:MovementType,C<:ChildCare} \n id::Int\n ...\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Now, we can define methods dispatching on parameters of the main type.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Composition is simpler in such a general case. Composition over inheritance. ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"A simple example of parametric approach will be demonstarted in the lab.","category":"page"},{"location":"lecture_03/lecture/#Interfaces:-inheritance/subtyping-without-a-hierarchy-tree","page":"Lecture","title":"Interfaces: inheritance/subtyping without a hierarchy tree","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"In OOP languages such as Java, interfaces have a dedicated keyword such that compiler can check correctes of the interface implementation. ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"In Julia, interfaces can be achived by defining ordinary functions. Not so strict validation by the compiler as in other languages. Freedom...","category":"page"},{"location":"lecture_03/lecture/#Example:-Iterators","page":"Lecture","title":"Example: Iterators","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Many fundamental objects can be iterated: Arrays, Tuples, Data collections...","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"They do not have any common \"predecessor\". They are almost \"primitive\" types.\nthey share just the property of being iterable\nwe do not want to modify them in any way","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Example: of interface Iterators defined by \"duck typing\" via two functions.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Required methods Brief description\niterate(iter) Returns either a tuple of the first item and initial state or nothing if empty\niterate(iter, state) Returns either a tuple of the next item and next state or nothing if no items remain","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Defining these two methods for any object/collection C will make the following work:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"for o in C\n # do something\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"The compiler will not check if both functions exist.\nIf one is missing, it will complain about it when it needs it\nThe error message may be less informative than in the case of formal definition","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Note:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"even iterators may have different features: they can be finite or infinite\nfor finite iterators we can define useful functions (collect)\nhow to pass this information in an extensible way?","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Poor solution: if statements.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"function collect(iter)\n if iter isa Tuple...\n\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"The compiler can do that for us.","category":"page"},{"location":"lecture_03/lecture/#Traits:-cherry-picking-subtyping","page":"Lecture","title":"Traits: cherry picking subtyping","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Trait mechanism in Julia is build using the existing tools: Type System and Multiple Dispatch.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Traits have a few key parts:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Trait types: the different traits a type can have.\nTrait function: what traits a type has.\nTrait dispatch: using the traits.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"From iterators:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"# trait types:\n\nabstract type IteratorSize end\nstruct SizeUnknown <: IteratorSize end\nstruct HasLength <: IteratorSize end\nstruct IsInfinite <: IteratorSize end\n\n# Trait function: Input is a Type, output is a Type\nIteratorSize(::Type{<:Tuple}) = HasLength()\nIteratorSize(::Type) = HasLength() # HasLength is the default\n\n# ...\n\n# Trait dispatch\nBitArray(itr) = gen_bitarray(IteratorSize(itr), itr)\ngen_bitarray(isz::IteratorSize, itr) = gen_bitarray_from_itr(itr)\ngen_bitarray(::IsInfinite, itr) = throw(ArgumentError(\"infinite-size iterable used in BitArray constructor\"))\n","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"What is needed to define for a new type that I want to iterate over? ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Do you still miss inheritance in the OOP style?","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Many packages automating this with more structure:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"https://github.com/andyferris/Traitor.jl\nhttps://github.com/mauro3/SimpleTraits.jl\nhttps://github.com/tk3369/BinaryTraits.jl","category":"page"},{"location":"lecture_03/lecture/#Functional-tools:-Partial-evaluation","page":"Lecture","title":"Functional tools: Partial evaluation","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"It is common to create a new function which \"just\" specify some parameters.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"_prod(x) = reduce(*,x)\n_sum(x) = reduce(+,x)","category":"page"},{"location":"lecture_03/lecture/#Functional-tools:-Closures","page":"Lecture","title":"Functional tools: Closures","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"tip: Closure (lexical closure, function closure)\nA technique for implementing lexically scoped name binding in a language with first-class functions. Operationally, a closure is a record storing a function together with an environment.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"originates in functional programming\nnow widespread in many common languages, Python, Matlab, etc..\nmemory management relies on garbage collector in general (can be optimized by compiler)","category":"page"},{"location":"lecture_03/lecture/#Example","page":"Lecture","title":"Example","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"function adder(x)\n return y->x+y\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"creates a function that \"closes\" the argument x. Try: f=adder(5); f(3).","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"x = 30;\nfunction adder()\n return y->x+y\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"creates a function that \"closes\" variable x.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"f = adder(10)\nf(1)\ng = adder()\ng(1)\n","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Such function can be passed as an argument: together with the closed data.","category":"page"},{"location":"lecture_03/lecture/#Implementation-of-closures-in-julia:-documentation","page":"Lecture","title":"Implementation of closures in julia: documentation","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Closure is a record storing a function together with an environment. The environment is a mapping associating each free variable of the function (variables that are used locally, but defined in an enclosing scope) with the value or reference to which the name was bound when the closure was created.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"function adder(x)\n return y->x+y\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"is lowered to (roughly):","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"struct ##1{T}\n x::T\nend\n\n(_::##1)(y) = _.x + y\n\nfunction adder(x)\n return ##1(x)\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Note that the structure ##1 is not directly accessible. Try f.x and g.x.","category":"page"},{"location":"lecture_03/lecture/#Functor-Function-like-structure","page":"Lecture","title":"Functor = Function-like structure","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Each structure can have a method that is invoked when called as a function.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"(_::Sheep)()= println(\"🐑\")","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"You can think of it as sheep.default_method().","category":"page"},{"location":"lecture_03/lecture/#Coding-style","page":"Lecture","title":"Coding style","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"From Flux.jl:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"function train!(loss, ps, data, opt; cb = () -> ())\n ps = Params(ps)\n cb = runall(cb)\n @progress for d in data\n gs = gradient(ps) do\n loss(batchmemaybe(d)...)\n end\n update!(opt, ps, gs)\n cb()\n end\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Is this confusing? What can cb() do and what it can not?","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Note that function train! does not have many local variables. The important ones are arguments, i.e. exist in the scope from which the function was invoked.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"loss(x,y)=mse(model(x),y)\ncb() = @info \"training\" loss(x,y)\ntrain!(loss, ps, data, opt; cb=cb)","category":"page"},{"location":"lecture_03/lecture/#Usage","page":"Lecture","title":"Usage","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Usage of closures:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"callbacks: the function can also modify the enclosed variable.\nabstraction: partial evaluation ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"theorem: Beware: Performance of captured variables\nInference of types may be difficult in closures: https://github.com/JuliaLang/julia/issues/15276 ","category":"page"},{"location":"lecture_03/lecture/#Aditional-materials","page":"Lecture","title":"Aditional materials","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Functional desighn pattersn","category":"page"},{"location":"lecture_09/lecture/#Manipulating-Intermediate-Represenation-(IR)","page":"Lecture","title":"Manipulating Intermediate Represenation (IR)","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"using InteractiveUtils: @code_typed, @code_lowered, code_lowered","category":"page"},{"location":"lecture_09/lecture/#Generated-functions","page":"Lecture","title":"Generated functions","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Sometimes it is convenient to generate function once types of arguments are known. For example if we have function foo(args...), we can generate different body for different length of Tuple and types in args. Do we really need such thing, or it is just wish of curious programmer? Not really, as ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"we can deal with variability of args using normal control-flow logic if length(args) == 1 elseif ...\nwe can (automatically) generate (a possibly very large) set of functions foo specialized for each length of args (or combination of types of args) and let multiple dispatch to deal with this\nwe cannot deal with this situation with macros, because macros do not see types, only parsed AST, which is in this case always the same.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Generated functions allow to specialize the code for a given type of argumnets. They are like macros in the sense that they return expressions and not results. But unlike macros, the input is not expression or value of arguments, but their types (the arguments are of type Type). They are also called when compiler needs (which means at least once for each combination of arguments, but possibly more times due to code invalidation).","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Let's look at an example","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"@generated function genplus(x, y)\n println(\"generating genplus(x, y)\")\n @show (x, y, typeof(x), typeof(y))\n quote \n println(\"executing generated genplus(x, y)\")\n @show (x, y, typeof(x), typeof(y))\n x + y\n end\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"and observe the output","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"julia> genplus(1.0, 1.0) == 1.0 + 1.0\ngenerating genplus(x, y)\n(x, y, typeof(x), typeof(y)) = (Float64, Float64, DataType, DataType)\nexecuting generated genplus(x, y)\n(x, y, typeof(x), typeof(y)) = (1.0, 1.0, Float64, Float64)\ntrue\n\njulia> genplus(1.0, 1.0) == 1.0 + 1.0\nexecuting generated genplus(x, y)\n(x, y, typeof(x), typeof(y)) = (1.0, 1.0, Float64, Float64)\ntrue\n\njulia> genplus(1, 1) == 1 + 1\ngenerating genplus(x, y)\n(x, y, typeof(x), typeof(y)) = (Int64, Int64, DataType, DataType)\nexecuting generated genplus(x, y)\n(x, y, typeof(x), typeof(y)) = (1, 1, Int64, Int64)\ntrue","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"which shows that the body of genplus is called for each combination of types of parameters, but the generated code is called whenever genplus is called.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Generated functions has to be pure in the sense that they are not allowed to have side effects, for example modifying some global variables. Note that printing is not allowed in pure functions, as it modifies the global buffer. From the above example this rule does not seems to be enforced, but not obeying it can lead to unexpected errors mostly caused by not knowing when and how many times the functions will be called.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Finally, generated functions cannot call functions that has been defined after their definition.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"@generated function genplus(x, y)\n foo()\n :(x + y)\nend\n\nfoo() = println(\"foo\")\ngenplus(1,1)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Here, the applicable method is foo.","category":"page"},{"location":"lecture_09/lecture/#An-example-that-explains-everything.","page":"Lecture","title":"An example that explains everything.","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Consider a version of map applicable to NamedTuples with permuted names. Recall the behavior of normal map, which works if the names are in the same order.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"x = (a = 1, b = 2, c = 3)\ny = (a = 4, b = 5, c = 6)\nmap(+, x, y)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The same does not work with permuted names:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"x = (a = 1, b = 2, c = 3)\ny = (c = 6, b = 5, a = 4)\nmap(+, x, y)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"How to fix this? The usual approach would be to iterate over the keys in named tuples:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"function permuted_map(f, x::NamedTuple{KX}, y::NamedTuple{KY}) where {KX, KY}\n ks = tuple(intersect(KX,KY)...)\n NamedTuple{ks}(map(k -> f(x[k], y[k]), ks))\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"But, can we do better? Recall that in NamedTuples, we exactly know the position of the arguments, hence we should be able to directly match the corresponding arguments without using get. ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Since creation (and debugging) of generated functions is difficult, we start with a single-argument unrolled map.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"@generated function unrolled_map(f, x::NamedTuple{KX}) where {KX} \n vals = [:(f(getfield(x, $(QuoteNode(k))))) for k in KX]\n :(($(vals...),))\nend\nunrolled_map(e->e+1, x)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"We see that inserting a Symbol specifying the field in the NamedTuple is a bit tricky. It needs to be quoted, since $() which is needed to substitute k for its value \"peels\" one layer of the quoting. Compare this to","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"vals = [:(f(getfield(x, $(k)))) for k in KX]","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Since getting the field is awkward, we write syntactic sugar for that","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"_get(name, k) = :(getfield($(name), $(QuoteNode(k))))","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"with that, we proceed to a nicer two argument function which we have desired:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"@generated function unrolled_map(f, x::NamedTuple{KX}, y::NamedTuple{KY}) where {KX, KY} \n ks = tuple(intersect(KX,KY)...)\n _get(name, k) = :(getfield($(name), $(QuoteNode(k))))\n vals = [:(f($(_get(:x, k)), $(_get(:y, k)))) for k in ks]\n :(NamedTuple{$(ks)}(($(vals...),)))\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"We can check that the unrolled_map unrolls the map and generates just needed operations","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"@code_typed unrolled_map(+, x, y)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"and compare this to the code generated by the non-generated version permuted_map:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"@code_typed permuted_map(+, x, y)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"which is not shown here for the sake of conciseness.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"For fun, we can create a version which replaces the Symbol arguments directly by position numbers","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"@generated function unrolled_map(f, x::NamedTuple{KX}, y::NamedTuple{KY}) where {KX, KY} \n ks = tuple(intersect(KX,KY)...)\n _get(name, k, KS) = :(getfield($(name), $(findfirst(k .== KS))))\n vals = [:(f($(_get(:x, k, KX)), $(_get(:y, k, KY)))) for k in KX]\n :(NamedTuple{$(KX)}(($(vals...),)))\nend","category":"page"},{"location":"lecture_09/lecture/#Optionally-generated-functions","page":"Lecture","title":"Optionally generated functions","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Macro @generated is expanded to","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"julia> @macroexpand @generated function gentest(x)\n return :(x + x)\n end\n\n:(function gentest(x)\n if $(Expr(:generated))\n return $(Expr(:copyast, :($(QuoteNode(:(x + x))))))\n else\n $(Expr(:meta, :generated_only))\n return\n end\n end)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"which is a function with an if-condition, where the first branch $(Expr(:generated)) generates the expression :(x + x) and returns it. The other spits out an error saying that the function has only a generated version. This suggests the possibility (and reality) that one can implement two versions of the same function; A generated and a normal version. It is left up to the compiler to decide which one to use. It is entirely up to the author to ensure that both versions are the same. Which version will the compiler take? The last comment on 23168 (as of time of writing) states:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"\"Currently the @generated branch is always used. In the future, which branch is used will mostly depend on whether the JIT compiler is enabled and available, and if it's not available, then it will depend on how much we were able to compile before the compiler was taken away. So I think it will mostly be a concern for those that might need static compilation and JIT-less deployment.\"","category":"page"},{"location":"lecture_09/lecture/#Contextual-dispatch-/-overdubbing","page":"Lecture","title":"Contextual dispatch / overdubbing","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Imagine that under some circumstances (context), you would like to use alternative implementations of some functions. One of the most cited motivations for this is automatic differentiation, where you would like to take the code as-is and calculate gradients with respect to some variables. Other use cases of this approach are mentioned in Cassette.jl:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"\"Downstream applications for Cassette include dynamic code analysis (e.g. profiling, record and replay style debugging, etc.), JIT compilation to new hardware/software backends, automatic differentiation, interval constraint programming, automatic parallelization/rescheduling, automatic memoization, lightweight multistage programming, graph extraction, and more.\"","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"In theory, we can do all the above by directly modifying the code or introducing new types, but that may require a lot of coding and changing of foreign libraries.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The technique we desire is called contextual dispatch, which means that under some context, we invoke a different function. The library Casette.jl provides a high-level API for overdubbing, but it is interesting to see, how it works, as it shows, how we can \"interact\" with the lowered code before the code is typed.","category":"page"},{"location":"lecture_09/lecture/#Insertion-of-code","page":"Lecture","title":"Insertion of code","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Imagine that julia has compiled some function. For example ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"foo(x,y) = x * y + sin(x)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"and observe its lowered SSA format","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_lowered foo(1.0, 1.0)\nCodeInfo(\n1 ─ %1 = x * y\n│ %2 = Main.sin(x)\n│ %3 = %1 + %2\n└── return %3\n)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The lowered form is convenient, because on the left hand, there is always one variable and the right-hand side is simplified to have (mostly) a single call / expression. Moreover, in the lowered form, all control flow operations like if, for, while and exceptions are converted to Goto and GotoIfNot, which simplifies their handling. ","category":"page"},{"location":"lecture_09/lecture/#Codeinfo","page":"Lecture","title":"Codeinfo","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"We can access the lowered form by","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"ci = @code_lowered foo(1.0, 1.0)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"which returns an object of type CodeInfo containing many fields docs. To make the investigation slightly more interesting, we modify the function a bit to have local variables:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"function foo(x,y) \n z = x * y \n z + sin(x)\nend\n\nci = @code_lowered foo(1.0, 1.0)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The most important (and interesting) field is code:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"ci.code","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"It contains expressions corresponding to each line of the lowered form. You are free to access them (and modify them with care). Variables identified with underscore Int, for example _2, are slotted variables which are variables which have a name in the code, defined via input arguments or through an explicit assignment :(=). The names of slotted variables are stored in ci.slotnames and they are of type ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"typeof(ci.code[1].args[2].args[2])\nci.slotnames[ci.code[1].args[2].args[2].id]\nci.slotnames[ci.code[1].args[2].args[3].id]\nci.slotnames[ci.code[1].args[1].id]","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The remaining variables are identified by an integer with prefix %, where the number corresponds to the line (index in ci.code), in which the variable was created. For example the fourth line :(%2 + %3) adds the results of the second line :(_4) containing variable z and the third line :(Main.sin(_2)). The type of each slot variable is stored in slottypes, which provides some information about how the variable is used (see docs). Note that if you modify / introduce slot variables, the length of slotnames and slottypes has to match and it has to be equal to the maximum number of slotted variables.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"CodeInfo also contains information about the source code. Each item of ci.code has an identifier in ci.codelocs which is an index into ci.linetable containing Core.LineInfoNode identifying lines in the source code (or in the REPL). Notice that ci.linetable is generally shorter then ci.codelocs, as one line of source code can be translated to multiple lines in lowered code. ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The important feature of the lowered form is that we can freely edit (create new) CodeInfo and that generated functions can return a CodeInfo object instead of the AST. However, you need to explicitly write a return statement (see issue 25678).","category":"page"},{"location":"lecture_09/lecture/#Strategy-for-overdubbing","page":"Lecture","title":"Strategy for overdubbing","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"In overdubbing, our intention is to recursively dive into called function definitions and modify / change their code. In our example below, with which we will demonstrate the manual implementation (for educational purposes), our goal is to enclose each function call with statements that log the exection time. This means we would like to implement a simplified recording profiler. This functionality cannot be implemented by a macros, since macros do not allow us to dive into function definitions. For example, in our function foo, we would would not be able to dive into the definition of sin (not that this is a terribly good idea, but the point should be clear).","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The overdubbing pattern works as follows.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"We define a @generated function overdub(f, args...) which takes as a first argument a function f and then its arguments.\nIn the function overdub we retrieve the CodeInfo for f(args...), which is possible as we know types of the arguments at this time.\nWe modify the the CodeInfo of f(args...) according to our liking. Importantly, we replace all function calls some_fun(some_args...) with overdub(some_fun, some_args...) which establishes the recursive pattern.\nModify the arguments of the CodeInfo of f(args...) to match overdub(f, args..).\nReturn the modified CodeInfo.","category":"page"},{"location":"lecture_09/lecture/#The-profiler","page":"Lecture","title":"The profiler","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The implementation of the simplified logging profiler is straightforward and looks as follows.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"module LoggingProfiler\nstruct Calls\n stamps::Vector{Float64} # contains the time stamps\n event::Vector{Symbol} # name of the function that is being recorded\n startstop::Vector{Symbol} # if the time stamp corresponds to start or to stop\n i::Ref{Int}\nend\n\nfunction Calls(n::Int)\n Calls(Vector{Float64}(undef, n+1), Vector{Symbol}(undef, n+1), Vector{Symbol}(undef, n+1), Ref{Int}(0))\nend\n\nfunction Base.show(io::IO, calls::Calls)\n offset = 0\n if calls.i[] >= length(calls.stamps)\n @warn \"The recording buffer was too small, consider increasing it\"\n end\n for i in 1:min(calls.i[], length(calls.stamps))\n offset -= calls.startstop[i] == :stop\n foreach(_ -> print(io, \" \"), 1:max(offset, 0))\n rel_time = calls.stamps[i] - calls.stamps[1]\n println(io, calls.event[i], \": \", rel_time)\n offset += calls.startstop[i] == :start\n end\nend\n\nglobal const to = Calls(100)\n\n\"\"\"\n record_start(ev::Symbol)\n\n record the start of the event, the time stamp is recorded after all counters are \n appropriately increased\n\"\"\"\nrecord_start(ev::Symbol) = record_start(to, ev)\nfunction record_start(calls, ev::Symbol)\n n = calls.i[] = calls.i[] + 1\n n > length(calls.stamps) && return \n calls.event[n] = ev\n calls.startstop[n] = :start\n calls.stamps[n] = time_ns()\nend\n\n\"\"\"\n record_end(ev::Symbol)\n\n record the end of the event, the time stamp is recorded before all counters are \n appropriately increased\n\"\"\"\nrecord_end(ev::Symbol) = record_end(to, ev::Symbol)\nfunction record_end(calls, ev::Symbol)\n t = time_ns()\n n = calls.i[] = calls.i[] + 1\n n > length(calls.stamps) && return \n calls.event[n] = ev\n calls.startstop[n] = :stop\n calls.stamps[n] = t\nend\n\nreset!() = to.i[] = 0\n\nfunction Base.resize!(calls::Calls, n::Integer)\n resize!(calls.stamps, n)\n resize!(calls.event, n)\n resize!(calls.startstop, n)\nend\n\n\nexportname(ex::GlobalRef) = QuoteNode(ex.name)\nexportname(ex::Symbol) = QuoteNode(ex)\nexportname(ex::Expr) = exportname(ex.args[1])\nexportname(i::Int) = QuoteNode(Symbol(\"Int(\",i,\")\"))\n\nfunction overdubbable(ex::Expr)\n ex.head != :call && return(false)\n a = ex.args[1]\n a != GlobalRef && return(true)\n a.mod != Core\nend \noverdubbable(ex) = false \n\n\nfunction timable(ex::Expr) \n ex.head != :call && return(false)\n length(ex.args) < 2 && return(false)\n ex.args[1] isa Core.GlobalRef && return(true)\n ex.args[1] isa Symbol && return(true)\n return(false)\nend\ntimable(ex) = false\n\nexport timable, exportname, overdubbable\nend","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The important functions are report_start and report_end which mark the beggining and end of the executed function. They differ mainly when time is recorded (on the end or on the start of the function call). The profiler has a fixed capacity to prevent garbage collection, which might be increased.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Let's now describe the individual parts of overdub before presenting it in its entirety. At first, we retrieve the codeinfo ci of the overdubbed function. For now, we will just assume we obtain it for example by","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"ci = @code_lowered foo(1.0, 1.0)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"we initialize the new CodeInfo object by emptying some dummy function as ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"dummy() = return\nnew_ci = code_lowered(dummy, Tuple{})[1]\nempty!(new_ci.code)\nempty!(new_ci.slotnames)\nempty!(new_ci.linetable)\nempty!(new_ci.codelocs)\nnew_ci","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Then, we need to copy the slot variables from the ci codeinfo of foo to the new codeinfo. Additionally, we have to add the arguments of overdub(f, args...) since the compiler sees overdub(f, args...) and not foo(x,y):","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"new_ci.slotnames = vcat([Symbol(\"#self#\"), :f, :args], ci.slotnames[2:end])\nnew_ci.slotflags = vcat([0x00, 0x00, 0x00], ci.slotflags[2:end])","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Above, we also filled the slotflags. Authors admit that names :f and :args in the above should be replaced by a gensymed name, but they do not anticipate this code to be used outside of this educative example where name-clashes might occur. We also copy information about the lines from the source code:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"foreach(s -> push!(new_ci.linetable, s), ci.linetable)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The most difficult part when rewriting CodeInfo objects is working with indexes, as the line numbers and left hand side variables are strictly ordered one by one and we need to properly change the indexes to reflect changes we made. We will therefore keep three lists","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"maps = (\n ssa = Dict{Int, Int}(),\n slots = Dict{Int, Any}(),\n goto = Dict{Int,Int}(),\n)\nnothing # hide","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"where ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"slots maps slot variables in ci to those in new_ci\nssa maps indexes of left-hand side assignments in ci to new_ci\ngoto maps lines to which GotoNode and GotoIfNot point to variables in ci to new_ci (in our profiler example, we need to ensure to jump on the beggining of logging of executions)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Mapping of slots can be initialized in advance, as it is a static shift by 2 :","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"maps.slots[1] = Core.SlotNumber(1)\nforeach(i -> maps.slots[i] = Core.SlotNumber(i + 2), 2:length(ci.slotnames)) ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"and we can check the correctness by","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"@assert all(ci.slotnames[i] == new_ci.slotnames[maps.slots[i].id] for i in 1:length(ci.slotnames)) #test that ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Equipped with that, we start rewriting the code of foo(x, y). We start by a small preample, where we assign values of args... to x, and y. For the sake of simplicity, we map the slotnames to either Core.SlotNumber or to Core.SSAValues which simplifies the rewriting logic a bit.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"newci_no = 0\nargs = (Float64, Float64)\nfor i in 1:length(args)\n newci_no +=1\n push!(new_ci.code, Expr(:call, Base.getindex, Core.SlotNumber(3), i))\n maps.slots[i+1] = Core.SSAValue(newci_no)\n push!(new_ci.codelocs, ci.codelocs[1])\nend","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Now we come to the pinnacle of rewriting the body of foo(x,y) while inserting calls to the profiler:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"for (ci_no, ex) in enumerate(ci.code)\n if timable(ex)\n fname = exportname(ex)\n push!(new_ci.code, Expr(:call, GlobalRef(LoggingProfiler, :record_start), fname))\n push!(new_ci.codelocs, ci.codelocs[ci_no])\n newci_no += 1\n maps.goto[ci_no] = newci_no\n ex = overdubbable(ex) ? Expr(:call, GlobalRef(Main, :overdub), ex.args...) : ex\n push!(new_ci.code, ex)\n push!(new_ci.codelocs, ci.codelocs[ci_no])\n newci_no += 1\n maps.ssa[ci_no] = newci_no\n push!(new_ci.code, Expr(:call, GlobalRef(LoggingProfiler, :record_end), fname))\n push!(new_ci.codelocs, ci.codelocs[ci_no])\n newci_no += 1\n else\n push!(new_ci.code, ex)\n push!(new_ci.codelocs, ci.codelocs[ci_no])\n newci_no += 1\n maps.ssa[ci_no] = newci_no\n end\nend","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"which yields","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"julia> new_ci.code\n15-element Vector{Any}:\n :((getindex)(_3, 1))\n :((getindex)(_3, 2))\n :(_4 = _2 * _3)\n :(_4)\n :(Main.LoggingProfiler.record_start(:sin))\n :(Main.overdub(Main.sin, _2))\n :(Main.LoggingProfiler.record_end(:sin))\n :(Main.LoggingProfiler.record_start(:+))\n :(Main.overdub(Main.:+, %2, %3))\n :(Main.LoggingProfiler.record_end(:+))\n :(return %4)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The important parts are:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Depending on the type of expressions (controlled by timable) we decide, if a function's execution time should be recorded.\nfname = exportname(ex) obtains the name of the profiled function call.\npush!(new_ci.code, Expr(:call, GlobalRef(LoggingProfiler, :record_start), fname)) records the start of the exection.\nmaps.goto[ci_ssa_no] = ssa_no updates the map from the code line number in ci to the one in new_ci.\nmaps.ssa[ci_ssa_no] = ssa_no updates the map from the SSA line number in ci to new_ci.\nex = overdubbable(ex) ? Expr(:call, GlobalRef(Main, :overdub), ex.args...) : ex modifies the function call (expression in general) to recurse the overdubbing.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Finally, we need to change the names of slot variables (Core.SlotNumber) and variables indexed by the SSA (Core.SSAValue).","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"for i in length(args)+1:length(new_ci.code)\n new_ci.code[i] = remap(new_ci.code[i], maps)\nend","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"where remap is defined by the following block of code","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"remap(ex::Expr, maps) = Expr(ex.head, remap(ex.args, maps)...)\nremap(args::AbstractArray, maps) = map(a -> remap(a, maps), args)\nremap(c::Core.GotoNode, maps) = Core.GotoNode(maps.goto[c.label])\nremap(c::Core.GotoIfNot, maps) = Core.GotoIfNot(remap(c.cond, maps), maps.goto[c.dest])\nremap(r::Core.ReturnNode, maps) = Core.ReturnNode(remap(r.val, maps))\nremap(a::Core.SlotNumber, maps) = maps.slots[a.id]\nremap(a::Core.SSAValue, maps) = Core.SSAValue(maps.ssa[a.id])\nremap(a::Core.NewvarNode, maps) = Core.NewvarNode(maps.slots[a.slot.id])\nremap(a::GlobalRef, maps) = a\nremap(a::QuoteNode, maps) = a\nremap(ex, maps) = ex","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"warn: Warn\nRetrieving the code properlyConsider the following function:function test(x::T) where T<:Union{Float64, Float32}\n x < T(pi)\nend\n\njulia> ci = @code_lowered test(1.0)\nCodeInfo(\n1 ─ %1 = ($(Expr(:static_parameter, 1)))(Main.pi)\n│ %2 = x < %1\n└── return %2\n)the Expr(:static_parameter, 1) in the first line of code obtains the type parameter T of the function test. Since this information is not accessible in the CodeInfo, it might render our tooling useless. The needed hook is Base.Meta.partially_inline! which partially inlines this into the CodeInfo object. The code to retrieve the CodeInfo adapted from IRTools is a little involved:function retrieve_code_info(sigtypes, world = Base.get_world_counter())\n S = Tuple{map(s -> Core.Compiler.has_free_typevars(s) ? typeof(s.parameters[1]) : s, sigtypes)...}\n _methods = Base._methods_by_ftype(S, -1, world)\n if isempty(_methods) \n @info(\"method $(sigtypes) does not exist\")\n return(nothing)\n end\n type_signature, raw_static_params, method = _methods[1]\n mi = Core.Compiler.specialize_method(method, type_signature, raw_static_params, false)\n ci = Base.isgenerated(mi) ? Core.Compiler.get_staged(mi) : Base.uncompressed_ast(method)\n Base.Meta.partially_inline!(ci.code, [], method.sig, Any[raw_static_params...], 0, 0, :propagate)\n ci\nendbutjulia> ci = retrieve_code_info((typeof(test), Float64))\nCodeInfo(\n @ REPL[5]:2 within `test'\n1 ─ %1 = ($(QuoteNode(Float64)))(Main.pi)\n│ %2 = x < %1\n└── return %2\n)it performs the needed inlining of Float64.","category":"page"},{"location":"lecture_09/lecture/#Implementing-the-profiler-with-IRTools","page":"Lecture","title":"Implementing the profiler with IRTools","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The above implementation of the profiler has shown, that rewriting IR manually is doable, but requires a lot of careful book-keeping. IRTools.jl makes our life much simpler, as they take away all the needed book-keeping and let us focus on what is important.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"using IRTools\nfunction foo(x, y)\n z = x * y\n z + sin(y)\nend;\nir = @code_ir foo(1.0, 1.0)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"We can see that at first sight, the representation of the lowered code in IRTools is similar to that of CodeInfo. Some notable differences:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"SlotNumber are converted to SSAValues\nSSA form is divided into blocks by GotoNode and GotoIfNot in the parsed CodeInfo\nSSAValues do not need to be ordered. The reordering is deffered to the moment when one converts IRTools.Inner.IR back to the CodeInfo.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Let's now use the IRTools to insert the timing statements into the code for foo:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"using IRTools: xcall, insert!, insertafter!\n\nir = @code_ir foo(1.0, 1.0)\nfor (v, ex) in ir\n if timable(ex.expr)\n fname = exportname(ex.expr)\n insert!(ir, v, xcall(LoggingProfiler, :record_start, fname))\n insertafter!(ir, v, xcall(LoggingProfiler, :record_end, fname))\n end\nend\n\njulia> ir\n1: (%1, %2, %3)\n %7 = Main.LoggingProfiler.record_start(:*)\n %4 = %2 * %3\n %8 = Main.LoggingProfiler.record_end(:*)\n %9 = Main.LoggingProfiler.record_start(:sin)\n %5 = Main.sin(%3)\n %10 = Main.LoggingProfiler.record_end(:sin)\n %11 = Main.LoggingProfiler.record_start(:+)\n %6 = %4 + %5\n %12 = Main.LoggingProfiler.record_end(:+)\n return %6","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Observe that the statements are on the right places but they are not ordered. We can turn the ir object into an anonymous function","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"f = IRTools.func(ir)\nLoggingProfiler.reset!()\nf(nothing, 1.0, 1.0)\nLoggingProfiler.to","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"where we can observe that our profiler is working as it should. But this is not yet our final goal. Originally, our goal was to recursivelly dive into the nested functions. IRTools offers a macro @dynamo, which is similar to @generated but simplifies our job by allowing to return the IRTools.Inner.IR object and it also taking care of properly renaming the arguments. With that we write","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"using IRTools: @dynamo\nprofile_fun(f::Core.IntrinsicFunction, args...) = f(args...)\nprofile_fun(f::Core.Builtin, args...) = f(args...)\n\n@dynamo function profile_fun(f, args...)\n ir = IRTools.Inner.IR(f, args...)\n for (v, ex) in ir\n if timable(ex.expr)\n fname = exportname(ex.expr)\n insert!(ir, v, xcall(LoggingProfiler, :record_start, fname))\n insertafter!(ir, v, xcall(LoggingProfiler, :record_end, fname))\n end\n end\n for (x, st) in ir\n recursable(st.expr) || continue\n ir[x] = xcall(profile_fun, st.expr.args...)\n end\n return ir\nend","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"where the first pass is as it was above and the ir[x] = xcall(profile_fun, st.expr.args...) ensures that the profiler will recursively call itself. recursable is a filter defined as below, which is used to prevent profiling itself (and possibly other things).","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"recursable(gr::GlobalRef) = gr.name ∉ [:profile_fun, :record_start, :record_end]\nrecursable(ex::Expr) = ex.head == :call && recursable(ex.args[1])\nrecursable(ex) = false","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Additionally, the first two definitions of profile_fun for Core.IntrinsicFunction and for Core.Builtin prevent trying to dive into functions which do not have a Julia IR. And that's all. The full code is ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"using IRTools\nusing IRTools: var, xcall, insert!, insertafter!, func, recurse!, @dynamo\ninclude(\"loggingprofiler.jl\")\nLoggingProfiler.resize!(LoggingProfiler.to, 10000)\n\nfunction timable(ex::Expr) \n ex.head != :call && return(false)\n length(ex.args) < 2 && return(false)\n ex.args[1] isa Core.GlobalRef && return(true)\n ex.args[1] isa Symbol && return(true)\n return(false)\nend\ntimable(ex) = false\n\nfunction recursable_fun(ex::GlobalRef)\n ex.name ∈ (:profile_fun, :record_start, :record_end) && return(false)\n iswhite(recursable_list, ex) && return(true)\n isblack(recursable_list, ex) && return(false)\n return(isempty(recursable_list) ? true : false)\nend\n\nrecursable_fun(ex::IRTools.Inner.Variable) = true\n\nfunction recursable(ex::Expr) \n ex.head != :call && return(false)\n isempty(ex.args) && return(false)\n recursable(ex.args[1])\nend\n\nrecursable(ex) = false\n\nexportname(ex::GlobalRef) = QuoteNode(ex.name)\nexportname(ex::Symbol) = QuoteNode(ex)\nexportname(ex::Expr) = exportname(ex.args[1])\nexportname(i::Int) = QuoteNode(Symbol(\"Int(\",i,\")\"))\n\nprofile_fun(f::Core.IntrinsicFunction, args...) = f(args...)\nprofile_fun(f::Core.Builtin, args...) = f(args...)\n\n@dynamo function profile_fun(f, args...)\n ir = IRTools.Inner.IR(f, args...)\n for (v, ex) in ir\n if timable(ex.expr)\n fname = exportname(ex.expr)\n insert!(ir, v, xcall(LoggingProfiler, :record_start, fname))\n insertafter!(ir, v, xcall(LoggingProfiler, :record_end, fname))\n end\n end\n for (x, st) in ir\n recursable(st.expr) || continue\n ir[x] = xcall(profile_fun, st.expr.args...)\n end\n # recurse!(ir)\n return ir\nend\n\nmacro record(ex)\n esc(Expr(:call, :profile_fun, ex.args...))\nend\n\nLoggingProfiler.reset!()\n@record foo(1.0, 1.0)\nLoggingProfiler.to","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"where you should notice the long time the first execution of @record foo(1.0, 1.0) takes. This is caused by the compiler specializing for every function into which we dive into. The second execution of @record foo(1.0, 1.0) is fast. It is also interesting to observe how the time of the compilation is logged by the profiler. The output of the profiler to is not shown here due to the length of the output.","category":"page"},{"location":"lecture_09/lecture/#Petite-Zygote","page":"Lecture","title":"Petite Zygote","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"IRTools.jl were created for Zygote.jl –- Julia's source-to-source AD system currently powering Flux.jl. An interesting aspect of Zygote was to recognize that TensorFlow is in its nutshell a compiler, PyTorch is an interpreter. So the idea was to let Julia's compiler compile the gradient and perform optimizations that are normally performed with normal code. Recall that a lot of research went into how to generate efficient code and it is reasonable to use this research. Zygote.jl provides mainly reversediff, but there was an experimental support for forwarddiff.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"One of the questions when developing an AD engine is where and how to create a computation graph. Recall that in TensorFlow, you specify it through a domain specific language, in PyTorch it generated on the fly. Mike Innes' idea was use SSA form provided by the julia compiler. ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_lowered foo(1.0, 1.0)\nCodeInfo(\n1 ─ z = x * y\n│ %2 = z\n│ %3 = Main.sin(y)\n│ %4 = %2 + %3\n└── return %4\n)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"It is very easy to differentiate each line, as they correspond to single expressions (or function calls) and importantly, each variable is assigned exactly once. The strategy to use it for AD would as follows.","category":"page"},{"location":"lecture_09/lecture/#Strategy","page":"Lecture","title":"Strategy","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"We assume to have a set of AD rules (e.g. ChainRules), which for a given function returns its evaluation and pullback. If Zygote.jl is tasked with computing the gradient.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"If a rule exists for this function, directly return the rule.\nIf not, deconstruct the function into a sequence of functions using CodeInfo / IR representation\nReplace statements by calls to obtain the evaluation of the statements and the pullback.\nChain pullbacks in reverse order.\nReturn the function evaluation and the chained pullback.","category":"page"},{"location":"lecture_09/lecture/#Simplified-implementation","page":"Lecture","title":"Simplified implementation","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The following code is adapted from this example","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"using IRTools, ChainRules\nusing IRTools: @dynamo, IR, Pipe, finish, substitute, return!, block, blocks,\n returnvalue, arguments, isexpr, xcall, self, stmt\n\nstruct Pullback{S,T}\n data::T\nend\n\nPullback{S}(data) where S = Pullback{S,typeof(data)}(data)\n\nfunction primal(ir, T = Any)\n pr = Pipe(ir)\n calls = []\n ret = []\n for (v, st) in pr\n ex = st.expr\n if isexpr(ex, :call)\n t = insert!(pr, v, stmt(xcall(Main, :forward, ex.args...), line = st.line))\n pr[v] = xcall(:getindex, t, 1)\n J = push!(pr, xcall(:getindex, t, 2))\n push!(calls, v)\n push!(ret, J)\n end\n end\n pb = Expr(:call, Pullback{T}, xcall(:tuple, ret...))\n return!(pr, xcall(:tuple, returnvalue(block(ir, 1)), pb))\n return finish(pr), calls\nend\n\n@dynamo function forward(m...)\n ir = IR(m...)\n ir == nothing && return :(error(\"Non-differentiable function \", repr(args[1])))\n length(blocks(ir)) == 1 || error(\"control flow is not supported\")\n return primal(ir, Tuple{m...})[1]\nend\n","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"where ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"the generated function forward calls primal to perform AD manual chainrule\nactual chainrule is performed in the for loop\nevery function call is replaced xcall(Main, :forward, ex.args...), which is the recursion we have observed above. stmt allows to insert information about lines in the source code).\nthe output of the forward is the value of the function, and pullback, the function calculating gradient with respect to its inputs.\npr[v] = xcall(:getindex, t, 1) fixes the output of the overwritten function call to be the output of forward(...)\nthe next line logs the pullback \nExpr(:call, Pullback{T}, xcall(:tuple, ret...)) will serve to call generated function which will assemble the pullback in the right order","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Let's now observe how the the IR of foo is transformed","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"ir = IR(typeof(foo), Float64, Float64)\njulia> primal(ir)[1]\n1: (%1, %2, %3)\n %4 = Main.forward(Main.:*, %2, %3)\n %5 = Base.getindex(%4, 1)\n %6 = Base.getindex(%4, 2)\n %7 = Main.forward(Main.sin, %3)\n %8 = Base.getindex(%7, 1)\n %9 = Base.getindex(%7, 2)\n %10 = Main.forward(Main.:+, %5, %8)\n %11 = Base.getindex(%10, 1)\n %12 = Base.getindex(%10, 2)\n %13 = Base.tuple(%6, %9, %12)\n %14 = (Pullback{Any, T} where T)(%13)\n %15 = Base.tuple(%11, %14)\n return %15","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Every function call was transformed into the sequence of forward(...) and obtaining first and second item from the returned typle.\nLine %14 constructs the Pullback, which (as will be seen shortly below) will allow to generate the pullback for the generated function\nLine %15 generates the returned tuple, where the first item is the function value (computed at line %11) and pullback (constructed at libe %15).","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"We define few AD rules by specializing forward with calls from ChainRules","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"forward(::typeof(sin), x) = ChainRules.rrule(sin, x)\nforward(::typeof(*), x, y) = ChainRules.rrule(*, x, y)\nforward(::typeof(+), x, y) = ChainRules.rrule(+, x, y)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Zygote implements this inside the generated function, such that whatever is added to ChainRules is automatically reflected. The process is not as trivial (see has_chain_rule) and for the brevity is not shown here. ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"We now obtain the value and the pullback of function foo as ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"julia> v, pb = forward(foo, 1.0, 1.0);","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The pullback contains in data field with individual jacobians that have been collected in ret in primal function.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"pb.data[1]\npb.data[2]\npb.data[3]","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The function for which the Jacobian has been created is stored in type parameter S of the Pullback type. The pullback for foo is generated in another generated function, as Pullback struct is a functor. This is an interesting design pattern, which allows us to return closure from a generated function. ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Let's now investigate the code generating code for pullback.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"\n_sum() = 0\n_sum(x) = x\n_sum(x...) = xcall(:+, x...)\n\nfunction pullback(pr)\n ir = empty(pr)\n grads = Dict()\n grad(x) = _sum(get(grads, x, [])...)\n grad(x, x̄) = push!(get!(grads, x, []), x̄)\n grad(returnvalue(block(pr, 1)), IRTools.argument!(ir))\n data = push!(ir, xcall(:getfield, self, QuoteNode(:data)))\n _, pbs = primal(pr)\n pbs = Dict(pbs[i] => push!(ir, xcall(:getindex, data, i)) for i = 1:length(pbs))\n for v in reverse(keys(pr))\n ex = pr[v].expr\n isexpr(ex, :call) || continue\n Δs = push!(ir, Expr(:call, pbs[v], grad(v)))\n for (i, x) in enumerate(ex.args)\n grad(x, push!(ir, xcall(:getindex, Δs, i)))\n end\n end\n return!(ir, xcall(:tuple, [grad(x) for x in arguments(pr)]...))\nend\n\n@dynamo function (pb::Pullback{S})(Δ) where S\n return pullback(IR(S.parameters...))\nend","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Let's walk how the reverse is constructed for pr = IR(typeof(foo), Float64, Float64)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"ir = empty(pr)\ngrads = Dict()\ngrad(x) = _sum(get(grads, x, [])...)\ngrad(x, x̄) = push!(get!(grads, x, []), x̄)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"construct the empty ir for the constructed pullback, defines Dict where individual contributors of the gradient with respect to certain variable will be stored, and two function for pushing statements to to grads. The next statement","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"grad(returnvalue(block(pr, 1)), IRTools.argument!(ir))","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"pushes to grads statement that the gradient of the output of the primal pr is provided as an argument of the pullback IRTools.argument!(ir). ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"data = push!(ir, xcall(:getfield, self, QuoteNode(:data)))\n_, pbs = primal(pr)\npbs = Dict(pbs[i] => push!(ir, xcall(:getindex, data, i)) for i = 1:length(pbs))","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"sets data to the data field of the Pullback structure containing pullback functions. Then it create a dictionary pbs, where the output of each call in the primal (identified by the line) is mapped to the corresponding pullback, which is now a line in the IR representation. The IR so far looks as ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"1: (%1)\n %2 = Base.getfield(IRTools.Inner.Self(), :data)\n %3 = Base.getindex(%2, 1)\n %4 = Base.getindex(%2, 2)\n %5 = Base.getindex(%2, 3)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"and pbs contains ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"julia> pbs\nDict{IRTools.Inner.Variable, IRTools.Inner.Variable} with 3 entries:\n %6 => %5\n %4 => %3\n %5 => %4","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"says that the pullback of a function producing variable at line %6 in the primal is stored at variable %5 in the contructed pullback. The real deal comes in the for loop ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"for v in reverse(keys(pr))\n ex = pr[v].expr\n isexpr(ex, :call) || continue\n Δs = push!(ir, Expr(:call, pbs[v], grad(v)))\n for (i, x) in enumerate(ex.args)\n grad(x, push!(ir, xcall(:getindex, Δs, i)))\n end\nend","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"which iterates the primal pr in the reverse order and for every call, it inserts statement to calls the appropriate pullback Δs = push!(ir, Expr(:call, pbs[v], grad(v))) and adds gradients with respect to the inputs to values accumulating corresponding gradient in the loop for (i, x) in enumerate(ex.args) ... The last line","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"return!(ir, xcall(:tuple, [grad(x) for x in arguments(pr)]...))","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"puts statements accumulating gradients with respect to individual variables to the ir.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The final generated IR code looks as","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"julia> pullback(IR(typeof(foo), Float64, Float64))\n1: (%1)\n %2 = Base.getfield(IRTools.Inner.Self(), :data)\n %3 = Base.getindex(%2, 1)\n %4 = Base.getindex(%2, 2)\n %5 = Base.getindex(%2, 3)\n %6 = (%5)(%1)\n %7 = Base.getindex(%6, 1)\n %8 = Base.getindex(%6, 2)\n %9 = Base.getindex(%6, 3)\n %10 = (%4)(%9)\n %11 = Base.getindex(%10, 1)\n %12 = Base.getindex(%10, 2)\n %13 = (%3)(%8)\n %14 = Base.getindex(%13, 1)\n %15 = Base.getindex(%13, 2)\n %16 = Base.getindex(%13, 3)\n %17 = %12 + %16\n %18 = Base.tuple(0, %15, %17)\n return %18","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"and it calculates the gradient with respect to the input as","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"julia> pb(1.0)\n(0, 1.0, 1.5403023058681398)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"where the first item is gradient with parameters of the function itself.","category":"page"},{"location":"lecture_09/lecture/#Conclusion","page":"Lecture","title":"Conclusion","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The above examples served to demonstrate that @generated functions offers extremely powerful paradigm, especially if coupled with manipulation of intermediate representation. Within few lines of code, we have implemented reasonably powerful profiler and reverse AD engine. Importantly, it has been done without a single-purpose engine or tooling. ","category":"page"},{"location":"lecture_10/hw/#hw09","page":"Homework","title":"Homework 9: Accelerating 1D convolution with threads","text":"","category":"section"},{"location":"lecture_10/hw/#How-to-submit","page":"Homework","title":"How to submit","text":"","category":"section"},{"location":"lecture_10/hw/","page":"Homework","title":"Homework","text":"Put all the code of inside hw.jl. Zip only this file (not its parent folder) and upload it to BRUTE. You should not not import anything but Base.Threads or just Threads. ","category":"page"},{"location":"lecture_10/hw/","page":"Homework","title":"Homework","text":"
\n
Homework (2 points)
\n
","category":"page"},{"location":"lecture_10/hw/","page":"Homework","title":"Homework","text":"Implement multithreaded discrete 1D convolution operator[1] without padding (output will be shorter). The required function signature: thread_conv1d(x, w), where x is the signal array and w the kernel. For testing correctness of the implementation you can use the following example of a step function and it's derivative realized by kernel [-1, 1]:","category":"page"},{"location":"lecture_10/hw/","page":"Homework","title":"Homework","text":"using Test\n@test all(thread_conv1d(vcat([0.0, 0.0, 1.0, 1.0, 0.0, 0.0]), [-1.0, 1.0]) .≈ [0.0, -1.0, 0.0, 1.0, 0.0])","category":"page"},{"location":"lecture_10/hw/","page":"Homework","title":"Homework","text":"[1]: Discrete convolution with finite support https://en.wikipedia.org/wiki/Convolution#Discrete_convolution","category":"page"},{"location":"lecture_10/hw/","page":"Homework","title":"Homework","text":"Your parallel implementation will be tested both in sequential and two threaded mode with the following inputs","category":"page"},{"location":"lecture_10/hw/","page":"Homework","title":"Homework","text":"using Random\nRandom.seed!(42)\nx = rand(10_000_000)\nw = [1.0, 2.0, 4.0, 2.0, 1.0]\n@btime thread_conv1d($x, $w);","category":"page"},{"location":"lecture_10/hw/","page":"Homework","title":"Homework","text":"On your local machine you should be able to achieve 0.6x reduction in execution time with two threads, however the automatic eval system is a noisy environment and therefore we require only 0.8x reduction therein. This being said, please reach out to us, if you encounter any issues.","category":"page"},{"location":"lecture_10/hw/","page":"Homework","title":"Homework","text":"HINTS:","category":"page"},{"location":"lecture_10/hw/","page":"Homework","title":"Homework","text":"start with single threaded implementation\ndon't forget to reverse the kernel\n@threads macro should be all you need\nfor testing purposes create a simple script, that you can run with julia -t 1 and julia -t 2","category":"page"},{"location":"lecture_10/hw/","page":"Homework","title":"Homework","text":"
\n","category":"page"},{"location":"lecture_07/hw/#hw07","page":"Homework","title":"Homework 7: Creating world in 3 days/steps","text":"","category":"section"},{"location":"lecture_07/hw/#How-to-submit","page":"Homework","title":"How to submit","text":"","category":"section"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"Put all the code of inside hw.jl. Zip only this file (not its parent folder) and upload it to BRUTE. You should assume that only","category":"page"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"using Ecosystem","category":"page"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"will be put before your file is executed, but do not include them in your solution. The version of Ecosystem pkg should be the same as in HW4.","category":"page"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"
\n
Homework (2 points)
\n
","category":"page"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"Create a macro @ecosystem that should be able to define a world given a list of statements @add # $species ${optional:sex}","category":"page"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"world = @ecosystem begin\n @add 10 Sheep female # adds 10 female sheep\n @add 2 Sheep male # adds 2 male sheep\n @add 100 Grass # adds 100 pieces of grass\n @add 3 Wolf # adds 5 wolf with random sex\nend","category":"page"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"@add should not be treated as a macro, but rather just as a syntax, that can be easily matched.","category":"page"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"As this is not a small task let's break it into 3 steps. (These intemediate steps will also be checked in BRUTE.)","category":"page"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"Define method default_config(::Type{T}) for each T in Grass, Wolf,..., which returns a named tuple of default parameters for that particular agent (you can choose the default values however you like).\nDefine method _add_agents(max_id, count::Int, species::Type{<:Species}) and _add_agents(max_id, count::Int, species::Type{<:AnimalSpecies}, sex::Sex) that return an array of count agents of species species with id going from max_id+1 to max_id+count. Default parameters should be constructed with default_config. Make sure you can handle even animals with random sex (@add 3 Wolf).\nDefine the underlying function _ecosystem(ex), which parses the block expression and creates a piece of code that constructs the world.","category":"page"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"You can test the macro (more precisely the _ecosystem function) with the following expression","category":"page"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"ex = :(begin\n @add 10 Sheep female\n @add 2 Sheep male\n @add 100 Grass\n @add 3 Wolf\nend)\ngenex = _ecosystem(ex)\nworld = eval(genex)","category":"page"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"
\n","category":"page"},{"location":"lecture_10/lab/#parallel_lab","page":"Lab","title":"Lab 10: Parallel computing","text":"","category":"section"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"In this lab we are going to introduce tools that Julia's ecosystem offers for different ways of parallel computing. As an ilustration for how capable Julia was/is consider the fact that it has joined (alongside C,C++ and Fortran) the so-called \"PetaFlop club\"[1], a list of languages capable of running at over 1PFLOPS.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"[1]: Blogpost \"Julia Joins Petaflop Club\" https://juliacomputing.com/media/2017/09/julia-joins-petaflop-club/","category":"page"},{"location":"lecture_10/lab/#Introduction","page":"Lab","title":"Introduction","text":"","category":"section"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Nowadays there is no need to convince anyone about the advantages of having more cores available for your computation be it on a laptop, workstation or a cluster. The trend can be nicely illustrated in the figure bellow: (Image: 42-cpu-trend) Image source[2]","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"[2]: Performance metrics trend of CPUs in the last 42years: https://www.karlrupp.net/2018/02/42-years-of-microprocessor-trend-data/","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"However there are some shortcomings when going from sequential programming, that we have to note","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"We don't think in parallel\nWe learn to write and reason about programs serially\nThe desire for parallelism often comes after you've written your algorithm (and found it too slow!)\nHarder to reason and therefore harder to debug\nThe number of cores is increasing, thus knowing how the program scales is crucial (not just that it runs better)\nBenchmarking parallel code, that tries to exhaust the processor pool is much more affected by background processes","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"warning: Shortcomings of parallelism\nParallel computing brings its own set of problems and not an insignificant overhead with data manipulation and communication, therefore try always to optimize your serial code as much as you can before advancing to parallel acceleration.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"warning: Disclaimer\nWith the increasing complexity of computer HW some statements may become outdated. Moreover we won't cover as many tips that you may encounter on a parallel programming specific course, which will teach you more in the direction of how to think in parallel, whereas here we will focus on the tools that you can use to realize the knowledge gained therein.","category":"page"},{"location":"lecture_10/lab/#Process-based-parallelism","page":"Lab","title":"Process based parallelism","text":"","category":"section"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"As the name suggest process based parallelism is builds on the concept of running code on multiple processes, which can run even on multiple machines thus allowing to scale computing from a local machine to a whole network of machines - a major difference from the other parallel concept of threads. In Julia this concept is supported within standard library Distributed and the scaling to cluster can be realized by 3rd party library ClusterManagers.jl.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Let's start simply with knowing how to start up additional Julia processes. There are two ways:","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"by adding processes using cmd line argument -p ##","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"julia -p 4","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"by adding processes after startup using the addprocs(##) function from std library Distributed","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"julia> using Distributed\njulia> addprocs(4) # returns a list of ids of individual processes\n4-element Vector{Int64}:\n 2\n 3\n 4\n 5\njulia> nworkers() # returns number of workers\n4\njulia> nprocs() # returns number of processes `nworkers() + 1`\n5","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"The result shown in a process manager such as htop:","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":".../julia-1.6.2/bin/julia --project \n.../julia-1.6.2/bin/julia -Cnative -J/home/honza/Apps/julia-1.6.2/lib/julia/sys.so -g1 --bind-to 127.0.0.1 --worker\n.../julia-1.6.2/bin/julia -Cnative -J/home/honza/Apps/julia-1.6.2/lib/julia/sys.so -g1 --bind-to 127.0.0.1 --worker\n.../julia-1.6.2/bin/julia -Cnative -J/home/honza/Apps/julia-1.6.2/lib/julia/sys.so -g1 --bind-to 127.0.0.1 --worker\n.../julia-1.6.2/bin/julia -Cnative -J/home/honza/Apps/julia-1.6.2/lib/julia/sys.so -g1 --bind-to 127.0.0.1 --worker","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Both of these result in total of 5 running processes - 1 controller, 4 workers - with their respective ids accessible via myid() function call. Note that the controller process has always id 1 and other processes are assigned subsequent integers, see for yourself with @everywhere macro, which runs easily code on all or a subset of processes.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"@everywhere println(myid())\n@everywhere [2,3] println(myid()) # select a subset of workers","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"The same way that we have added processes we can also remove them","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"julia> workers() # returns array of worker ids\n4-element Vector{Int64}:\n 2\n 3\n 4\n 5\njulia> rmprocs(2) # kills worker with id 2\nTask (done) @0x00007ff2d66a5e40\njulia> workers()\n3-element Vector{Int64}:\n 3\n 4\n 5","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"As we have seen from the htop/top output, added processes start with specific cmd line arguments, however they are not shared with any aliases that we may have defined, e.g. julia ~ julia --project=.. Therefore in order to use an environment, we have to first activate it on all processes","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"@everywhere begin\n using Pkg; Pkg.activate(@__DIR__) # @__DIR__ equivalent to a call to pwd()\nend","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"or we can load files containing this line on all processes with cmdline option -L ###.jl together with -p ##.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"There are generally two ways of working with multiple processes","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"using low level functionality - we specify what/where is loaded, what/where is being run and when we fetch results\n@everywhere to run everywhere and wait for completion\n@spawnat and remotecall to run at specific process and return Future (a reference to a future result - remote reference)\nfetch - fetching remote reference\npmap - for easily mapping a function over a collection\nusing high level functionality - define only simple functions and apply them on collections\nDistributedArrays' with DArrays\nTransducers.jl pipelines\nDagger.jl out-of-core and parallel computing","category":"page"},{"location":"lecture_10/lab/#Sum-with-processes","page":"Lab","title":"Sum with processes","text":"","category":"section"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Writing your own sum of an array function is a good way to show all the potential problems, you may encounter with parallel programming. For comparison here is the naive version that uses zero for initialization and @inbounds for removing boundschecks.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"function naive_sum(a)\n r = zero(eltype(a))\n for aᵢ in a\n r += aᵢ\n end\n r\nend","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Its performance will serve us as a sequential baseline.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"julia> using BenchmarkTools\njulia> a = rand(10_000_000); # 10^7\njulia> sum(a) ≈ naive_sum(a)\ntrue\njulia> @btime sum($a)\n5.011 ms (0 allocations: 0 bytes)\njulia> @btime naive_sum($a)\n11.786 ms (0 allocations: 0 bytes)","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Note that the built-in sum exploits single core parallelism with Single instruction, multiple data (SIMD instructions) and is thus faster.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Write a distributed/multiprocessing version of sum function dist_sum(a, np=nworkers()) without the help of DistributedArrays. Measure the speed up when doubling the number of workers (up to the number of logical cores - see note on hyper threading).","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"map builtin sum over chunks of the array using pmap\nthere are built in partition iterators Iterators.partition(array, chunk_size)\nchunk_size should relate to the number of available workers\npmap has the option to pass the ids of workers as the second argument pmap(f, WorkerPool([2,4]), collection)\npmap collects the partial results to the controller where it can be collected with another sum","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"using Distributed\naddprocs(4)\n\n@everywhere begin\n using Pkg; Pkg.activate(@__DIR__)\nend\n\nfunction dist_sum(a, np=nworkers())\n chunk_size = div(length(a), np)\n sum(pmap(sum, WorkerPool(workers()[1:np]), Iterators.partition(a, chunk_size)))\nend\n\ndist_sum(a) ≈ sum(a)\n@btime dist_sum($a)\n\n@time dist_sum(a, 1) # 74ms \n@time dist_sum(a, 2) # 46ms\n@time dist_sum(a, 4) # 49ms\n@time dist_sum(a, 8) # 35ms","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"As you can see the built-in pmap already abstracts quite a lot from the process and all the data movement is handled internally, however in order to show off how we can abstract even more, let's use the DistributedArrays.jl pkg.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Write a distributed/multiprocessing version of sum function dist_sum_lib(a, np=nworkers()) with the help of DistributedArrays. Measure the speed up when doubling the number of workers (up to the number of logical cores - see note on hyper threading).","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"chunking and distributing the data can be handled for us using the distribute function on an array (creates a DArray)\ndistribute has an option to specify on which workers should an array be distributed to\nsum function has a method for DArray\nremember to run using DistributedArrays on every process","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Setting up.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"using Distributed\naddprocs(8)\n\n@everywhere begin\n using Pkg; Pkg.activate(@__DIR__)\nend\n\n@everywhere begin\n using DistributedArrays\nend ","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"And the actual computation.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"adist = distribute(a) # distribute array to workers |> typeof - DArray\n@time adist = distribute(a) # we should not disregard this time\n@btime sum($adist) # call the built-in function (dispatch on DArrray)\n\nfunction dist_sum_lib(a, np=nworkers())\n adist = distribute(a, procs = workers()[1:np])\n sum(adist)\nend\n\ndist_sum_lib(a) ≈ sum(a)\n@btime dist_sum_lib($a)\n\n@time dist_sum_lib(a, 1) # 80ms \n@time dist_sum_lib(a, 2) # 54ms\n@time dist_sum_lib(a, 4) # 48ms\n@time dist_sum_lib(a, 8) # 33ms","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"In both previous examples we have included the data transfer time from the controller process, in practice however distributed computing is used in situations where the data may be stored on individual local machines. As a general rule of thumb we should always send only instruction what to do and not the actual data to be processed. This will be more clearly demonstrated in the next more practical example.","category":"page"},{"location":"lecture_10/lab/#lab10_dist_file_p","page":"Lab","title":"Distributed file processing","text":"","category":"section"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Distributed is often used in processing of files, such as the commonly encountered mapreduce jobs with technologies like Hadoop, Spark, where the files live on a distributed file system and a typical job requires us to map over all the files and gather some statistics such as histograms, sums and others. We will simulate this situation with the Julia's pkg codebase, which on a typical user installation can contain up to hundreds of thousand of .jl files (depending on how extensively one uses Julia).","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Write a distributed pipeline for computing a histogram of symbols found in AST by parsing Julia source files in your .julia/packages/ directory. We have already implemented most of the code that you will need (available as source code here).","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\npkg_processing.jl

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"using Markdown #hide\ncode = Markdown.parse(\"\"\"```julia\\n$(readchomp(\"./pkg_processing.jl\"))\\n```\"\"\") #hide\ncode","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Your task is to write a function that does the map and reduce steps, that will create and gather the dictionaries from different workers. There are two ways to do a map","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"either over directories inside .julia/packages/ - call it distributed_histogram_pkgwise\nor over all files obtained by concatenation of filter_jl outputs (NOTE that this might not be possible if the listing itself is expensive - speed or memory requirements) - call it distributed_histogram_filewise","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Measure if the speed up scales linearly with the number of processes by restricting the number of workers inside a pmap.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"for each file path apply tokenize to extract symbols and follow it with the update of a local histogram\ntry writing sequential version first\neither load ./pkg_processing.jl on startup with -L and -p options or include(\"./pkg_processing.jl\") inside @everywhere\nuse pmap to easily iterate in parallel over a collection - the result should be an array of histogram, which has to be merged on the controller node (use builtin mergewith! function in conjunction with reduce)\npmap supports do syntax","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"pmap(collection) do item\n do_something(item)\nend","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"pkg directory can be obtained with joinpath(DEPOT_PATH[1], \"packages\")","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"BONUS: What is the most frequent symbol in your codebase?","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Let's implement first a sequential version as it is much easier to debug.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"include(\"./pkg_processing.jl\")\n\nusing ProgressMeter\nfunction sequential_histogram(path)\n h = Dict{Symbol, Int}()\n @showprogress for pkg_dir in sample_all_installed_pkgs(path)\n for jl_path in filter_jl(pkg_dir)\n syms = tokenize(jl_path)\n for s in syms\n v = get!(h, s, 0)\n h[s] += 1\n end\n end\n end\n h\nend\npath = joinpath(DEPOT_PATH[1], \"packages\") # usually the first entry\n@time h = sequential_histogram(path) # 87s","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"First we try to distribute over package folders. TODO add the ability to run it only on some workers","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"using Distributed\naddprocs(8)\n\n@everywhere begin\n using Pkg; Pkg.activate(@__DIR__)\n # we have to realize that the code that workers have access to functions we have defined\n include(\"./pkg_processing.jl\") \nend\n\n\"\"\"\n merge_with!(h1, h2)\n\nMerges count dictionary `h2` into `h1` by adding the counts. Equivalent to `Base.mergewith!(+)`.\n\"\"\"\nfunction merge_with!(h1, h2)\n for s in keys(h2)\n get!(h1, s, 0)\n h1[s] += h2[s]\n end\n h1\nend\n\nusing ProgressMeter\nfunction distributed_histogram_pkgwise(path, np=nworkers())\n r = @showprogress pmap(WorkerPool(workers()[1:np]), sample_all_installed_pkgs(path)) do pkg_dir\n h = Dict{Symbol, Int}()\n for jl_path in filter_jl(pkg_dir)\n syms = tokenize(jl_path)\n for s in syms\n v = get!(h, s, 0)\n h[s] += 1\n end\n end\n h\n end\n reduce(merge_with!, r)\nend\npath = joinpath(DEPOT_PATH[1], \"packages\")\n\n@time h = distributed_histogram_pkgwise(path, 2) # 41.5s\n@time h = distributed_histogram_pkgwise(path, 4) # 24.0s\n@time h = distributed_histogram_pkgwise(path, 8) # 24.0s","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Second we try to distribute over all files.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"function distributed_histogram_filewise(path, np=nworkers())\n jl_files = reduce(vcat, filter_jl(pkg_dir) for pkg_dir in sample_all_installed_pkgs(path))\n r = @showprogress pmap(WorkerPool(workers()[1:np]), jl_files) do jl_path\n h = Dict{Symbol, Int}()\n syms = tokenize(jl_path)\n for s in syms\n v = get!(h, s, 0)\n h[s] += 1\n end\n h\n end\n reduce(merge_with!, r)\nend\npath = joinpath(DEPOT_PATH[1], \"packages\")\n@time h = distributed_histogram_pkgwise(path, 2) # 46.9s\n@time h = distributed_histogram_pkgwise(path, 4) # 24.8s\n@time h = distributed_histogram_pkgwise(path, 8) # 20.4s","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Here we can see that we have improved the timings a bit by increasing granularity of tasks.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"BONUS: You can do some analysis with DataFrames","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"using DataFrames\ndf = DataFrame(:sym => collect(keys(h)), :count => collect(values(h)));\nsort!(df, :count, rev=true);\ndf[1:50,:]","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_10/lab/#lab10_thread","page":"Lab","title":"Threading","text":"","category":"section"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"The number of threads for a Julia process can be set up in an environmental variable JULIA_NUM_THREADS or directly on Julia startup with cmd line option -t ## or --threads ##. If both are specified the latter takes precedence.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"julia -t 8","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"In order to find out how many threads are currently available, there exist the nthreads function inside Base.Threads library. There is also an analog to the Distributed myid example, called threadid.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"julia> using Base.Threads\njulia> nthreads()\n8\njulia> threadid()\n1","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"As opposed to distributed/multiprocessing programming, threads have access to the whole memory of Julia's process, therefore we don't have to deal with separate environment manipulation, code loading and data transfers. However we have to be aware of the fact that memory can be modified from two different places and that there may be some performance penalties of accessing memory that is physically further from a given core (e.g. caches of different core or different NUMA[3] nodes). Another significant difference from distributed computing is that we cannot spawn additional threads on the fly in the same way that we have been able to do with addprocs function.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"[3]: NUMA - https://en.wikipedia.org/wiki/Non-uniform_memory_access","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"info: Hyper threads\nIn most of today's CPUs the number of threads is larger than the number of physical cores. These additional threads are usually called hyper threads[4] or when talking about cores - logical cores. The technology relies on the fact, that for a given \"instruction\" there may be underutilized parts of the CPU core's machinery (such as one of many arithmetic units) and if a suitable work/instruction comes in it can be run simultaneously. In practice this means that adding more threads than physical cores may not be accompanied with the expected speed up.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"[4]: Hyperthreading - https://en.wikipedia.org/wiki/Hyper-threading","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"The easiest (not always yielding the correct result) way how to turn a code into multi threaded code is putting the @threads macro in front of a for loop, which instructs Julia to run the body on separate threads.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"julia> A = Array{Union{Int,Missing}}(missing, nthreads());\njulia> for i in 1:nthreads()\n A[threadid()] = threadid()\nend\njulia> A # only the first element is filled\n8-element Vector{Union{Missing, Int64}}:\n 1\n missing\n missing\n missing\n missing\n missing\n missing\n missing","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"julia> A = Array{Union{Int,Missing}}(missing, nthreads());\njulia> @threads for i in 1:nthreads()\n A[threadid()] = threadid()\nend\njulia> A # the expected results\n8-element Vector{Union{Missing, Int64}}:\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8","category":"page"},{"location":"lecture_10/lab/#Multithreaded-sum","page":"Lab","title":"Multithreaded sum","text":"","category":"section"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Armed with this knowledge let's tackle the problem of the simple sum.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"function threaded_sum_naive(a)\n r = zero(eltype(a))\n @threads for i in eachindex(a)\n @inbounds r += a[i]\n end\n return r\nend","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Comparing this with the built-in sum we see not an insignificant discrepancy (one that cannot be explained by reordering of computation) and moreover the timings show us some ridiculous overhead.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"julia> using BenchmarkTools\njulia> a = rand(10_000_000); # 10^7\njulia> sum(a), threaded_sum_naive(a)\n(5.000577175855193e6, 625888.2270955174)\njulia> @btime sum($a)\n 4.861 ms (0 allocations: 0 bytes)\njulia> @btime threaded_sum_naive($a)\n 163.379 ms (20000042 allocations: 305.18 MiB)","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Recalling what has been said above we have to be aware of the fact that the data can be accessed from multiple threads at once, which if not taken into an account means that each thread reads possibly outdated value and overwrites it with its own updated state. ","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"There are two solutions which we will tackle in the next two exercises. ","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Implement threaded_sum_atom, which uses Atomic wrapper around the accumulator variable r in order to ensure correct locking of data access. ","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"use atomic_add! as a replacement of r += A[i]\n\"collect\" the result by dereferencing variable r with empty bracket operator []","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"info: Side note on dereferencing\nIn Julia we can create references to a data types, which are guarranteed to point to correct and allocated type in memory, as long as a reference exists the memory is not garbage collected. These are constructed with Ref(x), Ref(a, 7) or Ref{T}() for reference to variable x, 7th element of array a and an empty reference respectively. Dereferencing aka asking about the underlying value is done using empty bracket operator [].x = 1 # integer\nrx = Ref(x) # reference to that particular integer `x`\nx == rx[] # dereferencing yields the same valueThere also exist unsafe references/pointers Ptr, however we should not really come into a contact with those.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"BONUS: Try chunking the array and calling sum on individual chunks to obtain some real speedup.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"function threaded_sum_atom(a)\n r = Atomic{eltype(a)}(zero(eltype(a)))\n @threads for i in eachindex(a)\n @inbounds atomic_add!(r, a[i])\n end\n return r[]\nend\n\njulia> sum(a) ≈ threaded_sum_atom(a)\ntrue\njulia> @btime threaded_sum_atom($a)\n 661.502 ms (42 allocations: 3.66 KiB)","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"That's better but far from the performance we need. ","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"BONUS: There is a fancier and faster way to do this by chunking the array","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"function threaded_sum_fancy_atom(a)\n r = Atomic{eltype(a)}(zero(eltype(a)))\n len, rem = divrem(length(a), nthreads())\n @threads for t in 1:nthreads()\n rₜ = zero(eltype(a))\n @simd for i in (1:len) .+ (t-1)*len\n @inbounds rₜ += a[i]\n end\n atomic_add!(r, rₜ)\n end\n # catch up any stragglers\n result = r[]\n @simd for i in length(a)-rem+1:length(a)\n @inbounds result += a[i]\n end\n return result\nend\n\njulia> sum(a) ≈ threaded_sum_fancy_atom(a)\ntrue\njulia> @btime threaded_sum_fancy_atom($a)\n 2.983 ms (42 allocations: 3.67 KiB)","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Finally we have beaten the \"sequential\" sum. The quotes are intentional, because the Base's implementation of a sum uses Single instruction, multiple data (SIMD) instructions as well, which allow to process multiple elements at once.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Implement threaded_sum_buffer, which uses an array of length nthreads() (we will call this buffer) for local aggregation of results of individual threads. ","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"use threadid() to index the buffer array\nsum the buffer array to obtain final result","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"function threaded_sum_buffer(a)\n R = zeros(eltype(a), nthreads())\n @threads for i in eachindex(a)\n @inbounds R[threadid()] += a[i]\n end\n r = zero(eltype(a))\n # sum the partial results from each thread\n for i in eachindex(R)\n @inbounds r += R[i]\n end\n return r\nend\n\njulia> sum(a) ≈ threaded_sum_buffer(a)\ntrue\njulia> @btime threaded_sum_buffer($a)\n 2.750 ms (42 allocations: 3.78 KiB)","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Though this implementation is cleaner and faster, there is possible drawback with this implementation, as the buffer R lives in a continuous part of the memory and each thread that accesses it brings it to its caches as a whole, thus invalidating the values for the other threads, which it in the same way.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Seeing how multithreading works on a simple example, let's apply it on the \"more practical\" case of the Symbol histogram from exercise above.","category":"page"},{"location":"lecture_10/lab/#lab10_dist_file_t","page":"Lab","title":"Multithreaded file processing","text":"","category":"section"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Write a multithreaded analog of the file processing pipeline from exercise above. Again the task is to write the map and reduce steps, that will create and gather the dictionaries from different workers. There are two ways to map","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"either over directories inside .julia/packages/ - threaded_histogram_pkgwise\nor over all files obtained by concatenation of filter_jl outputs - threaded_histogram_filewise","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Compare the speedup with the version using process based parallelism.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"create a separate dictionary for each thread in order to avoid the need for atomic operations\n","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"BONUS: In each of the cases count how many files/pkgs each thread processed. Would the dynamic scheduler help us in this situation?","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Setup is now much simpler.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"using Base.Threads\ninclude(\"./pkg_processing.jl\") \npath = joinpath(DEPOT_PATH[1], \"packages\")","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Firstly the version with folder-wise parallelism.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"function threaded_histogram_pkgwise(path)\n ht = [Dict{Symbol, Int}() for _ in 1:nthreads()]\n @threads for pkg_dir in sample_all_installed_pkgs(path)\n h = ht[threadid()]\n for jl_path in filter_jl(pkg_dir)\n syms = tokenize(jl_path)\n for s in syms\n v = get!(h, s, 0)\n h[s] += 1\n end\n end\n end\n reduce(mergewith!(+), ht)\nend\n\njulia> @time h = threaded_histogram_pkgwise(path)\n 26.958786 seconds (81.69 M allocations: 10.384 GiB, 4.58% gc time)","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Secondly the version with file-wise parallelism.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"function threaded_histogram_filewise(path)\n jl_files = reduce(vcat, filter_jl(pkg_dir) for pkg_dir in sample_all_installed_pkgs(path))\n ht = [Dict{Symbol, Int}() for _ in 1:nthreads()]\n @threads for jl_path in jl_files\n h = ht[threadid()]\n syms = tokenize(jl_path)\n for s in syms\n v = get!(h, s, 0)\n h[s] += 1\n end\n end\n reduce(mergewith!(+), ht)\nend\n\njulia> @time h = threaded_histogram_filewise(path)\n 29.677184 seconds (81.66 M allocations: 10.411 GiB, 4.13% gc time)","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_10/lab/#Task-switching","page":"Lab","title":"Task switching","text":"","category":"section"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"There is a way how to run \"multiple\" things at once, which does not necessarily involve either threads or processes. In Julia this concept is called task switching or asynchronous programming, where we fire off our requests in a short time and let the cpu/os/network handle the distribution. As an example which we will try today is querying a web API, which has some variable latency. In the usuall sequantial fashion we can always post queries one at a time, however generally the APIs can handle multiple request at a time, therefore in order to better utilize them, we can call them asynchronously and fetch all results later, in some cases this will be faster.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"info: Burst requests\nIt is a good practice to check if an API supports some sort of batch request, because making a burst of single request might lead to a worse performance for others and a possible blocking of your IP/API key.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Consider following functions","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"function a()\n for i in 1:10\n sleep(1)\n end\nend\n\nfunction b()\n for i in 1:10\n @async sleep(1)\n end\nend\n\nfunction c()\n @sync for i in 1:10\n @async sleep(1)\n end\nend","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"How much time will the execution of each of them take?","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\nSolution

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"@time a() # 10s\n@time b() # ~0s\n@time c() # >~1s","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Choose one of the free web APIs and query its endpoint using the HTTP.jl library. Implement both sequential and asynchronous version. Compare them on an burst of 10 requests.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"use HTTP.request for GET requests on your chosen API, e.g. r = HTTP.request(\"GET\", \"https://catfact.ninja/fact\") for random cat fact\nconverting body of a response can be done simply by constructing a String out of it - String(r.body)\nin order to parse a json string use JSON.jl's parse function\nJulia offers asyncmap - asynchronous map","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"using HTTP, JSON\n\nfunction query_cat_fact()\n r = HTTP.request(\"GET\", \"https://catfact.ninja/fact\")\n j = String(r.body)\n d = JSON.parse(j)\n d[\"fact\"]\nend\n\n# without asyncmap\nfunction get_cat_facts_async(n)\n facts = Vector{String}(undef, n)\n @sync for i in 1:10\n @async facts[i] = query_cat_fact()\n end\n facts\nend\n\nget_cat_facts_async(n) = asyncmap(x -> query_cat_fact(), Base.OneTo(n))\nget_cat_facts(n) = map(x -> query_cat_fact(), Base.OneTo(n))\n\n@time get_cat_facts_async(10) # ~0.15s\n@time get_cat_facts(10) # ~1.1s","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_10/lab/#Resources","page":"Lab","title":"Resources","text":"","category":"section"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"parallel computing course by Julia Computing","category":"page"},{"location":"lecture_10/lecture/#Parallel-programming-with-Julia","page":"Lecture","title":"Parallel programming with Julia","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Julia offers different levels of parallel programming","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"distributed processing, where jobs are split among different Julia processes\nmulti-threadding, where jobs are split among multiple threads within the same processes\nSIMD instructions\nTask switching.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"In this lecture, we will focus mainly on the first two, since SIMD instructions are mainly used for low-level optimization (such as writing your own very performant BLAS library), and task switching is not a true paralelism, but allows to run a different task when one task is waiting for example for IO.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"The most important lesson is that before you jump into the parallelism, be certain you have made your sequential code as fast as possible.","category":"page"},{"location":"lecture_10/lecture/#Process-level-paralelism","page":"Lecture","title":"Process-level paralelism","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Process-level paralelism means we run several instances of Julia (in different processes) and they communicate between each other using inter-process communication (IPC). The implementation of IPC differs if parallel julia instances share the same machine, or they are on different machines spread over the network. By default, different processes do not share any libraries or any variables. They are loaded clean and it is up to the user to set-up all needed code and data.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Julia's default modus operandi is a single main instance controlling several workers. This main instance has myid() == 1, worker processes receive higher numbers. Julia can be started with multiple workers from the very beggining, using -p switch as ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"julia -p n","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"where n is the number of workers, or you can add workers after Julia has been started by","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using Distributed\naddprocs(n)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"You can also remove workers using rmprocs. When Julia is started with -p, Distributed library is loaded by default on main worker. Workers can be on the same physical machines, or on different machines. Julia offer integration via ClusterManagers.jl with most schedulling systems.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"If you want to evaluate piece of code on all workers including main process, a convenience macro @everywhere is offered.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere @show myid()","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"As we have mentioned, workers are loaded without libraries. We can see that by running","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere InteractiveUtils.varinfo()","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"which fails, but after loading InteractiveUtils everywhere","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using Statistics\n@everywhere begin \n\tusing InteractiveUtils\n\tprintln(InteractiveUtils.varinfo(;imported = true))\nend","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"we see that Statistics was loaded only on the main process. Thus, there is not magical sharing of data and code. With @everywhere macro we can define function and variables, and import libraries on workers as ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tfoo(x, y) = x * y + sin(y)\n\tfoo(x) = foo(x, myid())\n\tx = rand()\nend\n@everywhere @show foo(1.0)\n@everywhere @show x","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"The fact that x has different values on different workers and master demonstrates again the independency of processes. While we can set up everything using @everywhere macro, we can also put all the code for workers into a separate file, e.g. worker.jl and load it on all workers using -L worker.jl.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Julia's multi-processing model is based on message-passing paradigm, but the abstraction is more akin to procedure calls. This means that users are saved from prepending messages with headers and implementing logic deciding which function should be called for thich header. Instead, we can schedulle an execution of a function on a remote worker and return the control immeadiately to continue in our job. A low-level function providing this functionality is remotecall(fun, worker_id, args...). For example ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tfunction delayed_foo(x, y, n )\n\t\tsleep(n)\n\t\tprintln(\"woked up\")\n\t\tfoo(x, y)\n\tend\nend\nr = remotecall(delayed_foo, 2, 1, 1, 60)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"returns immediately, even though the function will take at least 60 seconds. r does not contain result of foo(1, 1), but a struct Future, which is a remote reference in Julia's terminology. It points to data located on some machine, indicates, if they are available and allows to fetch them from the remote worker. fetch is blocking, which means that the execution is blocked until data are available (if they are never available, the process can wait forever.) The presence of data can be checked using isready, which in case of Future returned from remote_call indicate that the computation has finished.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"isready(r)\nfetch(r) == foo(1, 1)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"An advantage of the remote reference is that it can be freely shared around processes and the result can be retrieved on different node then the one which issued the call.s","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"r = remotecall(delayed_foo, 2, 1, 1, 60)\nremotecall(r -> println(\"value: \",fetch(r), \" retrieved on \", myid()) , 3, r)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"An interesting feature of fetch is that it re-throw an exception raised on a different process.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tfunction exfoo()\n\t\tthrow(\"Exception from $(myid())\")\n\tend\nend\nr = @spawnat 2 exfoo()","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"where we have used @spawnat instead of remote_call. It is higher level alternative executing a closure around the expression (in this case exfoo()) on a specified worker, in this case 2. Coming back to the example, when we fetch the result r, the exception is throwed on the main process, not on the worker","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"fetch(r)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@spawnat can be executed with :any to signal that the user does not care, where the function will be executed and it will be left up to Julia.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"r = @spawnat :any foo(1,1)\nfetch(r)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Finally, if you would for some reason need to wait for the computed value, you can use ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"remotecall_fetch(foo, 2, 1, 1)","category":"page"},{"location":"lecture_10/lecture/#Running-example:-Julia-sets","page":"Lecture","title":"Running example: Julia sets","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Our example for explaining mechanisms of distributed computing will be Julia set fractals, as they can be easily paralelized. The example is adapted from Eric Aubanel. Some fractals (Julia set, Mandelbrot) are determined by properties of some complex-valued functions. Julia set counts, how many iteration is required for f(z) = z^2+c to be bigger than two in absolute value, f(z) =2. The number of iterations can then be mapped to the pixel's color, which creates a nice visualization we know.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"function juliaset_pixel(z₀, c)\n z = z₀\n for i in 1:255\n abs2(z)> 4.0 && return (i - 1)%UInt8\n z = z*z + c\n end\n return UInt8(255)\nend","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"A nice property of fractals like Julia set is that the computation can be easily paralelized, since the value of each pixel is independent from the remaining. In our experiments, the level of granulity will be one column, since calculation of single pixel is so fast, that thread or process switching will have much higher overhead.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"function juliaset_column!(img, c, n, j)\n x = -2.0 + (j-1)*4.0/(n-1)\n for i in 1:n\n y = -2.0 + (i-1)*4.0/(n-1)\n @inbounds img[i,j] = juliaset_pixel(x+im*y, c)\n end\n nothing\nend","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"To calculate full image","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"function juliaset(x, y, n=1000)\n c = x + y*im\n img = Array{UInt8,2}(undef,n,n)\n for j in 1:n\n juliaset_column!(img, c, n, j)\n end\n return img\nend","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"and run it and view it","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using Plots\nfrac = juliaset(-0.79, 0.15)\nplot(heatmap(1:size(frac,1),1:size(frac,2), frac, color=:Spectral))","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"or with GLMakie","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using GLMakie\nfrac = juliaset(-0.79, 0.15)\nheatmap(frac)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"To observe the execution length, we will use BenchmarkTools.jl ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using BenchmarkTools\njulia> @btime juliaset(-0.79, 0.15);\n 39.822 ms (2 allocations: 976.70 KiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Let's now try to speed-up the computation using more processes. We first make functions available to workers","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tfunction juliaset_pixel(z₀, c)\n\t z = z₀\n\t for i in 1:255\n\t abs2(z)> 4.0 && return (i - 1)%UInt8\n\t z = z*z + c\n\t end\n\t return UInt8(255)\n\tend\n\n\tfunction juliaset_column!(img, c, n, colj, j)\n\t x = -2.0 + (j-1)*4.0/(n-1)\n\t for i in 1:n\n\t y = -2.0 + (i-1)*4.0/(n-1)\n\t @inbounds img[i,colj] = juliaset_pixel(x+im*y, c)\n\t end\n\t nothing\n\tend\nend","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"For the actual parallelisation, we split the computation of the whole image into bands, such that each worker computes a smaller portion.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tfunction juliaset_columns(c, n, columns)\n\t img = Array{UInt8,2}(undef, n, length(columns))\n\t for (colj, j) in enumerate(columns)\n\t juliaset_column!(img, c, n, colj, j)\n\t end\n\t img\n\tend\nend\n\nfunction juliaset_spawn(x, y, n = 1000)\n c = x + y*im\n columns = Iterators.partition(1:n, div(n, nworkers()))\n r_bands = [@spawnat w juliaset_columns(c, n, cols) for (w, cols) in enumerate(columns)]\n slices = map(fetch, r_bands)\n reduce(hcat, slices)\nend","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"we observe some speed-up over the serial version, but not linear in terms of number of workers","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"julia> @btime juliaset(-0.79, 0.15);\n 38.699 ms (2 allocations: 976.70 KiB)\n\njulia> @btime juliaset_spawn(-0.79, 0.15);\n 21.521 ms (480 allocations: 1.93 MiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"In the above example, we spawn one function on each worker and collect the results. In essence, we are performing map over bands. Julia offers for this usecase a parallel version of map pmap. With that, our example can look like","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"function juliaset_pmap(x, y, n = 1000, np = nworkers())\n c = x + y*im\n columns = Iterators.partition(1:n, div(n, np))\n slices = pmap(cols -> juliaset_columns(c, n, cols), columns)\n reduce(hcat, slices)\nend\n\njulia> @btime juliaset_pmap(-0.79, 0.15);\n 17.597 ms (451 allocations: 1.93 MiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"which has slightly better timing then the version based on @spawnat and fetch (as explained below in section about Threads, the parallel computation of Julia set suffers from each pixel taking different time to compute, which can be relieved by dividing the work into more parts:","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"julia> @btime juliaset_pmap(-0.79, 0.15, 1000, 16);\n 12.686 ms (1439 allocations: 1.96 MiB)","category":"page"},{"location":"lecture_10/lecture/#Shared-memory","page":"Lecture","title":"Shared memory","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"When main and all workers are located on the same process, and the OS supports sharing memory between processes (by sharing memory pages), we can use SharedArrays to avoid sending the matrix with results.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin\n\tusing SharedArrays\n\tfunction juliaset_shared(x, y, n=1000)\n\t c = x + y*im\n\t img = SharedArray(Array{UInt8,2}(undef,n,n))\n\t @sync @distributed for j in 1:n\n\t juliaset_column!(img, c, n, j, j)\n\t end\n\t return img\n\tend \nend\n\njulia> @btime juliaset_shared(-0.79, 0.15);\n 19.088 ms (963 allocations: 1017.92 KiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"The allocation of the Shared Array mich be costly, let's try to put the allocation outside of the loop","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"img = SharedArray(Array{UInt8,2}(undef,1000,1000))\nfunction juliaset_shared!(img, x, y, n=1000)\n c = x + y*im\n @sync @distributed for j in 1:n\n juliaset_column!(img, c, n, j, j)\n end\n return img\nend \n\njulia> @btime juliaset_shared!(img, -0.79, 0.15);\n 17.399 ms (614 allocations: 27.61 KiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"but both versions are not akin. It seems like the alocation of SharedArray costs approximately 2ms.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@distributed for (Distributed.pfor) does not allows to supply, as it splits the for cycle to nworkers() processes. Above we have seen that more splits is better","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tfunction juliaset_columns!(img, c, n, columns)\n\t for (colj, j) in enumerate(columns)\n\t juliaset_column!(img, c, n, colj, j)\n\t end\n\tend\nend\n\nimg = SharedArray(Array{UInt8,2}(undef,1000,1000))\nfunction juliaset_shared!(img, x, y, n=1000, np = nworkers())\n c = x + y*im\n columns = Iterators.partition(1:n, div(n, np))\n pmap(cols -> juliaset_columns!(img, c, n, cols), columns)\n return img\nend \n\njulia> @btime juliaset_shared!(img, -0.79, 0.15, 1000, 16);\n 11.760 ms (1710 allocations: 85.98 KiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Which is almost 1ms faster than without used of pre-allocated SharedArray. Notice the speedup is now 38.699 / 11.76 = 3.29×","category":"page"},{"location":"lecture_10/lecture/#Synchronization-/-Communication-primitives","page":"Lecture","title":"Synchronization / Communication primitives","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"The orchestration of a complicated computation might be difficult with relatively low-level remote calls. A producer / consumer paradigm is a synchronization paradigm that uses queues. Consumer fetches work intructions from the queue and pushes results to different queue. Julia supports this paradigm with Channel and RemoteChannel primitives. Importantly, putting to and taking from queue is an atomic operation, hence we do not have take care of race conditions. The code for the worker might look like","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tfunction juliaset_channel_worker(instructions, results)\n\t\twhile isready(instructions)\n\t\t\tc, n, cols = take!(instructions)\n\t\t\tput!(results, (cols, juliaset_columns(c, n, cols)))\n\t\tend\n\tend\nend","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"The code for the main will look like","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"function juliaset_channels(x, y, n = 1000, np = nworkers())\n\tc = x + y*im\n\tcolumns = Iterators.partition(1:n, div(n, np))\n\tinstructions = RemoteChannel(() -> Channel(np))\n\tforeach(cols -> put!(instructions, (c, n, cols)), columns)\n\tresults = RemoteChannel(()->Channel(np))\n\trfuns = [@spawnat i juliaset_channel_worker(instructions, results) for i in workers()]\n\n\timg = Array{UInt8,2}(undef, n, n)\n\tfor i in 1:np\n\t\tcols, impart = take!(results)\n\t\timg[:,cols] .= impart;\n\tend\n\timg\nend\n\njulia> @btime juliaset_channels(-0.79, 0.15);","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"The execution time is much higher then what we have observed in the previous cases and changing the number of workers does not help much. What went wrong? The reason is that setting up the infrastructure around remote channels is a costly process. Consider the following alternative, where (i) we let workers to run endlessly and (ii) the channel infrastructure is set-up once and wrapped into an anonymous function","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tfunction juliaset_channel_worker(instructions, results)\n\t\twhile true\n\t\t c, n, cols = take!(instructions)\n\t\t put!(results, (cols, juliaset_columns(c, n, cols)))\n\t\tend\n\tend\nend\n\nfunction juliaset_init(x, y, n = 1000, np = nworkers())\n c = x + y*im\n columns = Iterators.partition(1:n, div(n, np))\n T = Tuple{ComplexF64,Int64,UnitRange{Int64}}\n instructions = RemoteChannel(() -> Channel{T}(np))\n T = Tuple{UnitRange{Int64},Array{UInt8,2}}\n results = RemoteChannel(()->Channel{T}(np))\n foreach(p -> remote_do(juliaset_channel_worker, p, instructions, results), workers())\n function compute()\n img = Array{UInt8,2}(undef, n, n)\n foreach(cols -> put!(instructions, (c, n, cols)), columns)\n for i in 1:np\n cols, impart = take!(results)\n img[:,cols] .= impart;\n end\n img\n end \nend\n\nt = juliaset_init(-0.79, 0.15)\njulia> @btime t();\n 17.697 ms (776 allocations: 1.94 MiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"with which we obtain the comparable speed to the pmap approach.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\nremote_do vs remote_callInstead of @spawnat (remote_call) we can also use remote_do as foreach(p -> remote_do(juliaset_channel_worker, p, instructions, results), workers), which executes the function juliaset_channel_worker at worker p with parameters instructions and results but does not return Future handle to receive the future results.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\nChannel and RemoteChannelAbstractChannel has to implement the interface put!, take!, fetch, isready and wait, i.e. it should behave like a queue. Channel is an implementation if an AbstractChannel that facilitates a communication within a single process (for the purpose of multi-threadding and task switching). Channel can be easily created by Channel{T}(capacity), which can be infinite. The storage of a channel can be seen in data field, but a direct access will of course break all guarantees like atomicity of take! and put!. For communication between proccesses, the <:AbstractChannel has to be wrapped in RemoteChannel. The constructor for RemoteChannel(f::Function, pid::Integer=myid()) has a first argument a function (without arguments) which constructs the Channel (or something like that) on the remote machine identified by pid and returns the RemoteChannel. The storage thus resides on the machine specified by pid and the handle provided by the RemoteChannel can be freely passed to any process. (For curious, ProcessGroup Distributed.PGRP contains an information about channels on machines.) ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"In the above example, juliaset_channel_worker defined as ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"function juliaset_channel_worker(instructions, results)\n\twhile true\n\t c, n, cols = take!(instructions)\n\t put!(results, (cols, juliaset_columns(c, n, cols)))\n\tend\nend","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"runs forever due to the while true loop. ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Julia does not provide by default any facility to kill the remote execution except sending ctrl-c to the remote worker as interrupt(pids::Integer...). To stop the computation, we usually extend the type accepted by the instructions channel to accept some stopping token (e.g. :stop) and stop.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tfunction juliaset_channel_worker(instructions, results)\n\t\twhile true\n\t\t\ti = take!(instructions)\n\t\t\ti === :stop && break\n\t\t\tc, n, cols = i\n\t\t\tput!(results, (cols, juliaset_columns(c, n, cols)))\n\t\tend\n\t\tprintln(\"worker $(myid()) stopped\")\n\t\tput!(results, :stop)\n\tend\nend\n\nfunction juliaset_init(x, y, n = 1000, np = nworkers())\n c = x + y*im\n columns = Iterators.partition(1:n, div(n, np))\n instructions = RemoteChannel(() -> Channel(np))\n results = RemoteChannel(()->Channel(np))\n foreach(p -> remote_do(juliaset_channel_worker, p, instructions, results), workers())\n function compute()\n img = Array{UInt8,2}(undef, n, n)\n foreach(cols -> put!(instructions, (c, n, cols)), columns)\n for i in 1:np\n cols, impart = take!(results)\n img[:,cols] .= impart;\n end\n img\n end \nend\n\nt = juliaset_init(-0.79, 0.15)\nt()\nforeach(i -> put!(t.instructions, :stop), workers())","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"In the above example we paid the price of introducing type instability into the channels, which now contain types Any instead of carefully constructed tuples. But the impact on the overall running time is negligible","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"t = juliaset_init(-0.79, 0.15)\njulia> @btime t()\n 17.551 ms (774 allocations: 1.94 MiB)\nforeach(i -> put!(t.instructions, :stop), workers())","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"In some use-cases, the alternative can be to put all jobs to the RemoteChannel before workers are started, and then stop the workers when the remote channel is empty as ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tfunction juliaset_channel_worker(instructions, results)\n\t\twhile !isready(instructions)\n\t\t c, n, cols = take!(instructions)\n\t\t put!(results, (cols, juliaset_columns(c, n, cols)))\n\t\tend\n\tend\nend","category":"page"},{"location":"lecture_10/lecture/#Sending-data","page":"Lecture","title":"Sending data","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Sending parameters of functions and receiving results from a remotely called functions migh incur a significant cost. ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Try to minimize the data movement as much as possible. A prototypical example is","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"A = rand(1000,1000);\nBref = @spawnat :any A^2;","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"and","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Bref = @spawnat :any rand(1000,1000)^2;","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"It is not only volume of data (in terms of the number of bytes), but also a complexity of objects that are being sent. Serialization can be very time consuming, an efficient converstion to something simple might be worth","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using BenchmarkTools\n@everywhere begin \n\tusing Random\n\tv = [randstring(rand(1:20)) for i in 1:1000];\n\tp = [i => v[i] for i in 1:1000]\n\td = Dict(p)\n\n\tsend_vec() = v\n\tsend_dict() = d\n\tsend_pairs() = p\n\tcustom_serialization() = (length.(v), join(v, \"\"))\nend\n\n@btime remotecall_fetch(send_vec, 2);\n@btime remotecall_fetch(send_dict, 2);\n@btime remotecall_fetch(send_pairs, 2);\n@btime remotecall_fetch(custom_serialization, 2);","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Some type of objects cannot be properly serialized and deserialized","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"a = IdDict(\n\t:a => rand(1,1),\n\t)\nb = remotecall_fetch(identity, 2, a)\na[:a] === a[:a]\na[:a] === b[:a]","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"If you need to send the data to worker, i.e. you want to define (overwrite) a global variable there","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tg = rand()\n\tshow_secret() = println(\"secret of \", myid(), \" is \", g)\nend\n@everywhere show_secret()\n\nfor i in workers()\n\tremotecall_fetch(g -> eval(:(g = $(g))), i, g)\nend\n@everywhere show_secret()","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"which is implemented in the ParallelDataTransfer.jl with other variants, but in general, this construct should be avoided.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Alternatively, you can overwrite a global variable","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tg = rand()\n\tshow_secret() = println(\"secret of \", myid(), \" is \", g)\n\tfunction set_g(x) \n\t\tglobal g\n\t\tg = x\n\t\tnothing\n\tend\nend\n\n@everywhere show_secret()\nremote_do(set_g, 2, 2)\n@everywhere show_secret()","category":"page"},{"location":"lecture_10/lecture/#Practical-advices","page":"Lecture","title":"Practical advices","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Recall that (i) workers are started as clean processes and (ii) they might not share the same environment with the main process. The latter is due to the possibility of remote machines to have a different directory structure. ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tusing Pkg\n\tprintln(Pkg.project().path)\nend","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Our advices earned by practice are:","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"to have shared directory (shared home) with code and to share the location of packages\nto place all code for workers to one file, let's call it worker.jl (author of this includes the code for master as well).\nput to the beggining of worker.jl code activating specified environment as (or specify environmnet for all workers in environment variable as export JULIA_PROJECT=\"$PWD\")","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using Pkg\nPkg.activate(@__DIR__)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"and optionally","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Pkg.resolve()\nPkg.instantiate()","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"run julia as","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"julia -p ?? -L worker.jl main.jl","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"where main.jl is the script to be executed on the main node. Or","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"julia -p ?? -L worker.jl -e \"main()\"","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"where main() is the function defined in worker.jl to be executed on the main node.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"A complete example can be seen in juliaset_p.jl.","category":"page"},{"location":"lecture_10/lecture/#Multi-threadding","page":"Lecture","title":"Multi-threadding","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"So far, we have been able to decrese the computation from 39ms to something like 13ms. Can we improve? Let's now turn our attention to multi-threadding, where we will not pay the penalty for IPC. Moreover, the computation of Julia set is multi-thread friendly, as all the memory can be pre-allocatted. We slightly modify our code to accept different methods distributing the work among slices in the pre-allocated matrix. To start Julia with support of multi-threadding, run it with julia -t n, where n is the number of threads. It is reccomended to set n to number of physical cores, since in hyper-threadding two threads shares arithmetic units of a single core, and in applications for which Julia was built, they are usually saturated.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using BenchmarkTools\nfunction juliaset_pixel(z₀, c)\n z = z₀\n for i in 1:255\n abs2(z)> 4.0 && return (i - 1)%UInt8\n z = z*z + c\n end\n return UInt8(255)\nend\n\nfunction juliaset_column!(img, c, n, j)\n x = -2.0 + (j-1)*4.0/(n-1)\n for i in 1:n\n y = -2.0 + (i-1)*4.0/(n-1)\n @inbounds img[i,j] = juliaset_pixel(x+im*y, c)\n end\n nothing\nend\n\nfunction juliaset(x, y, n=1000)\n c = x + y*im\n img = Array{UInt8,2}(undef,n,n)\n for j in 1:n\n juliaset_column!(img, c, n, j)\n end\n return img\nend\n\njulia> @btime juliaset(-0.79, 0.15, 1000);\n 38.932 ms (2 allocations: 976.67 KiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Let's now try to speed-up the calculation using multi-threadding. Julia v0.5 has introduced multi-threadding with static-scheduller with a simple syntax: just prepend the for-loop with a Threads.@threads macro. With that, the first multi-threaded version will looks like","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"function juliaset_static(x, y, n=1000)\n c = x + y*im\n img = Array{UInt8,2}(undef,n,n)\n Threads.@threads :static for j in 1:n\n juliaset_column!(img, c, n, j)\n end\n return img\nend","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"with benchmark","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"julia> \t@btime juliaset_static(-0.79, 0.15, 1000);\n 15.751 ms (27 allocations: 978.75 KiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Although we have used four-threads, and the communication overhead should be next to zero, the speed improvement is 24. Why is that? ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"To understand bettern what is going on, we have improved the profiler we have been developing last week. The logging profiler logs time of entering and exitting every function call of every thread, which is useful to understand, what is going on. The api is not yet polished, but it will do its job. Importantly, to prevent excessive logging, we ask to log only some functions.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using LoggingProfiler\nfunction juliaset_static(x, y, n=1000)\n c = x + y*im\n img = Array{UInt8,2}(undef,n,n)\n Threads.@threads :dynamic for j in 1:n\n LoggingProfiler.@recordfun juliaset_column!(img, c, n, j)\n end\n return img\nend\n\nLoggingProfiler.initbuffer!(1000)\njuliaset_static(-0.79, 0.15, 1000);\nLoggingProfiler.recorded()\nLoggingProfiler.adjustbuffer!()\njuliaset_static(-0.79, 0.15, 1000)\nLoggingProfiler.export2svg(\"/tmp/profile.svg\")\nLoggingProfiler.export2luxor(\"profile.png\")","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"(Image: profile.png) From the visualization of the profiler we can see not all threads were working the same time. Thread 1 and 4 were working less that Thread 2 and 3. The reason is that the static scheduller partition the total number of columns (1000) into equal parts, where the total number of parts is equal to the number of threads, and assign each to a single thread. In our case, we will have four parts each of size 250. Since execution time of computing value of each pixel is not the same, threads with a lot zero iterations will finish considerably faster. This is the incarnation of one of the biggest problems in multi-threadding / schedulling. A contemprary approach is to switch to dynamic schedulling, which divides the problem into smaller parts, and when a thread is finished with one part, it assigned new not-yet computed part.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"From 1.5, one can specify the scheduller for Threads.@thread [scheduller] for construct to be either :static and / or :dynamic. The :dynamic is compatible with the partr dynamic scheduller. From 1.8, :dynamic is default, but the range is dividided into nthreads() parts, which is the reason why we do not see an improvement.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Dynamic scheduller is also supported using by Threads.@spawn macro. The prototypical approach used for invocation is the fork-join model, where one recursivelly partitions the problems and wait in each thread for the other","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"function juliaset_recspawn!(img, c, n, lo=1, hi=n, ntasks=128)\n if hi - lo > n/ntasks-1\n mid = (lo+hi)>>>1\n finish = Threads.@spawn juliaset_recspawn!(img, c, n, lo, mid, ntasks)\n juliaset_recspawn!(img, c, n, mid+1, hi, ntasks)\n wait(finish)\n return\n end\n for j in lo:hi\n juliaset_column!(img, c, n, j)\n end\n nothing\nend","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Measuring the time we observe four-times speedup, which corresponds to the number of threads.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"function juliaset_forkjoin(x, y, n=1000, ntasks = 16)\n c = x + y*im\n img = Array{UInt8,2}(undef,n,n)\n juliaset_recspawn!(img, c, n, 1, n, ntasks)\n return img\nend\n\njulia> @btime juliaset_forkjoin(-0.79, 0.15);\n 10.326 ms (142 allocations: 986.83 KiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"This is so far our fastest construction with speedup 38.932 / 10.326 = 3.77×.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Unfortunatelly, the LoggingProfiler does not handle task migration at the moment, which means that we cannot visualize the results. Due to task switching overhead, increasing the granularity might not pay off.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"4 tasks: 16.262 ms (21 allocations: 978.05 KiB)\n8 tasks: 10.660 ms (45 allocations: 979.80 KiB)\n16 tasks: 10.326 ms (142 allocations: 986.83 KiB)\n32 tasks: 10.786 ms (238 allocations: 993.83 KiB)\n64 tasks: 10.211 ms (624 allocations: 1021.89 KiB)\n128 tasks: 10.224 ms (1391 allocations: 1.05 MiB)\n256 tasks: 10.617 ms (2927 allocations: 1.16 MiB)\n512 tasks: 11.012 ms (5999 allocations: 1.38 MiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using FLoops, FoldsThreads\nfunction juliaset_folds(x, y, n=1000, basesize = 2)\n c = x + y*im\n img = Array{UInt8,2}(undef,n,n)\n @floop ThreadedEx(basesize = basesize) for j in 1:n\n juliaset_column!(img, c, n, j)\n end\n return img\nend\n\njulia> @btime juliaset_folds(-0.79, 0.15, 1000);\n 10.253 ms (3960 allocations: 1.24 MiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"where basesize is the size of the smallest part allocated to a single thread, in this case 2 columns.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"julia> @btime juliaset_folds(-0.79, 0.15, 1000);\n 10.575 ms (52 allocations: 980.12 KiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"function juliaset_folds(x, y, n=1000, basesize = 2)\n c = x + y*im\n img = Array{UInt8,2}(undef,n,n)\n @floop DepthFirstEx(basesize = basesize) for j in 1:n\n juliaset_column!(img, c, n, j)\n end\n return img\nend\njulia> @btime juliaset_folds(-0.79, 0.15, 1000);\n 10.421 ms (3582 allocations: 1.20 MiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"We can identify the best smallest size of the work basesize and measure its influence on the time","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"map(2 .^ (0:7)) do bs \n\tt = @belapsed juliaset_folds(-0.79, 0.15, 1000, $(bs));\n\t(;basesize = bs, time = t)\nend |> DataFrame","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":" Row │ basesize time\n │ Int64 Float64\n─────┼─────────────────────\n 1 │ 1 0.0106803\n 2 │ 2 0.010267\n 3 │ 4 0.0103081\n 4 │ 8 0.0101652\n 5 │ 16 0.0100204\n 6 │ 32 0.0100097\n 7 │ 64 0.0103293\n 8 │ 128 0.0105411","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"We observe that the minimum is for basesize = 32, for which we got 3.8932× speedup. ","category":"page"},{"location":"lecture_10/lecture/#Garbage-collector-is-single-threadded","page":"Lecture","title":"Garbage collector is single-threadded","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Keep reminded that while threads are very easy very convenient to use, there are use-cases where you might be better off with proccess, even though there will be some communication overhead. One such case happens when you need to allocate and free a lot of memory. This is because Julia's garbage collector is single-threadded (in 1.10 it is now partially multi-threaded). Imagine a task of making histogram of bytes in a directory. For a fair comparison, we will use Transducers, since they offer thread and process based paralelism","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using Transducers\n@everywhere begin \n\tfunction histfile(filename)\n\t\th = Dict{UInt8,Int}()\n\t\tforeach(open(read, filename, \"r\")) do b \n\t\t\th[b] = get(h, b, 0) + 1\n\t\tend\n\t\th\n\tend\nend\n\nfiles = filter(isfile, readdir(\"/Users/tomas.pevny/Downloads/\", join = true))\n@elapsed foldxd(mergewith(+), files |> Map(histfile))\n150.863183701","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"and using the multi-threaded version of map","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@elapsed foldxt(mergewith(+), files |> Map(histfile))\n205.309952618","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"we see that the threadding is actually worse than process based paralelism despite us paying the price for serialization and deserialization of Dict. Needless to say that changing Dict to Vector as","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using Transducers\n@everywhere begin \n\tfunction histfile(filename)\n\t\th = Dict{UInt8,Int}()\n\t\tforeach(open(read, filename, \"r\")) do b \n\t\t\th[b] = get(h, b, 0) + 1\n\t\tend\n\t\th\n\tend\nend\nfiles = filter(isfile, readdir(\"/Users/tomas.pevny/Downloads/\", join = true))\n@elapsed foldxd(mergewith(+), files |> Map(histfile))\n86.44577969\n@elapsed foldxt(mergewith(+), files |> Map(histfile))\n105.32969331","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"is much better.","category":"page"},{"location":"lecture_10/lecture/#Locks-/-lock-free-multi-threadding","page":"Lecture","title":"Locks / lock-free multi-threadding","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Avoid locks.","category":"page"},{"location":"lecture_10/lecture/#Take-away-message","page":"Lecture","title":"Take away message","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"When deciding, what kind of paralelism to employ, consider following","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"for tightly coupled computation over shared data, multi-threadding is more suitable due to non-existing sharing of data between processes\nbut if the computation requires frequent allocation and freeing of memery, or IO, separate processes are multi-suitable, since garbage collectors are independent between processes\nMaking all cores busy while achieving an ideally linear speedup is difficult and needs a lot of experience and knowledge. Tooling and profilers supporting debugging of parallel processes is not much developped.\nTransducers thrives for (almost) the same code to support thread- and process-based paralelism.","category":"page"},{"location":"lecture_10/lecture/#Materials","page":"Lecture","title":"Materials","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"http://cecileane.github.io/computingtools/pages/notes1209.html\nhttps://lucris.lub.lu.se/ws/portalfiles/portal/61129522/julia_parallel.pdf\nhttp://igoro.com/archive/gallery-of-processor-cache-effects/\nhttps://www.csd.uwo.ca/~mmorenom/cs2101amoreno/ParallelcomputingwithJulia.pdf\nComplexity of thread schedulling https://www.youtube.com/watch?v=YdiZa0Y3F3c\nTapIR –- Teaching paralelism to Julia compiler https://www.youtube.com/watch?v=-JyK5Xpk7jE\nThreads: https://juliahighperformance.com/code/Chapter09.html\nProcesses: https://juliahighperformance.com/code/Chapter10.html\nAlan Adelman uses FLoops in https://www.youtube.com/watch?v=dczkYlOM2sg\nExamples: ?Heat equation? from [https://hpc.llnl.gov/training/tutorials/](introduction-parallel-computing-tutorial#Examples(https://hpc.llnl.gov/training/tutorials/)","category":"page"},{"location":"lecture_05/lab/#perf_lab","page":"Lab","title":"Lab 05: Practical performance debugging tools","text":"","category":"section"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Performance is crucial in scientific computing. There is a big difference if your experiments run one minute or one hour. We have already developed quite a bit of code, both in and outside packages, on which we are going to present some of the tooling that Julia provides for finding performance bottlenecks. Performance of your code or more precisely the speed of execution is of course relative (preference, expectation, existing code) and it's hard to find the exact threshold when we should start to care about it. When starting out with Julia, we recommend not to get bogged down by the performance side of things straightaway, but just design the code in the way that feels natural to you. As opposed to other languages Julia offers you to write the things \"like you are used to\" (depending on your background), e.g. for cycles are as fast as in C; vectorization of mathematical operators works the same or even better than in MATLAB, NumPy. ","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Once you have tested the functionality, you can start exploring the performance of your code by different means:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"manual code inspection - identifying performance gotchas (tedious, requires skill)\nautomatic code inspection - Jet.jl (probably not as powerful as in statically typed languages)\nbenchmarking - measuring variability in execution time, comparing with some baseline (only a statistic, non-specific)\nprofiling - measuring the execution time at \"each line of code\" (no easy way to handle advanced parallelism, ...)\nallocation tracking - similar to profiling but specifically looking at allocations (one sided statistic)","category":"page"},{"location":"lecture_05/lab/#Checking-type-stability","page":"Lab","title":"Checking type stability","text":"","category":"section"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Recall that type stable function is written in a way, that allows Julia's compiler to infer all the types of all the variables and produce an efficient native code implementation without the need of boxing some variables in a structure whose types is known only during runtime. Probably unbeknown to you we have already seen an example of type unstable function (at least in some situations) in the first lab, where we have defined the polynomial function:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"function polynomial(a, x)\n accumulator = 0\n for i in length(a):-1:1\n accumulator += x^(i-1) * a[i] # ! 1-based indexing for arrays\n end\n return accumulator\nend","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"The exact form of compiled code and also the type stability depends on the arguments of the function. Let's explore the following two examples of calling the function:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Integer number valued arguments","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"a = [-19, 7, -4, 6]\nx = 3\npolynomial(a, x)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Float number valued arguments","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"xf = 3.0\npolynomial(a, xf)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"The result they produce is the \"same\" numerically, however it differs in the output type. Though you have probably not noticed it, there should be a difference in runtime (assuming that you have run it once more after its compilation). It is probably a surprise to no one, that one of the methods that has been compiled is type unstable. This can be check with the @code_warntype macro:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"using InteractiveUtils #hide\n@code_warntype polynomial(a, x) # type stable\n@code_warntype polynomial(a, xf) # type unstable","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"We are getting a little ahead of ourselves in this lab, as understanding of these expressions is part of the future lecture. Anyway the output basically shows what the compiler thinks of each variable in the code, albeit for us in less readable form than the original code. The more red the color is of the type info the less sure the inferred type is. Our main focus should be on the return type of the function which is just at the start of the code with the keyword Body. In the first case the return type is an Int64, whereas in the second example the compiler is unsure whether the type is Float64 or Int64, marked as the Union type of the two. Fortunately for us this type instability can be fixed with a single line edit, but we will see later that it is not always the case.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"note: Type stability\nHaving a variable represented as Union of multiple types in a functions is a lesser evil than having Any, as we can at least enumerate statically the available options of functions to which to dynamically dispatch and in some cases there may be a low penalty.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Create a new function polynomial_stable, which is type stable and measure the difference in evaluation time. ","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"HINTS: ","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Ask for help on the one and zero keyword, which are often as a shorthand for these kind of functions.\nrun the function with the argument once before running @time or use @btime if you have BenchmarkTools readily available in your environment\nTo see some measurable difference with this simple function, a longer vector of coefficients may be needed.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"function polynomial_stable(a, x)\n accumulator = zero(x)\n for i in length(a):-1:1\n accumulator += x^(i-1) * a[i]\n end\n accumulator\nend","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"@code_warntype polynomial_stable(a, x) # type stable\n@code_warntype polynomial_stable(a, xf) # type stable","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"polynomial(a, xf) #hide\npolynomial_stable(a, xf) #hide\n@time polynomial(a, xf)\n@time polynomial_stable(a, xf)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Only really visible when evaluating multiple times.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"julia> using BenchmarkTools\n\njulia> @btime polynomial($a, $xf)\n 31.806 ns (0 allocations: 0 bytes)\n128.0\n\njulia> @btime polynomial_stable($a, $xf)\n 28.522 ns (0 allocations: 0 bytes)\n128.0","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Difference only a few nanoseconds.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Note: Recalling homework from lab 1. Adding zero also extends this function to the case of x being a matrix, see ? menu.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Code stability issues are something unique to Julia, as its JIT compilation allows it to produce code that contains boxed variables, whose type can be inferred during runtime. This is one of the reasons why interpreted languages are slow to run but fast to type. Julia's way of solving it is based around compiling functions for specific arguments, however in order for this to work without the interpreter, the compiler has to be able to infer the types.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"There are other problems (such as unnecessary allocations), that you can learn to spot in your code, however the code stability issues are by far the most commonly encountered problems among beginner users of Julia wanting to squeeze more out of it.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"note: Advanced tooling\nSometimes @code_warntype shows that the function's return type is unstable without any hints to the possible problem, fortunately for such cases a more advanced tools such as Cthuhlu.jl or JET.jl have been developed.","category":"page"},{"location":"lecture_05/lab/#Benchmarking-with-BenchmarkTools","page":"Lab","title":"Benchmarking with BenchmarkTools","text":"","category":"section"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"In the last exercise we have encountered the problem of timing of code to see, if we have made any progress in speeding it up. Throughout the course we will advertise the use of the BenchmarkTools package, which provides an easy way to test your code multiple times. In this lab we will focus on some advanced usage tips and gotchas that you may encounter while using it. ","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"There are few concepts to know in order to understand how the pkg works","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"evaluation - a single execution of a benchmark expression (default 1)\nsample - a single time/memory measurement obtained by running multiple evaluations (default 1e5)\ntrial - experiment in which multiple samples are gathered ","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"The result of a benchmark is a trial in which we collect multiple samples of time/memory measurements, which in turn may be composed of multiple executions of the code in question. This layering of repetition is required to allow for benchmarking code at different runtime magnitudes. Imagine having to benchmark operations which are faster than the act of measuring itself - clock initialization, dispatch of an operation and subsequent time subtraction.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"The number of samples/evaluations can be set manually, however most of the time won't need to know about them, due to an existence of a tuning method tune!, which tries to run the code once to estimate the correct ration of evaluation/samples. ","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"The most commonly used interface of Benchmarkools is the @btime macro, which returns an output similar to the regular @time macro however now aggregated over samples by taking their minimum (a robust estimator for the location parameter of the time distribution, should not be considered an outlier - usually the noise from other processes/tasks puts the results to the other tail of the distribution and some miraculous noisy speedups are uncommon. In order to see the underlying sampling better there is also the @benchmark macro, which runs in the same way as @btime, but prints more detailed statistics which are also returned in the Trial type instead of the actual code output.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"julia> @btime sum($(rand(1000)))\n 174.274 ns (0 allocations: 0 bytes)\n504.16236531044757\n\njulia> @benchmark sum($(rand(1000)))\nBenchmarkTools.Trial: 10000 samples with 723 evaluations.\n Range (min … max): 174.274 ns … 364.856 ns ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 174.503 ns ┊ GC (median): 0.00%\n Time (mean ± σ): 176.592 ns ± 7.361 ns ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n █▃ ▃▃ ▁\n █████████▇█▇█▇▇▇▇▇▆▆▇▆▆▆▆▆▆▅▆▆▅▅▅▆▆▆▆▅▅▅▅▅▅▅▅▆▅▅▅▄▄▅▅▄▄▅▃▅▅▄▅ █\n 174 ns Histogram: log(frequency) by time 206 ns <\n\n Memory estimate: 0 bytes, allocs estimate: 0.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"danger: Interpolation ~ `$` in BenchmarkTools\nIn the previous example we have used the interpolation signs $ to indicate that the code inside should be evaluated once and stored into a local variable. This allows us to focus only on the benchmarking of code itself instead of the input generation. A more subtle way where this is crops up is the case of using previously defined global variable, where instead of data generation we would measure also the type inference at each evaluation, which is usually not what we want. The following list will help you decide when to use interpolation.@btime sum($(rand(1000))) # rand(1000) is stored as local variable, which is used in each evaluation\n@btime sum(rand(1000)) # rand(1000) is called in each evaluation\nA = rand(1000)\n@btime sum($A) # global variable A is inferred and stored as local, which is used in each evaluation\n@btime sum(A) # global variable A has to be inferred in each evaluation","category":"page"},{"location":"lecture_05/lab/#Profiling","page":"Lab","title":"Profiling","text":"","category":"section"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Profiling in Julia is part of the standard library in the Profile module. It implements a fairly simple sampling based profiler, which in a nutshell asks at regular intervals, where the code execution is currently at. As a result we get an array of stacktraces (= chain of function calls), which allow us to make sense of where the execution spent the most time. The number of samples, that can be stored and the period in seconds can be checked after loading Profile into the session with the init() function.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"using Profile\nProfile.init()","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"The same function, but with keyword arguments, can be used to change these settings, however these settings are system dependent. For example on Windows, there is a known issue that does not allow to sample faster than at 0.003s and even on Linux based system this may not do much. There are some further caveat specific to Julia:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"When running profile from REPL, it is usually dominated by the interactive part which spawns the task and waits for it's completion.\nCode has to be run before profiling in order to filter out all the type inference and interpretation stuff. (Unless compilation is what we want to profile.)\nWhen the execution time is short, the sampling may be insufficient -> run multiple times.","category":"page"},{"location":"lecture_05/lab/#Polynomial-with-scalars","page":"Lab","title":"Polynomial with scalars","text":"","category":"section"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Let's look at our favorite polynomial function or rather it's type stable variant polynomial_stable under the profiling lens.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"# clear the last trace (does not have to be run on fresh start)\nProfile.clear()\n\n@profile polynomial_stable(a, xf)\n\n# text based output of the profiler\n# not shown here because it is not incredibly informative\nProfile.print()","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Unless the machine that you run the code on is really slow, the resulting output contains nothing or only some internals of Julia's interactive REPL. This is due to the fact that our polynomial function take only few nanoseconds to run. When we want to run profiling on something, that takes only a few nanoseconds, we have to repeatedly execute the function.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"function run_polynomial_stable(a, x, n) \n for _ in 1:n\n polynomial_stable(a, x)\n end\nend\n\na = rand(-10:10, 10) # using longer polynomial\n\nrun_polynomial_stable(a, xf, 10) #hide\nProfile.clear()\n@profile run_polynomial_stable(a, xf, Int(1e5))\nProfile.print()","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"In order to get more of a visual feel for profiling, there are packages that allow you to generate interactive plots or graphs. In this lab we will use ProfileSVG.jl, which does not require any fancy IDE or GUI libraries.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"@profview run_polynomial_stable(a, xf, Int(1e5))","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"(Image: poly_stable)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Let's compare this with the type unstable situation.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"First let's define the function that allows us to run the polynomial multiple times.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"function run_polynomial(a, x, n) \n for _ in 1:n\n polynomial(a, x)\n end\nend","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"@profview run_polynomial(a, xf, Int(1e5)) # clears the profile for us","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"(Image: poly_unstable)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Other options for viewing profiler outputs","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"ProfileView - close cousin of ProfileSVG, spawns GTK window with interactive FlameGraph\nVSCode - always imported @profview macro, flamegraphs (js extension required), filtering, one click access to source code \nPProf - serializes the profiler output to protobuffer and loads it in pprof web app, graph visualization of stacktraces","category":"page"},{"location":"lecture_05/lab/#horner","page":"Lab","title":"Applying fixes","text":"","category":"section"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"We have noticed that no matter if the function is type stable or unstable the majority of the computation falls onto the power function ^ and there is a way to solve this using a clever technique called Horner schema[1], which uses distributive and associative rules to convert the sum of powers into an incremental multiplication of partial results.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Rewrite the polynomial function using the Horner schema/method[1]. Moreover include the type stability fixes from polynomial_stable You should get more than 3x speedup when measured against the old implementation (measure polynomial against polynomial_stable.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"BONUS: Profile the new method and compare the differences in traces.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"[1]: Explanation of the Horner schema can be found on https://en.wikipedia.org/wiki/Horner%27s_method.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"function polynomial(a, x)\n accumulator = a[end] * one(x)\n for i in length(a)-1:-1:1\n accumulator = accumulator * x + a[i]\n end\n accumulator \nend","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Speed up:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"49ns -> 8ns ~ 6x on integer valued input \n59ns -> 8ns ~ 7x on real valued input","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"julia> @btime polynomial($a, $x)\n 8.008 ns (0 allocations: 0 bytes)\n97818\n\njulia> @btime polynomial_stable($a, $x)\n 49.173 ns (0 allocations: 0 bytes)\n97818\n\njulia> @btime polynomial($a, $xf)\n 8.008 ns (0 allocations: 0 bytes)\n97818.0\n\njulia> @btime polynomial_stable($a, $xf)\n 58.773 ns (0 allocations: 0 bytes)\n97818.0","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"These numbers will be different on different HW.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"BONUS: The profile trace does not even contain the calling of mathematical operators and is mainly dominated by the iteration utilities. In this case we had to increase the number of runs to 1e6 to get some meaningful trace.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"@profview run_polynomial(a, xf, Int(1e6))","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"(Image: poly_horner)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"","category":"page"},{"location":"lecture_05/lab/#Where-to-find-source-code?","page":"Lab","title":"Where to find source code?","text":"","category":"section"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"As most of Julia is written in Julia itself it is sometimes helpful to look inside for some details or inspiration. The code of Base and stdlib pkgs is located just next to Julia's installation in the ./share/julia subdirectory","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"./julia-1.6.2/\n ├── bin\n ├── etc\n │ └── julia\n ├── include\n │ └── julia\n │ └── uv\n ├── lib\n │ └── julia\n ├── libexec\n └── share\n ├── appdata\n ├── applications\n ├── doc\n │ └── julia # offline documentation (https://docs.julialang.org/en/v1/)\n └── julia\n ├── base # base library\n ├── stdlib # standard library\n └── test","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Other packages installed through Pkg interface are located in the .julia/ directory which is located in your $HOMEDIR, i.e. /home/$(user)/.julia/ on Unix based systems and /Users/$(user)/.julia/ on Windows.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"~/.julia/\n ├── artifacts\n ├── compiled\n ├── config # startup.jl lives here\n ├── environments\n ├── logs\n ├── packages # packages are here\n └── registries","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"If you are using VSCode, the paths visible in the REPL can be clicked through to he actual source code. Moreover in that environment the documentation is usually available upon hovering over code.","category":"page"},{"location":"lecture_05/lab/#Setting-up-benchmarks-to-our-liking","page":"Lab","title":"Setting up benchmarks to our liking","text":"","category":"section"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"In order to control the number of samples/evaluation and the amount of time given to a given benchmark, we can simply append these as keyword arguments to @btime or @benchmark in the following way","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"julia> @benchmark sum($(rand(1000))) evals=100 samples=10 seconds=1\nBenchmarkTools.Trial: 10 samples with 100 evaluations.\n Range (min … max): 174.580 ns … 188.750 ns ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 175.420 ns ┊ GC (median): 0.00%\n Time (mean ± σ): 176.585 ns ± 4.293 ns ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n █ \n █▅▁█▁▅▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▅ ▁\n 175 ns Histogram: frequency by time 189 ns <\n\n Memory estimate: 0 bytes, allocs estimate: 0.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"which runs the code repeatedly for up to 1s, where each of the 10 samples in the trial is composed of 10 evaluations. Setting up these parameters ourselves creates a more controlled environment in which performance regressions can be more easily identified.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Another axis of customization is needed when we are benchmarking mutable operations such as sort!, which sorts an array in-place. One way of achieving a consistent benchmark is by omitting the interpolation such as","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"julia> @benchmark sort!(rand(1000))\nBenchmarkTools.Trial: 10000 samples with 1 evaluation.\n Range (min … max): 27.250 μs … 95.958 μs ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 29.875 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 30.340 μs ± 2.678 μs ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n ▃▇█▄▇▄ \n ▁▁▁▂▃▆█████████▆▅▃▄▃▃▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▂\n 27.2 μs Histogram: frequency by time 41.3 μs <\n\n Memory estimate: 7.94 KiB, allocs estimate: 1.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"however now we are again measuring the data generation as well. A better way of doing such timing is using the built in setup keyword, into which you can put a code that has to be run before each sample and which won't be measured.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"julia> @benchmark sort!(y) setup=(y=rand(1000))\nBenchmarkTools.Trial: 10000 samples with 7 evaluations.\n Range (min … max): 7.411 μs … 25.869 μs ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 7.696 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 7.729 μs ± 305.383 ns ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n ▂▄▅▆█▇▇▆▄▃ \n ▁▁▁▁▂▂▃▄▅▆████████████▆▅▃▂▂▂▁▁▁▁▁▁▁▁▁▂▂▁▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▃\n 7.41 μs Histogram: frequency by time 8.45 μs <\n\n Memory estimate: 0 bytes, allocs estimate: 0.","category":"page"},{"location":"lecture_05/lab/#Ecosystem-debugging","page":"Lab","title":"Ecosystem debugging","text":"","category":"section"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Let's now apply what we have learned so far on the much bigger codebase of our Ecosystem.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"include(\"ecosystems/lab04/Ecosystem.jl\")\n\nfunction make_counter()\n n = 0\n counter() = n += 1\nend\n\nfunction create_world()\n n_grass = 1_000\n n_sheep = 40\n n_wolves = 4\n\n nextid = make_counter()\n\n World(vcat(\n [Grass(nextid()) for _ in 1:n_grass],\n [Sheep(nextid()) for _ in 1:n_sheep],\n [Wolf(nextid()) for _ in 1:n_wolves],\n ))\nend\nworld = create_world();\nnothing # hide","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Use @profview and @code_warntype to find the type unstable and slow parts of our simulation.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Precompile everything by running one step of our simulation and run the profiler like this:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"world_step!(world)\n@profview for i=1:100 world_step!(world) end","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"You should get a flamegraph similar to the one below: (Image: lab04-ecosystem)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Red bars indicate type instabilities. The bars stacked on top of them are high, narrow and not filling the whole width, indicating that the problem is pretty serious. In our case the worst offender is the filter method inside find_food and find_mate functions. In both cases the bars on top of it are narrow and not the full with, meaning that not that much time has been really spend working, but instead inferring the types in the function itself during runtime.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"As a reminder, this is the find_food function:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"# original\nfunction find_food(a::Animal, w::World)\n as = filter(x -> eats(a,x), w.agents |> values |> collect)\n isempty(as) ? nothing : sample(as)\nend","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Just from looking at that piece of code its not obvious what is the problem, however the red color indicates that the code may be type unstable. Let's see if that is the case by evaluation the function with some isolated inputs.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"using InteractiveUtils # hide\nw = Wolf(4000)\nfind_food(w, world)\n@code_warntype find_food(w, world)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Indeed we see that the return type is not inferred precisely but ends up being just the Union{Nothing, Agent}, this is better than straight out Any, which is the union of all types but still, julia has to do dynamic dispatch here, which is slow.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"The underlying issue here is that we are working array of type Vector{Agent}, where Agent is abstract, which does not allow the compiler to specialize the code for the loop body.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_05/lab/#Different-Ecosystem.jl-versions","page":"Lab","title":"Different Ecosystem.jl versions","text":"","category":"section"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"In order to fix the type instability in the Vector{Agent} we somehow have to rethink our world such that we get a vector of a concrete type. Optimally we would have one vector for each type of agent that populates our world. Before we completely redesign how our world works we can try a simple hack that might already improve things. Instead of letting julia figure our which types of agents we have (which could be infinitely many), we can tell the compiler at least that we have only three of them: Wolf, Sheep, and Grass.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"We can do this with a tiny change in the constructor of our World:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"function World(agents::Vector{<:Agent})\n ids = [a.id for a in agents]\n length(unique(ids)) == length(agents) || error(\"Not all agents have unique IDs!\")\n\n # construct Dict{Int,Union{Animal{Wolf}, Animal{Sheep}, Plant{Grass}}}\n # instead of Dict{Int,Agent}\n types = unique(typeof.(agents))\n dict = Dict{Int,Union{types...}}(a.id => a for a in agents)\n\n World(dict, maximum(ids))\nend","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Run the benchmark script provided here to get timings for find_food and reproduce! for the original ecosystem.\nRun the same benchmark with the modified World constructor.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Which differences can you observe? Why is one version faster than the other?","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"It turns out that with this simple change we can already gain a little bit of speed:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":" find_food reproduce!\nAnimal{A} & Dict{Int,Agent} 43.917 μs 439.666 μs\nAnimal{A} & Dict{Int,Union{...}} 12.208 μs 340.041 μs","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"We are gaining performance here because for small Unions of types the julia compiler can precompile the multiple available code branches. If we have just a Dict of Agents this is not possible.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"This however, does not yet fix our type instabilities completely. We are still working with Unions of types which we can see again using @code_warntype:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"include(\"ecosystems/animal_S_world_DictUnion/Ecosystem.jl\")\n\nfunction make_counter()\n n = 0\n counter() = n += 1\nend\n\nfunction create_world()\n n_grass = 1_000\n n_sheep = 40\n n_wolves = 4\n\n nextid = make_counter()\n\n World(vcat(\n [Grass(nextid()) for _ in 1:n_grass],\n [Sheep(nextid()) for _ in 1:n_sheep],\n [Wolf(nextid()) for _ in 1:n_wolves],\n ))\nend\nworld = create_world();","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"using InteractiveUtils # hide\nw = Wolf(4000)\nfind_food(w, world)\n@code_warntype find_food(w, world)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Julia still has to perform runtime dispatch on the small Union of Agents that is in our dictionary. To avoid this we could create a world that - instead of one plain dictionary - works with a tuple of dictionaries with one entry for each type of agent. Our world would then look like this:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"# pseudocode:\nworld ≈ (\n :Grass => Dict{Int, Plant{Grass}}(...),\n :Sheep => Dict{Int, Animal{Sheep}}(...),\n :Wolf => Dict{Int, Animal{Wolf}}(...)\n)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"In order to make this work we have to touch our ecosystem code in a number of places, mostly related to find_food and reproduce!. You can find a working version of the ecosystem with a world based on NamedTuples here. With this slightly more involved update we can gain another bit of speed:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":" find_food reproduce!\nAnimal{A} & Dict{Int,Agent} 43.917 μs 439.666 μs\nAnimal{A} & Dict{Int,Union{...}} 12.208 μs 340.041 μs\nAnimal{A} & NamedTuple{Dict,...} 8.639 μs 273.103 μs","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"And type stable code!","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"include(\"ecosystems/animal_S_world_NamedTupleDict/Ecosystem.jl\")\n\nfunction make_counter()\n n = 0\n counter() = n += 1\nend\n\nfunction create_world()\n n_grass = 1_000\n n_sheep = 40\n n_wolves = 4\n\n nextid = make_counter()\n\n World(vcat(\n [Grass(nextid()) for _ in 1:n_grass],\n [Sheep(nextid()) for _ in 1:n_sheep],\n [Wolf(nextid()) for _ in 1:n_wolves],\n ))\nend\nworld = create_world();","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"using InteractiveUtils # hide\nw = Wolf(4000)\nfind_food(w, world)\n@code_warntype find_food(w, world)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"The last optimization we can do is to move the Sex of our animals from a field into a parametric type. Our world would then look like below:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"# pseudocode:\nworld ≈ (\n :Grass => Dict{Int, Plant{Grass}}(...),\n :SheepFemale => Dict{Int, Animal{Sheep,Female}}(...),\n :SheepMale => Dict{Int, Animal{Sheep,Male}}(...),\n :WolfFemale => Dict{Int, Animal{Wolf,Female}}(...)\n :WolfMale => Dict{Int, Animal{Wolf,Male}}(...)\n)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"This should give us a lot of speedup in the reproduce! function, because we will not have to filter for the correct sex anymore, but instead can just pick the NamedTuple that is associated with the correct type of mate. Unfortunately, changing the type signature of Animal essentially means that we have to touch every line of code of our original ecosystem. However, the gain we get for it is quite significant:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":" find_food reproduce!\nAnimal{A} & Dict{Int,Agent} 43.917 μs 439.666 μs\nAnimal{A} & Dict{Int,Union{...}} 12.208 μs 340.041 μs\nAnimal{A} & NamedTuple{Dict,...} 8.639 μs 273.103 μs\nAnimal{A,S} & NamedTuple{Dict,...} 7.823 μs 77.646 ns\nAnimal{A,S} & Dict{Int,Union{...}} 13.416 μs 6.436 ms","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"The implementation of the new version with two parametric types can be found here. The completely blue (i.e. type stable) @profview of this version of the Ecosystem is quite satisfying to see","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"(Image: neweco)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"The same is true for the output of @code_warntype","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"include(\"ecosystems/animal_ST_world_NamedTupleDict/Ecosystem.jl\")\n\nfunction make_counter()\n n = 0\n counter() = n += 1\nend\n\nfunction create_world()\n n_grass = 1_000\n n_sheep = 40\n n_wolves = 4\n\n nextid = make_counter()\n\n World(vcat(\n [Grass(nextid()) for _ in 1:n_grass],\n [Sheep(nextid()) for _ in 1:n_sheep],\n [Wolf(nextid()) for _ in 1:n_wolves],\n ))\nend\nworld = create_world();\nnothing # hide","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"using InteractiveUtils # hide\nw = Wolf(4000)\nfind_food(w, world)\n@code_warntype find_food(w, world)","category":"page"},{"location":"lecture_05/lab/#Useful-resources","page":"Lab","title":"Useful resources","text":"","category":"section"},{"location":"lecture_12/lab/#lab12","page":"Lab","title":"Lab 12 - Differential Equations","text":"","category":"section"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"In this lab you will implement a simple solver for ordinary differential equations (ODE) as well as a less verbose version of the GaussNums that were introduced in the lecture.","category":"page"},{"location":"lecture_12/lab/#Euler-ODE-Solver","page":"Lab","title":"Euler ODE Solver","text":"","category":"section"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"In this first part you will implement your own, simple, ODE framwork (feel free to make it a package;) in which you can easily specify different ODE solvers. The API is heavily inspired by DifferentialEquations.jl, so if you ever need to use it, you will already have a feeling for how it works.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Like in the lecture, we want to be able to specify an ODE like below.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"function lotkavolterra(x,θ)\n α, β, γ, δ = θ\n x₁, x₂ = x\n\n dx₁ = α*x₁ - β*x₁*x₂\n dx₂ = δ*x₁*x₂ - γ*x₂\n\n [dx₁, dx₂]\nend\nnothing # hide","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"In the lecture we then solved it with a solve function that received all necessary arguments to fully specify how the ODE should be solved. The number of necessary arguments to solve can quickly become very large, so we will introduce a new API for solve which will always take only two arguments: solve(::ODEProblem, ::ODESolver). The solve function will only do some book-keeping and call the solver until the ODE is solved for the full tspan.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"The ODEProblem will contain all necessary parameters to fully specify the ODE that should be solved. In our case that is the function f that defines the ODE itself, initial conditions u0, ODE parameters θ, and the time domain of the ODE tspan:","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"struct ODEProblem{F,T<:Tuple{Number,Number},U<:AbstractVector,P<:AbstractVector}\n f::F\n tspan::T\n u0::U\n θ::P\nend","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"The solvers will all be subtyping the abstract type ODESolver. The Euler solver from the lecture will need one field dt which specifies its time step:","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"abstract type ODESolver end\n\nstruct Euler{T} <: ODESolver\n dt::T\nend","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Overload the call-method of Euler","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"(solver::Euler)(prob::ODEProblem, u, t)","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"such that calling the solver with an ODEProblem will perform one step of the Euler solver and return updated ODE varialbes u1 and the corresponding timestep t1.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"function (solver::Euler)(prob::ODEProblem, u, t)\n f, θ, dt = prob.f, prob.θ, solver.dt\n (u + dt*f(u,θ), t+dt)\nend","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"# define ODEProblem\nθ = [0.1,0.2,0.3,0.2]\nu0 = [1.0,1.0]\ntspan = (0.,100.)\nprob = ODEProblem(lotkavolterra,tspan,u0,θ)\n\n# run one solver step\nsolver = Euler(0.2)\n(u1,t1) = solver(prob,u0,0.)","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Implement the function solve(::ODEProblem,::ODESolver) which calls the solver as many times as are necessary to solve the ODE for the full time domain. solve should return a vector of timesteps and a corresponding matrix of variables.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"function solve(prob::ODEProblem, solver::ODESolver)\n t = prob.tspan[1]; u = prob.u0\n us = [u]; ts = [t]\n while t < prob.tspan[2]\n (u,t) = solver(prob, u, t)\n push!(us,u)\n push!(ts,t)\n end\n ts, reduce(hcat,us)\nend\nnothing # hide","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"You can load the true solution and compare it in a plot like below. The file that contains the correct solution is located here: lotkadata.jld2.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"using JLD2\nusing Plots\n\ntrue_data = load(\"lotkadata.jld2\")\n\np1 = plot(true_data[\"t\"], true_data[\"u\"][1,:], lw=4, ls=:dash, alpha=0.7, color=:gray, label=\"x Truth\")\nplot!(p1, true_data[\"t\"], true_data[\"u\"][2,:], lw=4, ls=:dash, alpha=0.7, color=:gray, label=\"y Truth\")\n\n(t,X) = solve(prob, Euler(0.2))\n\nplot!(p1,t,X[1,:], color=1, lw=3, alpha=0.8, label=\"x Euler\")\nplot!(p1,t,X[2,:], color=2, lw=3, alpha=0.8, label=\"y Euler\")","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"As you can see in the plot above, the Euler method quickly becomes quite inaccurate because we make a step in the direction of the tangent which inevitably leads us away from the perfect solution as shown in the plot below. (Image: euler)","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"In the homework you will implement a Runge-Kutta solver to get a much better accuracy with the same step size.","category":"page"},{"location":"lecture_12/lab/#Automating-GaussNums","page":"Lab","title":"Automating GaussNums","text":"","category":"section"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Next you will implement your own uncertainty propagation. In the lecture you have already seen the new number type that we need for this:","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"struct GaussNum{T<:Real} <: Real\n μ::T\n σ::T\nend","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise (tiny)
\n
","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Overload the ± (type: \\pm) symbol to define GaussNums like this: 2.0 ± 1.0. Additionally, overload the show function such that GaussNums are printed with the ± as well.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"±(x,y) = GaussNum(x,y)\nBase.show(io::IO, x::GaussNum) = print(io, \"$(x.μ) ± $(x.σ)\")","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Recall, that for a function f(bm x) with N inputs, the uncertainty sigma_f is defined by","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"sigma_f = sqrtsum_i=1^N left( fracdfdx_isigma_i right)^2","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"To make GaussNums work for arithmetic operations we could manually implement all desired functions as we started doing in the lecture. With the autodiff package Zygote we can automate the generation of these functions. In the next two exercises you will implement a macro @register that takes a function and defines the corresponding uncertainty propagation rule according to the equation above.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Implement a helper function uncertain(f, args::GaussNum...) which takes a function f and its args and returns the resulting GaussNum with an uncertainty defined by the equation above.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Hint: You can compute the gradient of a function with Zygote, for example:","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"using Zygote;\nf(x,y) = x*y;\nZygote.gradient(f, 2., 3.)","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"function uncertain(f, args::GaussNum...)\n μs = [x.μ for x in args]\n dfs = Zygote.gradient(f,μs...)\n σ = mapreduce(+, zip(dfs,args)) do (df,x)\n (df * x.σ)^2\n end |> sqrt\n GaussNum(f(μs...), σ)\nend\nnothing # hide","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Now you can propagate uncertainties through any function like this:","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"x1 = 2.0 ± 2.0\nx2 = 2.0 ± 2.0\nuncertain(*, x1, x2)","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"You can verify the correctness of your implementation by comparing to the manual implementation from the lecture.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"For convenience, implement the macro @register which will define the uncertainty propagation rule for a given function. E.g. for the function * the macro should generate code like below","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Base.:*(args::GaussNum...) = uncertain(*, args...)","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Hint: If you run into trouble with module names of functions you can make use of","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"getmodule(f) = first(methods(f)).module\ngetmodule(*)","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"function _register(func::Symbol)\n mod = getmodule(eval(func))\n :($(mod).$(func)(args::GaussNum...) = uncertain($func, args...))\nend\n\nfunction _register(funcs::Expr)\n Expr(:block, map(_register, funcs.args)...)\nend\n\nmacro register(funcs)\n _register(funcs)\nend\nnothing # hide","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Lets register some arithmetic functions and see if they work","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"@register *\nx1 * x2\n@register - +\nx1 + x2\nx1 - x2","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"To finalize the definition of our new GaussNum we can define conversion and promotion rules such that we do not have to define things like","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"+(x::GaussNum, y::Real) = ...\n+(x::Real, y::GaussNum) = ...","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Define convert and promote_rules such that you can perform arithmetic operations on GaussNums and other Reals.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Hint: When converting a normal number to a GaussNum you can set the standard deviation to zero.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Base.convert(::Type{T}, x::T) where T<:GaussNum = x\nBase.convert(::Type{GaussNum{T}}, x::Number) where T = GaussNum(x,zero(T))\nBase.promote_rule(::Type{GaussNum{T}}, ::Type{S}) where {T,S} = GaussNum{T}\nBase.promote_rule(::Type{GaussNum{T}}, ::Type{GaussNum{T}}) where T = GaussNum{T}","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"You can test if everything works by adding/multiplying floats to GuassNums.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"1.0±1.0 + 2.0\n[1.0±0.001, 2.0]","category":"page"},{"location":"lecture_12/lab/#Propagating-Uncertainties-through-ODEs","page":"Lab","title":"Propagating Uncertainties through ODEs","text":"","category":"section"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"With our newly defined GaussNum we can easily propagate uncertainties through our ODE solvers without changing a single line of their code. Try it!","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"θ = [0.1±0.001, 0.2, 0.3, 0.2]\nu0 = [1.0±0.1, 1.0±0.1]\ntspan = (0.,100.)\ndt = 0.1\nprob = ODEProblem(lotkavolterra,tspan,u0,θ)\n\nt, X = solve(prob, Euler(0.1))","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Create a plot that takes a Vector{<:GaussNum} and plots the mean surrounded by the uncertainty.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"mu(x::GaussNum) = x.μ\nsig(x::GaussNum) = x.σ\n\nfunction uncertainplot(t, x::Vector{<:GaussNum})\n p = plot(\n t,\n mu.(x) .+ sig.(x),\n xlabel = \"x\",\n ylabel = \"y\",\n fill = (mu.(x) .- sig.(x), :lightgray, 0.5),\n linecolor = nothing,\n primary = false, # no legend entry\n )\n \n # add the data to the plots\n plot!(p, t, mu.(X[1,:])) \n\n return p\nend","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"uncertainplot(t, X[1,:])","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Unfortunately, with this approach, we would have to define things like uncertainplot! by hand. To make plotting GaussNums more pleasant we can make use of the @recipe macro from Plots.jl. It allows to define plot recipes for custom types (without having to depend on Plots.jl). Additionally, it makes it easiert to support all the different ways of creating plots (e.g. via plot or plot!, and with support for all keyword args) without having to overload tons of functions manually. If you want to read more about plot recipies in the docs of RecipesBase.jl. An example of a recipe for vectors of GaussNums could look like this:","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"@recipe function plot(ts::AbstractVector, xs::AbstractVector{<:GaussNum})\n # you can set a default value for an attribute with `-->`\n # and force an argument with `:=`\n μs = [x.μ for x in xs]\n σs = [x.σ for x in xs]\n @series begin\n :seriestype := :path\n # ignore series in legend and color cycling\n primary := false\n linecolor := nothing\n fillcolor := :lightgray\n fillalpha := 0.5\n fillrange := μs .- σs\n # ensure no markers are shown for the error band\n markershape := :none\n # return series data\n ts, μs .+ σs\n end\n ts, μs\nend\n\n# now we can easily plot multiple things on to of each other\np1 = plot(t, X[1,:], label=\"x\", lw=3)\nplot!(p1, t, X[2,:], label=\"y\", lw=3)","category":"page"},{"location":"lecture_12/lab/#References","page":"Lab","title":"References","text":"","category":"section"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"MIT18-330S12: Chapter 5\nRK2 derivation","category":"page"},{"location":"lecture_06/lecture/#introspection","page":"Lecture","title":"Language introspection","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"What is metaprogramming? A high-level code that writes high-level code by Steven Johnson.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Why do we need metaprogramming? ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"In general, we do not need it, as we can do whatever we need without it, but it can help us to remove a boilerplate code. \nAs an example, consider a @show macro, which just prints the name of the variable (or the expression) and its evaluation. This means that instead of writing println(\"2+exp(4) = \", 2+exp(4)) we can just write @show 2+exp(4).\nWe have seen @time or @benchmark, which is difficult to implement using normal function, since when you pass 2+exp(4) as a function argument, it will be automatically evaluated. You need to pass it as an expression, that can be evaluated within the function.\n@chain macro from Chain.jl improves over native piping |>\nWe have seen @forward macro implementing encapsulation.\nMacros are used to insert compilation directives not accessible through the syntax, e.g. @inbounds.\nA chapter on its own is definition of Domain Specific Languages.","category":"page"},{"location":"lecture_06/lecture/#Translation-stages-from-source-code-to-machine-code","page":"Lecture","title":"Translation stages from source code to machine code","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Julia (as any modern compiler) uses several stages to convert source code to native code. ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Julia's steps consists from [1] \nparse the source code to abstract syntax tree (AST) –- parse function \nexpand macros –- macroexpand function\nsyntax desugaring\nstamentize controlflow\nresolve scopes \ngenerate intermediate representation (IR) (\"goto\" form) –- expand or code_lowered functions\ntop-level evaluation, method sorting –- methods\ntype inference\ninlining and high level optimization –- code_typed\nLLVM IR generation –- code_llvm\nLLVM optimizer, native code generation –- code_native\nsteps 3-6 are done in inseparable stage\nJulia's IR is in static single assignment form","category":"page"},{"location":"lecture_06/lecture/#Example:-Fibonacci-numbers","page":"Lecture","title":"Example: Fibonacci numbers","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Consider for example a function computing the Fibonacci numbers[1]","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"function nextfib(n)\n\ta, b = one(n), one(n)\n\twhile b < n\n\t\ta, b = b, a + b\n\tend\n\treturn b\nend","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"[1]: From StackOverflow ","category":"page"},{"location":"lecture_06/lecture/#Parsing","page":"Lecture","title":"Parsing","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"The first thing the compiler does is that it will parse the source code (represented as a string) to the abstract syntax tree (AST). We can inspect the results of this stage as ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> parsed_fib = Meta.parse(\n\"\"\"\n\tfunction nextfib(n)\n\t\ta, b = one(n), one(n)\n\t\twhile b < n\n\t\t\ta, b = b, a + b\n\t\tend\n\t\treturn b\n\tend\"\"\")\n:(function nextfib(n)\n #= none:1 =#\n #= none:2 =#\n (a, b) = (one(n), one(n))\n #= none:3 =#\n while b < n\n #= none:4 =#\n (a, b) = (b, a + b)\n end\n #= none:6 =#\n return b\n end)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"AST is a tree representation of the source code, where the parser has already identified individual code elements function call, argument blocks, etc. The parsed code is represented by Julia objects, therefore it can be read and modified by Julia from Julia at your wish (this is what is called homo-iconicity of a language the itself being derived from Greek words homo- meaning \"the same\" and icon meaning \"representation\"). Using TreeView","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"using TreeView, TikzPictures\ng = tikz_representation(walk_tree(parsed_fib))\nTikzPictures.save(SVG(\"parsed_fib.svg\"), g)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"(Image: parsed_fib.svg)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"We can see that the AST is indeed a tree, with function being the root node (caused by us parsing a function). Each inner node represents a function call with children of the inner node being its arguments. An interesting inner node is the Block representing a sequence of statements, where we can also see information about lines in the source code inserted as comments. Lisp-like S-Expression can be printed using Meta.show_sexpr(parsed_fib).","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"(:function, (:call, :nextfib, :n), (:block,\n :(#= none:1 =#),\n :(#= none:2 =#),\n (:(=), (:tuple, :a, :b), (:tuple, (:call, :one, :n), (:call, :one, :n))),\n :(#= none:3 =#),\n (:while, (:call, :<, :b, :n), (:block,\n :(#= none:4 =#),\n (:(=), (:tuple, :a, :b), (:tuple, :b, (:call, :+, :a, :b)))\n )),\n :(#= none:6 =#),\n (:return, :b)\n ))","category":"page"},{"location":"lecture_06/lecture/#Expanding-macros","page":"Lecture","title":"Expanding macros","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"If we insert a \"useless\" macro to nextfib, for example @show b, we see that the macro is not expanded and it is left there as-is.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> parsed_fib = Meta.parse(\n\"\"\"\n function nextfib(n)\n a, b = one(n), one(n)\n while b < n\n a, b = b, a + b\n end\n @show b\n return b\n end\"\"\")\n:(function nextfib(n)\n #= none:1 =#\n #= none:2 =#\n (a, b) = (one(n), one(n))\n #= none:3 =#\n while b < n\n #= none:4 =#\n (a, b) = (b, a + b)\n #= none:5 =#\n end\n #= none:6 =#\n #= none:6 =# @show b\n #= none:7 =#\n return b\n end)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"and we can ask for expansion of the macro","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> macroexpand(Main, parsed_fib)\n:(function nextfib(n)\n #= none:1 =#\n #= none:2 =#\n (a, b) = (one(n), one(n))\n #= none:3 =#\n while b < n\n #= none:4 =#\n (a, b) = (b, a + b)\n #= none:5 =#\n end\n #= none:6 =#\n begin\n Base.println(\"b = \", Base.repr(begin\n #= show.jl:1047 =#\n local var\"#62#value\" = b\n end))\n var\"#62#value\"\n end\n #= none:7 =#\n return b\n end)","category":"page"},{"location":"lecture_06/lecture/#Lowering","page":"Lecture","title":"Lowering","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"The next stage is lowering, where AST is converted to Static Single Assignment Form (SSA), in which \"each variable is assigned exactly once, and every variable is defined before it is used\". Loops and conditionals are transformed into gotos and labels using a single unless/goto construct (this is not exposed in user-level Julia).","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_lowered nextfib(3)\nCodeInfo(\n1 ─ %1 = Main.one(n)\n│ %2 = Main.one(n)\n│ a = %1\n└── b = %2\n2 ┄ %5 = b < n\n└── goto #4 if not %5\n3 ─ %7 = b\n│ %8 = a + b\n│ a = %7\n│ b = %8\n└── goto #2\n4 ─ return b\n)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"or alternatively lowered_fib = Meta.lower(@__MODULE__, parsed_fib). ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"We can see that ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"compiler has introduced a lot of variables \nwhile (and for) loops has been replaced by a goto, where goto can be conditional","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"For inserted debugging information, there is an option to pass keyword argument debuginfo=:source. ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_lowered debuginfo=:source nextfib(3)\nCodeInfo(\n @ none:2 within `nextfib'\n1 ─ %1 = Main.one(n)\n│ %2 = Main.one(n)\n│ a = %1\n└── b = %2\n @ none:3 within `nextfib'\n2 ┄ %5 = b < n\n└── goto #4 if not %5\n @ none:4 within `nextfib'\n3 ─ %7 = b\n│ %8 = a + b\n│ a = %7\n│ b = %8\n└── goto #2\n @ none:6 within `nextfib'\n4 ─ return b\n)","category":"page"},{"location":"lecture_06/lecture/#Code-Typing","page":"Lecture","title":"Code Typing","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Code typing is the process in which the compiler attaches types to variables and tries to infer types of objects returned from called functions. If the compiler fails to infer the returned type, it will give the variable type Any, in which case a dynamic dispatch will be used in subsequent operations with the variable. Inspecting typed code is therefore important for detecting type instabilities (the process can be difficult and error prone, fortunately, new tools like Jet.jl may simplify this task). The output of typing can be inspected using @code_typed macro. If you know the types of function arguments, aka function signature, you can call directly function InteractiveUtils.code_typed(nextfib, (typeof(3),)).","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_typed nextfib(3)\nCodeInfo(\n1 ─ nothing::Nothing\n2 ┄ %2 = φ (#1 => 1, #3 => %6)::Int64\n│ %3 = φ (#1 => 1, #3 => %2)::Int64\n│ %4 = Base.slt_int(%2, n)::Bool\n└── goto #4 if not %4\n3 ─ %6 = Base.add_int(%3, %2)::Int64\n└── goto #2\n4 ─ return %2\n) => Int64","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"We can see that ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"some calls have been inlined, e.g. one(n) was replaced by 1 and the type was inferred as Int. \nThe expression b < n has been replaced with its implementation in terms of the slt_int intrinsic (\"signed integer less than\") and the result of this has been annotated with return type Bool. \nThe expression a + b has been also replaced with its implementation in terms of the add_int intrinsic and its result type annotated as Int64. \nAnd the return type of the entire function body has been annotated as Int64.\nThe phi-instruction %2 = φ (#1 => 1, #3 => %6) is a selector function, which returns the value depending on from which branch do you come from. In this case, variable %2 will have value 1, if the control was transfered from block #1 and it will have value copied from variable %6 if the control was transferreed from block 3 see also. The φ stands from phony variable.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"When we have called @code_lower, the role of types of arguments was in selecting - via multiple dispatch - the appropriate function body among different methods. Contrary in @code_typed, the types of parameters determine the choice of inner methods that need to be called (again with multiple dispatch). This process can trigger other optimization, such as inlining, as seen in the case of one(n) being replaced with 1 directly, though here this replacement is hidden in the φ function. ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Note that the same view of the code is offered by the @code_warntype macro, which we have seen in the previous lecture. The main difference from @code_typed is that it highlights type instabilities with red color and shows only unoptimized view of the code. You can view the unoptimized code with a keyword argument optimize=false:","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_typed optimize=false nextfib(3)\nCodeInfo(\n1 ─ %1 = Main.one(n)::Core.Const(1)\n│ %2 = Main.one(n)::Core.Const(1)\n│ (a = %1)::Core.Const(1)\n└── (b = %2)::Core.Const(1)\n2 ┄ %5 = (b < n)::Bool\n└── goto #4 if not %5\n3 ─ %7 = b::Int64\n│ %8 = (a + b)::Int64\n│ (a = %7)::Int64\n│ (b = %8)::Int64\n└── goto #2\n4 ─ return b\n) => Int64","category":"page"},{"location":"lecture_06/lecture/#Lowering-to-LLVM-IR","page":"Lecture","title":"Lowering to LLVM IR","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Julia uses the LLVM compiler framework to generate machine code. LLVM stands for low-level virtual machine and it is basis of many modern compilers (see wiki). We can see the textual form of code lowered to LLVM IR by invoking ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_llvm debuginfo=:source nextfib(3)\n; @ REPL[10]:1 within `nextfib'\ndefine i64 @julia_nextfib_890(i64 signext %0) {\ntop:\n br label %L2\n\nL2: ; preds = %L2, %top\n %value_phi = phi i64 [ 1, %top ], [ %1, %L2 ]\n %value_phi1 = phi i64 [ 1, %top ], [ %value_phi, %L2 ]\n; @ REPL[10]:3 within `nextfib'\n; ┌ @ int.jl:83 within `<'\n %.not = icmp slt i64 %value_phi, %0\n; └\n; @ REPL[10]:4 within `nextfib'\n; ┌ @ int.jl:87 within `+'\n %1 = add i64 %value_phi1, %value_phi\n; └\n; @ REPL[10]:3 within `nextfib'\n br i1 %.not, label %L2, label %L8\n\nL8: ; preds = %L2\n; @ REPL[10]:6 within `nextfib'\n ret i64 %value_phi\n}","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"LLVM code can be tricky to understand first, but one gets used to it. Notice references to the source code, which help with orientation. From the code above, we may infer","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"code starts by jumping to label L2, from where it reads values of two variables to two \"registers\" value_phi and value_phi1 (variables in LLVM starts with %). \nBoth registers are treated as int64 and initialized by 1. \n[ 1, %top ], [ %value_phi, %L2 ] means that values are initialized as 1 if you come from the label top and as value value_phi if you come from %2. This is the LLVM's selector (phony φ).\nicmp slt i64 %value_phi, %0 compares the variable %value_phi to the content of variable %0. Notice the anotation that we are comparing Int64.\n%1 = add i64 %value_phi1, %value_phi adds two variables %value_phi1 and %value_phi. Note again than we are using Int64 addition. \nbr i1 %.not, label %L2, label %L8 implements a conditional jump depending on the content of %.not variable. \nret i64 %value_phi returns the value indicating it to be an Int64.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"It is not expected you will be directly operating on the LLVM code, though there are libraries which does that. For example Enzyme.jl performs automatic differentiation of LLVM code, which has the benefit of being able to take a gradient through setindex!.","category":"page"},{"location":"lecture_06/lecture/#Producing-the-native-vode","page":"Lecture","title":"Producing the native vode","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Native code The last stage is generation of the native code, which Julia executes. The native code depends on the target architecture (e.g. x86, ARM). As in previous cases there is a macro for viewing the compiled code @code_native","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_native debuginfo=:source nextfib(3)\n\t.section\t__TEXT,__text,regular,pure_instructions\n; ┌ @ REPL[10]:1 within `nextfib'\n\tmovl\t$1, %ecx\n\tmovl\t$1, %eax\n\tnopw\t(%rax,%rax)\nL16:\n\tmovq\t%rax, %rdx\n\tmovq\t%rcx, %rax\n; │ @ REPL[10]:4 within `nextfib'\n; │┌ @ int.jl:87 within `+'\n\taddq\t%rcx, %rdx\n\tmovq\t%rdx, %rcx\n; │└\n; │ @ REPL[10]:3 within `nextfib'\n; │┌ @ int.jl:83 within `<'\n\tcmpq\t%rdi, %rax\n; │└\n\tjl\tL16\n; │ @ REPL[10]:6 within `nextfib'\n\tretq\n\tnopw\t%cs:(%rax,%rax)\n; └","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"and the output is used mainly for debugging / inspection. ","category":"page"},{"location":"lecture_06/lecture/#Looking-around-the-language","page":"Lecture","title":"Looking around the language","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Language introspection is very convenient for investigating, how things are implemented and how they are optimized / compiled to the native code.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"note: Reminder `@which`\nThough we have already used it quite a few times, recall the very useful macro @which, which identifies the concrete function called in a function call. For example @which mapreduce(sin, +, [1,2,3,4]). Note again that the macro here is a convenience macro to obtain types of arguments from the expression. Under the hood, it calls InteractiveUtils.which(function_name, (Base.typesof)(args...)). Funny enough, you can call @which InteractiveUtils.which(+, (Base.typesof)(1,1)) to inspect, where which is defined.","category":"page"},{"location":"lecture_06/lecture/#Broadcasting","page":"Lecture","title":"Broadcasting","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Broadcasting is not a unique concept in programming languages (Python/Numpy, MATLAB), however its implementation in Julia allows to easily fuse operations. For example ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"x = randn(100)\nsin.(x) .+ 2 .* cos.(x) .+ x","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"is all computed in a single loop. We can inspect, how this is achieved in the lowered code:","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> Meta.@lower sin.(x) .+ 2 .* cos.(x) .+ x\n:($(Expr(:thunk, CodeInfo(\n @ none within `top-level scope'\n1 ─ %1 = Base.broadcasted(sin, x)\n│ %2 = Base.broadcasted(cos, x)\n│ %3 = Base.broadcasted(*, 2, %2)\n│ %4 = Base.broadcasted(+, %1, %3)\n│ %5 = Base.broadcasted(+, %4, x)\n│ %6 = Base.materialize(%5)\n└── return %6\n))))","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Notice that we have not used the usual @code_lowered macro, because the statement to be lowered is not a function call. In these cases, we have to use @code_lowered, which can handle more general program statements. On these cases, we cannot use @which either, as that applies to function calls only as well.","category":"page"},{"location":"lecture_06/lecture/#Generators","page":"Lecture","title":"Generators","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Meta.@lower [x for x in 1:4]\n:($(Expr(:thunk, CodeInfo(\n @ none within `top-level scope'\n1 ─ %1 = 1:4\n│ %2 = Base.Generator(Base.identity, %1)\n│ %3 = Base.collect(%2)\n└── return %3\n))))","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"from which we see that the Generator is implemented using the combination of a Base.collect, which is a function collecting items of a sequence and Base.Generator(f,x), which implements an iterator, which applies function f on elements of x over which is being iterated. So an almost magical generators have instantly lost their magic.","category":"page"},{"location":"lecture_06/lecture/#Closures","page":"Lecture","title":"Closures","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"adder(x) = y -> y + x\n\njulia> @code_lowered adder(5)\nCodeInfo(\n1 ─ %1 = Main.:(var\"#8#9\")\n│ %2 = Core.typeof(x)\n│ %3 = Core.apply_type(%1, %2)\n│ #8 = %new(%3, x)\n└── return #8\n)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Core.apply_type is a one way of constructing an object briefly mentioned in description of object allocation.","category":"page"},{"location":"lecture_06/lecture/#The-effect-of-type-instability","page":"Lecture","title":"The effect of type-instability","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"struct Wolf\n\tname::String\n\tenergy::Int\nend\n\nstruct Sheep\n\tname::String\n\tenergy::Int\nend\n\nsound(wolf::Wolf) = println(wolf.name, \" has howled.\")\nsound(sheep::Sheep) = println(sheep.name, \" has baaed.\")\nstable_pack = (Wolf(\"1\", 1), Wolf(\"2\", 2), Sheep(\"3\", 3))\nunstable_pack = [Wolf(\"1\", 1), Wolf(\"2\", 2), Sheep(\"3\", 3)]\n@code_typed map(sound, stable_pack)\n@code_typed map(sound, unstable_pack)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\nCthulhu.jlCthulhu.jl is a library (tool) which simplifies the above, where we want to iteratively dive into functions called in some piece of code (typically some function). Cthulhu is different from te normal debugger, since the debugger is executing the code, while Cthulhu is just lower_typing the code and presenting functions (with type of arguments inferred).","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"using Cthulhu\n@descend map(sound, unstable_pack)","category":"page"},{"location":"lecture_06/lecture/#General-notes-on-metaprogramming","page":"Lecture","title":"General notes on metaprogramming","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"According to an excellent talk by Steven Johnson, you should use metaprogramming sparingly, because on one hand it's very powerful, but on the other it is generally difficult to read and it can lead to unexpected errors. Julia allows you to interact with the compiler at two different levels.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"After the code is parsed to AST, you can modify it directly or through macros.\nWhen SSA form is being typed, you can create custom functions using the concept of generated functions or directly emit intermediate representation.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"More functionalities are coming through the JuliaCompilerPlugins project, but we will not talk about them (yet), as they are not mature yet. ","category":"page"},{"location":"lecture_06/lecture/#What-is-Quotation?","page":"Lecture","title":"What is Quotation?","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"When we are doing metaprogramming, we need to somehow tell the compiler that the next block of code is not a normal block of code to be executed, but that it should be interpreted as data and in any sense it should not be evaluated. Quotation refers to exactly this syntactic sugar. In Julia, quotation is achieved either through :(...) or quote ... end.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Notice the difference between","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"1 + 1 ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"and ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":":(1 + 1)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"The type returned by the quotation depends on what is quoted. Observe the returned type of the following quoted code","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":":(1) |> typeof\n:(:x) |> typeof\n:(1 + x) |> typeof\nquote\n 1 + x\n x + 1\nend |> typeof","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"All of these snippets are examples of the quoted code, but only :(1 + x) and the quote block produce objects of type Expr. An interesting return type is the QuoteNode, which allows to insert piece of code which should contain elements that should not be interpolated. Most of the time, quoting returns Expressions.","category":"page"},{"location":"lecture_06/lecture/#Expressions","page":"Lecture","title":"Expressions","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Abstract Syntax Tree, the output of Julia's parser, is expressed using Julia's own datastructures, which means that you can freely manipulate it (and constructed) from the language itself. This property is called homoiconicity. Julia's compiler allows you to intercept compilation just after it has parsed the source code. Before we will take advantage of this power, we should get familiar with the strucute of the AST.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"The best way to inspect the AST is through the combination ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Meta.parse, which parses the source code to AST, \ndump which print AST to terminal, \neval which evaluates the AST within the current module.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Let's start by investigating a very simple statement 1 + 1, whose AST can be constructed either by Meta.parse(\"1 + 1\") or :(1 + 1) or quote 1+1 end (the last one includes also the line information metadata).","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> p = :(1+1)\n:(1 + 1)\n\njulia> typeof(p)\nExpr\n\njulia> dump(p)\nExpr\n head: Symbol call\n args: Array{Any}((3,))\n 1: Symbol +\n 2: Int64 1\n 3: Int64 1","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"The parsed code p is of type Expr, which according to Julia's help[2] is a type representing compound expressions in parsed julia code (ASTs). Each expression consists: of a head Symbol identifying which kind of expression it is (e.g. a call, for loop, conditional statement, etc.), and subexpressions (e.g. the arguments of a call). The subexpressions are stored in a Vector{Any} field called args. If you recall the figure above, where AST was represented as a tree, head gives each node the name name args are either some parameters of the node, or they point to childs of that node. The interpretation of the node depends on the its type stored in head (note that the word type used here is not in the Julia sense).","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"[2]: Help: Core.Expr","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"info: `Symbol` type\nWhen manipulations of expressions, we encounter the term Symbol. Symbol is the smallest atom from which the program (in AST representation) is built. It is used to identify an element in the language, for example variable, keyword or function name. Symbol is not a string, since string represents itself, whereas Symbol can represent something else (a variable). An illustrative example[3] goes as follows.julia> eval(:foo)\nERROR: foo not defined\n\njulia> foo = \"hello\"\n\"hello\"\n\njulia> eval(:foo)\n\"hello\"\n\njulia> eval(\"foo\")\n\"foo\"which shows that what the symbol :foo evaluates to depends on what – if anything – the variable foo is bound to, whereas \"foo\" always just evaluates to \"foo\".Symbols can be constructed either by prepending any string with : or by calling Symbol(...), which concatenates the arguments and create the symbol out of it. All of the following are symbolsjulia> :+\n:+\n\njulia> :function\n:function\n\njulia> :call\n:call\n\njulia> :x\n:x\n\njulia> Symbol(:Very,\"_twisted_\",:symbol,\"_definition\")\n:Very_twisted_symbol_definition\n\njulia> Symbol(\"Symbol with blanks\")\nSymbol(\"Symbol with blanks\")Symbols therefore allows us to operate with a piece of code without evaluating it.In Julia, symbols are \"interned strings\", which means that compiler attaches each string a unique identifier (integer), such that it can quickly compare them. Compiler uses Symbols exclusively and the important feature is that they can be quickly compared. This is why people like to use them as keys in Dict.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"[3]: An example provided by Stefan Karpinski.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"info: `Expr`essions\nFrom Julia's help[2]:Expr(head::Symbol, args...)A type representing compound expressions in parsed julia code (ASTs). Each expression consists of a head Symbol identifying which kind of expression it is (e.g. a call, for loop, conditional statement, etc.), and subexpressions (e.g. the arguments of a call). The subexpressions are stored in a Vector{Any} field called args. The expression is simple yet very flexible. The head Symbol tells how the expression should be treated and arguments provide all needed parameters. Notice that the structure is also type-unstable. This is not a big deal, since the expression is used to generate code, hence it is not executed repeatedly.","category":"page"},{"location":"lecture_06/lecture/#Construct-code-from-scratch","page":"Lecture","title":"Construct code from scratch","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Since Expr is a Julia structure, we can construct it manually as we can construct any other structure","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> Expr(:call, :+, 1 , 1) |> dump\nExpr\n head: Symbol call\n args: Array{Any}((3,))\n 1: Symbol +\n 2: Int64 1\n 3: Int64 1","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"yielding to the same structure as we have created above. Expressions can be evaluated using eval, as has been said. to programmatically evaluate our expression, let's do ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"e = Expr(:call, :+, 1, 1)\neval(e)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"We are free to use variables (identified by symbols) inside the expression ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"e = Expr(:call, :+, :x, 5)\neval(e)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"but unless they are not defined within the scope, the expression cannot produce a meaningful result","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"x = 3\neval(e)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":":(1 + sin(x)) == Expr(:call, :+, 1, Expr(:call, :sin, :x))","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Since the expression is a Julia structure, we are free to manipulate it. Let's for example substitutue x in e = :(x + 5) with 2x.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"e = :(x + 5)\ne.args = map(e.args) do a \n\ta == :x ? Expr(:call, :*, 2, :x) : a \nend","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"or ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"e = :(x + 5)\ne.args = map(e.args) do a \n\ta == :x ? :(2*x) : a \nend","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"and verify that the results are correct.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> dump(e)\nExpr\n head: Symbol call\n args: Array{Any}((3,))\n 1: Symbol +\n 2: Expr\n head: Symbol call\n args: Array{Any}((3,))\n 1: Symbol *\n 2: Int64 2\n 3: Symbol x\n 3: Int64 5\n\njulia> eval(e)\n11","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"As already mentioned, the manipulation of Expression can be arbitrary. In the above example, we have been operating directly on the arguments. But what if x would be deeper in the expression, as for example in 2(3 + x) + 2(2 - x)? We can implement the substitution using multiple dispatch as we would do when implementing any other function in Julia.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"substitue_x(x::Symbol) = x == :x ? :(2*x) : x\nsubstitue_x(e::Expr) = Expr(e.head, map(substitue_x, e.args)...)\nsubstitue_x(u) = u","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"which works as promised.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> e = :(2(3 + 2x) + 2(2 - x))\n:(2 * (3 + x) + 2 * (2 - x))\njulia> f = substitue_x(e)\n:(2 * (3 + 2x) + 2 * (2 - 2x))","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"or we can replace the sin function","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"replace_sin(x::Symbol) = x == :sin ? :cos : x\nreplace_sin(e::Expr) = Expr(e.head, map(replace_sin, e.args)...)\nreplace_sin(u) = u","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"replace_sin(:(1 + sin(x)))","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Sometimes, we want to operate on a block of code as opposed to single line expressions. Recall that a block of code is defined-quoted with quote ... end. Let us see how replace_x can handle the following example:","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"e = quote \n\ta = x + 3\n\tb = 2 - x\n\t2a + 2b\nend","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> replace_x(e) |> Base.remove_linenums!\nquote\n a = 2x + 3\n b = 2 - 2x\n 2a + 2b\nend\n\njulia> replace_x(e) |> eval\n10","category":"page"},{"location":"lecture_06/lecture/#Brittleness-of-code-manipulation","page":"Lecture","title":"Brittleness of code manipulation","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"When we are manipulating the AST or creating new expressions from scratch, there is no syntactic validation performed by the parser. It is therefore very easy to create AST which does not make any sense and cannot be compiled. We have already seen that we can refer to variables that were not defined yet (this makes perfect sense). The same goes with functions (which also makes a lot of sense).","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"e = :(g() + 5)\neval(e)\ng() = 5\neval(e)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"But we can also introduce keywords which the language does not know. For example ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"e = Expr(:my_keyword, 1, 2, 3)\n:($(Expr(:my_keyword, 1, 2, 3)))\n\njulia> e.head\n:my_keyword\n\njulia> e.args\n3-element Vector{Any}:\n 1\n 2\n 3\n\njulia> eval(e)\nERROR: syntax: invalid syntax (my_keyword 1 2 3)\nStacktrace:\n [1] top-level scope\n @ none:1\n [2] eval\n @ ./boot.jl:360 [inlined]\n [3] eval(x::Expr)\n @ Base.MainInclude ./client.jl:446\n [4] top-level scope\n @ REPL[8]:1","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"notice that error is not related to undefined variable / function, but the invalid syntax. This also demonstrates the role of head in Expr. More on Julia AST can be found in the developer documentation.","category":"page"},{"location":"lecture_06/lecture/#Alternative-way-to-look-at-code","page":"Lecture","title":"Alternative way to look at code","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Meta.parse(\"x[3]\") |> dump","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"We can see a new Symbol ref as a head and the position 3 of variable x.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Meta.parse(\"(1,2,3)\") |> dump","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Meta.parse(\"1/2/3\") |> dump","category":"page"},{"location":"lecture_06/lecture/#Code-generation","page":"Lecture","title":"Code generation","text":"","category":"section"},{"location":"lecture_06/lecture/#Using-metaprogramming-in-inheritance-by-encapsulation","page":"Lecture","title":"Using metaprogramming in inheritance by encapsulation","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Recall that Julia (at the moment) does not support inheritance, therefore the only way to adopt functionality of some object and extend it is through encapsulation. Assuming we have some object T, we wrap that object into a new structure. Let's work out a concrete example, where we define the our own matrix. ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"struct MyMatrix{T} <: AbstractMatrix{T}\n\tx::Matrix{T}\nend","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Now, to make it useful, we should define all the usual methods, like size, length, getindex, setindex!, etc. We can list methods defined with Matrix as an argument methodswith(Matrix) (recall this will load methods that are defined with currently loaded libraries). Now, we would like to overload them. To minimize the written code, we can write","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"import Base: setindex!, getindex, size, length\nfor f in [:setindex!, :getindex, :size, :length]\n\teval(:($(f)(A::MyMatrix, args...) = $(f)(A.x, args...)))\nend","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"which we can verify now that it works as expected ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> a = MyMatrix([1 2 ; 3 4])\n2×2 MyMatrix{Int64}:\n 1 2\n 3 4\n\njulia> a[4]\n4\n\njulia> a[3] = 0\n0\n\njulia> a\n2×2 MyMatrix{Int64}:\n 1 0\n 3 4","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"In this way, Julia acts as its own preprocessor. The above look can be equally written as ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"for f in [:setindex!, :getindex, :size, :length]\n println(\"$(f)(A::MyMatrix, args...) = $(f)(A.x, args...)\")\nend","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"for f in [:setindex!, :getindex, :size, :length]\n\ts = \"Base.$(f)(A::MyMatrix, args...) = $(f)(A.x, args...)\"\n\tprintln(s)\n\teval(Meta.parse(s))\nend","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"for f in [:setindex!, :getindex, :size, :length] \t@eval f(A::MyMatrix, args...) = f(A.x, args...) end","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Notice that we have just hand-implemented parts of @forward macro from MacroTools, which does exactly this.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"","category":"page"},{"location":"lecture_06/lecture/#Resources","page":"Lecture","title":"Resources","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Introduction to Julia by Jeff Bezanson on first JuliaCon\nJulia's manual on metaprogramming\nDavid P. Sanders' workshop @ JuliaCon 2021 \nSteven Johnson's keynote talk @ JuliaCon 2019\nJames Nash's Is Julia Aot or JIT @ JuliaCon 2017\nAndy Ferris's workshop @ JuliaCon 2018\nFrom Macros to DSL by John Myles White \nNotes on JuliaCompilerPlugin","category":"page"},{"location":"lecture_09/lab/#Lab-09-Generated-Functions-and-IR","page":"Lab","title":"Lab 09 - Generated Functions & IR","text":"","category":"section"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"In this lab you will practice two advanced meta programming techniques:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Generated functions can help you write specialized code for certain kinds of parametric types with more flexibility and/or less code.\nIRTools.jl is a package that simplifies the manipulation of lowered and typed Julia code","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"using BenchmarkTools","category":"page"},{"location":"lecture_09/lab/#@generated-Functions","page":"Lab","title":"@generated Functions","text":"","category":"section"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Remember the three most important things about generated functions:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"They return quoted expressions (like macros).\nYou have access to type information of your input variables.\nThey have to be pure","category":"page"},{"location":"lecture_09/lab/#A-faster-polynomial","page":"Lab","title":"A faster polynomial","text":"","category":"section"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Throughout this course we have come back to our polynomial function which evaluates a polynomial based on the Horner schema. Below you can find a version of the function that operates on a tuple of length N.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"function polynomial(x, p::NTuple{N}) where N\n acc = p[N]\n for i in N-1:-1:1\n acc = x*acc + p[i]\n end\n acc\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Julia has its own implementation of this function called evalpoly. If we compare the performance of our polynomial and Julia's evalpoly we can observe a pretty big difference:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"x = 2.0\np = ntuple(float,20);\n\n@btime polynomial($x,$p)\n@btime evalpoly($x,$p)","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Julia's implementation uses a generated function which specializes on different tuple lengths (i.e. it unrolls the loop) and eliminates the (small) overhead of looping over the tuple. This is possible, because the length of the tuple is known during compile time. You can check the difference between polynomial and evalpoly yourself via the introspectionwtools you know - e.g. @code_lowered.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Rewrite the polynomial function as a generated function with the signature","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"genpoly(x::Number, p::NTuple{N}) where N","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Hints:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Remember that you have to generate a quoted expression inside your generated function, so you will need things like :($expr1 + $expr2).\nYou can debug the expression you are generating by omitting the @generated macro from your function.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"@generated function genpoly(x, p::NTuple{N}) where N\n ex = :(p[$N])\n for i in N-1:-1:1\n ex = :(x*$ex + p[$i])\n end\n ex\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"You should get the same performance as evalpoly (and as @poly from Lab 7 with the added convenience of not having to spell out all the coefficients in your code like: p = @poly 1 2 3 ...).","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"@btime genpoly($x,$p)","category":"page"},{"location":"lecture_09/lab/#Fast,-Static-Matrices","page":"Lab","title":"Fast, Static Matrices","text":"","category":"section"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Another great example that makes heavy use of generated functions are static arrays. A static array is an array of fixed size which can be implemented via an NTuple. This means that it will be allocated on the stack, which can buy us a lot of performance for smaller static arrays. We define a StaticMatrix{T,C,R,L} where the paramteric types represent the matrix element type T (e.g. Float32), the number of rows R, the number of columns C, and the total length of the matrix L=C*R (which we need to set the size of the NTuple).","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"struct StaticMatrix{T,R,C,L} <: AbstractArray{T,2}\n data::NTuple{L,T}\nend\n\nfunction StaticMatrix(x::AbstractMatrix{T}) where T\n (R,C) = size(x)\n StaticMatrix{T,R,C,C*R}(x |> Tuple)\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"As a warm-up, overload the Base functions size, length, getindex(x::StaticMatrix,i::Int), and getindex(x::Solution,r::Int,c::Int).","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Base.size(x::StaticMatrix{T,R,C}) where {T,R,C} = (R,C)\nBase.length(x::StaticMatrix{T,R,C,L}) where {T,R,C,L} = L\nBase.getindex(x::StaticMatrix, i::Int) = x.data[i]\nBase.getindex(x::StaticMatrix{T,R,C}, r::Int, c::Int) where {T,R,C} = x.data[R*(c-1) + r]","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"You can check if everything works correctly by comparing to a normal Matrix:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"x = rand(2,3)\nx[1,2]\na = StaticMatrix(x)\na[1,2]","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Overload matrix multiplication between two static matrices","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Base.:*(x::StaticMatrix{T,K,M},y::StaticMatrix{T,M,N})","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"with a generated function that creates an expression without loops. Below you can see an example for an expression that would be generated from multiplying two 2times 2 matrices.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":":(StaticMatrix{T,2,2,4}((\n (x[1,1]*y[1,1] + x[1,2]*y[2,1]),\n (x[2,1]*y[1,1] + x[2,2]*y[2,1]),\n (x[1,1]*y[1,2] + x[1,2]*y[2,2]),\n (x[2,1]*y[1,2] + x[2,2]*y[2,2])\n)))","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Hints:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"You can get output like above by leaving out the @generated in front of your overload.\nIt might be helpful to implement matrix multiplication in a normal Julia function first.\nYou can construct an expression for a sum of multiple elements like below.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Expr(:call,:+,1,2,3)\nExpr(:call,:+,1,2,3) |> eval","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"@generated function Base.:*(x::StaticMatrix{T,K,M}, y::StaticMatrix{T,M,N}) where {T,K,M,N}\n zs = map(Iterators.product(1:K, 1:N) |> collect |> vec) do (k,n)\n Expr(:call, :+, [:(x[$k,$m] * y[$m,$n]) for m=1:M]...)\n end\n z = Expr(:tuple, zs...)\n :(StaticMatrix{$T,$K,$N,$(K*N)}($z))\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"You can check that your matrix multiplication works by multiplying two random matrices. Which one is faster?","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"a = rand(2,3)\nb = rand(3,4)\nc = StaticMatrix(a)\nd = StaticMatrix(b)\na*b\nc*d","category":"page"},{"location":"lecture_09/lab/#OptionalArgChecks.jl","page":"Lab","title":"OptionalArgChecks.jl","text":"","category":"section"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"The package OptionalArgChecks.jl makes is possible to add checks to a function which can then be removed by calling the function with the @skip macro. For example, we can check if the input to a function f is an even number","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"function f(x::Number)\n iseven(x) || error(\"Input has to be an even number!\")\n x\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"If you are doing more involved argument checking it can take quite some time to perform all your checks. However, if you want to be fast and are completely sure that you are always passing in the correct inputs to your function, you might want to remove them in some cases. Hence, we would like to transform the IR of the function above","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"using IRTools\nusing IRTools: @code_ir\n@code_ir f(1)","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"To some thing like this","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"transformed_f(x::Number) = x\n@code_ir transformed_f(1)","category":"page"},{"location":"lecture_09/lab/#Marking-Argument-Checks","page":"Lab","title":"Marking Argument Checks","text":"","category":"section"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"As a first step we will implement a macro that marks checks which we might want to remove later by surrounding it with :meta expressions. This will make it easy to detect which part of the code can be removed. A :meta expression can be created like this","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Expr(:meta, :mark_begin)\nExpr(:meta, :mark_end)","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"and they will not be evaluated but remain in your IR. To surround an expression with two meta expressions you can use a :block expression:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"ex = :(x+x)\nExpr(:block, :(print(x)), ex, :(print(x)))","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Define a macro @mark that takes an expression and surrounds it with two meta expressions marking the beginning and end of a check. Hints","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Defining a function _mark(ex::Expr) which manipulates your expressions can help a lot with debugging your macro.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"function _mark(ex::Expr)\n return Expr(\n :block,\n Expr(:meta, :mark_begin),\n esc(ex),\n Expr(:meta, :mark_end),\n )\nend\n\nmacro mark(ex)\n _mark(ex)\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"If you have defined a _mark function you can test that it works like this","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"_mark(:(println(x)))","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"The complete macro should work like below","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"function f(x::Number)\n @mark @show x\n x\nend;\n@code_ir f(2)\nf(2)","category":"page"},{"location":"lecture_09/lab/#Removing-Argument-Checks","page":"Lab","title":"Removing Argument Checks","text":"","category":"section"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Now comes tricky part for which we need IRTools.jl. We want to remove all lines that are between our two meta blocks. You can delete the line that corresponds to a certain variable with the delete! and the var functions. E.g. deleting the line that defines variable %4 works like this:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"using IRTools: delete!, var\n\nir = @code_ir f(2)\ndelete!(ir, var(4))","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Write a function skip(ir::IR) which deletes all lines between the meta expression :mark_begin and :mark_end.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Hints You can check whether a statement is one of our meta expressions like this:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"ismarkbegin(e::Expr) = Meta.isexpr(e,:meta) && e.args[1]===:mark_begin\nismarkbegin(Expr(:meta,:mark_begin))","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"ismarkend(e::Expr) = Meta.isexpr(e,:meta) && e.args[1]===:mark_end\n\nfunction skip(ir)\n delete_line = false\n for (x,st) in ir\n isbegin = ismarkbegin(st.expr)\n isend = ismarkend(st.expr)\n\n if isbegin\n delete_line = true\n end\n \n if delete_line\n delete!(ir,x)\n end\n\n if isend\n delete_line = false\n end\n end\n ir\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Your function should transform the IR of f like below.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"ir = @code_ir f(2)\nir = skip(ir)\nusing IRTools: func\nfunc(ir)(nothing, 2) # no output from @show!","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"However, if we have a slightly more complicated IR like below this version of our function will fail. It actually fails so badly that running func(ir)(nothing,2) after skip will cause the build of this page to crash, so we cannot show you the output here ;).","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"function g(x)\n @mark iseven(x) && println(\"even\")\n x\nend\n\nir = @code_ir g(2)\nir = skip(ir)","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"The crash is due to %4 not existing anymore. We can fix this by emptying the block in which we found the :mark_begin expression and branching to the block that contains :mark_end (unless they are in the same block already). If some (branching) code in between remained, it should then be removed by the compiler because it is never reached.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Use the functions IRTools.block, IRTools.branches, IRTools.empty!, and IRTools.branch! to modify skip such that it also empties the :mark_begin block, and adds a branch to the :mark_end block (unless they are the same block).","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Hints","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"block gets you the block of IR in which a given variable is if you call e.g. block(ir,var(4)).\nempty! removes all statements in a block.\nbranches returns all branches of a block.\nbranch!(a,b) creates a branch from the end of block a to the beginning block b","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"using IRTools: block, branch!, empty!, branches\nfunction skip(ir)\n delete_line = false\n orig = nothing\n for (x,st) in ir\n isbegin = ismarkbegin(st.expr)\n isend = ismarkend(st.expr)\n\n if isbegin\n delete_line = true\n end\n\n # this part is new\n if isbegin\n orig = block(ir,x)\n elseif isend\n dest = block(ir,x)\n if orig != dest\n empty!(branches(orig))\n branch!(orig,dest)\n end\n end\n \n if delete_line\n delete!(ir,x)\n end\n\n if isend\n delete_line = false\n end\n end\n ir\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"The result should construct valid IR for our g function.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"g(2)\nir = @code_ir g(2)\nir = skip(ir)\nfunc(ir)(nothing,2)","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"And it should not break when applying it to f.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"f(2)\nir = @code_ir f(2)\nir = skip(ir)\nfunc(ir)(nothing,2)","category":"page"},{"location":"lecture_09/lab/#Recursively-Removing-Argument-Checks","page":"Lab","title":"Recursively Removing Argument Checks","text":"","category":"section"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"The last step to finalize the skip function is to make it work recursively. In the current version we can handle functions that contain @mark statements, but we are not going any deeper than that. Nested functions will not be touched:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"foo(x) = bar(baz(x))\n\nfunction bar(x)\n @mark iseven(x) && println(\"The input is even.\")\n x\nend\n\nfunction baz(x)\n @mark x<0 && println(\"The input is negative.\")\n x\nend\n\nnothing # hide","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"ir = @code_ir foo(-2)\nir = skip(ir)\nfunc(ir)(nothing,-2)","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"For recursion we will use the macro IRTools.@dynamo which will make recursion of our skip function a lot easier. Additionally, it will save us from all the func(ir)(nothing, args...) statements. To use @dynamo we have to slightly modify how we call skip:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"@dynamo function skip(args...)\n ir = IR(args...)\n \n # same code as before that modifies `ir`\n # ...\n\n return ir\nend\n\n# now we can call `skip` like this\nskip(f,2)","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Now we can easily use skip in recursion, because we can just pass the arguments of an expression like this:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"using IRTools: xcall\n\nfor (x,st) in ir\n isexpr(st.expr,:call) || continue\n ir[x] = xcall(skip, st.expr.args...)\nend","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"The function xcall will create an expression that calls skip with the given arguments and returns Expr(:call, skip, args...). Note that you can modify expressions of a given variable in the IR via setindex!.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Modify skip such that it uses @dynamo and apply it recursively to all :call expressions that you ecounter while looping over the given IR. This will dive all the way down to Core.Builtins and Core.IntrinsicFunctions which you cannot maniuplate anymore (because they are written in C). You have to end the recursion at these places which can be done via multiple dispatch of skip on Builtins and IntrinsicFunctions.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Once you are done with this you can also define a macro such that you can conveniently call @skip with an expression:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"skip(f,2)\n@skip f(2)","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"using IRTools: @dynamo, xcall, IR\n\n# this is where we want to stop recursion\nskip(f::Core.IntrinsicFunction, args...) = f(args...)\nskip(f::Core.Builtin, args...) = f(args...)\n\n@dynamo function skip(args...)\n ir = IR(args...)\n delete_line = false\n orig = nothing\n for (x,st) in ir\n isbegin = ismarkbegin(st.expr)\n isend = ismarkend(st.expr)\n\n if isbegin\n delete_line = true\n end\n\n if isbegin\n orig = block(ir,x)\n elseif isend\n dest = block(ir,x)\n if orig != dest\n empty!(branches(orig))\n branch!(orig,dest)\n end\n end\n \n if delete_line\n delete!(ir,x)\n end\n\n if isend\n delete_line = false\n end\n\n # this part is new\n if haskey(ir,x) && Meta.isexpr(st.expr,:call)\n ir[x] = xcall(skip, st.expr.args...)\n end\n end\n return ir\nend\n\nmacro skip(ex)\n ex.head == :call || error(\"Input expression has to be a `:call`.\")\n return xcall(skip, ex.args...)\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"@code_ir foo(2)\n@code_ir skip(foo,2)\nfoo(-2)\nskip(foo,-2)\n@skip foo(-2)","category":"page"},{"location":"lecture_09/lab/#References","page":"Lab","title":"References","text":"","category":"section"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Static matrices with @generated functions blog post\nOptionalArgChecks.jl\nIRTools Dynamo","category":"page"},{"location":"lecture_02/hw/#Homework-2:-Predator-Prey-Agents","page":"Homework","title":"Homework 2: Predator-Prey Agents","text":"","category":"section"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"In this lab you will continue working on your agent simulation. If you did not manage to finish the homework, do not worry, you can use this script which contains all the functionality we developed in the lab.","category":"page"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"projdir = dirname(Base.active_project())\ninclude(joinpath(projdir,\"src\",\"lecture_02\",\"Lab02Ecosystem.jl\"))","category":"page"},{"location":"lecture_02/hw/#How-to-submit?","page":"Homework","title":"How to submit?","text":"","category":"section"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"Put all your code (including your or the provided solution of lab 2) in a script named hw.jl. Zip only this file (not its parent folder) and upload it to BRUTE. Your file cannot contain any package dependencies. For example, having a using Plots in your code will cause the automatic evaluation to fail.","category":"page"},{"location":"lecture_02/hw/#Counting-Agents","page":"Homework","title":"Counting Agents","text":"","category":"section"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"To monitor the different populations in our world we need a function that counts each type of agent. For Animals we simply have to count how many of each type are currently in our World. In the case of Plants we will use the fraction of size(plant)/max_size(plant) as a measurement quantity.","category":"page"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"
\n
Compulsory Homework (2 points)
\n
","category":"page"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"Implement a function agent_count that can be called on a single Agent and returns a number between (01) (i.e. always 1 for animals; and size(plant)/max_size(plant) for plants).\nAdd a method for a vector of agents Vector{<:Agent} which sums all agent counts.\nAdd a method for a World which returns a dictionary that contains pairs of Symbols and the agent count like below:","category":"page"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"agent_count(p::Plant) = p.size / p.max_size\nagent_count(::Animal) = 1\nagent_count(as::Vector{<:Agent}) = sum(agent_count,as)\n\nfunction agent_count(w::World)\n function op(d::Dict,a::A) where A<:Agent\n n = nameof(A)\n if n in keys(d)\n d[n] += agent_count(a)\n else\n d[n] = agent_count(a)\n end\n return d\n end\n foldl(op, w.agents |> values |> collect, init=Dict{Symbol,Real}())\nend","category":"page"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"grass1 = Grass(1,5,5);\nagent_count(grass1)\n\ngrass2 = Grass(2,1,5);\nagent_count([grass1,grass2]) # one grass is fully grown; the other only 20% => 1.2\n\nsheep = Sheep(3,10.0,5.0,1.0,1.0);\nwolf = Wolf(4,20.0,10.0,1.0,1.0);\nworld = World([grass1, grass2, sheep, wolf]);\nagent_count(world)","category":"page"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"Hint: You can get the name of a type by using the nameof function:","category":"page"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"nameof(Grass)","category":"page"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"Use as much dispatch as you can! ;)","category":"page"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"
","category":"page"},{"location":"lecture_01/hw/#Homework-1:-Extending-polynomial-the-other-way","page":"Homework","title":"Homework 1: Extending polynomial the other way","text":"","category":"section"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"
\n
Homework (2 points)
\n
","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"Extend the original polynomial function to the case where x is a square matrix. Create a function called circlemat, that returns nxn matrix A(n) with the following elements","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"leftA(n)right_ij = \nbegincases\n 1 textif (i = j-1 land j 1) lor (i = n land j=1) \n 1 textif (i = j+1 land j n) lor (i = 1 land j=n) \n 0 text otherwise\nendcases","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"and evaluate the polynomial","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"f(A) = I + A + A^2 + A^3","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":", at point A = A(10).","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"HINTS for matrix definition: You can try one of these options:","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"create matrix with all zeros with zeros(n,n), use two nested for loops going in ranges 1:n and if condition with logical or ||, and && \nemploy array comprehension with nested loops [expression for i in 1:n, j in 1:n] and ternary operator condition ? true branch : false","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"HINTS for polynomial extension:","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"extend the original example (one with for-loop) to initialize the accumulator variable with matrix of proper size (use size function to get the dimension), using argument typing for x is preferred to distinguish individual implementations <: AbstractMatrix","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"or","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"test later defined polynomial methods, that may work out of the box","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"
\n","category":"page"},{"location":"lecture_01/hw/#How-to-submit?","page":"Homework","title":"How to submit?","text":"","category":"section"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"Put all the code for the exercise above in a file called hw.jl and upload it to BRUTE. If you have any questions, write an email to one of the lab instructors of the course.","category":"page"},{"location":"lecture_01/hw/#Voluntary","page":"Homework","title":"Voluntary","text":"","category":"section"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"
\n
Exercise (voluntary)
\n
","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"Install GraphRecipes and Plots packages into the environment defined during the lecture and figure out, how to plot the graph defined by adjacency matrix A from the homework.","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"HINTS:","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"There is help command inside the the pkg mod of the REPL. Type ? add to find out how to install a package. Note that both pkgs are registered.\nFollow a guide in the Plots pkg's documentation, which is accessible through docs icon on top of the README in the GitHub repository. Direct link.","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"Activate the environment in pkg mode, if it is not currently active.","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"pkg> activate .","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"Installing pkgs is achieved using the add command. Running ] ? add returns a short piece of documentation for this command:","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"pkg> ? add\n[...]\n Examples\n\n pkg> add Example # most commonly used for registered pkgs (installs usually the latest release)\n pkg> add Example@0.5 # install with some specific version (realized through git tags)\n pkg> add Example#master # install from master branch directly\n pkg> add Example#c37b675 # install from specific git commit\n pkg> add https://github.com/JuliaLang/Example.jl#master # install from specific remote repository (when pkg is not registered)\n pkg> add git@github.com:JuliaLang/Example.jl.git # same as above but using the ssh protocol\n pkg> add Example=7876af07-990d-54b4-ab0e-23690620f79a # when there are multiple pkgs with the same name","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"As the both Plots and GraphRecipes are registered and we don't have any version requirements, we will use the first option.","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"pkg> add Plots\npkg> add GraphRecipes","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"This process downloads the pkgs and triggers some build steps, if for example some binary dependencies are needed. The process duration depends on the \"freshness\" of Julia installation and the size of each pkg. With Plots being quite dependency heavy, expect few minutes. After the installation is complete we can check the updated environment with the status command.","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"pkg> status","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"The plotting itself as easy as calling the graphplot function on our adjacency matrix.","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"using GraphRecipes, Plots\nA = [ 0 1 0 0 0 0 0 0 0 1; 1 0 1 0 0 0 0 0 0 0; 0 1 0 1 0 0 0 0 0 0; 0 0 1 0 1 0 0 0 0 0; 0 0 0 1 0 1 0 0 0 0; 0 0 0 0 1 0 1 0 0 0; 0 0 0 0 0 1 0 1 0 0; 0 0 0 0 0 0 1 0 1 0; 0 0 0 0 0 0 0 1 0 1; 1 0 0 0 0 0 0 0 1 0]# hide\ngraphplot(A)","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"graphplot(A) #hide","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"

","category":"page"},{"location":"lecture_04/lecture/#pkg_lecture","page":"Lecture","title":"Package development","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Organization of the code is more important with the increasing size of the project and the number of contributors and users. Moreover, it will become essential when different codebases are expected to be combined and reused. ","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Julia was designed from the beginning to encourage code reuse across different codebases as possible\nJulia ecosystem lives on a namespace. From then, it builds projects and environments.","category":"page"},{"location":"lecture_04/lecture/#Namespaces-and-modules","page":"Lecture","title":"Namespaces and modules","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Namespace logically separate fragments of source code so that they can be developed independently without affecting each other. If I define a function in one namespace, I will still be able to define another function in a different namespace even though both functions have the same name.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"prevents confusion when common words are used in different meaning:\nToo general name of functions \"create\", \"extend\", \"loss\", \nor data \"X\", \"y\" (especially in mathematics, think of π)\nmay not be an issue if used with different types\nModules is Julia syntax for a namespace","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Example:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"module MySpace\nfunction test1()\n println(\"test1\")\nend\nfunction test2()\n println(\"test2\")\nend\nexport test1\n#include(\"filename.jl\")\nend","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Function include copies content of the file to this location (will be part of the module).","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Creates functions:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"MySpace.test1\nMySpace.test2","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"For easier manipulation, these functions can be \"exported\" to be exposed to the outer world (another namespace).","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Keyword: using exposes the exported functions and structs:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"using .MySpace","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"The dot means that the module was defined in this scope.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Keyword: import imports function with availability to redefine it.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Combinations:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"usecase results\nusing MySpace MySpace.test1\n MySpace.test2\n test1\nusing MySpace: test1 test1\nimport MySpace MySpace.test1*\n MySpace.test2*\nimport MySpace: test1 test1*\nimport MySpace: test2 test2*","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"symbol \"*\" denotes functions that can be redefined","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":" using .MySpace: test1\n test1()=println(\"new test\")\n import .MySpace: test1\n test1()=println(\"new test\")","category":"page"},{"location":"lecture_04/lecture/#Conflicts:","page":"Lecture","title":"Conflicts:","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"When importing/using functions with name that is already imported/used from another module:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"the imported functions/structs are invalidated. \nboth functions has to be acessed by their full names.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Resolution:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"It may be easier to cherry pick only the functions we need (rather than importing all via using)\nrename some function using keyword as\nimport MySpace2: test1 as t1","category":"page"},{"location":"lecture_04/lecture/#Submodules","page":"Lecture","title":"Submodules","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Modules can be used or included within other modules:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"module A\n a=1;\nend\nmodule B\n module C\n c = 2\n end\n b = C.c # you can read from C (by reference)\n using ..A: a\n # a= b # but not write to A\nend;","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"REPL of Julia is a module called \"Main\". ","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"modules are not copied, but referenced, i.e. B.b===B.C.c\nincluding one module twice (from different packages) is not a problem\nJulia 1.9 has the ability to change the contextual module in the REPL: REPL.activate(TestPackage)","category":"page"},{"location":"lecture_04/lecture/#Revise.jl","page":"Lecture","title":"Revise.jl","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"The fact that Julia can redefine a function in a Module by importing it is used by package Revise.jl to synchronize REPL with a module or file.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"So far, we have worked in REPL. If you have a file that is loaded and you want to modify it, you would need to either:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"reload the whole file, or\ncopy the changes to REPL","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Revise.jl does the latter automatically.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Example demo:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"using Revise.jl\nincludet(\"example.jl\")","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Works with: ","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"any package loaded with import or using, \nscript loaded with includet, \nBase julia itself (with Revise.track(Base))\nstandard libraries (with, e.g., using Unicode; Revise.track(Unicode))","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Does not work with variables!","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"How it works: monitors source code for changes and then does:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"for def in setdiff(oldexprs, newexprs)\n # `def` is an expression that defines a method.\n # It was in `oldexprs`, but is no longer present in `newexprs`--delete the method.\n delete_methods_corresponding_to_defexpr(mod, def)\nend\nfor def in setdiff(newexprs, oldexprs)\n # `def` is an expression for a new or modified method. Instantiate it.\n Core.eval(mod, def)\nend","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"See Modern Julia Workflows for more hints","category":"page"},{"location":"lecture_04/lecture/#Namespaces-and-scoping","page":"Lecture","title":"Namespaces & scoping","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Every module introduces a new global scope. ","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Global scope\nNo variable or function is expected to exist outside of it\nEvery module is equal to a global scope (no single \"global\" exists)\nThe REPL has a global module called Main\nLocal scope\nVariables in Julia do not need to be explicitly declared, they are created by assignments: x=1. \nIn local scope, the compiler checks if variable x does not exist outside. We have seen:\nx=1\nf(y)=x+y\nThe rules for local scope determine how to treat assignment of x. If local x exists, it is used, if it does not:\nin hard scope: new local x is created\nin soft scope: checks if x exists outside (global)\nif not: new local x is created\nif yes: the split is REPL/non-interactive:\nREPL: global x is used (convenience, as of 1.6)\nnon-interactive: local x is created","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"keyword local and global can be used to specify which variable to use","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"From documentation:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Construct Scope type Allowed within\nmodule, baremodule global global\nstruct local (soft) global\nfor, while, try local (soft) global, local\nmacro local (hard) global\nfunctions, do blocks, let blocks, comprehensions, generators local (hard) global, local","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Question:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"x=1\nf()= x=3\nf()\n@show x;","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"x = 1\nfor _ = 1:1\n x=3\nend\n@show x;","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Notice that if does not introduce new scope","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"if true \n branch_taken = true \nelse\n branch_not_taken = true \nend","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"tip: do-block\nLet's assume a function, which takes as a first argument a function g(f::Function, args...) = println(\"f called on $(args) evaluates to \", f(args...))We can use g as g(+, 1, 2), or with a lambda function g(x -> x^2, 2). But sometimes, it might be useful to the lambda function to span multiple lines. This can be achieved by a do block as g(1,2,3) do a,b,c\n a*b + c\nend","category":"page"},{"location":"lecture_04/lecture/#Packages","page":"Lecture","title":"Packages","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Package is a source tree with a standard layout. It provides a module and thus can be loaded with include or using.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Minimimal package:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"PackageName/\n├── src/\n│ └── PackageName.jl\n├── Project.toml","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Contains:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Project.toml file describing basic properties:\nName, does not have to be unique (federated package sources)\nUUID, has to be unique (generated automatically)\noptionally [deps], [targets],...\nfile src/PackageName.jl that defines module PackageName which is executed when loaded.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Many other optional directories:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"directory tests/, (almost mandatory)\ndirectory docs/ (common)\ndirectory scripts/, examples/,... (optional)","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"tip: Type-Piracy\n\"Type piracy\" refers to the practice of extending or redefining methods in Base or other packages on types that you have not defined. In extreme cases, you can crash Julia (e.g. if your method extension or redefinition causes invalid input to be passed to a ccall). Type piracy can complicate reasoning about code, and may introduce incompatibilities that are hard to predict and diagnose.module A\nimport Base.*\n*(x::Symbol, y::Symbol) = Symbol(x,y)\nend","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"The package typically loads other modules that form package dependencies.","category":"page"},{"location":"lecture_04/lecture/#Project-environments","page":"Lecture","title":"Project environments","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Is a package that does not contain Name and UUID in Project.toml. It's used when you don't need to create a package for your work. It's created by activate some/path in REPL package mode. ","category":"page"},{"location":"lecture_04/lecture/#Project-Manifest","page":"Lecture","title":"Project Manifest","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Both package and environment can contain an additional file Manifest.toml. This file tracks full dependency tree of a project including versions of the packages on which it depends.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"for example:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"# This file is machine-generated - editing it directly is not advised\n\n[[AbstractFFTs]]\ndeps = [\"LinearAlgebra\"]\ngit-tree-sha1 = \"485ee0867925449198280d4af84bdb46a2a404d0\"\nuuid = \"621f4979-c628-5d54-868e-fcf4e3e8185c\"\nversion = \"1.0.1\"\n\n[[AbstractTrees]]\ngit-tree-sha1 = \"03e0550477d86222521d254b741d470ba17ea0b5\"\nuuid = \"1520ce14-60c1-5f80-bbc7-55ef81b5835c\"\nversion = \"0.3.4\"","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Content of files Project.toml and Manifest.toml are maintained by PackageManager.","category":"page"},{"location":"lecture_04/lecture/#Package-manager","page":"Lecture","title":"Package manager","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Handles both packages and projects:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"creating a project ]generate PkgName\nadding an existing project add PkgName or add https://github.com/JuliaLang/Example.jl\nNames are resolved by Registrators (public or private).\nremoving ]rm PkgName\nupdating ]update\ndeveloping ]dev http://... \nadd treats packages as being finished, version handling pkg manager. Precompiles!\ndev leaves all operations on the package to the user (git versioning, etc.). Always read content of files","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"By default these operations are related to environment .julia/environments/v1.9","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"E.g. running and updating will update packages in Manifest.toml in this directory. What if the update breaks functionality of some project package that uses special features?","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"There can and should be more than one environment!","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Project environments are based on files with installed packages.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"switching by ]activate Path - creates Project.toml if not existing\nfrom that moment, all package modifications will be relevant only to this project!\nwhen switching to a new project ]instantiate will prepare (download and precompile) the environment\ncreates Manifest.toml = list of all exact versions of all packages \nwhich Packages are visible is determined by LOAD_PATH\ntypically contaings default libraries and default environment\nit is different for REPL and Pkg.tests ! No default env. in tests. ","category":"page"},{"location":"lecture_04/lecture/#Package-hygiene-workflow","page":"Lecture","title":"Package hygiene - workflow","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"theorem: Potential danger\nPackage dependencies may not be compatible: package A requires C@<0.2\npackage B requires C@>0.3\nwhat should happen when ]add A and add B?","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"keep your \"@v#.#\" as clean as possible (recommended are only debugging/profiling packages)\nuse packages as much as you can, even for short work with scripts ]activate .\nadding a package existing elsewhere is cheap (global cache)\nif do you not wish to store any files just test random tricks of a cool package: ]activate --temp","category":"page"},{"location":"lecture_04/lecture/#Package-development-with-Revise","page":"Lecture","title":"Package development with Revise","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Developing a package with interactive test/development:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Create a package/module at one directory MainPackage\nCreate a script at another directory MainScript, and activate it ]activate .\ndev MainPackage in the MainScript environment\nRevise.jl will watch the MainPackage so it is always up to date\nin dev mode you have full control over commits etc.","category":"page"},{"location":"lecture_04/lecture/#Package-Extensions","page":"Lecture","title":"Package Extensions","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Some functionality of a package that depends on external packages may not be always needed. A typical example is plotting and visualization that may reguire heavy visualization packages. These are completely unnecessary e.g. in distributed server number crunching.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"The extension is a new module depending on: i) the base package, and ii) the conditioning package. It will not be compiled if the conditioning package is not loaded. Once the optional package is loaded, the extension will be automatically compiled and loaded.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"New feature since Julia 1.9:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"new directory in project tree: ext/ the extending module is stored here\nnew section in Project.toml called [extensions] listing extension names and their conditioning packages","category":"page"},{"location":"lecture_04/lecture/#Unit-testing,-/test","page":"Lecture","title":"Unit testing, /test","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Without explicit keywords for checking constructs (think missing functions in interfaces), the good quality of the code is guaranteed by detailed unit testing.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"each package should have directory /test\nfile /test/runtest.jl is run by the command ]test of the package manager\nthis file typically contains include of other tests\nno formal structure of tests is prescribed\ntest files are just ordinary julia scripts\nuser is free to choose what to test and how (freedom x formal rules)\ntesting functionality is supported by macros @test and @teststet\n@testset \"trigonometric identities\" begin\n θ = 2/3*π\n @test sin(-θ) ≈ -sin(θ)\n @test cos(-θ) ≈ cos(θ)\n @test sin(2θ) ≈ 2*sin(θ)*cos(θ)\n @test cos(2θ) ≈ cos(θ)^2 - sin(θ)^2\nend;","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Testset is a collection of tests that will be run and summarized in a common report.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Testsets can be nested: testsets in testsets\ntests can be in loops or functions\nfor i=1:10\n @test a[i]>0\nend\nUseful macro ≈ checks for equality with given tolerance\na=5+1e-8\n@test a≈5\n@test a≈5 atol=1e-10","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"@testset resets RNG to Random.GLOBAL_SEED before and after the test for repeatability \nThe same results of RNG are not guaranteed between Julia versions!\nTest coverage: package Coverage.jl\nCan be run automatically by continuous integration, e.g. GitHub actions\nintegration in VSCode test via package TestItems.jl ","category":"page"},{"location":"lecture_04/lecture/#Documentation-and-Style,-/docs","page":"Lecture","title":"Documentation & Style, /docs","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"A well written package is reusable if it is well documented. ","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"The simpliest kind of documentation is the docstring:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"\"Auxiliary function for printing a hello\"\nhello()=println(\"hello\")\n\n\"\"\"\nMore complex function that adds π to input:\n- x is the input argument (itemize)\n\nCan be written in latex: ``x \\leftarrow x + \\pi``\n\"\"\"\naddπ(x) = x+π","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Yieds:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"tip: Renders as\nMore complex function that adds π to input:x is the input argument (itemize)Can be written in latex: x leftarrow x + pi","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Structure of the document","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"PackageName/\n├── src/\n│ └── SourceFile.jl\n├── docs/\n│ ├── build/\n│ ├── src/\n│ └── make.jl\n...","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Where the line-by-line documentation is in the source files.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"/docs/src folder can contain more detailed information: introductory pages, howtos, tutorials, examples\nrunning make.jl controls which pages are generated in what form (html or latex) documentation in the /build directory\nautomated with GitHub actions","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Documentation is generated by the julia code.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"code in documentation can be evaluated","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"x=3\n@show x","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"documentation can be added by code:\nstruct MyType\n value::String\nend\n\nDocs.getdoc(t::MyType) = \"Documentation for MyType with value $(t.value)\"\n\nx = MyType(\"x\")\ny = MyType(\"y\")\nSee ?x and ?y. \nIt uses the same very standard building blocks: multiple dispatch.","category":"page"},{"location":"lecture_04/lecture/#Precompilation","page":"Lecture","title":"Precompilation","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"By default, every package is precompiled when loading and stored in compiled form in a cache.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"If it defines methods that extend previously defined (e.g. from Base), it may affect already loaded packages which need to be recompiled as well. May take time.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Julia has a tracking mechanism that stores information about the whole graph of dependencies. ","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Faster code can be achieved by the precompile directive:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"module FSum\n\nfsum(x) = x\nfsum(x,p...) = x+fsum(p...)\n\nprecompile(fsum,(Float64,Float64,Float64))\nend","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Can be investigated using MethodAnalysis.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"using MethodAnalysis\nmi =methodinstances(fsum)","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Useful packages:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"PackageCompiler.jl has three main purposes:\nCreating custom sysimages for reduced latency when working locally with packages that has a high startup time.\nCreating \"apps\" which are a bundle of files including an executable that can be sent and run on other machines without Julia being installed on that machine.\nCreating a relocatable C library bundle form of Julia code.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"AutoSysimages.jl allows easy generation of precompiles images - reduces package loading","category":"page"},{"location":"lecture_04/lecture/#Additional-material","page":"Lecture","title":"Additional material","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Modern Julia Workflows","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"These notes are from poking around Core.Compiler to see, how they are different from working just with CodeInfo and IRTools.jl. Notes are mainly around IRCode. Why there is a Core.Compiler.IRCode when there was Core.CodeInfo? Seems to be historical reasons. At the beginning, Julia did not have any intermediate representation and code directly emitted LLVM. Then, it has received an CodeInfo as in intermediate representation. IRCode seems like an evolution of CodeInfo. Core.Compiler works mostly with IRCode, but the IRCode can be converted to the CodeInfo and the other way around. IRCode seems to be designed more for implementation of various optimisation phases. Personal experience tells me it is much nicer to work with even on the low level. ","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"Throughout the explanation, we assume that Core.Compiler was imported as CC to decrease the typing load.","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"Let's play with a simple silly function ","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"\nfunction foo(x,y) \n z = x * y \n z + sin(x)\nend","category":"page"},{"location":"lecture_09/ircode/#IRCode","page":"-","title":"IRCode","text":"","category":"section"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"We can obtain CC.IRCode","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"import Core.Compiler as CC\n(ir, rt) = only(Base.code_ircode(foo, (Float64, Float64), optimize_until = \"compact 1\"))","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"which returns Core.Compiler.IRCode in ir and return-type Float64 in rt. The output might look like ","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"julia> (ir, rt) = only(Base.code_ircode(foo, (Float64, Float64), optimize_until = \"compact 1\"))\n 1─ %1 = (_2 * _3)::Float64 \n │ %2 = Main.sin(_2)::Float64 \n │ %3 = (%1 + %2)::Float64 \n └── return %3 \n => Float64\n","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"Options of optimize_until are compact 1, compact 2, nothing. I do not see a difference between compact 2 and compact 2.","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"The IRCode structure is defined as","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"struct IRCode\n stmts::InstructionStream\n argtypes::Vector{Any}\n sptypes::Vector{VarState}\n linetable::Vector{LineInfoNode}\n cfg::CFG\n new_nodes::NewNodeStream\n meta::Vector{Expr}\nend","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"where","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"stmts is a stream of instruction (more in this below)\nargtypes holds types of arguments of the function whose IRCode we have obtained\nsptypes is a vector of VarState. It seems to be related to parameters of types\nlinetable is a table of unique lines in the source code from which statements \ncfg holds control flow graph, which contains building blocks and jumps between them\nnew_nodes is an infrastructure that can be used to insert new instructions to the existing IRCode . The idea behind is that since insertion requires a renumbering all statements, they are put in a separate queue. They are put to correct position with a correct SSANumber by calling compact!.\nmeta is something.","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"Before going further, let's take a look on InstructionStream defined as ","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"struct InstructionStream\n inst::Vector{Any}\n type::Vector{Any}\n info::Vector{CallInfo}\n line::Vector{Int32}\n flag::Vector{UInt8}\nend","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"where ","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"inst is a vector of instructions, stored as Expressions. The allowed fields in head are described here\ntype is the type of the value returned by the corresponding statement\nCallInfo is ???some info???\nline is an index into IRCode.linetable identifying from which line in source code the statement comes from\nflag are some flags providing additional information about the statement.","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"- `0x01 << 0` = statement is marked as `@inbounds`\n- `0x01 << 1` = statement is marked as `@inline`\n- `0x01 << 2` = statement is marked as `@noinline`\n- `0x01 << 3` = statement is within a block that leads to `throw` call\n- `0x01` << 4 = statement may be removed if its result is unused, in particular it is thus be both pure and effect free\n- `0x01 << 5-6 = `\n- `0x01 << 7 = ` has out-of-band info","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"For the above foo function, the InstructionStream looks like","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"julia> DataFrame(flag = ir.stmts.flag, info = ir.stmts.info, inst = ir.stmts.inst, line = ir.stmts.line, type = ir.stmts.type)\n4×5 DataFrame\n Row │ flag info inst line type\n │ UInt8 CallInfo Any Int32 Any\n─────┼────────────────────────────────────────────────────────────────────────\n 1 │ 112 MethodMatchInfo(MethodLookupResu… _2 * _3 1 Float64\n 2 │ 80 MethodMatchInfo(MethodLookupResu… Main.sin(_2) 2 Float64\n 3 │ 112 MethodMatchInfo(MethodLookupResu… %1 + %2 2 Float64\n 4 │ 0 NoCallInfo() return %3 2 Any","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"We can index into the statements as ir.stmts[1], which provides a \"view\" into the vector. To obtain the first instruction, we can do ir.stmts[1][:inst].","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"The IRCode is typed, but the fields can contain Any. It is up to the user to provide corrrect types of the output and there is no helper functions to perform typing. A workaround is shown in the Petite Diffractor project. Julia's sections of the manual https://docs.julialang.org/en/v1/devdocs/ssair/ and seems incredibly useful. The IR form they talk about seems to be Core.Compiler.IRCode. ","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"It seems to be that it is possible to insert IR instructions into the it structure by queuing that to the field stmts and then call compact!, which would perform the heavy machinery of relabeling everything.","category":"page"},{"location":"lecture_09/ircode/#Example-of-modifying-the-function-through-IRCode","page":"-","title":"Example of modifying the function through IRCode","text":"","category":"section"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"Below is an MWE that tries to modify the IRCode of a function and execute it. The goal is to change the function foo to fooled.","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"import Core.Compiler as CC\nusing Core: SSAValue, GlobalRef, ReturnNode\n\nfunction foo(x,y) \n z = x * y \n z + sin(x)\nend\n\nfunction fooled(x,y) \n z = x * y \n z + sin(x) + cos(y)\nend\n\n(ir, rt) = only(Base.code_ircode(foo, (Float64, Float64), optimize_until = \"compact 1\"));\nnr = CC.insert_node!(ir, 2, CC.NewInstruction(Expr(:call, Core.GlobalRef(Main, :cos), Core.Argument(3)), Float64))\nnr2 = CC.insert_node!(ir, 4, CC.NewInstruction(Expr(:call, GlobalRef(Main, :+), SSAValue(3), nr), Float64))\nCC.setindex!(ir.stmts[4], ReturnNode(nr2), :inst)\nir = CC.compact!(ir)\nirfooled = Core.OpaqueClosure(ir)\nirfooled(1.0, 2.0) == fooled(1.0, 2.0)","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"So what we did?","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"(ir, rt) = only(Base.code_ircode(foo, (Float64, Float64), optimize_until = \"compact 1\")) obtain the IRCode of the function foo when called with both arguments being Float64. rt contains the return type of the \nA new instruction cos is inserted to the ir by Core.Compiler.insert_node!, which takes as an argument an IRCode, position (2 in our case), and new instruction. The new instruction is created by NewInstruction accepting as an input expression Expr and a return type. Here, we force it to be Float64, but ideally it should be inferred. (This would be the next stage). Or, may-be, we can run it through type inference? . The new instruction is added to the ir.new_nodes instruction stream and obtain a new SSAValue returned in nr, which can be then used further.\nWe add one more instruction + that uses output of the instruction we add in step 2, nr and SSAValue from statement 3 of the original IR (at this moment, the IR is still numbered with respect to the old IR, the renumbering will happen later.) The output of this second instruction is returned in nr2.\nThen, we rewrite the return statement to return nr2 instead of SSAValue(3).\nir = CC.compact!(ir) is superimportant since it moves the newly added statements from ir.new_stmts to ir.stmts and importantly renumbers SSAValues. Even though the function is mutating, the mutation here is meant that the argument is changed, but the new correct IRCode is returned and therefore has to be reassigned.\nThe function is created through OpaqueClosure.\nThe last line certifies that the function do what it should do.","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"There is no infrastructure to make the above manipulation transparent, like is the case of @generated function and codeinfo. It is possible to hook through generated function by converting the IRCode to untyped CodeInfo, in which case you do not have to bother with typing.","category":"page"},{"location":"lecture_09/ircode/#How-to-obtain-code-info-the-proper-way?","page":"-","title":"How to obtain code info the proper way?","text":"","category":"section"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"This is the way code info is obtained in the diffractor.","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"mthds = Base._methods_by_ftype(sig, -1, world)\nmatch = only(mthds)\n\nmi = Core.Compiler.specialize_method(match)\nci = Core.Compiler.retrieve_code_info(mi, world)","category":"page"},{"location":"lecture_09/ircode/#CodeInfo","page":"-","title":"CodeInfo","text":"","category":"section"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"IRTools.jl are great for modifying CodeInfo. I have found two tools for modifying IRCode and I wonder if they have been abandoned because they were both dead ends or because of lack of human labor. I am also aware of Also, this is quite cool play with IRStuff.","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"Resources","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"https://vchuravy.dev/talks/licm/\nCompilerPluginTools \nCodeInfoTools.jl.\nTKF's CodeInfo.jl is nice for visualization of the IRCode\nDiffractor is an awesome source of howto. For example function my_insert_node! in src/stage1/hacks.jl\nhttps://nbviewer.org/gist/tkf/d4734be24d2694a3afd669f8f50e6b0f/00_notebook.ipynb\nhttps://github.com/JuliaCompilerPlugins/Mixtape.jl","category":"page"},{"location":"lecture_07/lab/#macro_lab","page":"Lab","title":"Lab 07: Macros","text":"","category":"section"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"A little reminder from the lecture, a macro in its essence is a function, which ","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"takes as an input an expression (parsed input)\nmodifies the expressions in arguments\ninserts the modified expression at the same place as the one that is parsed.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"In this lab we are going to use what we have learned about manipulation of expressions and explore avenues of where macros can be useful","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"convenience (@repeat, @show)\nperformance critical code generation (@poly)\nalleviate tedious code generation (@species, @eats)\njust as a syntactic sugar (@ecosystem)","category":"page"},{"location":"lecture_07/lab/#Show-macro","page":"Lab","title":"Show macro","text":"","category":"section"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Let's start with dissecting \"simple\" @show macro, which allows us to demonstrate advanced concepts of macros and expression manipulation.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"x = 1\n@show x + 1\nlet y = x + 1 # creates a temporary local variable\n println(\"x + 1 = \", y)\n y # show macro also returns the result\nend\n\n# assignments should create the variable\n@show x = 3\nlet y = x = 2 \n println(\"x = 2 = \", y)\n y\nend\nx # should be equal to 2","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"The original Julia's implementation is not dissimilar to the following macro definition:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"macro myshow(ex)\n quote\n println($(QuoteNode(ex)), \" = \", repr(begin local value = $(esc(ex)) end))\n value\n end\nend","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Testing it gives us the expected behavior","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"@myshow xx = 1 + 1\nxx # should be defined","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"In this \"simple\" example, we had to use the following concepts mentioned already in the lecture:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"QuoteNode(ex) is used to wrap the expression inside another layer of quoting, such that when it is interpolated into :() it stays being a piece of code instead of the value it represents - TRUE QUOTING\nesc(ex) is used in case that the expression contains an assignment, that has to be evaluated in the top level module Main (we are escaping the local context) - ESCAPING\n$(QuoteNode(ex)) and $(esc(ex)) is used to evaluate an expression into another expression. INTERPOLATION\nlocal value = is used in order to return back the result after evaluation","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Lastly, let's mention that we can use @macroexpand to see how the code is manipulated in the @myshow macro","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"@macroexpand @show x + 1","category":"page"},{"location":"lecture_07/lab/#Repeat-macro","page":"Lab","title":"Repeat macro","text":"","category":"section"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"In the profiling/performance labs we have sometimes needed to run some code multiple times in order to gather some samples and we have tediously written out simple for loops inside functions such as this","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"function run_polynomial(n, a, x)\n for _ in 1:n\n polynomial(a, x)\n end\nend","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"We can remove this boilerplate code by creating a very simple macro that does this for us.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Define macro @repeat that takes two arguments, first one being the number of times a code is to be run and the other being the actual code.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"julia> @repeat 3 println(\"Hello!\")\nHello!\nHello!\nHello!","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Before defining the macro, it is recommended to write the code manipulation functionality into a helper function _repeat, which helps in organization and debugging of macros.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"_repeat(3, :(println(\"Hello!\"))) # testing \"macro\" without defining it","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"use $ interpolation into a for loop expression; for example given ex = :(1+x) we can interpolate it into another expression :($ex + y) -> :(1 + x + y)\nif unsure what gets interpolated use round brackets :($(ex) + y)\nmacro is a function that creates code that does what we want","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"BONUS: What happens if we call @repeat 3 x = 2? Is x defined?","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"macro repeat(n::Int, ex)\n return _repeat(n, ex)\nend\n\nfunction _repeat(n::Int, ex)\n :(for _ in 1:$n\n $ex\n end)\nend\n\n_repeat(3, :(println(\"Hello!\")))\n@repeat 3 println(\"Hello!\")","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Even if we had used escaping the expression x = 2 won't get evaluated properly due to the induced scope of the for loop. In order to resolve this we would have to specially match that kind of expression and generate a proper syntax withing the for loop global $ex. However we may just warn the user in the docstring that the usage is disallowed. ","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Note that this kind of repeat macro is also defined in the Flux.jl machine learning framework, wherein it's called @epochs and is used for creating training loop.","category":"page"},{"location":"lecture_07/lab/#lab07_polymacro","page":"Lab","title":"Polynomial macro","text":"","category":"section"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"This is probably the last time we are rewriting the polynomial function, though not quite in the same way. We have seen in the last lab, that some optimizations occur automatically, when the compiler can infer the length of the coefficient array, however with macros we can generate optimized code directly (not on the same level - we are essentially preparing already unrolled/inlined code).","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Ideally we would like to write some macro @poly that takes a polynomial in a mathematical notation and spits out an anonymous function for its evaluation, where the loop is unrolled. ","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Example usage:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"p = @poly x 3x^2+2x^1+10x^0 # the first argument being the independent variable to match\np(2) # return the value","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"However in order to make this happen, let's first consider much simpler case of creating the same but without the need for parsing the polynomial as a whole and employ the fact that macro can have multiple arguments separated by spaces.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"p = @poly 3 2 10\np(2)","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Create macro @poly that takes multiple arguments and creates an anonymous function that constructs the unrolled code. Instead of directly defining the macro inside the macro body, create helper function _poly with the same signature that can be reused outside of it.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Recall Horner's method polynomial evaluation from previous labs:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"function polynomial(a, x)\n accumulator = a[end] * one(x)\n for i in length(a)-1:-1:1\n accumulator = accumulator * x + a[i]\n #= accumulator = muladd(x, accumulator, a[i]) =# # equivalent\n end\n accumulator \nend","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"you can use muladd function as replacement for ac * x + a[i]\nthink of the accumulator variable as the mathematical expression that is incrementally built (try to write out the Horner's method[1] to see it)\nyou can nest expression arbitrarily\nthe order of coefficients has different order than in previous labs (going from high powers of x last to them being first)\nuse evalpoly to check the correctness","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"using Test\np = @poly 3 2 10\n@test p(2) == evalpoly(2, [10,2,3]) # reversed coefficients","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"[1]: Explanation of the Horner schema can be found on https://en.wikipedia.org/wiki/Horner%27s_method.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"using InteractiveUtils #hide\nmacro poly(a...)\n return _poly(a...)\nend\n\nfunction _poly(a...)\n N = length(a)\n ex = :($(a[1]))\n for i in 2:N\n ex = :(muladd(x, $ex, $(a[i]))) # equivalent of :(x * $ex + $(a[i]))\n end\n :(x -> $ex)\nend\n\np = @poly 3 2 10\np(2) == evalpoly(2, [10,2,3])\n@code_lowered p(2) # can show the generated code","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Moving on to the first/harder case, where we need to parse the mathematical expression.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Create macro @poly that takes two arguments first one being the independent variable and second one being the polynomial written in mathematical notation. As in the previous case this macro should define an anonymous function that constructs the unrolled code. ","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"julia> p = @poly x 3x^2+2x^1+10x^0 # the first argument being the independent variable to match","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"though in general we should be prepared for some edge cases, assume that we are really strict with the syntax allowed (e.g. we really require spelling out x^0, even though it is mathematically equivalent to 1)\nreuse the _poly function from the previous exercise\nuse the MacroTools.jl to match/capture a_*$v^(n_), where v is the symbol of independent variable, this is going to be useful in the following steps\nget maximal rank of the polynomial\nget coefficient for each power","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"note: `MacroTools.jl`\nThough not the most intuitive, MacroTools.jl pkg help us with writing custom macros. We will use two utilities@captureThis macro is used to match a pattern in a single expression and return values of particular spots. For examplejulia> using MacroTools\njulia> @capture(:[1, 2, 3, 4, 5, 6, 7], [1, a_, 3, b__, c_])\ntrue\n\njulia> a, b, c\n(2,[4,5,6],7)postwalk/prewalkIn order to extend @capture to more complicated expression trees, we can used either postwalk or prewalk to walk the AST and match expression along the way. For examplejulia> using MacroTools: prewalk, postwalk\njulia> ex = quote\n x = f(y, g(z))\n return h(x)\n end\n\njulia> postwalk(ex) do x\n @capture(x, fun_(arg_)) && println(\"Function: \", fun, \" with argument: \", arg)\n x\n end;\nFunction: g with argument: z\nFunction: h with argument: xNote that the x or the iteration is required, because by default postwalk/prewalk replaces currently read expression with the output of the body of do block.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"using MacroTools\nusing MacroTools: postwalk, prewalk\n\nmacro poly(v::Symbol, p::Expr)\n a = Tuple(reverse(_get_coeffs(v, p)))\n return _poly(a...)\nend\n\nfunction _max_rank(v, p)\n mr = 0\n postwalk(p) do x\n if @capture(x, a_*$v^(n_))\n mr = max(mr, n)\n end\n x\n end\n mr\nend\n\nfunction _get_coeffs(v, p)\n N = _max_rank(v, p) + 1\n coefficients = zeros(N)\n postwalk(p) do x\n if @capture(x, a_*$v^(n_))\n coefficients[n+1] = a\n end\n x\n end\n coefficients\nend","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Let's test it.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"p = @poly x 3x^2+2x^1+10x^0\np(2) == evalpoly(2, [10,2,3])\n@code_lowered p(2) # can show the generated code","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_07/lab/#Ecosystem-macros","page":"Lab","title":"Ecosystem macros","text":"","category":"section"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"There are at least two ways how we can make our life simpler when using our Ecosystem and EcosystemCore pkgs. Firstly, recall that in order to test our simulation we always had to write something like this:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"function create_world()\n n_grass = 500\n regrowth_time = 17.0\n\n n_sheep = 100\n Δenergy_sheep = 5.0\n sheep_reproduce = 0.5\n sheep_foodprob = 0.4\n\n n_wolves = 8\n Δenergy_wolf = 17.0\n wolf_reproduce = 0.03\n wolf_foodprob = 0.02\n\n gs = [Grass(id, regrowth_time) for id in 1:n_grass];\n ss = [Sheep(id, 2*Δenergy_sheep, Δenergy_sheep, sheep_reproduce, sheep_foodprob) for id in n_grass+1:n_grass+n_sheep];\n ws = [Wolf(id, 2*Δenergy_wolf, Δenergy_wolf, wolf_reproduce, wolf_foodprob) for id in n_grass+n_sheep+1:n_grass+n_sheep+n_wolves];\n World(vcat(gs, ss, ws))\nend\nworld = create_world();","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"which includes the tedious process of defining the agent counts, their parameters and last but not least the unique id manipulation. As part of the HW for this lecture you will be tasked to define a simple DSL, which can be used to define a world in a few lines.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Secondly, the definition of a new Animal or Plant, that did not have any special behavior currently requires quite a bit of repetitive code. For example defining a new plant type Broccoli goes as follows","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"abstract type Broccoli <: PlantSpecies end\nBase.show(io::IO,::Type{Broccoli}) = print(io,\"🥦\")\n\nEcosystemCore.eats(::Animal{Sheep},::Plant{Broccoli}) = true","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"and definition of a new animal like a Rabbit looks very similar","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"abstract type Rabbit <: AnimalSpecies end\nBase.show(io::IO,::Type{Rabbit}) = print(io,\"🐇\")\n\nEcosystemCore.eats(::Animal{Rabbit},p::Plant{Grass}) = size(p) > 0\nEcosystemCore.eats(::Animal{Rabbit},p::Plant{Broccoli}) = size(p) > 0","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"In order to make this code \"clearer\" (depends on your preference) we will create two macros, which can be called at one place to construct all the relations.","category":"page"},{"location":"lecture_07/lab/#New-Animal/Plant-definition","page":"Lab","title":"New Animal/Plant definition","text":"","category":"section"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Our goal is to be able to define new plants and animal species, while having a clear idea about their relations. For this we have proposed the following macros/syntax:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"@species Plant Broccoli 🥦\n@species Animal Rabbit 🐇\n@eats Rabbit [Grass => 0.5, Broccoli => 1.0, Mushroom => -1.0]","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Unfortunately the current version of Ecosystem and EcosystemCore, already contains some definitions of species such as Sheep, Wolf and Mushroom, which may collide with definitions during prototyping, therefore we have created a modified version of those pkgs, which will be provided in the lab.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"note: Testing relations\nWe can test the current definition with the following code that constructs \"eating matrix\"using Ecosystem\nusing Ecosystem.EcosystemCore\n\nfunction eating_matrix()\n _init(ps::Type{<:PlantSpecies}) = ps(1, 10.0)\n _init(as::Type{<:AnimalSpecies}) = as(1, 10.0, 1.0, 0.8, 0.7)\n function _check(s1, s2)\n try\n if s1 !== s2\n EcosystemCore.eats(_init(s1), _init(s2)) ? \"✅\" : \"❌\"\n else\n return \"❌\"\n end\n catch e\n if e isa MethodError\n return \"❔\"\n else\n throw(e)\n end\n end\n end\n\n animal_species = subtypes(AnimalSpecies)\n plant_species = subtypes(PlantSpecies)\n species = vcat(animal_species, plant_species)\n em = [_check(s, ss) for (s,ss) in Iterators.product(animal_species, species)]\n string.(hcat([\"🌍\", animal_species...], vcat(permutedims(species), em)))\nend\neating_matrix()\n 🌍 🐑 🐺 🌿 🍄\n 🐑 ❌ ❌ ✅ ✅\n 🐺 ✅ ❌ ❌ ❌","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Based on the following example syntax, ","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"@species Plant Broccoli 🥦\n@species Animal Rabbit 🐇","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"write macro @species inside Ecosystem pkg, which defines the abstract type, its show function and exports the type. For example @species Plant Broccoli 🥦 should generate code:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"abstract type Broccoli <: PlantSpecies end\nBase.show(io::IO,::Type{Broccoli}) = print(io,\"🥦\")\nexport Broccoli","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Define first helper function _species to inspect the macro's output. This is indispensable, as we are defining new types/constants and thus we may otherwise encounter errors during repeated evaluation (though only if the type signature changed).","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"_species(:Plant, :Broccoli, :🥦)\n_species(:Animal, :Rabbit, :🐇)","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"use QuoteNode in the show function just like in the @myshow example\nescaping esc is needed for the returned in order to evaluate in the top most module (Ecosystem/Main)\nideally these changes should be made inside the modified Ecosystem pkg provided in the lab (though not everything can be refreshed with Revise) - there is a file ecosystem_macros.jl just for this purpose\nmultiple function definitions can be included into a quote end block\ninterpolation works with any expression, e.g. $(typ == :Animal ? AnimalSpecies : PlantSpecies)","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"BONUS: Based on @species define also macros @animal and @plant with two arguments instead of three, where the species type is implicitly carried in the macro's name.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Macro @species","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"macro species(typ, name, icon)\n esc(_species(typ, name, icon))\nend\n\nfunction _species(typ, name, icon)\n quote\n abstract type $name <: $(typ == :Animal ? AnimalSpecies : PlantSpecies) end\n Base.show(io::IO, ::Type{$name}) = print(io, $(QuoteNode(icon)))\n export $name\n end\nend\n\n_species(:Plant, :Broccoli, :🥦)\n_species(:Animal, :Rabbit, :🐇)","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"And the bonus macros @plant and @animal","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"macro plant(name, icon)\n return :(@species Plant $name $icon)\nend\n\nmacro animal(name, icon)\n return :(@species Animal $name $icon)\nend","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"The next exercise applies macros to the agents eating behavior.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Define macro @eats inside Ecosystem pkg that assigns particular species their eating habits via eat! and eats functions. The macro should process the following example syntax","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"@eats Rabbit [Grass => 0.5, Broccoli => 1.0],","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"where Grass => 0.5 defines the behavior of the eat! function. The coefficient is used here as a multiplier for the energy balance, in other words the Rabbit should get only 0.5 of energy for a piece of Grass.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"ideally these changes should be made inside the modified Ecosystem pkg provided in the lab (though not everything can be refreshed with Revise) - there is a file ecosystem_macros.jl just for this purpose\nescaping esc is needed for the returned in order to evaluate in the top most module (Ecosystem/Main)\nyou can create an empty quote end block with code = Expr(:block) and push new expressions into its args incrementally\nuse dispatch to create specific code for the different combinations of agents eating other agents (there may be catch in that we have to first eval the symbols before calling in order to know if they are animals or plants)","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"note: Reminder of `EcosystemCore` `eat!` and `eats` functionality\nIn order to define that an Wolf eats Sheep, we have to define two methodsEcosystemCore.eats(::Animal{Wolf}, ::Animal{Sheep}) = true\n\nfunction EcosystemCore.eat!(ae::Animal{Wolf}, af::Animal{Sheep}, w::World)\n incr_energy!(ae, $(multiplier)*energy(af)*Δenergy(ae))\n kill_agent!(af, w)\nendIn order to define that an Sheep eats Grass, we have to define two methodsEcosystemCore.eats(::Animal{Sheep}, p::Plant{Grass}) = size(p)>0\n\nfunction EcosystemCore.eat!(a::Animal{Sheep}, p::Plant{Grass}, w::World)\n incr_energy!(a, $(multiplier)*size(p)*Δenergy(a))\n p.size = 0\nend","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"BONUS: You can try running the simulation with the newly added agents.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"macro eats(species::Symbol, foodlist::Expr)\n return esc(_eats(species, foodlist))\nend\n\n\nfunction _generate_eat(eater::Type{<:AnimalSpecies}, food::Type{<:PlantSpecies}, multiplier)\n quote\n EcosystemCore.eats(::Animal{$(eater)}, p::Plant{$(food)}) = size(p)>0\n function EcosystemCore.eat!(a::Animal{$(eater)}, p::Plant{$(food)}, w::World)\n incr_energy!(a, $(multiplier)*size(p)*Δenergy(a))\n p.size = 0\n end\n end\nend\n\nfunction _generate_eat(eater::Type{<:AnimalSpecies}, food::Type{<:AnimalSpecies}, multiplier)\n quote\n EcosystemCore.eats(::Animal{$(eater)}, ::Animal{$(food)}) = true\n function EcosystemCore.eat!(ae::Animal{$(eater)}, af::Animal{$(food)}, w::World)\n incr_energy!(ae, $(multiplier)*energy(af)*Δenergy(ae))\n kill_agent!(af, w)\n end\n end\nend\n\n_parse_eats(ex) = Dict(arg.args[2] => arg.args[3] for arg in ex.args if arg.head == :call && arg.args[1] == :(=>))\n\nfunction _eats(species, foodlist)\n cfg = _parse_eats(foodlist)\n code = Expr(:block)\n for (k,v) in cfg\n push!(code.args, _generate_eat(eval(species), eval(k), v))\n end\n code\nend\n\nspecies = :Rabbit \nfoodlist = :([Grass => 0.5, Broccoli => 1.0])\n_eats(species, foodlist)","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"","category":"page"},{"location":"lecture_07/lab/#Resources","page":"Lab","title":"Resources","text":"","category":"section"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"macros in Julia documentation","category":"page"},{"location":"lecture_07/lab/#Type{T}-type-selectors","page":"Lab","title":"Type{T} type selectors","text":"","category":"section"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"We have used ::Type{T} signature[2] at few places in the Ecosystem family of packages (and it will be helpful in the HW as well), such as in the show methods","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Base.show(io::IO,::Type{World}) = print(io,\"🌍\")","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"This particular example defines a method where the second argument is the World type itself and not an instance of a World type. As a result we are able to dispatch on specific types as values. ","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Furthermore we can use subtyping operator to match all types in a hierarchy, e.g. ::Type{<:AnimalSpecies} matches all animal species","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"[2]: https://docs.julialang.org/en/v1/manual/types/#man-typet-type","category":"page"},{"location":"projects/#projects","page":"Projects","title":"Projects","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"The goal of the project should be to create something, which is actually useful. Therefore we offer a lot of freedom in how the project will look like with the condition that you should spent around 60 hours on it (this number was derived as follows: each credit is worth 30 hours minus 13 lectures + labs minus 10 homeworks 2 hours each) and you should demonstrate some skills in solving the project. In general, we can distinguish three types of project depending on the beneficiary:","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"You benefit: Use / try to solve a well known problem using Julia language,\nOur group: work with your tutors on a topic researched in the AIC group, \nJulia community: choose an issue in a registered Julia project you like and fix it (documentation issues are possible but the resulting documentation should be very nice.).","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"The project should be of sufficient complexity that verify your skill of the language (to be agreed individually).","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"Below, we list some potential projects for inspiration.","category":"page"},{"location":"projects/#Implementing-new-things","page":"Projects","title":"Implementing new things","text":"","category":"section"},{"location":"projects/#Lenia-(Continuous-Game-of-Life)","page":"Projects","title":"Lenia (Continuous Game of Life)","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"Lenia is a continuous version of Conway's Game of Life. Implement a Julia version. For example, you could focus either on performance compared to the python version, or build nice visualizations with Makie.jl.","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"Nice tutorial from Conway to Lenia","category":"page"},{"location":"projects/#The-Equation-Learner-And-Its-Symbolic-Representation","page":"Projects","title":"The Equation Learner And Its Symbolic Representation","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"In many scientific and engineering one searches for interpretable (i.e. human-understandable) models instead of the black-box function approximators that neural networks provide. The equation learner (EQL) is one approach that can identify concise equations that describe a given dataset.","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"The EQL is essentially a neural network with different unary or binary activation functions at each indiviual unit. The network weights are regularized during training to obtain a sparse model which hopefully results in a model that represents a simple equation.","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"The goal of this project is to implement the EQL, and if there is enough time the improved equation learner (iEQL). The equation learners should be tested on a few toy problems (possibly inspired by the tasks in the papers). Finally, you will implement functionality that can transform the learned model into a symbolic, human readable, and exectuable Julia expression.","category":"page"},{"location":"projects/#Architecture-visualizer","page":"Projects","title":"Architecture visualizer","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"Create an extension of Flux / Lux and to visualize architecture of a neural network suitable for publication. Something akin PlotNeuralNet.","category":"page"},{"location":"projects/#Learning-Large-Language-Models-with-reduced-precition-(Mentor:-Tomas-Pevny)","page":"Projects","title":"Learning Large Language Models with reduced precition (Mentor: Tomas Pevny)","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"Large Language Models ((Chat) GPT, LLama, Falcon, Palm, ...) are huge. A recent trend is to perform optimization in reduced precision, for example in int8 instead of Float32. Such feature is currently missing in Julia ecosystem and this project should be about bringing this to the community (for an introduction, read these blogs LLM-int8 and emergent features, A gentle introduction to 8-bit Matrix Multiplication). The goal would be to implement this as an additional type of Number / Matrix and overload multiplication on CPU (and ideally on GPU) to make it transparent for neural networks? What I will learn? In this project, you will learn a lot about the (simplicity of) implementation of deep learning libraries and you will practice abstraction of Julia's types. You can furthermore learn about GPU Kernel programming and Transformers.jl library.","category":"page"},{"location":"projects/#Planning-algorithms-(Mentor:-Tomas-Pevny)","page":"Projects","title":"Planning algorithms (Mentor: Tomas Pevny)","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"Extend SymbolicPlanners.jl with the mm-ϵ variant of the bi-directional search MM: A bidirectional search algorithm that is guaranteed to meet in the middle. This pull request might be very helpful in understanding better the library.","category":"page"},{"location":"projects/#A-Rule-Learning-Algorithms-(Mentor:-Tomas-Pevny)","page":"Projects","title":"A Rule Learning Algorithms (Mentor: Tomas Pevny)","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"Rule-based models are simple and very interpretable models that have been around for a long time and are gaining popularity again. The goal of this project is to implement one of these algorithms","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"sequential covering","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"algorithm called RIPPER and evaluate it on a number of datasets.","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"Learning Certifiably Optimal Rule Lists for Categorical Data\nBoolean decision rules via column generation\nLearning Optimal Decision Trees with SAT\nA SAT-based approach to learn explainable decision sets","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"To increase the impact of the project, consider interfacing it with MLJ.jl","category":"page"},{"location":"projects/#Parallel-optimization-(Mentor:-Tomas-Pevny)","page":"Projects","title":"Parallel optimization (Mentor: Tomas Pevny)","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"Implement one of the following algorithms to train neural networks in parallel. Can be implemented in a separate package or consider extending FluxDistributed.jl. Do not forget to verify that the method actually works!!!","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"Hogwild!\nLocal sgd with periodic averaging: Tighter analysis and adaptive synchronization\nDistributed optimization for deep learning with gossip exchange","category":"page"},{"location":"projects/#Solve-issues-in-existing-projects:","page":"Projects","title":"Solve issues in existing projects:","text":"","category":"section"},{"location":"projects/#Create-Yao-backend-for-quantum-simulation-(Mentor:-Niklas-Heim)","page":"Projects","title":"Create Yao backend for quantum simulation (Mentor: Niklas Heim)","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"The recently published quantum programming library Qadence needs a Julia backend. The tricky quantum parts are already implemented in a library called Yao.jl. The goal of this project is to take the Qadence (Python) representation and translate it to Yao.jl (Julia). You will work with the Python/Julia interfacing library PythonCall.jl to realize this and benchmark the Julia backend in the end to assess if it is faster than the existing python implementation.","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"If this sounds interesting, talk to Niklas.","category":"page"},{"location":"projects/#Address-issues-in-markov-decision-processes-(Mentor:-Jan-Mrkos)","page":"Projects","title":"Address issues in markov decision processes (Mentor: Jan Mrkos)","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"Fix type stability issue in MCTS.jl, prepare benchmarks, and evaluate the impact of the changes. Details can be found in this issue. This project will require learnind a little bit about Markov Decision Processes if you don't know them already.","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"If it sounds interesting, get in touch with lecturer/lab assistant, who will connect you with Jan Mrkos.","category":"page"},{"location":"projects/#Extend-HMil-library-with-Retentative-networks-(mentor-Tomas-Pevny)","page":"Projects","title":"Extend HMil library with Retentative networks (mentor Tomas Pevny)","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"Retentative networks were recently proposed as a low-cost alternative to Transformer models without sacrificing performance (according to authors). By implementing Retentative Networks, te HMil library will be able to learn sequences (not just sets), which might nicely extend its applicability.","category":"page"},{"location":"projects/#Address-issues-in-HMil/JsonGrinder-library-(mentor-Simon-Mandlik)","page":"Projects","title":"Address issues in HMil/JsonGrinder library (mentor Simon Mandlik)","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"These are open source toolboxes that are used internally in Avast. Lots of general functionality is done, but some love is needed in polishing.","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"refactor the codebase using package extensions (e.g. for FillArrays)\nimprove compilation time (tracking down bottlenecks with SnoopCompile and using precompile directives from PrecompileTools.jl)","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"Or study new metric learning approach on application in animation description","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"apply machine learning on slides within presentation provide by PowToon","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"If it sounds interesting, get in touch with lecturer/lab assistant, who will connect you with Simon Mandlik.","category":"page"},{"location":"projects/#Project-requirements","page":"Projects","title":"Project requirements","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"The goal of the semestral project is to create a Julia pkg with reusable, properly tested and documented code. We have given you some options of topics, as well as the freedom to choose something that could be useful for your research or other subjects. In general we are looking for something where performance may be crucial such as data processing, optimization or equation solving.","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"In practice the project should follow roughly this tree structure","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":".\n├── scripts\n│\t├── run_example.jl\t\t\t# one or more examples showing the capabilities of the pkg\n│\t├── Project.toml \t\t\t# YOUR_PROJECT should be added here with develop command with rel path\n│\t└── Manifest.toml \t\t\t# should be committed as it allows to reconstruct the environment exactly\n├── src\n│\t├── YOUR_PROJECT.jl \t\t# ideally only some top level code such as imports and exports, rest of the code included from other files\n│\t├── src1.jl \t\t\t\t# source files structured in some logical chunks\n│\t└── src2.jl\n├── test\n│\t├── runtest.jl # contains either all the tests or just includes them from other files\n│\t├── Project.toml \t\t\t# lists some additional test dependencies\n│\t└── Manifest.toml \t\t# usually not committed to git as it is generated on the fly\n├── README.md \t\t\t\t\t# describes in short what the pkg does and how to install pkg (e.g. some external deps) and run the example\n├── Project.toml \t\t\t\t# lists all the pkg dependencies\n└── Manifest.toml \t\t\t\t# usually not committed to git as the requirements may be to restrictive","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"The first thing that we will look at is README.md, which should warn us if there are some special installation steps, that cannot be handled with Julia's Pkg system. For example if some 3rd party binary dependency with license is required. Secondly we will try to run tests in the test folder, which should run and not fail and should cover at least some functionality of the pkg. Thirdly and most importantly we will instantiate environment in scripts and test if the example runs correctly. Lastly we will focus on documentation in terms of code readability, docstrings and inline comments. ","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"Only after all this we may look at the extent of the project and it's difficulty, which may help us in deciding between grades. ","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"Nice to have things, which are not strictly required but obviously improves the score.","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"Ideally the project should be hosted on GitHub, which could have the continuous integration/testing set up.\nInclude some benchmark and profiling code in your examples, which can show us how well you have dealt with the question of performance.\nSome parallelization attempts either by multi-processing, multi-threadding, or CUDA. Do not forget to show the improvement.\nDocumentation with a webpage using Documenter.jl.","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"Here are some examples of how the project could look like:","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"ImageInspector","category":"page"},{"location":"lecture_01/outline/#Course-outline","page":"Outline","title":"Course outline","text":"","category":"section"},{"location":"lecture_01/outline/","page":"Outline","title":"Outline","text":"Introduction\nType system\nuser: tool for abstraction\ncompiler: tool for memory layout\nDesign patterns (mental setup)\nJulia is a type-based language\nmultiple-dispatch generalizes OOP and FP\nPackages\nway how to organize code\ncode reuse (alternative to libraries)\nexperiment reproducibility\nBenchmarking\nhow to measure code efficiency\nIntrospection\nunderstand how the compiler process the data\nMacros\nautomate writing of boring the boilerplate code\ngood macro create cleaner code\nAutomatic Differentiation\nTheory: difference between the forward and backward mode\nImplementation techniques\nIntermediate representation\nhow to use internal the representation of the code \nexample in automatic differentiation\nParallel computing\nthreads, processes\nGraphics card coding\ntypes for GPU\nspecifics of architectures\nOrdinary Differential Equations\nsimple solvers\nerror propagation\nData driven ODE\ncombine ODE with optimization\nautomatic differentiation (adjoints)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"using Plots","category":"page"},{"location":"lecture_08/lecture/#Automatic-Differentiation","page":"Lecture","title":"Automatic Differentiation","text":"","category":"section"},{"location":"lecture_08/lecture/#Motivation","page":"Lecture","title":"Motivation","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"It supports a lot of modern machine learning by allowing quick differentiation of complex mathematical functions. The 1st order optimization methods are ubiquitous in finding parameters of functions (not only in deep learning).\nAD is interesting to study from both mathermatical and implementation perspective, since different approaches comes with different trade-offs. Julia offers many implementations (some of them are not maintained anymore), as it showed to implement (simple) AD is relatively simple.\nWe (authors of this course) believe that it is good to understand (at least roughly), how the methods work in order to use them effectively in your work.\nJulia is unique in the effort separating definitions of AD rules from AD engines that use those rules to perform the AD and the backend which executes the rules This allows authors of generic libraries to add new rules that would be compatible with many frameworks. See juliadiff.org for a list.","category":"page"},{"location":"lecture_08/lecture/#Theory","page":"Lecture","title":"Theory","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"The differentiation is routine process, as most of the time we break complicated functions down into small pieces that we know, how to differentiate and from that to assemble the gradient of the complex function back. Thus, the essential piece is the differentiation of the composed function f mathbbR^n rightarrow mathbbR^m","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"f(x) = f_1(f_2(f_3(ldots f_n(x)))) = (f_1 circ f_2 circ ldots circ f_n)(x)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"which is computed by chainrule. Before we dive into the details, let's define the notation, which for the sake of clarity needs to be precise. The gradient of function f(x) with respect to x at point x_0 is denoted as leftfracpartial fpartial xright_x^0","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"For a composed function f(x) the gradient with respect to x at point x_0 is equal to","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"leftfracpartial fpartial xright_x^0 = leftfracf_1partial y_1right_y_1^0 times leftfracf_2partial y_2right_y_2^0 times ldots times leftfracf_npartial y_nright_y_n^0","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"where y_i denotes the input of function f_i and","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"beginalignat*2\ny_i^0 = left(f_i+1 circ ldots circ f_nright) (x^0) \ny_n^0 = x^0 \ny_0^0 = f(x^0) \nendalignat*","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"How leftfracf_ipartial y_iright_y_i^0 looks like? ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"If f_i mathbbR rightarrow mathbbR, then fracf_ipartial y_i in mathbbR is a real number mathbbR and we live in a high-school world, where it was sufficient to multiply real numbers.\nIf f_i mathbbR^m_i rightarrow mathbbR^n_i, then mathbfJ_i = leftfracf_ipartial y_iright_y_i^0 in mathbbR^n_im_i is a matrix with m_i rows and n_i columns. ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"The computation of gradient fracpartial fpartial x theoretically boils down to ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"computing Jacobians leftmathbfJ_iright_i=1^n \nmultiplication of Jacobians as it holds that leftfracpartial fpartial xright_y_0 = J_1 times J_2 times ldots times J_n. ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"The complexity of the computation (at least one part of it) is therefore therefore determined by the Matrix multiplication, which is generally expensive, as theoretically it has complexity at least O(n^23728596) but in practice a little bit more as the lower bound hides the devil in the O notation. The order in which the Jacobians are multiplied has therefore a profound effect on the complexity of the AD engine. While determining the optimal order of multiplication of sequence of matrices is costly, in practice, we recognize two important cases.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Jacobians are multiplied from right to left as J_1 times (J_2 times ( ldots times (J_n-1 times J_n) ldots)) which has the advantage when the input dimension of f mathbbR^n rightarrow mathbbR^m is smaller than the output dimension, n m. - referred to as the FORWARD MODE\nJacobians are multiplied from left to right as ( ldots ((J_1 times J_2) times J_3) times ldots ) times J_n which has the advantage when the input dimension of f mathbbR^n rightarrow mathbbR^m is larger than the output dimension, n m. - referred to as the BACKWARD MODE","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"The ubiquitous in machine learning to minimization of a scalar (loss) function of a large number of parameters. Also notice that for f of certain structures, it pays-off to do a mixed-mode AD, where some parts are done using forward diff and some parts using reverse diff. ","category":"page"},{"location":"lecture_08/lecture/#Example","page":"Lecture","title":"Example","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Let's workout an example","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"z = xy + sin(x)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"How it maps to the notation we have used above? Particularly, what are f_1 f_2 ldots f_n and the corresponding y_i_i=1^n, such that (f_1 circ f_2 circ ldots circ f_n)(xy) = xy + sin(x) ?","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"beginalignat*6\nf_1mathbbR^2 rightarrow mathbbR quadf_1(y_1) = y_11 + y_12 quad y_0 = (xy + sin(x)) \nf_2mathbbR^3 rightarrow mathbbR^2 quadf_2(y_2) = (y_21y_22 y_23) quad y_1 = (xy sin(x))\nf_3 mathbbR^2 rightarrow mathbbR^3 quadf_3(y_3) = (y_31 y_32 sin(y_31)) quad y_2 = (x y sin(x))\nendalignat*","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"The corresponding jacobians are ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"beginalignat*4\nf_1(y_1) = y_11 + y_12 quad mathbfJ_1 = beginbmatrix 1 1 endbmatrix \nf_2(y_2) = (y_21y_22 y_23) quad mathbfJ_2 = beginbmatrix y_2 2 0 y_21 0 0 1 endbmatrix\nf_3(y_3) = (y_31 y_32 sin(y_31)) quad mathbfJ_3 = beginbmatrix 1 0 cos(y_31) 0 1 0 endbmatrix \nendalignat*","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"and for the gradient it holds that","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"beginbmatrix fracpartial f(x y)partialx fracpartial f(xy)partialy endbmatrix = mathbfJ_3 times mathbfJ_2 times mathbfJ_1 = beginbmatrix 1 0 cos(x) 0 1 0 endbmatrix times beginbmatrix y 0 x 0 0 1 endbmatrix times beginbmatrix 1 1 endbmatrix = beginbmatrix y cos(x) x 0 endbmatrix times beginbmatrix 1 1 endbmatrix = beginbmatrix y + cos(x) x endbmatrix","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Note that from theoretical point of view this decomposition of a function is not unique, however as we will see later it usually given by the computational graph in a particular language/environment.","category":"page"},{"location":"lecture_08/lecture/#Calculation-of-the-Forward-mode","page":"Lecture","title":"Calculation of the Forward mode","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"In theory, we can calculate the gradient using forward mode as follows Initialize the Jacobian of y_n with respect to x to an identity matrix, because as we have stated above y^0_n = x, i.e. fracpartial y_npartial x = mathbbI. Iterate i from n down to 1 as","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"calculate the next intermediate output as y^0_i-1 = f_i(y^0_i) \ncalculate Jacobian J_i = leftfracf_ipartial y_iright_y^0_i\npush forward the gradient as leftfracpartial y_i-1partial xright_x = J_i times leftfracpartial y_npartial xright_x","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Notice that ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"on the very end, we are left with y = y^0_0 and with fracpartial y_0partial x, which is the gradient we wanted to calculate;\nif y is a scalar, then fracpartial y_0partial x is a matrix with single row\nthe Jacobian and the output of the function is calculated in one sweep.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"The above is an idealized computation. The real implementation is a bit different, as we will see later.","category":"page"},{"location":"lecture_08/lecture/#Implementation-of-the-forward-mode-using-Dual-numbers","page":"Lecture","title":"Implementation of the forward mode using Dual numbers","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Forward modes need to keep track of the output of the function and of the derivative at each computation step in the computation of the complicated function f. This can be elegantly realized with a dual number, which are conceptually similar to complex numbers, but instead of the imaginary number i dual numbers use epsilon in its second component:","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"x = v + dot v epsilon","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"where (vdot v) in mathbb R and by definition epsilon^2=0 (instead of i^2=-1 in complex numbers). What are the properties of these Dual numbers?","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"beginalign\n(v + dot v epsilon) + (u + dot u epsilon) = (v + u) + (dot v + dot u)epsilon \n(v + dot v epsilon)(u + dot u epsilon) = vu + (udot v + dot u v)epsilon + dot v dot u epsilon^2 = vu + (udot v + dot u v)epsilon \nfracv + dot v epsilonu + dot u epsilon = fracv + dot v epsilonu + dot u epsilon fracu - dot u epsilonu - dot u epsilon = fracvu - frac(dot u v - u dot v)epsilonu^2\nendalign","category":"page"},{"location":"lecture_08/lecture/#How-are-dual-numbers-related-to-differentiation?","page":"Lecture","title":"How are dual numbers related to differentiation?","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Let's evaluate the above equations at (v dot v) = (v 1) and (u dot u) = (u 0) we obtain ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"beginalign\n(v + dot v epsilon) + (u + dot u epsilon) = (v + u) + 1epsilon \n(v + dot v epsilon)(u + dot u epsilon) = vu + uepsilon\nfracv + dot v epsilonu + dot u epsilon = fracvu + frac1u epsilon\nendalign","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"and notice that terms (1 u frac1u) corresponds to gradient of functions (u+v uv fracvu) with respect to v. We can repeat it with changed values of epsilon as (v dot v) = (v 0) and (u dot u) = (u 1) and we obtain","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"beginalign\n(v + dot v epsilon) + (u + dot u epsilon) = (v + u) + 1epsilon \n(v + dot v epsilon)(u + dot u epsilon) = vu + vepsilon\nfracv + dot v epsilonu + dot u epsilon = fracvu - fracvu^2 epsilon\nendalign","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"meaning that at this moment we have obtained gradients with respect to u.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"All above functions (u+v uv fracuv) are of mathbbR^2 rightarrow mathbbR, therefore we had to repeat the calculations twice to get gradients with respect to both inputs. This is inline with the above theory, where we have said that if input dimension is larger then output dimension, the backward mode is better. But consider a case, where we have a function ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"f(v) = (v + 5 5*v 5 v) ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"which is mathbbR rightarrow mathbbR^3. In this case, we obtain the Jacobian 1 5 -frac5v^2 in a single forward pass (whereas the reverse would require three passes over the backward calculation, as will be seen later).","category":"page"},{"location":"lecture_08/lecture/#Does-dual-numbers-work-universally?","page":"Lecture","title":"Does dual numbers work universally?","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Let's first work out polynomial. Let's assume the polynomial","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"p(v) = sum_i=1^n p_iv^i","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"and compute its value at v + dot v epsilon (note that we know how to do addition and multiplication)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"beginsplit\np(v) = \n sum_i=0^n p_i(v + dotv epsilon )^i = \n sum_i=0^n leftp_i sum_j=0^nbinomijv^i-j(dot v epsilon)^iright = \n p_0 + sum_i=1^n leftp_i sum_j=0^1binomijv^i-j(dot v epsilon)^jright = \n = p_0 + sum_i=1^n p_i(v^i + i v^i-1 dot v epsilon ) \n = p(v) + left(sum_i=1^n ip_i v^i-1right) dot v epsilon\nendsplit","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"where in the multiplier of dotv epsilon: sum_i=1^n ip_i v^i - 1, we recognize the derivative of p(v) with respect to v. This proves that Dual numbers can be used to calculate the gradient of polynomials.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Let's now consider a general function fmathbbR rightarrow mathbbR. Its value at point v + dot v epsilon can be approximated using Taylor expansion at function at point v as","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"f(v+dot v epsilon) = sum_i=0^infty fracf^i(v)dot v^iepsilon^ni\n = f(v) + f(v)dot vepsilon","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"where all higher order terms can be dropped because epsilon^i=0 for i1. This shows that we can calculate the gradient of f at point v by calculating its value at f(v + epsilon) and taking the multiplier of epsilon.","category":"page"},{"location":"lecture_08/lecture/#Implementing-Dual-number-with-Julia","page":"Lecture","title":"Implementing Dual number with Julia","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"To demonstrate the simplicity of Dual numbers, consider following definition of Dual numbers, where we define a new number type and overload functions +, -, *, and /. In Julia, this reads:","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"struct Dual{T<:Number} <: Number\n x::T\n d::T\nend\n\nBase.:+(a::Dual, b::Dual) = Dual(a.x+b.x, a.d+b.d)\nBase.:-(a::Dual, b::Dual) = Dual(a.x-b.x, a.d-b.d)\nBase.:/(a::Dual, b::Dual) = Dual(a.x/b.x, (a.d*b.x - a.x*b.d)/b.x^2) # recall (a/b) = a/b + (a'b - ab')/b^2 ϵ\nBase.:*(a::Dual, b::Dual) = Dual(a.x*b.x, a.d*b.x + a.x*b.d)\n\n# Let's define some promotion rules\nDual(x::S, d::T) where {S<:Number, T<:Number} = Dual{promote_type(S, T)}(x, d)\nDual(x::Number) = Dual(x, zero(typeof(x)))\nDual{T}(x::Number) where {T} = Dual(T(x), zero(T))\nBase.promote_rule(::Type{Dual{T}}, ::Type{S}) where {T<:Number,S<:Number} = Dual{promote_type(T,S)}\nBase.promote_rule(::Type{Dual{T}}, ::Type{Dual{S}}) where {T<:Number,S<:Number} = Dual{promote_type(T,S)}\n\n# and define api for forward differentionation\nforward_diff(f::Function, x::Real) = _dual(f(Dual(x,1.0)))\n_dual(x::Dual) = x.d\n_dual(x::Vector) = _dual.(x)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"And let's test the Babylonian Square Root (an algorithm to compute sqrt x):","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"babysqrt(x, t=(1+x)/2, n=10) = n==0 ? t : babysqrt(x, (t+x/t)/2, n-1)\n\nforward_diff(babysqrt, 2) \nforward_diff(babysqrt, 2) ≈ 1/(2sqrt(2))\nforward_diff(x -> [1 + x, 5x, 5/x], 2) ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"We now compare the analytic solution to values computed by the forward_diff and byt he finite differencing","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"f(x) = sqrtx qquad f(x) = frac12sqrtx","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"using FiniteDifferences\nforward_dsqrt(x) = forward_diff(babysqrt,x)\nanalytc_dsqrt(x) = 1/(2babysqrt(x))\nforward_dsqrt(2.0)\nanalytc_dsqrt(2.0)\ncentral_fdm(5, 1)(babysqrt, 2.0)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"plot(0.0:0.01:2, babysqrt, label=\"f(x) = babysqrt(x)\", lw=3)\nplot!(0.1:0.01:2, analytc_dsqrt, label=\"Analytic f'\", ls=:dot, lw=3)\nplot!(0.1:0.01:2, forward_dsqrt, label=\"Dual Forward Mode f'\", lw=3, ls=:dash)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"","category":"page"},{"location":"lecture_08/lecture/#Takeaways","page":"Lecture","title":"Takeaways","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Forward mode f is obtained simply by pushing a Dual through babysqrt\nTo make the forward diff work in Julia, we only need to overload a few operators for forward mode AD to work on any function. Therefore the name of the approach is called operator overloading.\nFor vector valued function we can use Hyperduals\nForward diff can differentiation through the setindex! (called each time an element is assigned to a place in array, e.g. x = [1,2,3]; x[2] = 1)\nForwardDiff is implemented in ForwardDiff.jl, which might appear to be neglected, but the truth is that it is very stable and general implementation.\nForwardDiff does not have to be implemented through Dual numbers. It can be implemented similarly to ReverseDiff through multiplication of Jacobians, which is what is the community work on now (in Diffractor, Zygote with rules defined in ChainRules).","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"","category":"page"},{"location":"lecture_08/lecture/#Reverse-mode","page":"Lecture","title":"Reverse mode","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"In reverse mode, the computation of the gradient follow the opposite order. We initialize the computation by setting mathbfJ_0 = fracpartial ypartial y_0 which is again an identity matrix. Then we compute Jacobians and multiplications in the opposite order. The problem is that to calculate J_i we need to know the value of y_i^0, which cannot be calculated in the reverse pass. The backward pass therefore needs to be preceded by the forward pass, where y_i^0_i=1^n are calculated.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"The complete reverse mode algorithm therefore proceeds as ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Forward pass: iterate i from n down to 1 as\ncalculate the next intermediate output as y^0_i-1 = f_i(y^0_i) \nBackward pass: iterate i from 1 down to n as\ncalculate Jacobian J_i = leftfracf_ipartial y_iright_y_i^0 at point y_i^0\npull back the gradient as leftfracpartial f(x)partial y_iright_y^0_i = leftfracpartial y_0partial y_i-1right_y^0_i-1 times J_i","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"The need to store intermediate outs has a huge impact on memory requirements, which particularly on GPU is a big deal. Recall few lectures ago we have been discussing how excessive memory allocations can be damaging for performance, here we are given an algorithm where the excessive allocation is by design.","category":"page"},{"location":"lecture_08/lecture/#Tricks-to-decrease-memory-consumptions","page":"Lecture","title":"Tricks to decrease memory consumptions","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Define custom rules over large functional blocks. For example while we can auto-grad (in theory) matrix product, it is much more efficient to define make a matrix multiplication as one large function, for which we define Jacobians (note that by doing so, we can dispatch on Blas). e.g","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"beginalignat*2\n mathbfC = mathbfA * mathbfB \n fracpartialmathbfCpartial mathbfA = mathbfB \n fracpartialmathbfCpartial mathbfB = mathbfA^mathrmT \nendalignat*","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"When differentiating Invertible functions, calculate intermediate outputs from the output. This can lead to huge performance gain, as all data needed for computations are in caches. \nCheckpointing does not store intermediate ouputs after larger sequence of operations. When they are needed for forward pass, they are recalculated on demand.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Most reverse mode AD engines does not support mutating values of arrays (setindex! in julia). This is related to the memory consumption, where after every setindex! you need in theory save the full matrix. Enzyme differentiating directly LLVM code supports this, since in LLVM every variable is assigned just once. ForwardDiff methods does not suffer this problem, as the gradient is computed at the time of the values.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\nReverse mode AD was first published in 1976 by Seppo Linnainmaa[1], a finnish computer scientist. It was popularized in the end of 80s when applied to training multi-layer perceptrons, which gave rise to the famous backpropagation algorithm[2], which is a special case of reverse mode AD.[1]: Linnainmaa, S. (1976). Taylor expansion of the accumulated rounding error. BIT Numerical Mathematics, 16(2), 146-160.[2]: Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\nThe terminology in automatic differentiation is everything but fixed. The community around ChainRules.jl went a great length to use something reasonable. They use pullback for a function realizing vector-Jacobian product in the reverse-diff reminding that the gradient is pulled back to the origin of the computation. The use pushforward to denote the same operation in the ForwardDiff, as the gradient is push forward through the computation.","category":"page"},{"location":"lecture_08/lecture/#Implementation-details-of-reverse-AD","page":"Lecture","title":"Implementation details of reverse AD","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Reverse-mode AD needs to record operations over variables when computing the value of a differentiated function, such that it can walk back when computing the gradient. This record is called tape, but it is effectively a directed acyclic graph. The construction of the tape can be either explicit or implicit. The code computing the gradient can be produced by operator-overloading or code-rewriting techniques. This give rise of four different takes on AD, and Julia has libraries for alll four.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Yota.jl: explict tape, code-rewriting\nTracker.jl, AutoGrad.jl: implict tape, operator overloading\nReverseDiff.jl: explict tape, operator overloading\nZygote.jl: implict tape, code-rewriting","category":"page"},{"location":"lecture_08/lecture/#Graph-based-AD","page":"Lecture","title":"Graph-based AD","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"In Graph-based approach, we start with a complete knowledge of the computation graph (which is known in many cases like classical neural networks) and augment it with nodes representing the computation of the computation of the gradient (backward path). We need to be careful to add all edges representing the flow of information needed to calculate the gradient. Once the computation graph is augmented, we can find the subgraph needed to compute the desired node(s). ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Recall the example from the beginning of the lecture f(x y) = sin(x) + xy, let's observe, how the extension of the computational graph will look like. The computation graph of function f looks like","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"(Image: diff graph)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"where arrows rightarrow denote the flow of operations and we have denoted the output of function f as z and outputs of intermediate nodes as h_i standing for hidden.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"We start from the top and add a node calculating fracpartial zpartial h_3 which is an identity, needed to jump-start the differentiation. ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"(Image: diff graph)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"We connect it with the output of h_3, even though technically in this case it is not needed, as the z = h_3. We then add a node calculating fracpartial h_3partial h_2 for which we only need information about h_2 and mark it in the graph (again, this edge can be theoretically dropped due to being equal to one regardless the inputs). Following the chain rule, we need to combine fracpartial h_3partial h_2 with fracpartial zpartial h_3 to compute fracpartial zpartial h_2 which we note in the graph.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"(Image: diff graph)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"We continue with the same process with fracpartial h_3partial h_1, which we again combine with fracpartial zpartial h_1 to obtain fracpartial zpartial h_1. Continuing the reverse diff we obtain the final graph","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"(Image: diff graph) ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"containing the desired nodes fracpartial zpartial x and fracpartial zpartial y. This computational graph can be passed to the compiler to compute desired values.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"This approach to AD has been taken for example by Theano and by TensorFlow. In Tensorflow when you use functions like tf.mul( a, b ) or tf.add(a,b), you are not performing the computation in Python, but you are building the computational graph shown as above. You can then compute the values using tf.run with a desired inputs, but you are in fact computing the values in a different interpreter / compiler then in python.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Advantages:","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Knowing the computational graph in advance is great, as you can do expensive optimization steps to simplify the graph. \nThe computational graph have a simple semantics (limited support for loops, branches, no objects), and the compiler is therefore simpler than the compiler of full languages.\nSince the computation of gradient augments the graph, you can run the process again to obtain higher order gradients. \nTensorFlow allows you to specialize on sizes of Tensors, which means that it knows precisely how much memory you will need and where, which decreases the number of allocations. This is quite important in GPU.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Disadvantages:","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"You are restricted to fixed computation graph. It is generally difficult to implement if or while, and hence to change the computation according to values computed during the forward pass.\nDevelopment and debugging can be difficult, since you are not developing the computation graph in the host language.\nExploiting within computation graph parallelism might be difficult.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Comments:","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"DaggerFLux.jl use this approach to perform model-based paralelism, where parts of the computation graph (and especially parameters) can reside on different machines.\nUmlaut.jl allows to easily obtain the tape through tracing of the execution of a function, which can be then used to implement the AD as described above (see Yota's documentation for complete example).","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"using Umlaut\ng(x, y) = x * y\nf(x, y) = g(x, y)+sin(x)\ntape = trace(f, 1.0, 2.0)[2]","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Yota.jl use the tape to generate the gradient as ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"tape = Yota.gradtape(f, 1.0, 2.0; seed=1.0)\nUmlaut.to_expr(tape)","category":"page"},{"location":"lecture_08/lecture/#Tracking-based-AD","page":"Lecture","title":"Tracking-based AD","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Alternative to static-graph based methods are methods, which builds the graph during invocation of functions and then use this dynamically built graph to know, how to compute the gradient. The dynamically built graph is frequently called tape. This approach is used by popular libraries like PyTorch, AutoGrad, and Chainer in Python ecosystem, or by Tracker.jl (Flux.jl's former AD backend), ReverseDiff.jl, and AutoGrad.jl (Knet.jl's AD backend) in Julia. This type of AD systems is also called operator overloading, since in order to record the operations performed on the arguments we need to replace/wrap the original implementation.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"How do we build the tracing? Let's take a look what ReverseDiff.jl is doing. It defines TrackedArray (it also defines TrackedReal, but TrackedArray is more interesting) as","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"struct TrackedArray{T,N,V<:AbstractArray{T,N}} <: AbstractArray{T,N}\n value::V\n deriv::Union{Nothing,V}\n tape::Vector{Any}\n string_tape::String\nend","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"where in","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"value it stores the value of the array\nderiv will hold the gradient of the tracked array\ntape of will log operations performed with the tracked array, such that we can calculate the gradient as a sum of operations performed over the tape.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"What do we need to store on the tape? Let's denote as a the current TrackedArray. The gradient with respect to some output z is equal to fracpartial zpartial a = sum_g_i fracpartial zpartial g_i times fracpartial g_ipartial a where g_i is the output of any function (in the computational graph) where a was a direct input. The InstructionTape will therefore contain a reference to g_i (which has to be of TrackedArray and where we know fracpartial zpartial g_i will be stored in deriv field) and we also need to a method calculating fracpartial g_ipartial a, which can be stored as an anonymous function will accepting the grad as an argument.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"TrackedArray(a::AbstractArray, string_tape::String = \"\") = TrackedArray(a, similar(a) .= 0, [], string_tape)\nTrackedMatrix{T,V} = TrackedArray{T,2,V} where {T,V<:AbstractMatrix{T}}\nTrackedVector{T,V} = TrackedArray{T,1,V} where {T,V<:AbstractVector{T}}\nBase.show(io::IO, ::MIME\"text/plain\", a::TrackedArray) = show(io, a)\nBase.show(io::IO, a::TrackedArray) = print(io, \"TrackedArray($(size(a.value)))\")\nvalue(A::TrackedArray) = A.value\nvalue(A) = A\ntrack(A, string_tape = \"\") = TrackedArray(A, string_tape)\ntrack(a::Number, string_tape) = TrackedArray(reshape([a], 1, 1), string_tape)\n\nimport Base: +, *\nfunction *(A::TrackedMatrix, B::TrackedMatrix)\n a, b = value.((A, B))\n C = TrackedArray(a * b, \"($(A.string_tape) * $(B.string_tape))\")\n push!(A.tape, (C, ∂C -> ∂C * b'))\n push!(B.tape, (C, ∂C -> a' * ∂C))\n C\nend\n\nfunction *(A::TrackedMatrix, B::AbstractMatrix)\n a, b = value.((A, B))\n C = TrackedArray(a * b, \"($(A.string_tape) * B)\")\n push!(A.tape, (C, ∂C -> ∂C * b'))\n C\nend\n\nfunction *(A::Matrix, B::TrackedMatrix)\n a, b = value.((A, B))\n C = TrackedArray(a * b, \"A * $(B.string_tape)\")\n push!(A.tape, (C, ∂C -> ∂C * b'))\n C\nend\n\nfunction +(A::TrackedMatrix, B::TrackedMatrix)\n C = TrackedArray(value(A) + value(B), \"($(A.string_tape) + $(B.string_tape))\")\n push!(A.tape, (C, ∂C -> ∂C))\n push!(B.tape, (C, ∂C -> ∂C))\n C\nend\n\nfunction msin(A::TrackedMatrix)\n a = value(A)\n C = TrackedArray(sin.(a), \"sin($(A.string_tape))\")\n push!(A.tape, (C, ∂C -> cos.(a) .* ∂C))\n C\nend","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Let's observe that the operations are recorded on the tape as they should","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"a = rand()\nb = rand()\nA = track(a, \"A\")\nB = track(b, \"B\")\n# R = A * B + msin(A)\nC = A * B \nA.tape\nB.tape\nC.string_tape\nR = C + msin(A)\nA.tape\nB.tape\nR.string_tape","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Let's now implement a function that will recursively calculate the gradient of a term of interest. It goes over its childs, if they not have calculated the gradients, calculate it, otherwise it adds it to its own after if not, ask them to calculate the gradient and otherwise ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"function accum!(A::TrackedArray)\n isempty(A.tape) && return(A.deriv)\n A.deriv .= sum(g(accum!(r)) for (r, g) in A.tape)\n empty!(A.tape)\n A.deriv\nend","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"We can calculate the gradient by initializing the gradient of the result to vector of ones simulating the sum function","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"using FiniteDifferences\nR.deriv .= 1\naccum!(A)[1]\n∇a = grad(central_fdm(5,1), a -> a*b + sin(a), a)[1]\nA.deriv[1] ≈ ∇a\naccum!(B)[1]\n∇b = grad(central_fdm(5,1), b -> a*b + sin(a), b)[1]\nB.deriv[1] ≈ ∇b","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"The api function for computing the grad might look like","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"function trackedgrad(f, args...)\n args = track.(args)\n o = f(args...)\n fill!(o.deriv, 1)\n map(accum!, args)\nend","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"where we should assert that the output dimension is 1. In our implementation we dirtily expect the output of f to be summed to a scalar.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Let's compare the results to those computed by FiniteDifferences","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"A = rand(4,4)\nB = rand(4,4)\ntrackedgrad(A -> A * B + msin(A), A)[1]\ngrad(central_fdm(5,1), A -> sum(A * B + sin.(A)), A)[1]\ntrackedgrad(A -> A * B + msin(A), B)[1]\ngrad(central_fdm(5,1), A -> sum(A * B + sin.(A)), B)[1]","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"To make the above AD system really useful, we would need to ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Add support for TrackedReal, which is straightforward (we might skip the anonymous function, as the derivative of a scalar function is always a number).\nWe would need to add a lot of rules, how to work with basic values. This is why the the approach is called operator overloading since you need to overload a lot of functions (or methods or operators).","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"For example to add all combinations for +, we would need to add following rules.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"function +(A::TrackedMatrix, B::TrackedMatrix)\n C = TrackedArray(value(A) + value(B), \"($(A.string_tape) + $(B.string_tape))\")\n push!(A.tape, (C, ∂C -> ∂C ))\n push!(B.tape, (C, ∂C -> ∂C))\n C\nend\n\nfunction +(A::AbstractMatrix, B::TrackedMatrix)\n C = TrackedArray(A * value(B), \"(A + $(B.string_tape))\")\n push!(B.tape, (C, ∂C -> ∂C))\n C\nend","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Advantages:","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Debugging and development is nicer, as AD is implemented in the same language.\nThe computation graph, tape, is dynamic, which makes it simpler to take the gradient in the presence of if and while.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Disadvantages:","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"The computation graph is created and differentiated during every computation, which might be costly. In most deep learning applications, this overhead is negligible in comparison to time of needed to perform the operations itself (ReverseDiff.jl allows to compile the tape).\nThe compiler has limited options for optimization, since the tape is created during the execution.\nSince computation graph is dynamic, it cannot be optimized as the static graph, the same holds for the memory allocations. ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"A more complete example which allow to train feed-forward neural network on GPU can be found here.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\nThe difference between tracking and graph-based AD systems is conceptually similar to interpreted and compiled programming languages. Tracking AD systems interpret the time while computing the gradient, while graph-based AD systems compile the computation of the gradient.","category":"page"},{"location":"lecture_08/lecture/#ChainRules","page":"Lecture","title":"ChainRules","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"From our discussions about AD systems so far we see that while the basic, engine, part is relatively straightforward, the devil is in writing the rules prescribing the computation of gradients. These rules are needed for every system whether it is graph based, tracking, or Wengert list based. ForwardDiff also needs a rule system, but rules are a bit different (as they are pushing the gradient forward rather than pulling it back). It is obviously a waste of effort for each AD system to have its own set of rules. Therefore the community (initiated by Catherine Frames White backed by Invenia) have started to work on a unified system to express differentiation rules, such that they can be shared between systems. So far, they are supported by Zygote.jl, Nabla.jl, ReverseDiff.jl and Diffractor.jl, suggesting that the unification approach is working (but not by Enzyme.jl).","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"The definition of reverse diff rules follows the idea we have nailed above (we refer readers interested in forward diff rules to official documentation).","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"ChainRules defines the reverse rules for function foo in a function rrule with the following signature","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"function rrule(::typeof(foo), args...; kwargs...)\n ...\n return y, pullback\nend","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"where","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"the first argument ::typeof(foo) allows to dispatch on the function for which the rules is written\nthe output of function foo(args...) is returned as the first argument\npullback(Δy) takes the gradient of upstream functions with respect to the output of foo(args) and returns it multiplied by the jacobian of the output of foo(args) with respect to parameters of the function itself (recall the function can have parameters, as it can be a closure or a functor), and with respect to the arguments.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"function pullback(Δy)\n ...\n return ∂self, ∂args...\nend","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Notice that key-word arguments are not differentiated. This is a design decision with the explanation that parametrize the function, but most of the time, they are not differentiable.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"ChainRules.jl provides support for lazy (delayed) computation using Thunk. Its argument is a function, which is not evaluated until unthunk is called. There is also a support to signal that gradient is zero using ZeroTangent (which can save valuable memory) or to signal that the gradient does not exist using NoTangent. ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"How can we use ChainRules to define rules for our AD system? Let's first observe the output","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"using ChainRulesCore, ChainRules\nr, g = rrule(*, rand(2,2), rand(2,2))\ng(r)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"With that, we can extend our AD system as follows","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"import Base: *, +, -\nfor f in [:*, :+, :-]\n @eval function $(f)(A::TrackedMatrix, B::AbstractMatrix)\n C, pullback = rrule($(f), value(A), B)\n C = track(C)\n push!(A.tape, (C, Δ -> pullback(Δ)[2]))\n C\n end\n\n @eval function $(f)(A::AbstractMatrix, B::TrackedMatrix)\n C, pullback = rrule($(f), A, value(B))\n C = track(C)\n push!(B.tape, (C, Δ -> pullback(Δ)[3]))\n C\n end\n\n @eval function $(f)(A::TrackedMatrix, B::TrackedMatrix)\n C, pullback = rrule($(f), value(A), value(B))\n C = track(C)\n push!(A.tape, (C, Δ -> pullback(Δ)[2]))\n push!(B.tape, (C, Δ -> pullback(Δ)[3]))\n C\n end\nend","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"and we need to modify our accum! code to unthunk if needed","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"function accum!(A::TrackedArray)\n isempty(A.tape) && return(A.deriv)\n A.deriv .= sum(unthunk(g(accum!(r))) for (r, g) in A.tape)\nend","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"A = rand(4,4)\nB = rand(4,4)\ngrad(A -> (A * B + msin(A))*B, A)[1]\ngradient(A -> sum(A * B + sin.(A)), A)[1]\ngrad(A -> A * B + msin(A), B)[1]\ngradient(A -> sum(A * B + sin.(A)), B)[1]","category":"page"},{"location":"lecture_08/lecture/#Source-to-source-AD-using-Wengert","page":"Lecture","title":"Source-to-source AD using Wengert","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Recall the compile stages of julia and look, how the lowered code for","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"f(x,y) = x*y + sin(x)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"looks like","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_lowered f(1.0, 1.0)\nCodeInfo(\n1 ─ %1 = x * y\n│ %2 = Main.sin(x)\n│ %3 = %1 + %2\n└── return %3\n)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"This form is particularly nice for automatic differentiation, as we have on the left hand side always a single variable, which means the compiler has provided us with a form, on which we know, how to apply AD rules.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"What if we somehow be able to talk to the compiler and get this form from him?","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"simplest viable implementation","category":"page"},{"location":"lecture_08/lecture/#Sources-for-this-lecture","page":"Lecture","title":"Sources for this lecture","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Mike Innes' diff-zoo\nWrite Your Own StS in One Day\nBuild your own AD with Umlaut\nZygote.jl Paper and Zygote.jl Internals\nKeno's Talk\nChris' Lecture\nAutomatic-Differentiation-Based-on-Computation-Graph","category":"page"},{"location":"lecture_08/lab/#Lab-08-Reverse-Mode-Differentiation","page":"Lab","title":"Lab 08 - Reverse Mode Differentiation","text":"","category":"section"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"(Image: descend)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"In the lecture you have seen how to implement forward-mode automatic differentiation (AD). Assume you want to find the derivative fracdfdx of the function fmathbb R^2 rightarrow mathbb R","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"f(x,y) = x*y + sin(x)\nnothing # hide","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"If we have rules for *, +, and sin we could simply seed the function with Dual(x,one(x)) and read out the derivative fracdfdx from the Dual that is returned by f. If we are also interested in the derivative fracdfdy we will have to run f again, this time seeding the second argument with Dual(y,one(y)). Hence, we have to evaluate f twice if we want derivatives w.r.t to both its arguments which means that forward differentiation scales as O(N) where N is the number of inputs to f.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"dfdx = f(Dual(x,one(x)), Dual(y,zero(y)))\ndfdy = f(Dual(x,zero(x)), Dual(y,one(y)))","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Reverse-mode AD can compute gradients of functions with many inputs and one output in one go. This is great because very often we want to optimize loss functions which are exactly that: Functions with many input variables and one loss output.","category":"page"},{"location":"lecture_08/lab/#Reverse-Mode-AD","page":"Lab","title":"Reverse Mode AD","text":"","category":"section"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"With functions fmathbb R^Nrightarrowmathbb R^M and gmathbb R^Lrightarrow mathbb R^N with an input vector bm x we can define the composition of f and g as","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"bm z = (f circ g)(bm x) qquad textwhere qquad bm y=g(bm x) qquad bm z = f(bm y)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"The multivariate chainrule reads","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"leftfracpartial z_ipartial x_jright_bm x =\n sum_k=1^N leftfracpartial z_ipartial y_kright_bm y\n leftfracpartial y_kpartial x_iright_bm x","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"If you want to read about where this comes from you can check here or here. It is essentially one row of the Jacobian matrix J. Note that in order to compute the derivative we always have to know the input to the respective function, because we can only compute the derivative at a specific point (denoted by the _x _ notation). For our example","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"z = f(xy) = xy + sin(x)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"with the sub-functions g(xy)=xy and h(x)=sin(x) we get","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"leftfrac dfdxright_xy\n = leftfrac dfdgright_g(xy)cdot leftfrac dgdxright_xy\n + leftfrac dfdhright_h(x)cdot leftfrac dhdxright_x\n = 1 cdot y _y + 1cdotcos(x)_x","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"You can see that, in order to implement reverse-mode AD we have to trace and remember all inputs to our intermediate functions during the forward pass such that we can compute their gradients during the backward pass. The simplest way of doing this is by dynamically building a computation graph which tracks how each input variable affects its output variables. The graph below represents the computation of our function f.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"z = x*y + sin(x)\n\n# as a Wengert list # Partial derivatives\na = x*y # da/dx = y; da/dy = x\nb = sin(x) # db/dx = cos(x)\nz = a + b # dz/da = 1; dz/db = 1","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"(Image: graph)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"In the graph you can see that the variable x can directly affect b and a. Hence, x has two children a and b. During the forward pass we build the graph, keeping track of which input affects which output. Additionally we include the corresponding local derivatives (which we can already compute). To implement a dynamically built graph we can introduce a new number type TrackedReal which has three fields:","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"data contains the value of this node in the computation graph as obtained in the forward pass.\ngrad is initialized to nothing and will later hold the accumulated gradients (the sum in the multivariate chain rule)\nchildren is a Dict that keeps track which output variables are affected by the current node and also stores the corresponding local derivatives fracpartial fpartial g_k.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"mutable struct TrackedReal{T<:Real}\n data::T\n grad::Union{Nothing,T}\n children::Dict\n # this field is only need for printing the graph. you can safely remove it.\n name::String\nend\n\ntrack(x::Real,name=\"\") = TrackedReal(x,nothing,Dict(),name)\n\nfunction Base.show(io::IO, x::TrackedReal)\n t = isempty(x.name) ? \"(tracked)\" : \"(tracked $(x.name))\"\n print(io, \"$(x.data) $t\")\nend","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"The backward pass is nothing more than the application of the chainrule. To compute the derivative. Assuming we know how to compute the local derivatives fracpartial fpartial g_k for simple functions such as +, *, and sin, we can write a simple function that implements the gradient accumulation from above via the chainrule","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"leftfracpartial fpartial x_iright_bm x =\n sum_k=1^N leftfracpartial fpartial g_kright_bm g(bm x)\n leftfracpartial g_kpartial x_iright_bm x","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"We just have to loop over all children, collect the local derivatives, and recurse:","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"function accum!(x::TrackedReal)\n if isnothing(x.grad)\n x.grad = sum(w*accum!(v) for (v,w) in x.children)\n end\n x.grad\nend\nnothing # hide","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"where w corresponds to fracpartial fpartial g_k and accum!(v) corresponds to fracpartial g_kpartial x_i. At this point we have already implemented the core functionality of our first reverse-mode AD! The only thing left to do is implement the reverse rules for basic functions. Via recursion the chainrule is applied until we arrive at the final output z. This final output has to be seeded (just like with forward-mode) with fracpartial zpartial z=1.","category":"page"},{"location":"lecture_08/lab/#Writing-Reverse-Rules","page":"Lab","title":"Writing Reverse Rules","text":"","category":"section"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Lets start by overloading the three functions +, *, and sin that we need to build our computation graph. First, we have to track the forward computation and then we register the output z as a child of its inputs by using z as a key in the dictionary of children. The corresponding value holds the derivatives, in the case of multiplication case we simply have","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"z = a cdot b","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"for which the derivatives are","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"fracpartial zpartial a=b qquad\nfracpartial zpartial b=a","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Knowing the derivatives of * at a given point we can write our reverse rule","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"function Base.:*(a::TrackedReal, b::TrackedReal)\n z = track(a.data * b.data, \"*\")\n a.children[z] = b.data # dz/da=b\n b.children[z] = a.data # dz/db=a\n z\nend","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Creating two tracked numbers and adding them results in","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"x = track(2.0)\ny = track(3.0)\nz = x*y\nx.children\ny.children","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Implement the two remaining rules for + and sin by overloading the appropriate methods like we did for *. First you have to compute the tracked forward pass, and then register the local derivatives in the children of your input variables. Remember to return the tracked result of the forward pass in the end.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"function Base.:+(a::TrackedReal{T}, b::TrackedReal{T}) where T\n z = track(a.data + b.data, \"+\")\n a.children[z] = one(T)\n b.children[z] = one(T)\n z\nend\n\nfunction Base.sin(x::TrackedReal)\n z = track(sin(x.data), \"sin\")\n x.children[z] = cos(x.data)\n z\nend","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_08/lab/#Forward-and-Backward-Pass","page":"Lab","title":"Forward & Backward Pass","text":"","category":"section"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"To visualize that with reverse-mode AD we really do save computation we can visualize the computation graph at different stages. We start with the forward pass and keep observing x","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"using AbstractTrees\nAbstractTrees.children(v::TrackedReal) = v.children |> keys |> collect\nfunction AbstractTrees.printnode(io::IO,v::TrackedReal)\n print(io,\"$(v.name) data: $(round(v.data,digits=2)) grad: $(v.grad)\")\nend","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"x = track(2.0,\"x\");\ny = track(3.0,\"y\");\na = x*y;\nprint_tree(x)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"We can see that we x now has one child a which has the value 2.0*3.0==6.0. All the gradients are still nothing. Computing another value that depends on x will add another child.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"b = sin(x)\nprint_tree(x)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"In the final step we compute z which does not mutate the children of x because it does not depend directly on it. The result z is added as a child to both a and b.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"z = a + b\nprint_tree(x)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"For the backward pass we have to seed the initial gradient value of z and call accum! on the variable that we are interested in.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"z.grad = 1.0\ndx = accum!(x)\ndx ≈ y.data + cos(x.data)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"By accumulating the gradients for x, the gradients in the sub-tree connected to x will be evaluated. The parts of the tree that are only connected to y stay untouched.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"print_tree(x)\nprint_tree(y)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"If we now accumulate the gradients over y we re-use the gradients that are already computed. In larger computations this will save us a lot of effort!","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"info: Info\nThis also means that we have to re-build the graph for every new set of inputs!","category":"page"},{"location":"lecture_08/lab/#Optimizing-2D-Functions","page":"Lab","title":"Optimizing 2D Functions","text":"","category":"section"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Implement a function gradient(f, args::Real...) which takes a function f and its corresponding arguments (as Real numbers) and outputs the corresponding gradients","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"function gradient(f, args::Real...)\n ts = track.(args)\n y = f(ts...)\n y.grad = 1.0\n accum!.(ts)\nend\nnothing # hide","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"f(x,y) = x*y + sin(x)\ngradient(f, 2.0, 3.0)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"As an example we can find a local minimum of the function g (slightly modified to show you that we can now actually do automatic differentiation).","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"g(x,y) = y*y + sin(x)\n\nusing Plots\ncolor_scheme = cgrad(:RdYlBu_5, rev=true)\ncontour(-4:0.1:4, -2:0.1:2, g, fill=true, c=color_scheme, xlabel=\"x\", ylabel=\"y\")","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"We can find a local minimum of g by starting at an initial point (x_0y_0) and taking small steps in the opposite direction of the gradient","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"beginalign\nx_i+1 = x_i - lambda fracpartial fpartial x_i \ny_i+1 = y_i - lambda fracpartial fpartial y_i\nendalign","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"where lambda is the learning rate that has to be tuned manually.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Implement a function descend performs one step of Gradient Descent (GD) on a function f with an arbitrary number of inputs. For GD you also have to specify the learning rate lambda so the function signature should look like this","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"descend(f::Function, λ::Real, args::Real...)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"function descend(f::Function, λ::Real, args::Real...)\n Δargs = gradient(f, args...)\n args .- λ .* Δargs\nend\nnothing # hide","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Running one descend step should result in two new inputs with a smaller output for g","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"g(1.0, 1.0)\n(x,y) = descend(g, 0.2, 1.0, 1.0)\ng(x,y)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"You can minimize a g starting from an initial value. Below is a code snippet that performs a number of descend steps on two different initial points and creates an animation of each step of the GD algorithm.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"function minimize(f::Function, args::T...; niters=20, λ=0.01) where T<:Real\n paths = ntuple(_->Vector{T}(undef,niters), length(args))\n for i in 1:niters\n args = descend(f, λ, args...)\n @info f(args...)\n for j in 1:length(args)\n paths[j][i] = args[j]\n end\n end\n paths\nend\n\nxs1, ys1 = minimize(g, 1.5, -2.4, λ=0.2, niters=34)\nxs2, ys2 = minimize(g, 1.8, -2.4, λ=0.2, niters=16)\n\np1 = contour(-4:0.1:4, -2:0.1:2, g, fill=true, c=color_scheme, xlabel=\"x\", ylabel=\"y\")\nscatter!(p1, [xs1[1]], [ys1[1]], mc=:black, marker=:star, ms=7, label=\"Minimum\")\nscatter!(p1, [xs2[1]], [ys2[1]], mc=:black, marker=:star, ms=7, label=false)\nscatter!(p1, [-π/2], [0], mc=:red, marker=:star, ms=7, label=\"Initial Point\")\nscatter!(p1, xs1[1:1], ys1[1:1], mc=:black, label=\"GD Path\", xlims=(-4,4), ylims=(-2,2))\n\n@gif for i in 1:max(length(xs1), length(xs2))\n if i <= length(xs1)\n scatter!(p1, xs1[1:i], ys1[1:i], mc=:black, lw=3, xlims=(-4,4), ylims=(-2,2), label=false)\n end\n if i <= length(xs2)\n scatter!(p1, xs2[1:i], ys2[1:i], mc=:black, lw=3, label=false)\n end\n p1\nend","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"(Image: descend)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"At this point you can move to the homework of this lab. If you want to know how to generalize this simple reverse AD to work with functions that operate on Arrays, feel free to continue with the remaining volutary part of the lab.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"","category":"page"},{"location":"lecture_08/lab/#Naively-Vectorized-Reverse-AD","page":"Lab","title":"Naively Vectorized Reverse AD","text":"","category":"section"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"A naive solution to use our TrackedReal number type to differentiate functions that operate on vectors is to just use Array{<:TrackedReal}. Unfortunately, this means that we have to replace the fast BLAS matrix operations with our own matrix multiplication methods that know how to deal with TrackedReals. This results in large performance hits and your task during the rest of the lab is to implement a smarter solution to this problem.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"using LinearAlgebra\nBase.zero(::TrackedReal{T}) where T = TrackedReal(zero(T))\nLinearAlgebra.adjoint(x::TrackedReal) = x\ntrack(x::Array) = track.(x)\naccum!(xs::Array{<:TrackedReal}) = accum!.(xs)\n\nconst VecTracked = AbstractVector{<:TrackedReal}\nconst MatTracked = AbstractMatrix{<:TrackedReal}\n\nLinearAlgebra.dot(xs::VecTracked, ys::VecTracked) = mapreduce(*, +, xs, ys)\nBase.:*(X::MatTracked, y::VecTracked) = map(x->dot(x,y), eachrow(X))\nBase.:*(X::MatTracked, Y::MatTracked) = mapreduce(y->X*y, hcat, eachcol(Y))\nBase.sum(xs::AbstractArray{<:TrackedReal}) = reduce(+,xs)\n\nfunction reset!(x::TrackedReal)\n x.grad = nothing\n reset!.(keys(x.children))\n x.children = Dict()\nend\n\nX = rand(2,3)\nY = rand(3,2)\n\nfunction run()\n Xv = track(X)\n Yv = track(Y)\n z = sum(Xv * Yv)\n z.grad = 1.0\n accum!(Yv)\nend","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"julia> using BenchmarkTools\njulia> @benchmark run()\nBenchmarkTools.Trial: 10000 samples with 1 evaluation.\n Range (min … max): 44.838 μs … 8.404 ms ┊ GC (min … max): 0.00% … 98.78%\n Time (median): 48.680 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 53.048 μs ± 142.403 μs ┊ GC (mean ± σ): 4.61% ± 1.71%\n\n ▃▆█▃ \n ▂▁▁▂▂▃▆████▇▅▄▄▄▄▄▅▅▄▄▄▄▃▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂ ▃\n 44.8 μs Histogram: frequency by time 66.7 μs <\n\n Memory estimate: 26.95 KiB, allocs estimate: 392.","category":"page"},{"location":"lecture_08/lab/#Reverse-AD-with-TrackedArrays","page":"Lab","title":"Reverse AD with TrackedArrays","text":"","category":"section"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"To make use of the much faster BLAS methods we have to implement a custom array type which will offload the heavy matrix multiplications to the normal matrix methods. Start with a fresh REPL and possibly a new file that only contains the definition of our TrackedReal:","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"mutable struct TrackedReal{T<:Real}\n data::T\n grad::Union{Nothing,T}\n children::Dict\nend\n\ntrack(x::Real) = TrackedReal(x, nothing, Dict())\nnothing # hide","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Define a new TrackedArray type which subtypes and AbstractArray{T,N} and contains the three fields: data, grad, and children. Which type should grad have?","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Additionally define track(x::Array), and forward size, length, and eltype to x.data (maybe via metaprogrammming? ;).","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"mutable struct TrackedArray{T,N,A<:AbstractArray{T,N}} <: AbstractArray{T,N}\n data::A\n grad::Union{Nothing,A}\n children::Dict\nend\n\ntrack(x::Array) = TrackedArray(x, nothing, Dict())\ntrack(x::Union{TrackedArray,TrackedReal}) = x\n\nfor f in [:size, :length, :eltype]\n\teval(:(Base.$(f)(x::TrackedArray, args...) = $(f)(x.data, args...)))\nend\n\n# only needed for hashing in the children dict...\nBase.getindex(x::TrackedArray, args...) = getindex(x.data,args...)\n\n# pretty print TrackedArray\nBase.show(io::IO, x::TrackedArray) = print(io, \"Tracked $(x.data)\")\nBase.print_array(io::IO, x::TrackedArray) = Base.print_array(io, x.data)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Creating a TrackedArray should work like this:","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"track(rand(2,2))","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"function accum!(x::Union{TrackedReal,TrackedArray})\n if isnothing(x.grad)\n x.grad = sum(λ(accum!(Δ)) for (Δ,λ) in x.children)\n end\n x.grad\nend","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"To implement the first rule for * i.e. matrix multiplication we would first have to derive it. In the case of general matrix multiplication (which is a function (R^Ntimes M R^Mtimes L) rightarrow R^Ntimes L) we are not dealing with simple derivatives anymore, but with a so-called pullback which takes a wobble in the output space R^Ntimes L and returns a wiggle in the input space (either R^Ntimes M or R^Mtimes L).","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Luckily ChainRules.jl has a nice guide on how to derive array rules, so we will only state the solution for the reverse rule such that you can implement it. They read:","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"bar A = barOmega B^T qquad bar B = A^TbarOmega","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Where barOmega is the given output wobble, which in the simplest case can be the seeded value of the final node. The crucial problem to note here is that the two rules rely in barOmega being multiplied from different sides. This information would be lost if would just store B^T as the pullback for A. Hence we will store our pullbacks as closures:","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Ω̄ -> Ω̄ * B'\nΩ̄ -> A' * Ω̄","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Define the pullback for matrix multiplication i.e. Base.:*(A::TrackedArray,B::TrackedArray) by computing the primal and storing the partials as closures.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"function Base.:*(X::TrackedArray, Y::TrackedArray)\n Z = track(X.data * Y.data)\n X.children[Z] = Δ -> Δ * Y.data'\n Y.children[Z] = Δ -> X.data' * Δ\n Z\nend","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"X = rand(2,3) |> track\nY = rand(3,2) |> track\nZ = X*Y\nf = X.children[Z]\nΩ̄ = ones(size(Z)...)\nf(Ω̄)\nΩ̄*Y.data'","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Implement rules for sum, +, -, and abs2.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"function Base.sum(x::TrackedArray)\n z = track(sum(x.data))\n x.children[z] = Δ -> Δ*ones(eltype(x), size(x)...)\n z\nend\n\nfunction Base.:+(X::TrackedArray, Y::TrackedArray)\n Z = track(X.data + Y.data)\n X.children[Z] = Δ -> Δ\n Y.children[Z] = Δ -> Δ\n Z\nend\n\nfunction Base.:-(X::TrackedArray, Y::TrackedArray)\n Z = track(X.data - Y.data)\n X.children[Z] = Δ -> Δ\n Y.children[Z] = Δ -> -Δ\n Z\nend\n\nfunction Base.abs2(x::TrackedArray)\n y = track(abs2.(x.data))\n x.children[y] = Δ -> Δ .* 2x.data\n y\nend","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"X = rand(2,3)\nY = rand(3,2)\nfunction run()\n Xv = track(X)\n Yv = track(Y)\n z = sum(Xv * Yv)\n z.grad = 1.0\n accum!(Yv)\nend\nnothing # hide","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"julia> using BenchmarkTools\njulia> @benchmark run()\nBenchmarkTools.Trial: 10000 samples with 6 evaluations.\n Range (min … max): 5.797 μs … 1.618 ms ┊ GC (min … max): 0.00% … 98.97%\n Time (median): 6.530 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 7.163 μs ± 22.609 μs ┊ GC (mean ± σ): 4.42% ± 1.40%\n\n ▆█▇▇▇▆▅▄▃▃▂▂▂▁▁ ▁▁ ▂\n █████████████████████▇▇▇▆▆▅▅▅▅▆▅▄▅▅▄▁▃▁▁▄▁▃▁▁▁▃▃▄▁▁▁▄▁▃▁▅▄ █\n 5.8 μs Histogram: log(frequency) by time 15.8 μs <\n\n Memory estimate: 3.08 KiB, allocs estimate: 31.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Even for this tiny example we are already 10 times faster than with the naively vectorized approach!","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"In order to implement a full neural network we need two more rules. One for the non-linearity and one for concatentation of individual training points to a batch.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"σ(x::Real) = 1/(1+exp(-x))\nσ(x::AbstractArray) = σ.(x)\nfunction σ(x::TrackedArray)\n z = track(σ(x.data))\n d = z.data\n x.children[z] = Δ -> Δ .* d .* (1 .- d)\n z\nend\n\nfunction Base.hcat(xs::TrackedArray...)\n y = track(hcat(data.(xs)...))\n stops = cumsum([size(x,2) for x in xs])\n starts = vcat([1], stops[1:end-1] .+ 1)\n for (start,stop,x) in zip(starts,stops,xs)\n x.children[y] = function (Δ)\n δ = if ndims(x) == 1\n Δ[:,start]\n else\n ds = map(_ -> :, size(x)) |> Base.tail |> Base.tail\n Δ[:, start:stop, ds...]\n end\n δ\n end\n end\n y\nend","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"You can see a full implementation of our tracing based AD here and a simple implementation of a Neural Network that can learn an approximation to the function g here. Running the latter script will produce an animation that shows how the network is learning.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"(Image: anim)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"This lab is heavily inspired by Rufflewind","category":"page"},{"location":"lecture_03/hw/#Homework-3","page":"Homework","title":"Homework 3","text":"","category":"section"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"In this homework we will implement a function find_food and practice the use of closures. The solution of lab 3 can be found here. You can use this file and add the code that you write for the homework to it.","category":"page"},{"location":"lecture_03/hw/#How-to-submit?","page":"Homework","title":"How to submit?","text":"","category":"section"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"Put all your code (including your or the provided solution of lab 2) in a script named hw.jl. Zip only this file (not its parent folder) and upload it to BRUTE.","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"projdir = dirname(Base.active_project())\ninclude(joinpath(projdir,\"src\",\"lecture_03\",\"Lab03Ecosystem.jl\"))\n\nfunction find_food(a::Animal, w::World)\n as = filter(x -> eats(a,x), w.agents |> values |> collect)\n isempty(as) ? nothing : rand(as)\nend\n\neats(::Animal{Sheep},g::Plant{Grass}) = g.size > 0\neats(::Animal{Wolf},::Animal{Sheep}) = true\neats(::Agent,::Agent) = false\n\nfunction every_nth(f::Function, n::Int)\n i = 1\n function callback(args...)\n # display(i) # comment this out to see out the counter increases\n if i == n\n f(args...)\n i = 1\n else\n i += 1\n end\n end\nend\n\nnothing # hide","category":"page"},{"location":"lecture_03/hw/#Agents-looking-for-food","page":"Homework","title":"Agents looking for food","text":"","category":"section"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"
\n
Homework:
\n
","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"Implement a method find_food(a::Animal, w::World) returns one randomly chosen agent from all w.agents that can be eaten by a or nothing if no food could be found. This means that if e.g. the animal is a Wolf you have to return one random Sheep, etc.","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"Hint: You can write a general find_food method for all animals and move the parts that are specific to the concrete animal types to a separate function. E.g. you could define a function eats(::Animal{Wolf}, ::Animal{Sheep}) = true, etc.","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"You can check your solution with the public test:","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"sheep = Sheep(1,pf=1.0)\nworld = World([Grass(2), sheep])\nfind_food(sheep, world) isa Plant{Grass}","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"
","category":"page"},{"location":"lecture_03/hw/#Callbacks-and-Closures","page":"Homework","title":"Callbacks & Closures","text":"","category":"section"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"
\n
Homework:
\n
","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"Implement a function every_nth(f::Function,n::Int) that takes an inner function f and uses a closure to construct an outer function g that only calls f every nth call to g. For example, if n=3 the inner function f be called at the 3rd, 6th, 9th ... call to g (not at the 1st, 2nd, 4th, 5th, 7th... call).","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"Hint: You can use splatting via ... to pass on an unknown number of arguments from the outer to the inner function.","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"
","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"You can use every_nth to log (or save) the agent count only every couple of steps of your simulation. Using every_nth will look like this:","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"w = World([Sheep(1), Grass(2), Wolf(3)])\n# `@info agent_count(w)` is executed only every 3rd call to logcb(w)\nlogcb = every_nth(w->(@info agent_count(w)), 3);\n\nlogcb(w); # x->(@info agent_count(w)) is not called\nlogcb(w); # x->(@info agent_count(w)) is not called\nlogcb(w); # x->(@info agent_count(w)) *is* called","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"","category":"page"},{"location":"lecture_05/lecture/#perf_lecture","page":"Lecture","title":"Benchmarking, profiling, and performance gotchas","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"This class is a short introduction to writing a performant code. As such, we cover","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"how to identify weak spots in the code\nhow to properly benchmark\ncommon performance anti-patterns\nJulia's \"performance gotchas\", by which we mean performance problems specific for Julia (typical caused by the lack of understanding of Julia or by a errors in conversion from script to functions)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Though recall the most important rule of thumb: Never optimize code from the very beginning. A much more productive workflow is ","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"write the code that is idiomatic and easy to understand\nmeticulously cover the code with unit test, such that you know that the optimized code works the same as the original\noptimize the code","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Premature optimization frequently backfires, because:","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"you might end-up optimizing wrong thing, i.e. you will not optimize performance bottleneck, but something very different\noptimized code can be difficult to read and reason about, which means it is more difficult to make it right.","category":"page"},{"location":"lecture_05/lecture/#Optimize-for-your-mode-of-operation","page":"Lecture","title":"Optimize for your mode of operation","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Let's for fun measure a difference in computation of a simple polynomial over elements of arrays between numpy, jax, default Julia, and Julia with LoopVectorization library. ","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"import numpy as np\nimport jax\nfrom jax import jit\nimport jax.numpy as jnp\njax.config.update(\"jax_enable_x64\", True)\n\n@jit\ndef f(x):\n return 3*x**3 + 2*x**2 + x + 1\n\ndef g(x):\n return 3*x**3 + 2*x**2 + x + 1\n\nx = np.random.rand(10)\nf(x)\nx = random.uniform(key, shape=(10,), dtype=jnp.float64)\ng(x)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function f(x)\n @. 3*x^3 + 2*x^2 + x + 1\nend\n\nusing LoopVectorization\nfunction f_turbo(x)\n @turbo @. 3*x^3 + 2*x^2 + x + 1\nend\n\nfunction f_tturbo(x)\n @tturbo @. 3*x^3 + 2*x^2 + x + 1\nend\n\nx = rand(10)\nf(x)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"A complete implementations can be found here: Julia and Python. Julia should be executed with multithreaded support, in the case of below image it used four threads on MacBook PRO with M1 processor with four performant and four energy efficient cores. Below figure shows the minimum execution time with respect to the","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"(Image: figure)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"It frequently happens that Julia newbies asks on forum that their code in Julia is slow in comparison to the same code in Python (numpy). Most of the time, they make trivial mistakes and it is very educative to go over their mistakes","category":"page"},{"location":"lecture_05/lecture/#Numpy-10x-faster-than-julia-what-am-i-doing-wrong?-(solved-julia-faster-now)-[1]","page":"Lecture","title":"Numpy 10x faster than julia what am i doing wrong? (solved julia faster now) [1]","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"[1]: Adapted from Julia's discourse thread","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function f(p) # line 1 \n t0,t1 = p # line 2\n m0 = [[cos(t0) - 1im*sin(t0) 0]; [0 cos(t0) + 1im*sin(t0)]] # line 3\n m1 = [[cos(t1) - 1im*sin(t1) 0]; [0 cos(t1) + 1im*sin(t1)]] # line 4\n r = m1*m0*[1. ; 0.] # line 5\n return abs(r[1])^2 # line 6\nend\n\nfunction g(p,n)\n return [f(p[:,i]) for i=1:n]\nend\n\ng(rand(2,3),3) # call to force jit compilation\n\nn = 10^6\np = 2*pi*rand(2,n)\n\n@elapsed g(p,n)\n","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Let's first use Profiler to identify, where the function spends most time.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"note: Note\nJulia's built-in profilerJulia's built-in profiler is part of the standard library in the Profile module implementing a fairly standard sampling based profiler. It a nutshell it asks at regular intervals, where the code execution is currently and marks it and collects this information in some statistics. This allows us to analyze, where these \"probes\" have occurred most of the time which implies those parts are those, where the execution of your function spends most of the time. As such, the profiler has two \"controls\", which is the delay between two consecutive probes and the maximum number of probes n (if the profile code takes a long time, you might need to increase it).using Profile\nProfile.init(; n = 989680, delay = 0.001))\n@profile g(p,n)\nProfile.clear()\n@profile g(p,n)Making sense of profiler's outputThe default Profile.print function shows the call-tree with count, how many times the probe occurred in each function sorted from the most to least. The output is a little bit difficult to read and orient in, therefore there are some visualization options.What are our options?ProfileView is the workhorse with a GTK based API and therefore recommended for those with working GTK\nProfileSVG is the ProfileView with the output exported in SVG format, which is viewed by most browser (it is also very convenient for sharing with others)\nPProf.jl is a front-end to Google's PProf profile viewer https://github.com/JuliaPerf/PProf.jl\nStatProfilerHTML https://github.com/tkluck/StatProfilerHTML.jlBy personal opinion I mostly use ProfileView (or ProfileSVG) as it indicates places of potential type instability, which as will be seen later is very useful feature. Profiling caveatsThe same function, but with keyword arguments, can be used to change these settings, however these settings are system dependent. For example on Windows, there is a known issue that does not allow to sample faster than at 0.003s and even on Linux based system this may not do much. There are some further caveat specific to Julia:When running profile from REPL, it is usually dominated by the interactive part which spawns the task and waits for it's completion.\nCode has to be run before profiling in order to filter out all the type inference and interpretation stuff. (Unless compilation is what we want to profile.)\nWhen the execution time is short, the sampling may be insufficient -> run multiple times.We will use ProfileSVG for its simplicity (especially installation). It shows the statistics in form of a flame graph which read as follows: , where . The hierarchy is expressed as functions on the bottom calls functions on the top. reads as follows:each function is represented by a horizontal bar\nfunction in the bottom calls functions above\nthe width of the bar corresponds to time spent in the function\nred colored bars indicate type instabilities\nfunctions in bottom bars calls functions on top of upper barsFunction name contains location in files and particular line number called. GTK version is even \"clickable\" and opens the file in default editor.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Let's use the profiler on the above function g to find potential weak spots","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"using Profile, ProfileSVG\nProfile.clear()\n@profile g(p, n)\nProfileSVG.save(\"profile.svg\")","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"The output can be seen here","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"We can see that the function is type stable and 2/3 of the time is spent in lines 3 and 4, which allocates arrays","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"[[cos(t0) - 1im*sin(t0) 0]; \n [0 cos(t0) + 1im*sin(t0)]]","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"and","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"[[cos(t1) - 1im*sin(t1) 0]; \n [0 cos(t1) + 1im*sin(t1)]]","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Since version 1.8, Julia offers a memory profiler, which helps to identify parts of the code allocating memory on heap. Unfortunately, ProfileSVG does not currently visualize its output, hence we are going to use PProf.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"using Profile, PProf\nProfile.Allocs.@profile g(p,n)\nPProf.Allocs.pprof(Profile.Allocs.fetch(), from_c=false)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"PProf by default shows outputs in call graph (how to read it can be found here), but also supports the flamegraph (fortunately). Investigating the output we found that most allocations are caused by concatenation of arrays on lines 3 and 4.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Scrutinizing the function f, we see that in every call, it has to allocate arrays m0 and m1 on the heap. The allocation on heap is expensive, because it might require interaction with the operating system and it potentially stress garbage collector. Can we avoid it? Repeated allocation can be frequently avoided by:","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"preallocating arrays (if the arrays are of the fixed dimensions)\nor allocating objects on stack, which does not involve interaction with OS (but can be used in limited cases.)","category":"page"},{"location":"lecture_05/lecture/#Adding-preallocation","page":"Lecture","title":"Adding preallocation","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function f!(m0, m1, p, u) \t\t\t\t\t\t\t\t\t\t# line 1 \n t0,t1 = p \t\t\t\t\t\t\t\t\t\t\t\t\t\t# line 2\n m0[1,1] = cos(t0) - 1im*sin(t0)\t\t\t\t\t\t\t\t\t# line 3\n m0[2,2] = cos(t0) + 1im*sin(t0)\t\t\t\t\t\t\t\t\t# line 4\n m1[1,1] = cos(t1) - 1im*sin(t1)\t\t\t\t\t\t\t\t\t# line 5\n m1[2,2] = cos(t1) + 1im*sin(t1)\t\t\t\t\t\t\t\t\t# line 6\n r = m1*m0*u \t\t\t\t\t\t\t\t\t\t\t\t\t# line 7\n return abs(r[1])^2 \t\t\t\t\t\t\t\t\t\t\t\t# line 8\nend\n\nfunction g2(p,n)\n u = [1. ; 0.]\n m0 = [[cos(p[1]) - 1im*sin(p[1]) 0]; [0 cos(p[1]) + 1im*sin(p[1])]]\t# line 3\n m1 = [[cos(p[2]) - 1im*sin(p[2]) 0]; [0 cos(p[2]) + 1im*sin(p[2])]]\n return [f!(m0, m1, p[:,i], u) for i=1:n]\nend","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"note: Note\nBenchmarkingThe simplest benchmarking can be as simple as writing repetitions = 100\nt₀ = time()\nfor in 1:100\n\tg(p, n)\nend\n(time() - t₀) / n where we add repetitions to calibrate for background processes that can step in the precise measurements (recall that your program is not allone). Writing the above for benchmarking is utterly boring. Moreover, you might want to automatically determine the number of repetitions (the shorter time the more repetitions you want), take care of compilation of the function outside measured loop, you might want to have more informative output, for example median, mean, and maximum time of execution, information about number of allocation, time spent in garbage collector, etc. This is in nutshell what BenchmarkTools.jl offers, which we consider an essential tool for anyone interesting in tuning its code.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"We will using macro @benchmark from BenchmarkTools.jl to observe the speedup we will get between g and g2.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"using BenchmarkTools\n\njulia> @benchmark g(p,n)\nBenchmarkTools.Trial: 5 samples with 1 evaluation.\n Range (min … max): 1.168 s … 1.199 s ┊ GC (min … max): 11.57% … 13.27%\n Time (median): 1.188 s ┊ GC (median): 11.91%\n Time (mean ± σ): 1.183 s ± 13.708 ms ┊ GC (mean ± σ): 12.10% ± 0.85%\n\n █ █ █ █ █\n █▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█ ▁\n 1.17 s Histogram: frequency by time 1.2 s <\n\n Memory estimate: 1.57 GiB, allocs estimate: 23000002.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @benchmark g2(p,n)\nBenchmarkTools.Trial: 11 samples with 1 evaluation.\n Range (min … max): 413.167 ms … 764.393 ms ┊ GC (min … max): 6.50% … 43.76%\n Time (median): 426.728 ms ┊ GC (median): 6.95%\n Time (mean ± σ): 460.688 ms ± 102.776 ms ┊ GC (mean ± σ): 12.85% ± 11.04%\n\n ▃█ █\n ██▇█▁▁▁▁▁▁▁▁▇▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▇ ▁\n 413 ms Histogram: frequency by time 764 ms <\n\n Memory estimate: 450.14 MiB, allocs estimate: 4000021.\n","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"We can see that we have approximately 3-fold improvement.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Let's profile again, not forgetting to use Profile.clear() to clear already stored probes.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Profile.clear()\n@profile g2(p,n)\nProfileSVG.save(\"/tmp/profile2.svg\")","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"What the profiler tells is now (clear here to see the output)? \t- we spend a lot of time in similar in matmul, which is again an allocation of results for storing output of multiplication on line 7 matrix r. \t- the trigonometric operations on line 3-6 are very costly \t- Slicing p always allocates a new array and performs a deep copy.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Let's get rid of memory allocations at the expense of the code clarity","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"using LinearAlgebra\n@inline function initm!(m, t)\n st, ct = sincos(t) \n @inbounds m[1,1] = Complex(ct, -st)\t\t\t\t\t\t\t\t\n @inbounds m[2,2] = Complex(ct, st) \t\t\t\t\t\t\t\t\nend\n\nfunction f1!(r1, r2, m0, m1, t0, t1, u) \t\t\t\t\t\n initm!(m0, t0)\n initm!(m1, t1)\n mul!(r1, m0, u)\n mul!(r2, m1, r1)\n return @inbounds abs(@inbounds r2[1])^2\nend\n\nfunction g3(p,n)\n u = [1. ; 0.]\n m0 = [cos(p[1]) - 1im*sin(p[1]) 0; 0 cos(p[1]) + 1im*sin(p[1])]\n m1 = [cos(p[2]) - 1im*sin(p[2]) 0; 0 cos(p[2]) + 1im*sin(p[2])]\n r1 = m0*u\n r2 = m1*r1\n return [f1!(r1, r2, m0, m1, p[1,i], p[2,i], u) for i=1:n]\nend","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @benchmark g3(p,n)\n Range (min … max): 193.922 ms … 200.234 ms ┊ GC (min … max): 0.00% … 1.67%\n Time (median): 195.335 ms ┊ GC (median): 0.00%\n Time (mean ± σ): 196.003 ms ± 1.840 ms ┊ GC (mean ± σ): 0.26% ± 0.61%\n\n █▁ ▁ ██▁█▁ ▁█ ▁ ▁ ▁ ▁ ▁ ▁ ▁▁ ▁ ▁ ▁\n ██▁▁█▁█████▁▁██▁█▁█▁█▁▁▁▁▁▁▁▁█▁▁▁█▁▁▁▁▁█▁▁▁██▁▁▁█▁▁█▁▁▁▁▁▁▁▁█ ▁\n 194 ms Histogram: frequency by time 200 ms <\n\n Memory estimate: 7.63 MiB, allocs estimate: 24.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Notice that now, we are about six times faster than the first solution, albeit passing the preallocated arrays is getting messy. Also notice that we spent a very little time in garbage collector. Running the profiler, ","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Profile.clear()\n@profile g3(p,n)\nProfileSVG.save(\"/tmp/profile3.svg\")","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"we see here that there is a very little what we can do now. May-be, remove bounds checks (more on this later) and make the code a bit nicer.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Let's look at solution from a Discourse","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"using StaticArrays, BenchmarkTools\n\nfunction f(t0,t1)\n cis0, cis1 = cis(t0), cis(t1)\n m0 = @SMatrix [ conj(cis0) 0 ; 0 cis0]\n m1 = @SMatrix [ conj(cis1) 0 ; 0 cis1]\n r = m1 * (m0 * @SVector [1. , 0.])\n return abs2(r[1])\nend\n\ng(p) = [f(p[1,i],p[2,i]) for i in axes(p,2)]","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @benchmark g(p)\n Range (min … max): 36.076 ms … 43.657 ms ┊ GC (min … max): 0.00% … 9.96%\n Time (median): 37.948 ms ┊ GC (median): 0.00%\n Time (mean ± σ): 38.441 ms ± 1.834 ms ┊ GC (mean ± σ): 1.55% ± 3.60%\n\n █▃▅ ▅▂ ▂\n ▅▇▇███████▅███▄████▅▅▅▅▄▇▅▇▄▇▄▁▄▇▄▄▅▁▄▁▄▁▄▅▅▁▁▅▁▁▅▄▅▄▁▁▁▁▁▅ ▄\n 36.1 ms Histogram: frequency by time 43.4 ms <\n\n Memory estimate: 7.63 MiB, allocs estimate: 2.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"We can see that it is six-times faster than ours while also being much nicer to read and having almost no allocations. Where is the catch? It uses StaticArrays which offers linear algebra primitices performant for vectors and matrices of small size. They are allocated on stack, therefore there is no pressure of GarbageCollector and the type is specialized on size of matrices (unlike regular matrices) works on arrays of an sizes. This allows the compiler to perform further optimizations like unrolling loops, etc.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"What we have learned so far?","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Profiler is extremely useful in identifying functions, where your code spends most time.\nMemory allocation (on heap to be specific) can be very bad for the performance. We can generally avoided by pre-allocation (if possible) or allocating on the stack (Julia offers increasingly larger number of primitives for hits. We have already seen StaticArrays, DataFrames now offers for example String3, String7, String15, String31).\nBenchmarking is useful for comparison of solutions","category":"page"},{"location":"lecture_05/lecture/#Replacing-deep-copies-with-shallow-copies-(use-view-if-possible)","page":"Lecture","title":"Replacing deep copies with shallow copies (use view if possible)","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Let's look at the following function computing mean of a columns","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function cmean(x::AbstractMatrix{T}) where {T}\n\to = zeros(T, size(x,1))\n\tfor i in axes(x, 2)\n\t\to .+= x[:,i]\t\t\t\t\t\t\t\t# line 4\n\tend\n\tn = size(x, 2)\n\tn > 0 ? o ./ n : o \nend\nx = randn(2, 10000)\n","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"@benchmark cmean(x)\nBenchmarkTools.Trial: 10000 samples with 1 evaluation.\n Range (min … max): 371.018 μs … 3.291 ms ┊ GC (min … max): 0.00% … 83.30%\n Time (median): 419.182 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 482.785 μs ± 331.939 μs ┊ GC (mean ± σ): 9.91% ± 12.02%\n\n ▃█▄▃▃▂▁ ▁\n ████████▇▆▅▃▁▁▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▃▇██ █\n 371 μs Histogram: log(frequency) by time 2.65 ms <\n\n Memory estimate: 937.59 KiB, allocs estimate: 10001.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"What we see that function is performing more than 10000 allocations. They come from x[:,i] which allocates a new memory and copies the content. In this case, this is completely unnecessary, as the content of the array x is never modified. We can avoid it by creating a view into an x, which you can imagine as a pointer to x which automatically adjust the bounds. Views can be constructed either using a function call view(x, axes...) or using a convenience macro @view which turns the usual notation x[...] to view(x, ...)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function view_cmean(x::AbstractMatrix{T}) where {T}\n\to = zeros(T, size(x,1))\n\tfor i in axes(x, 2)\n\t\to .+= @view x[:,i]\n\tend\n\tn = size(x,2)\n\tn > 0 ? o ./ n : o \nend","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"We obtain instantly a 10-fold speedup","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @benchmark view_cmean(x)\nBenchmarkTools.Trial: 10000 samples with 1 evaluation.\n Range (min … max): 36.802 μs … 166.260 μs ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 41.676 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 42.936 μs ± 9.921 μs ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n ▂ █▆█▆▂ ▁▁ ▁ ▁ ▂\n █▄█████████▇▅██▆█▆██▆▆▇▆▆▆▆▇▆▅▆▆▅▅▁▅▅▆▇▆▆▆▆▄▃▆▆▆▄▆▄▅▅▄▆▅▆▅▄▆ █\n 36.8 μs Histogram: log(frequency) by time 97.8 μs <\n\n Memory estimate: 96 bytes, allocs estimate: 1.","category":"page"},{"location":"lecture_05/lecture/#Traverse-arrays-in-the-right-order","page":"Lecture","title":"Traverse arrays in the right order","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Let's now compute rowmean using the function similar to cmean and since we have learnt from the above, we use the view to have non-allocating version","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function rmean(x::AbstractMatrix{T}) where {T}\n\to = zeros(T, size(x,2))\n\tfor i in axes(x, 1)\n\t\to .+= @view x[i,:]\n\tend\n\tn = size(x,1)\n\tn > 0 ? o ./ n : o \nend\nx = randn(10000, 2)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"x = randn(10000, 2)\n@benchmark rmean(x)\nBenchmarkTools.Trial: 10000 samples with 1 evaluation.\n Range (min … max): 44.165 μs … 194.395 μs ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 46.654 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 48.544 μs ± 10.940 μs ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n █▆█▇▄▁ ▁ ▂\n ██████▇▇▇▇▆▇▅██▇█▇█▇▆▅▄▄▅▅▄▄▄▄▂▄▅▆▅▅▅▆▅▅▅▆▄▆▄▄▅▅▄▅▄▄▅▅▅▅▄▄▃▅ █\n 44.2 μs Histogram: log(frequency) by time 108 μs <\n\n Memory estimate: 192 bytes, allocs estimate: 2.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"The above seems OK and the speed is comparable to our tuned cmean. But, can we actually do better? We have to realize that when we are accessing slices in the matrix x, they are not aligned in the memory. Recall that Julia is column major (like Fortran and unlike C and Python), which means that consecutive arrays of memory are along columns. i.e for a matrix with n rows and m columns they are aligned as ","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"1 | n + 1 | 2n + 1 | ⋯ | (m-1)n + 1\n2 | n + 2 | 2n + 2 | ⋯ | (m-1)n + 2\n3 | n + 3 | 2n + 3 | ⋯ | (m-1)n + 3\n⋮ | ⋮ | ⋮ | ⋯ | ⋮ \nn | 2n | 3n | ⋯ | mn","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"accessing non-consecutively is really bad for cache, as we have to load the memory into a cache line and use a single entry (in case of Float64 it is 8 bytes) out of it, discard it and load another one. If cache line has length 32 bytes, then we are wasting remaining 24 bytes. Therefore, we rewrite rmean to access the memory in consecutive blocks as follows, where we essentially sum the matrix column by columns.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function aligned_rmean(x::AbstractMatrix{T}) where {T}\n\to = zeros(T, size(x,2))\n\tfor i in axes(x, 2)\n\t\to[i] = sum(@view x[:, i])\n\tend\n\tn = size(x, 1)\n\tn > 0 ? o ./ n : o \nend\n\naligned_rmean(x) ≈ rmean(x)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @benchmark aligned_rmean(x)\nBenchmarkTools.Trial: 10000 samples with 10 evaluations.\n Range (min … max): 1.988 μs … 11.797 μs ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 2.041 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 2.167 μs ± 568.616 ns ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n █▇▄▂▂▁▁ ▁ ▂▁ ▂\n ██████████████▅▅▃▁▁▁▁▁▄▅▄▁▅▆▆▆▇▇▆▆▆▆▅▃▅▅▄▅▅▄▄▄▃▃▁▁▁▄▁▁▄▃▄▃▆ █\n 1.99 μs Histogram: log(frequency) by time 5.57 μs <\n\n Memory estimate: 192 bytes, allocs estimate: 2.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Running the benchmark shows that we have about 20x speedup and we are on par with Julia's built-in functions.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Remark tempting it might be, there is actually nothing we can do to speed-up the cmean function. This trouble is inherent to the processor design and you should be careful how you align things in the memory, such that it is performant in your project","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Detecting this type of inefficiencies is generally difficult, and requires processor assisted measurement. LIKWID.jl is a wrapper for a LIKWID library providing various processor level statistics, like throughput, cache misses","category":"page"},{"location":"lecture_05/lecture/#Type-stability","page":"Lecture","title":"Type stability","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Sometimes it happens that we create a non-stable code, which might be difficult to spot at first, for a non-trained eye. A prototypical example of such bug is as follows","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function poor_sum(x)\n s = 0\n for xᵢ in x\n s += xᵢ\n end\n s\nend","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"x = randn(10^8);\njulia> @benchmark poor_sum(x)\nBenchmarkTools.Trial: 23 samples with 1 evaluation.\n Range (min … max): 222.055 ms … 233.552 ms ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 225.259 ms ┊ GC (median): 0.00%\n Time (mean ± σ): 225.906 ms ± 3.016 ms ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n ▁ ▁ ▁▁█ ▁▁ ▁ ▁█ ▁ ▁ ▁ ▁ ▁ ▁▁▁▁ ▁ ▁\n █▁█▁███▁▁██▁▁█▁██▁█▁█▁█▁█▁█▁▁▁▁████▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁█ ▁\n 222 ms Histogram: frequency by time 234 ms <\n\n Memory estimate: 16 bytes, allocs estimate: 1.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Can we do better? Let's look what profiler says.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"using Profile, ProfileSVG\nProfile.clear()\n@profile poor_sum(x)\nProfileSVG.save(\"/tmp/profile4.svg\")","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"The profiler (output here) does not show any red, which means that according to the profilerthe code is type stable (and so does the @code_typed poor_sum(x) does not show anything bad.) Yet, we can see that the fourth line of the poor_sum function takes unusually long (there is a white area above, which means that the time spend in childs of that line (iteration and sum) does the sum to the time spent in the line, which is fishy). ","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"A close lookup on the code reveals that s is initialized as Int64, because typeof(0) is Int64. But then in the loop, we add to s a Float64 because x is Vector{Float64}, which means during the execution, the type s changes the type.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"So why nor compiler nor @code_typed(poor_sum(x)) warns us about the type instability? This is because of the optimization called small unions, where Julia can optimize \"small\" type instabilitites (recall the second lecture).","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"We can fix it for example by initializing x to be the zero of an element type of the array x (though this solution technically assumes x is an array, which means that poor_sum will not work for generators)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function stable_sum(x)\n s = zero(eltype(x))\n for xᵢ in x\n s += xᵢ\n end\n s\nend","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"But there is no difference, due to small union optimization (the above would kill any performance in older versions.)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @benchmark stable_sum(x)\nBenchmarkTools.Trial: 42 samples with 1 evaluation.\n Range (min … max): 119.491 ms … 123.062 ms ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 120.535 ms ┊ GC (median): 0.00%\n Time (mean ± σ): 120.687 ms ± 819.740 μs ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n █\n ▅▁▅▁▅▅██▅▁█▁█▁██▅▁█▅▅▁█▅▁█▁█▅▅▅█▁▁▁▁▁▁▁▅▁▁▁▁▁▅▁▅▁▁▁▁▁▁▅▁▁▁▁▁▅ ▁\n 119 ms Histogram: frequency by time 123 ms <\n\n Memory estimate: 16 bytes, allocs estimate: 1.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\nThe optimization of small unions has been added in Julia 1.0. If we compare the of the same function in Julia 0.6, the difference is strikingjulia> @time poor_sum(x)\n 1.863665 seconds (300.00 M allocations: 4.470 GiB, 4.29% gc time)\n9647.736705951513\njulia> @time stable_sum(x)\n 0.167794 seconds (5 allocations: 176 bytes)\n9647.736705951513The optimization of small unions is a big deal. It simplifies implementation of arrays with missing values, or allows to signal that result has not been produced by returning missing. In case of arrays with missing values, the type of element is Union{Missing,T} where T is the type of non-missing element.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"We can tell Julia that it is safe to vectorize the code. Julia tries to vectorize anyway, but @simd macro allows more aggressive operations, such as instruction reordering, which might change the output due imprecision of representation of real numbers in Floats.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function simd_sum(x)\n s = zero(eltype(x))\n @simd for xᵢ in x\n s += xᵢ\n end\n s\nend","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @benchmark simd_sum(x)\nBenchmarkTools.Trial: 90 samples with 1 evaluation.\n Range (min … max): 50.854 ms … 62.260 ms ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 54.656 ms ┊ GC (median): 0.00%\n Time (mean ± σ): 55.630 ms ± 3.437 ms ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n █ ▂ ▄ ▂ ▂ ▂ ▄\n ▄▆█▆▁█▄██▁▁█▆██▆▄█▁▆▄▁▆▆▄▁▁▆▁▁▁▁▄██▁█▁▁█▄▄▆▆▄▄▁▄▁▁▁▄█▁▆▁▆▁▆ ▁\n 50.9 ms Histogram: frequency by time 62.1 ms <\n\n Memory estimate: 16 bytes, allocs estimate: 1.","category":"page"},{"location":"lecture_05/lecture/#Untyped-global-variables-introduce-type-instability","page":"Lecture","title":"Untyped global variables introduce type instability","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function implicit_sum()\n\ts = zero(eltype(y))\n\t@simd for yᵢ in y\n\t\ts += yᵢ\n\tend\n\ts\nend","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> y = randn(10^8);\njulia> @benchmark implicit_sum()\nBenchmarkTools.Trial: 1 sample with 1 evaluation.\n Single result which took 10.837 s (11.34% GC) to evaluate,\n with a memory estimate of 8.94 GiB, over 499998980 allocations.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"What? The same function where I made the parameters to be implicit has just turned nine orders of magnitude slower? ","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Let's look what the profiler says","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Profile.clear()\ny = randn(10^4)\n@profile implicit_sum()\nProfileSVG.save(\"/tmp/profile5.svg\")","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"(output available here) which does not say anything except that there is a huge type-instability (red bar). In fact, the whole computation is dominated by Julia constantly determining the type of something.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"How can we determine, where is the type instability?","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"@code_typed implicit_sum() is \nCthulhu as @descend implicit_sum()\nJET available for Julia 1.7 onward","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\nJETJET is a code analyzer, which analyze the code without actually invoking it. The technique is called \"abstract interpretation\" and JET internally uses Julia's native type inference implementation, so it can analyze code as fast/correctly as Julia's code generation. JET internally traces the compiler's knowledge about types and detects, where the compiler cannot infer the type (outputs Any). Note that small unions are no longer considered type instability, since as we have seen above, the performance bottleneck is small. We can use JET as \tusing JET\n\t@report_opt implicit_sum()","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"All of these tools tells us that the Julia's compiler cannot determine the type of x. But why? I can just invoke typeof(x) and I know immediately the type of x. ","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"To understand the problem, you have to think about the compiler.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"You define function implicit_sum().\nIf you call implicit_sum and y does not exist, Julia will happily crash.\nIf you call implicit_sum and y exist, the function will give you the result (albeit slowly). At this moment, Julia has to specialize implicit_sum. It has two options how to behave with respect to y. \na. The compiler can assume that type of y is the current typeof(y) but that would mean that if a user redefines y and change the type, the specialization of the function implicit_sum will assume the wrong type of y and it can have unexpected results.\nb. The compiler take safe approach and determine the type of y inside the function implicit_sum and behave accordingly (recall that julia is dynamically typed). Yet, not knowing the type precisely is absolute disaster for performance. You can see this assumption for yourself by typing @code_typed implicit_sum().","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Notice the compiler dispatches on the name of the function and type of its arguments, hence, the compiler cannot create different versions of implicit_sum for different types of y, since it is not an argument, hence the dynamic resolution of types y inside implicit_sum function.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Julia takes the safe approach, which we can verify that although the implicit_sum was specialized (compiled) when y was Vector{Float64}, it works for other types","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"y = rand(Int, 1000)\nimplicit_sum() ≈ sum(y)\ny = map(x -> Complex(y...), zip(rand(1000), rand(1000)))\nimplicit_sum() ≈ sum(y)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"This means, using global variables inside functions without passing them as arguments ultimately leads to type-instability. What are the solutions?","category":"page"},{"location":"lecture_05/lecture/#Julia-1.7-and-below-Declaring-y-as-const","page":"Lecture","title":"Julia 1.7 and below => Declaring y as const","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"We can declare y as const, which tells the compiler that y will not change (and for the compiler mainly indicates that type of y will not change).","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Let's see that, but restart the julia before trying. After defining y as const, we see that the speed is the same as of simd_sum().","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @benchmark implicit_sum()\nBenchmarkTools.Trial: 99 samples with 1 evaluation.\n Range (min … max): 47.864 ms … 58.365 ms ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 50.042 ms ┊ GC (median): 0.00%\n Time (mean ± σ): 50.479 ms ± 1.598 ms ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n ▂ █▂▂▇ ▅ ▃\n ▃▁▃▁▁▁▁▇██████▅█▆██▇▅▆▁▁▃▅▃▃▁▃▃▁▃▃▁▁▃▁▃▁▁▁▁▁▁▁▁▁▁▁▁▁▃▁▁▁▁▁▃ ▁\n 47.9 ms Histogram: frequency by time 57.1 ms <\n\n Memory estimate: 0 bytes, allocs estimate: 0.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Also notice the difference in @code_typed implicit_sum()","category":"page"},{"location":"lecture_05/lecture/#Julia-1.8-and-above-Provide-type-to-y","page":"Lecture","title":"Julia 1.8 and above => Provide type to y","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Julia 1.8 added support for typed global variables which solves the above problem as can be seen from (do not forget to restart julia)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"y::Vector{Float64} = rand(10^8);\n``julia\n@benchmark implicit_sum()\n@code_typed implicit_sum()","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Unlike in const, we are free to change the bindings if it is possible to convert it to typeof(y)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"y = [1.0,2.0]\ntypeof(y)\ny = [1,2]\ntypeof(y)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"but y = [\"1\",\"2\"] will issue an error, since String has no default conversion rule to Float64 (you can overwrite this by defining Base.convert(::Type{Float64}, s::String) = parse(Float64, s) but it will likely lead to all kinds of shenanigans).","category":"page"},{"location":"lecture_05/lecture/#Barier-function","page":"Lecture","title":"Barier function","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Recall the reason, why the implicit_sum is so slow is that every time the function invokes getindex and +, it has to resolve types. The solution would be to limit the number of resolutions, which can done by passing all parameters to inner function as follows (do not forget to restart julia if you have defined y as const before).","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"using BenchmarkTools\nfunction barrier_sum()\n simd_sum(y)\nend\ny = randn(10^8);","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"@benchmark barrier_sum()\nBenchmarkTools.Trial: 93 samples with 1 evaluation.\n Range (min … max): 50.229 ms … 58.484 ms ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 53.882 ms ┊ GC (median): 0.00%\n Time (mean ± σ): 54.064 ms ± 2.892 ms ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n ▂▆█ ▆▄\n ▆█████▆▄█▆▄▆▁▄▄▄▄▁▁▄▁▄▄▆▁▄▄▄▁▁▄▁▁▄▁▁▆▆▁▁▄▄▁▄▆████▄▆▄█▆▄▄▄▄█ ▁\n 50.2 ms Histogram: frequency by time 58.4 ms <\n\n Memory estimate: 16 bytes, allocs estimate: 1.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"using JET\n@report_opt barrier_sum()","category":"page"},{"location":"lecture_05/lecture/#Checking-bounds-is-expensive","page":"Lecture","title":"Checking bounds is expensive","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"By default, julia checks bounds on every access to a location on an array, which can be difficult. Consider a following quicksort","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function qsort!(a,lo,hi)\n i, j = lo, hi\n while i < hi\n pivot = a[(lo+hi)>>>1]\n while i <= j\n while a[i] < pivot; i = i+1; end\n while a[j] > pivot; j = j-1; end\n if i <= j\n a[i], a[j] = a[j], a[i]\n i, j = i+1, j-1\n end\n end\n if lo < j; qsort!(a,lo,j); end\n lo, j = i, hi\n end\n return a\nend\n\nqsort!(a) = qsort!(a,1,length(a))","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"On lines 6 and 7 the qsort! accesses elements of array and upon every access julia checks bounds. We can signal to the compiler that it is safe not to check bounds using macro @inbounds as follows","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function inqsort!(a,lo,hi)\n i, j = lo, hi\n @inbounds while i < hi\n pivot = a[(lo+hi)>>>1]\n while i <= j\n while a[i] < pivot; i = i+1; end\n while a[j] > pivot; j = j-1; end\n if i <= j\n a[i], a[j] = a[j], a[i]\n i, j = i+1, j-1\n end\n end\n if lo < j; inqsort!(a,lo,j); end\n lo, j = i, hi\n end\n return a\nend\n\ninqsort!(a) = inqsort!(a,1,length(a))","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"We @benchmark to measure the impact","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> a = randn(1000);\njulia> @benchmark qsort!($(a))\nBenchmarkTools.Trial: 10000 samples with 4 evaluations.\n Range (min … max): 7.324 μs … 41.118 μs ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 7.415 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 7.666 μs ± 1.251 μs ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n ▇█ ▁ ▁ ▁ ▁ ▁\n ██▄█▆█▆▆█▃▇▃▁█▆▁█▅▃█▆▁▆▇▃▄█▆▄▆█▇▅██▄▃▃█▆▁▁▃▄▃▁▃▁▆▅▅▅▁▃▃▅▆▆ █\n 7.32 μs Histogram: log(frequency) by time 12.1 μs <\n\n Memory estimate: 0 bytes, allocs estimate: 0.\n\n\njulia> @benchmark inqsort!($(a))\nBenchmarkTools.Trial: 10000 samples with 7 evaluations.\n Range (min … max): 4.523 μs … 873.401 μs ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 4.901 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 5.779 μs ± 9.165 μs ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n ▄█▇▅▁▁▁▁ ▁▁▂▃▂▁▁ ▁ ▁\n █████████▆▆▆▆▆▇██████████▇▇▆▆▆▇█▇▆▅▅▆▇▅▅▆▅▅▅▇▄▅▆▅▃▅▅▆▅▄▄▃▅▅ █\n 4.52 μs Histogram: log(frequency) by time 14.8 μs <\n\n Memory estimate: 0 bytes, allocs estimate: 0.\n","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"and see that by not checking bounds, the code is 33% faster.","category":"page"},{"location":"lecture_05/lecture/#Boxing-in-closure","page":"Lecture","title":"Boxing in closure","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Recall closure is a function which contains some parameters contained ","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"An example of closure (adopted from JET.jl)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function abmult(r::Int)\n if r < 0\n r = -r\n end\n # the closure assigned to `f` make the variable `r` captured\n f = x -> x * r\n return f\nend;","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Another example of closure counting the error and printing it every steps","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function initcallback(; steps = 10)\n i = 0\n ts = time()\n y = 0.0\n cby = function evalcb(_y)\n i += 1.0\n y += _y\n if mod(i, steps) == 0\n l = y / steps\n y = 0.0\n println(i, \": loss: \", l,\" time per step: \",round((time() - ts)/steps, sigdigits = 2))\n ts = time()\n end\n end\n cby\nend\n\n\ncby = initcallback()\n\nfor i in 1:100\n cby(rand())\nend","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function simulation()\n cby = initcallback(;steps = 10000)\t#intentionally disable printing\n for i in 1:1000\n cby(sin(rand()))\n end\nend\n\n@benchmark simulation()","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"using Profile, ProfileSVG\nProfile.clear()\n@profile (for i in 1:100; simulation(); end)\nProfileSVG.save(\"/tmp/profile.svg\")","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"We see a red bars in lines 4 and 8 of evalcb, which indicates the type instability hindering the performance. Why they are there? The answer is tricky.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"In closures, as the name suggest, function closes over (or captures) some variables defined in the function outside the function that is returned. If these variables are of primitive types (think Int, Float64, etc.), the compiler assumes that they might be changed. Though when primitive types are used in calculations, the result is not written to the same memory location but to a new location and the name of the variable is made to point to this new variable location (this is called rebinding). We can demonstrate it on this example [2].","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"[2]: Invenia blog entry","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> x = [1];\n\njulia> objectid(x)\n0x79eedc509237c203\n\njulia> x .= [10]; # mutating contents\n\njulia> objectid(x)\n0x79eedc509237c203\n\njulia> y = 100;\n\njulia> objectid(y)\n0xdb216d4e5c739c77\n\njulia> y = y + 100; # rebinding the variable name\n\njulia> objectid(y)\n0xb642af5f06b41e88","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Since the inner function needs to point to the same location, julia uses Box container which can be seen as a translation, where the pointer inside the Box can change while the inner function contains the same pointer to the Box. This makes possible to change the captured variables and tracks changes in the point. Sometimes (it can happen many time) the compiler fails to determine that the captured variable is read only, and it wrap it (box it) in the Box wrapper, which makes it type unstable, as Box does not track types (it would be difficult as even the type can change in the inner function). This is what we can see in the first example of abmult. In the second example, the captured variable y and i changes and the compiler is right.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"What can we do?","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"The first difficulty is to even detect this case. We can spot it using @code_typed and of course JET.jl can do it and it will warn us. Above we have seen the effect of the profiler.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Using @code_typed","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_typed abmult(1)\nCodeInfo(\n1 ─ %1 = Core.Box::Type{Core.Box}\n│ %2 = %new(%1, r@_2)::Core.Box\n│ %3 = Core.isdefined(%2, :contents)::Bool\n└── goto #3 if not %3\n2 ─ goto #4\n3 ─ $(Expr(:throw_undef_if_not, :r, false))::Any\n4 ┄ %7 = Core.getfield(%2, :contents)::Any\n│ %8 = (%7 < 0)::Any\n└── goto #9 if not %8\n5 ─ %10 = Core.isdefined(%2, :contents)::Bool\n└── goto #7 if not %10\n6 ─ goto #8\n7 ─ $(Expr(:throw_undef_if_not, :r, false))::Any\n8 ┄ %14 = Core.getfield(%2, :contents)::Any\n│ %15 = -%14::Any\n└── Core.setfield!(%2, :contents, %15)::Any\n9 ┄ %17 = %new(Main.:(var\"#5#6\"), %2)::var\"#5#6\"\n└── return %17\n) => var\"#5#6\"","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Using Jet.jl (recall it requires the very latest Julia 1.7)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @report_opt abmult(1)\n═════ 3 possible errors found ═════\n┌ @ REPL[15]:2 r = Core.Box(:(_7::Int64))\n│ captured variable `r` detected\n└──────────────\n┌ @ REPL[15]:2 Main.<(%7, 0)\n│ runtime dispatch detected: Main.<(%7::Any, 0)\n└──────────────\n┌ @ REPL[15]:3 Main.-(%14)\n│ runtime dispatch detected: Main.-(%14::Any)\n└──────────────","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Sometimes, we do not have to do anything. For example the above example of evalcb function, we assume that all the other code in the simulation would take much more time so a little type instability is not important.\nAlternatively, we can explicitly use Ref instead of the Box, which are typed wrappers, but they are awkward to use. ","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function ref_abmult(r::Int)\n if r < 0\n r = -r\n end\n rr = Ref(r)\n f = x -> x * rr[]\n return f\nend;","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"We can see in @code_typed that the compiler is happy as it can resolve the types correctly","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_typed ref_abmult(1)\nCodeInfo(\n1 ─ %1 = Base.slt_int(r@_2, 0)::Bool\n└── goto #3 if not %1\n2 ─ %3 = Base.neg_int(r@_2)::Int64\n3 ┄ %4 = φ (#2 => %3, #1 => _2)::Int64\n│ %5 = %new(Base.RefValue{Int64}, %4)::Base.RefValue{Int64}\n│ %6 = %new(var\"#7#8\"{Base.RefValue{Int64}}, %5)::var\"#7#8\"{Base.RefValue{Int64}}\n└── return %6\n) => var\"#7#8\"{Base.RefValue{Int64}}","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Jet is also happy.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"\njulia> @report_opt ref_abmult(1)\nNo errors !\n","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"So when you use closures, you should be careful of the accidental boxing, since it can inhibit the speed of code. This is a big deal in Multithreadding and in automatic differentiation, both heavily uses closures. You can track the discussion here.","category":"page"},{"location":"lecture_05/lecture/#NamedTuples-are-more-efficient-that-Dicts","page":"Lecture","title":"NamedTuples are more efficient that Dicts","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"It happens a lot in scientific code, that some experiments have many parameters. It is therefore very convenient to store them in Dict, such that when adding a new parameter, we do not have to go over all defined functions and redefine them.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Imagine that we have a (nonsensical) simulation like ","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"settings = Dict(:stepsize => 0.01, :h => 0.001, :iters => 500, :info => \"info\")\nfunction find_min!(f, x, p)\n for i in 1:p[:iters]\n x̃ = x + p[:h]\n fx = f(x) # line 4\n x -= p[:stepsize] * (f(x̃) - fx)/p[:h] # line 5\n end\n x\nend","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Notice the parameter p is a Dict and that it can contain arbitrary parameters, which is useful. Hence, Dict is cool for passing parameters. Let's now run the function through the profiler","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"x₀ = rand()\nf(x) = x^2\nProfile.clear()\n@profile find_min!(f, x₀, settings)\nProfileSVG.save(\"/tmp/profile6.svg\")","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"from the profiler's output here we can see some type instabilities. Where they come from? The compiler does not have any information about types stored in settings, as the type of stored values are Any (caused by storing String and Int).","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> typeof(settings)\nDict{Symbol, Any}","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"The second problem is get operation on dictionaries is very time consuming operation (although technically it is O(1)), because it has to search the key in the list. Dicts are designed as a mutable container, which is not needed in our use-case, as the settings are static. For similar use-cases, Julia offers NamedTuple, with which we can construct settings as ","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"nt_settings = (;stepsize = 0.01, h=0.001, iters=500, :info => \"info\")","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"The NamedTuple is fully typed, but which we mean the names of fields are part of the type definition and fields are also part of type definition. You can think of it as a struct. Moreover, when accessing fields in NamedTuple, compiler knows precisely where they are located in the memory, which drastically reduces the access time. Let's see the effect in BenchmarkTools.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @benchmark find_min!(x -> x^2, x₀, settings)\nBenchmarkTools.Trial: 10000 samples with 1 evaluation.\n Range (min … max): 86.350 μs … 4.814 ms ┊ GC (min … max): 0.00% … 97.61%\n Time (median): 90.747 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 102.405 μs ± 127.653 μs ┊ GC (mean ± σ): 4.69% ± 3.75%\n\n ▅██▆▂ ▁▁ ▁ ▂\n ███████▇▇████▇███▇█▇████▇▇▆▆▇▆▇▇▇▆▆▆▆▇▆▇▇▅▇▆▆▆▆▄▅▅▄▅▆▆▅▄▅▃▅▃▅ █\n 86.4 μs Histogram: log(frequency) by time 209 μs <\n\n Memory estimate: 70.36 KiB, allocs estimate: 4002.\n\njulia> @benchmark find_min!(x -> x^2, x₀, nt_settings)\nBenchmarkTools.Trial: 10000 samples with 7 evaluations.\n Range (min … max): 4.179 μs … 21.306 μs ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 4.188 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 4.493 μs ± 1.135 μs ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n █▃▁ ▁ ▁ ▁ ▁\n ████▇████▄██▄█▃██▄▄▇▇▇▇▅▆▆▅▄▄▅▄▅▅▅▄▁▅▄▁▄▄▆▆▇▄▅▆▄▄▃▄▆▅▆▁▄▄▄ █\n 4.18 μs Histogram: log(frequency) by time 10.8 μs <\n\n Memory estimate: 16 bytes, allocs estimate: 1.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Checking the output with JET, there is no type instability anymore","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"@report_opt find_min!(f, x₀, nt_settings)\nNo errors !","category":"page"},{"location":"lecture_05/lecture/#Don't-use-IO-unless-you-have-to","page":"Lecture","title":"Don't use IO unless you have to","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"debug printing in performance critical code should be kept to minimum or using in memory/file based logger in stdlib Logging.jl","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function find_min!(f, x, p; verbose=true)\n\tfor i in 1:p[:iters]\n\t\tx̃ = x + p[:h]\n\t\tfx = f(x)\n\t\tx -= p[:stepsize] * (f(x̃) - fx)/p[:h]\n\t\tverbose && println(\"x = \", x, \" | f(x) = \", fx)\n\tend\n\tx\nend\n\n@btime find_min!($f, $x₀, $params_tuple; verbose=true)\n@btime find_min!($f, $x₀, $params_tuple; verbose=false)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"interpolation of strings is even worse https://docs.julialang.org/en/v1/manual/performance-tips/#Avoid-string-interpolation-for-I/O","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function find_min!(f, x, p; verbose=true)\n\tfor i in 1:p[:iters]\n\t\tx̃ = x + p[:h]\n\t\tfx = f(x)\n\t\tx -= p[:stepsize] * (f(x̃) - fx)/p[:h]\n\t\tverbose && println(\"x = $x | f(x) = $fx\")\n\tend\n\tx\nend\n@btime find_min!($f, $x₀, $params_tuple; verbose=true)","category":"page"},{"location":"lecture_01/lab/#Lab-01:-Introduction-to-Julia","page":"Lab","title":"Lab 01: Introduction to Julia","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"This lab should get everyone up to speed in the basics of Julia's installation, syntax and basic coding. For more detailed introduction you can check out Lectures 1-3 of the bachelor course.","category":"page"},{"location":"lecture_01/lab/#Testing-Julia-installation-(custom-setup)","page":"Lab","title":"Testing Julia installation (custom setup)","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"In order to proceed further let's run a simple script to see, that the setup described in chapter Installation is working properly. After spawning a terminal/cmdline run this command:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"julia ./test_setup.jl","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The script does the following ","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"\"Tests\" if Julia is added to path and can be run with julia command from anywhere\nPrints Julia version info\nChecks Julia version.\nChecks git configuration (name + email)\nCreates an environment configuration files\nInstalls a basic pkg called BenchmarkTools, which we will use for benchmarking a simple function later in the labs.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"There are some quality of life improvements over long term support versions of Julia and thus throughout this course we will use the latest stable release of Julia 1.6.x.","category":"page"},{"location":"lecture_01/lab/#Polynomial-evaluation-example","page":"Lab","title":"Polynomial evaluation example","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Let's consider a common mathematical example for evaluation of nth-degree polynomial","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"f(x) = a_nx^n + a_n-1x^n-1 + dots + a_0x^0","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"where x in mathbbR and veca in mathbbR^n+1.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The simplest way of writing this in a generic fashion is realizing that essentially the function f is really implicitly containing argument veca, i.e. f equiv f(veca x), yielding the following Julia code","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"function polynomial(a, x)\n accumulator = 0\n for i in length(a):-1:1\n accumulator += x^(i-1) * a[i] # ! 1-based indexing for arrays\n end\n return accumulator\nend\nnothing #hide","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Evaluate the code of the function called polynomial in Julia REPL and evaluate the function itself with the following arguments.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"a = [-19, 7, -4, 6] # list coefficients a from a^0 to a^n\nx = 3 # point of evaluation\nnothing #hide","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The simplest way is to just copy&paste into an already running terminal manually. As opposed to the default Python REPL, Julia can deal with the blocks of code and different indentation much better without installation of an ipython-like REPL. There are ways to make this much easier in different text editors/IDEs:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"VSCode - when using Julia extension is installed and .jl file is opened, Ctrl/Cmd+Enter will spawn Julia REPL\nSublime Text - Ctrl/Cmd+Enter with Send Code pkg (works well with Linux terminal or tmux, support for Windows is poor)\nVim - there is a Julia language plugin, which can be combine with vimcmdline to gain similar functionality","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Either way, you should see the following:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"function polynomial(a, x)\n accumulator = 0\n for i in length(a):-1:1\n accumulator += x^(i-1) * a[i] # ! 1-based indexing for arrays\n end\n return accumulator\nend","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Similarly we enter the arguments of the function a and x:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"a = [-19, 7, -4, 6]\nx = 3","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Function call intuitively takes the name of the function with round brackets as arguments, i.e. works in the same way as majority of programming languages. The result is printed unless a ; is added at the end of the statement.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"polynomial(a, x) # function call","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Thanks to the high level nature of Julia language it is often the case that examples written in pseudocode are almost directly rewritable into the language itself without major changes and the code can be thus interpreted easily.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"(Image: polynomial_explained)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Due to the existence of the end keyword, indentation is not necessary as opposed to other languages such as Python, however it is strongly recommended to use it, see style guide.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Though there are libraries/IDEs that allow us to step through Julia code (Debugger.jl link and VSCode link), here we will explore the code interactively in REPL by evaluating pieces of code separately.","category":"page"},{"location":"lecture_01/lab/#Basic-types,-assignments-and-variables","page":"Lab","title":"Basic types, assignments and variables","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"When defining a variable through an assignment we get the representation of the right side, again this is different from the default behavior in Python, where the output of assignments a = [-19, 7, -4, 6] or x = 3, prints nothing. Internally Julia returns the result of the display function.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"a = [-19, 7, -4, 6]\ndisplay(a) # should return the same thing as the line above","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"As you can see, the string that is being displayed contains information about the contents of a variable along with it's type in this case this is a Vector/Array of Int types. If the output of display is insufficient the type of variable can be checked with the typeof function:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"typeof(a)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Additionally for collection/iterable types such as Vector there is also the eltype function, which returns the type of elements in the collection.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"eltype(a)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"In most cases variables store just a reference to a place in memory either stack/heap (exceptions are primitive types such as Int, Float) and therefore creating an array a, \"storing\" the reference in b with an assignment and changing elements of b, e.g. b[1] = 2, changes also the values in a.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Create variables x and accumulator, storing floating point 3.0 and integer value 0 respectively. Check the type of variables using typeof function.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"x = 3.0\naccumulator = 0\ntypeof(x), typeof(accumulator)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_01/lab/#For-cycles-and-ranges","page":"Lab","title":"For cycles and ranges","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Moving further into the polynomial function we encounter the definition of a for cycle, with the de facto standard syntax","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"for iteration_variable in iterator\n # do something\nend","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"As an example of iterator we have used an instance of a range type ","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"r = length(a):-1:1\ntypeof(r)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"As opposed to Python, ranges in Julia are inclusive, i.e. they contain number from start to end - in this case running from 4 to 1 with negative step -1, thus counting down. This can be checked with the collect and/or length functions.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"collect(r)\nlength(r)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Create variable c containing an array of even numbers from 2 to 42. Furthermore create variable d that is different from c only at the 7th position, which will contain 13.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"HINT: Use collect function for creation of c and copy for making a copy of c.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"c = collect(2:2:42)\nd = copy(c)\nd[7] = 13\nd","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_01/lab/#Functions-and-operators","page":"Lab","title":"Functions and operators","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Let us now move from the function body to the function definition itself. From the picture at the top of the page, we can infer the general syntax for function definition:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"function function_name(arguments)\n # do stuff with arguments and define output value `something`\n return something\nend","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The return keyword can be omitted, if the last line being evaluated contains the result.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"By creating the function polynomial we have defined a variable polynomial, that from now on always refers to a function and cannot be reassigned to a different type, like for example Int.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"polynomial = 42","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"This is caused by the fact that each function defines essentially a new type, the same like Int ~ Int64 or Vector{Int}.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"typeof(polynomial)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"You can check that it is a subtype of the Function abstract type, with the subtyping operator <:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"typeof(polynomial) <: Function","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"These concepts will be expanded further in the type system lecture, however for now note that this construction is quite useful for example if we wanted to create derivative rules for our function derivativeof(::typeof(polynomial), ...).","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Looking at mathematical operators +, *, we can see that in Julia they are also standalone functions. ","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"+\n*","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The main difference from our polynomial function is that there are multiple methods, for each of these functions. Each one of the methods coresponds to a specific combination of arguments, for which the function can be specialized to using multiple dispatch. You can see the list by calling a methods function:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"julia> methods(+)\n# 190 methods for generic function \"+\": \n[1] +(x::T, y::T) where T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8} in Base at\n int.jl:87 \n[2] +(c::Union{UInt16, UInt32, UInt64, UInt8}, x::BigInt) in Base.GMP at gmp.jl:528 \n[3] +(c::Union{Int16, Int32, Int64, Int8}, x::BigInt) in Base.GMP at gmp.jl:534\n...","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"One other notable difference is that these functions allow using both infix and postfix notation a + b and +(a,b), which is a specialty of elementary functions such as arithmetic operators or set operation such as ∩, ∪, ∈. ","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The functionality of methods is complemented with the reverse lookup methodswith, which for a given type returns a list of methods that can be called with it as an argument.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"julia> methodswith(Int)\n[1] +(x::T, y::T) where T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8} in Base at int.jl:87\n[2] +(c::Union{Int16, Int32, Int64, Int8}, x::BigInt) in Base.GMP at gmp.jl:534\n[3] +(c::Union{Int16, Int32, Int64, Int8}, x::BigFloat) in Base.MPFR at mpfr.jl:384\n[4] +(x::BigFloat, c::Union{Int16, Int32, Int64, Int8}) in Base.MPFR at mpfr.jl:379\n[5] +(x::BigInt, c::Union{Int16, Int32, Int64, Int8}) in Base.GMP at gmp.jl:533\n...","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Define function called addone with one argument, that adds 1 to the argument.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"function addone(x)\n x + 1\nend\naddone(1) == 2","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_01/lab/#Calling-for-help","page":"Lab","title":"Calling for help","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"In order to better understand some keywords we have encountered so far, we can ask for help in the Julia's REPL itself with the built-in help terminal. Accessing help terminal can be achieved by writing ? with a query keyword after. This searches documentation of all the available source code to find the corresponding keyword. The simplest way to create documentation, that can be accessed in this way, is using so called docstrings, which are multiline strings written above function or type definition. ","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"\"\"\"\n polynomial(a, x)\n\nReturns value of a polynomial with coefficients `a` at point `x`.\n\"\"\"\nfunction polynomial(a, x)\n # function body\nend","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"More on this in lecture 4 about pkg development.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Lookup docstring for the basic functions that we have introduced in the previous exercises: typeof, eltype, length, collect, copy, methods and methodswith. ","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"BONUS: Try it with others, for example with the subtyping operator <:.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Example docstring for typeof function.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":" typeof(x)\n\n Get the concrete type of x.\n\n Examples\n ≡≡≡≡≡≡≡≡≡≡\n\n julia> a = 1//2;\n \n julia> typeof(a)\n Rational{Int64}\n \n julia> M = [1 2; 3.5 4];\n \n julia> typeof(M)\n Matrix{Float64} (alias for Array{Float64, 2})","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_01/lab/#Testing-waters","page":"Lab","title":"Testing waters","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"As the arguments of the polynomial functions are untyped, i.e. they do not specify the allowed types like for example polynomial(a, x::Number) does, the following exercise explores which arguments the function accepts, while giving expected result.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Choose one of the variables af to ac representing polynomial coefficients and try to evaluate it with the polynomial function at point x=3 as before. Lookup the type of coefficient collection variable itself with typeof and the items in the collection with eltype. In this case we allow you to consult your solution with the expandable solution bellow to find out more information about a particular example.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"af = [-19.0, 7.0, -4.0, 6.0]\nat = (-19, 7, -4, 6)\nant = (a₀ = -19, a₁ = 7, a₂ = -4, a₃ = 6)\na2d = [-19 -4; 7 6]\nac = [2i^2 + 1 for i in -2:1]\nnothing #hide","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"typeof(af), eltype(af)\npolynomial(af, x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"As opposed to the basic definition of a type the array is filled with Float64 types and the resulting value gets promoted as well to the Float64.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"typeof(at), eltype(at)\npolynomial(at, x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"With round brackets over a fixed length vector we get the Tuple type, which is so called immutable \"array\" of a fixed size (its elements cannot be changed, unless initialized from scratch). Each element can be of a different type, but here we have only one and thus the Tuple is aliased into NTuple. There are some performance benefits for using immutable structure, which will be discussed later.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Defining key=value pairs inside round brackets creates a structure called NamedTuple, which has the same properties as Tuple and furthermore its elements can be conveniently accessed by dot syntax, e.g. ant.a₀.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"typeof(ant), eltype(ant)\npolynomial(ant, x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Defining a 2D array is a simple change of syntax, which initialized a matrix row by row separated by ; with spaces between individual elements. The function returns the same result because linear indexing works in 2d arrays in the column major order.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"typeof(a2d), eltype(a2d)\npolynomial(a2d, x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The last example shows so called array comprehension syntax, where we define and array of known length using and for loop iteration. Resulting array/vector has integer elements, however even mixed type is possible yielding Any, if there isn't any other common supertype to promote every entry into. (Use ? to look what promote and promote_type does.)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"typeof(ac), eltype(ac)\npolynomial(ac, x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"So far we have seen that polynomial function accepts a wide variety of arguments, however there are some understandable edge cases that it cannot handle.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Consider first the vector/array of characters ach","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"ach = ['1', '2', '3', '4']","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"which themselves have numeric values (you can check by converting them to Int Int('1') or convert(Int, '1')). In spite of that, our untyped function cannot process such input, as there isn't an operation/method that would allow multiplication of Char and Int type. Julia tries to promote the argument types to some common type, however checking the promote_type(Int, Char) returns Any (union of all types), which tells us that the conversion is not possible automatically.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"typeof(ach), eltype(ach)\npolynomial(ach, x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"In the stacktrace we can see the location of each function call. If we include the function polynomial from some file poly.jl using include(\"poly.jl\"), we will see that the location changes from REPL[X]:10 to the actual file name.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"By swapping square brackets for round in the array comprehension ac above, we have defined so called generator/iterator, which as opposed to original variable ac does not allocate an array, only the structure that produces it.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"ag = (2i^2 + 1 for i in -2:1)\ntypeof(ag), eltype(ag)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"You may notice that the element type in this case is Any, which means that a function using this generator as an argument cannot specialize based on the type and has to infer it every time an element is generated/returned. We will touch on how this affects performance in one of the later lectures.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"polynomial(ag, x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The problem that we face during evaluation is that generator type is missing the getindex operation, as they are made for situations where the size of the collection may be unknown and the only way of obtaining particular elements is through sequential iteration. Generators can be useful for example when creating batches of data for a machine learning training. We can \"fix\" the situation using collect function, mentioned earlier, however that again allocates an array.","category":"page"},{"location":"lecture_01/lab/#Extending/limiting-the-polynomial-example","page":"Lab","title":"Extending/limiting the polynomial example","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Following up on the polynomial example, let's us expand it a little further in order to facilitate the arguments, that have been throwing exceptions. The first direction, which we will move forward to, is providing the user with more detailed error message when an incorrect type of coefficients has been provided.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Design an if-else condition such that the array of Char example throws an error with custom string message, telling the user what went wrong and printing the incorrect input alongside it. Confirm that we have not broken the functionality of other examples from previous exercise.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Throw the ArgumentError(msg) with throw function and string message msg. More details in help mode ? or at the end of this document.\nStrings are defined like this s = \"Hello!\"\nUse string interpolation to create the error message. It allows injecting an expression into a string with the $ syntax b = 1; s = \"Hellow Number $(b)\"\nCompare eltype of the coefficients with Char type.\nThe syntax for if-else:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"if condition\n println(\"true\") # true branch code\nelse\n println(\"false\") # false branch code\nend","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Not equal condition can be written as a != b.\nThrowing an exception automatically returns from the function. Use return inside one of the branches to return the correct value.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The simplest way is to wrap the whole function inside an if-else condition and returning only when the input is \"correct\" (it will still fail in some cases).","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"function polynomial(a, x)\n if eltype(a) != Char\n accumulator = 0\n for i in length(a):-1:1\n accumulator += x^(i-1) * a[i] # ! 1-based indexing for arrays\n end\n return accumulator\n else\n throw(ArgumentError(\"Invalid coefficients $(a) of type Char!\"))\n end\nend\nnothing #hide","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Now this should show our predefined error message. ","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"polynomial(ach, x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Testing on other examples should pass without errors and give the same output as before.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"polynomial(a, x)\npolynomial(af, x)\npolynomial(at, x)\npolynomial(ant, x)\npolynomial(a2d, x)\npolynomial(ac, x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The second direction concerns the limitation to index-able structures, which the generator example is not. For this we will have to rewrite the whole loop in a more functional programming approach using map, anonymous function and other concepts.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Rewrite the following code inside our original polynomial function with map, enumerate and anonymous function.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"accumulator = 0\nfor i in length(a):-1:1\n accumulator += x^(i-1) * a[i] # ! 1-based indexing for arrays\nend","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"note: Anonymous functions reminder\nx -> x + 1 # unless the reference is stored it cannot be called\nplusone = x -> x + 1 # the reference can be stored inside a variable\nplusone(x) # calling with the same syntax","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Use enumerate to obtain iterator over a that returns a tuple of ia = (i, aᵢ). With Julia 1-based indexing i starts also from 1 and goes up to length(a).\nPass this into a map with either in-place or predefined anonymous function that does the operation of x^(i-1) * aᵢ.\nUse sum to collect the resulting array into accumulator variable or directly into the return command.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"BONUS: Can you figure out how to use the mapreduce function here? See entry in the help mode ?.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Ordered from the longest to the shortest, here are three examples with the same functionality (and there are definitely many more). Using the map(iterable) do itervar ... end syntax, that creates anonymous function from the block of code.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"function polynomial(a, x)\n powers = map(enumerate(a)) do (i, aᵢ)\n x^(i-1) * aᵢ\n end\n accumulator = sum(powers)\n return accumulator\nend\nnothing #hide","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Using the default syntax for map and storing the anonymous into a variable","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"function polynomial(a, x)\n polypow(i,aᵢ) = x^(i-1) * aᵢ\n powers = map(polypow, enumerate(a))\n return sum(powers)\nend\nnothing #hide","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"As the function polypow is used only once, there is no need to assign it to a local variable. Note the sightly awkward additional parenthesis in the argument of the lambda function.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"function polynomial(a, x)\n powers = map(((i,aᵢ),) -> x^(i-1) * aᵢ, enumerate(a))\n sum(powers)\nend\nnothing #hide","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Checking the behavior on all the inputs.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"polynomial(a, x)\npolynomial(af, x)\npolynomial(at, x)\npolynomial(ant, x)\npolynomial(a2d, x)\npolynomial(ach, x)\npolynomial(ac, x)\npolynomial(ag, x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"BONUS: You may have noticed that in the example above, the powers variable is allocating an additional, unnecessary vector. With the current, scalar x, this is not such a big deal. But in your homework you will generalize this function to matrix inputs of x, which means that powers becomes a vector of (potentially very large) matrices. This is a very natural use case for the mapreduce: function:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"polynomial(a, x) = mapreduce(+, enumerate(a), init=zero(x)) do (i, aᵢ)\n x^(i-1) * aᵢ\nend\n\npolynomial(a, x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Let's unpack what is happening here. If the function mapreduce(f, op, itr) is called with op=+ it returns the same result as sum(map(f, itr)). In contrast to sum(map(f, itr)) (which allocates a vector as a result of map and then sums) mapreduce applies f to an element in itr and immediately accumulates the result with the given op=+.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"polynomial(a, x) = sum(ia -> x^(ia[1]-1) * ia[2], enumerate(a))\nnothing #hide","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_01/lab/#How-to-use-code-from-other-people","page":"Lab","title":"How to use code from other people","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The script that we have run at the beginning of this lab has created two new files inside the current folder:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"./\n ├── Manifest.toml\n └── Project.toml","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Every folder with a toml file called Project.toml, can be used by Julia's pkg manager into setting so called environment, which contains a list of pkgs to be installed. Setting up or more often called activating an environment can be done either before starting Julia itself by running julia with the --project XXX flag or from within the Julia REPL, by switching to Pkg mode with ] key (similar to the help mode activated by pressing ?) and running command activate.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"So far we have used the general environment (depending on your setup), which by default does not come with any 3rd party packages and includes only the base and standard libraries - already quite powerful on its own. ","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"In order to find which environment is currently active, run the following:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"pkg> status","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The output of such command usually indicates the general environment located at .julia/ folder (${HOME}/.julia/ or ${APPDATA}/.julia/ in case of Unix/Windows based systems respectively)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"pkg> status\nStatus `~/.julia/environments/v1.6/Project.toml` (empty project)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Generally one should avoid working in the general environment, with the exception of some generic pkgs, such as PkgTemplates.jl, which is used for generating library templates/folder structure like the one above (link), more on this in the lecture on pkg development. ","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Activate the environment inside the current folder and check that the BenchmarkTools package has been installed. Use BenchmarkTools pkg's @btime to benchmark our polynomial function with the following arguments.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"aexp = ones(10) ./ factorial.(0:9)\nx = 1.1\nnothing #hide","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"In pkg mode use the command activate and status to check the presence. \nIn order to import the functionality from other package, lookup the keyword using in the repl help mode ?. \nThe functionality that we want to use is the @btime macro (it acts almost like a function but with a different syntax @macro arg1 arg2 arg3 ...). More on macros in lecture 7.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"BONUS: Compare the output of polynomial(aexp, x) with the value of exp(x), which it approximates.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"note: Broadcasting\nIn the assignment's code, we are using quite ubiquitous concept in Julia called broadcasting or simply the dot-syntax - represented here by ./, factorial.. This concept allows to map both simple arithmetic operations as well as custom functions across arrays, with the added benefit of increased performance, when the broadcasting system can merge operations into a more efficient code. More information can be found in the official documentation or section of our bachelor course.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"There are other options to import a function/macro from a different package, however for now let's keep it simple with the using Module syntax, that brings to the REPL, all the variables/function/macros exported by the BenchmarkTools pkg. If @btime is exported, which it is, it can be accessed without specification i.e. just by calling @btime without the need for BenchmarkTools.@btime. More on the architecture of pkg/module loading in the package developement lecture.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"julia> using BenchmarkTools\n\njulia> @btime polynomial(aexp, x)\n 97.119 ns (1 allocation: 16 bytes)\n3.004165230550543","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The output gives us the time of execution averaged over multiple runs (the number of samples is defined automatically based on run time) as well as the number of allocations and the output of the function, that is being benchmarked.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"BONUS: The difference between our approximation and the \"actual\" function value computed as a difference of the two. ","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"polynomial(aexp, x) - exp(x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The apostrophes in the previous sentence are on purpose, because implementation of exp also relies on a finite sum, though much more sophisticated than the basic Taylor expansion.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_01/lab/#Discussion-and-future-directions","page":"Lab","title":"Discussion & future directions","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Instead of if-else statements that would throw an error for different types, in Julia, we generally see the pattern of typing the function in a way, that for other than desirable types MethodError is emitted with the information about closest matching methods. This is part of the design process in Julia of a function and for the particular functionality of the polynomial example, we can look into the Julia itself, where it has been implemented in the evalpoly function","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"methods(evalpoly)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Another avenue, that we have only touched with the BenchmarkTools, is performance and will be further explored in the later lectures.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"With the next lecture focused on typing in Julia, it is worth noting that polynomials lend themselves quite nicely to a definition of a custom type, which can help both readability of the code as well further extensions.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"struct Polynom{C}\n coefficients::{C}\nend\n\nfunction (p:Polynom)(x)\n polynomial(p.coefficients, x)\nend","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"","category":"page"},{"location":"lecture_01/lab/#Useful-resources","page":"Lab","title":"Useful resources","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Getting Started tutorial from JuliaLang documentation - Docs\nConverting syntax between MATLAB ↔ Python ↔ Julia - Cheatsheet\nBachelor course for refreshing your knowledge - Course\nStylistic conventions - Style Guide\nReserved keywords - List\nOfficial cheatsheet with basic syntax - link","category":"page"},{"location":"lecture_01/lab/#lab_errors","page":"Lab","title":"Various errors and how to read them","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"This section summarizes most commonly encountered types of errors in Julia and how to resolve them or at least understand, what has gone wrong. It expands a little bit the official documentation, which contains the complete list with examples. Keep in mind again, that you can use help mode in the REPL to query error types as well.","category":"page"},{"location":"lecture_01/lab/#MethodError","page":"Lab","title":"MethodError","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"This type of error is most commonly thrown by Julia's multiple dispatch system with a message like no method matching X(args...), seen in two examples bellow.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"2 * 'a' # many candidates\ngetindex((i for i in 1:4), 3) # no candidates","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Both of these examples have a short stacktrace, showing that the execution failed on the top most level in REPL, however if this code is a part of some function in a separate file, the stacktrace will reflect it. What this error tells us is that the dispatch system could not find a method for a given function, that would be suitable for the type of arguments, that it has been given. In the first case Julia offers also a list of candidate methods, that match at least some of the arguments","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"When dealing with basic Julia functions and types, this behavior can be treated as something given and though one could locally add a method for example for multiplication of Char and Int, there is usually a good reason why Julia does not support such functionality by default. On the other hand when dealing with user defined code, this error may suggest the developer, that either the functions are too strictly typed or that another method definition is needed in order to satisfy the desired functionality.","category":"page"},{"location":"lecture_01/lab/#InexactError","page":"Lab","title":"InexactError","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"This type of error is most commonly thrown by the type conversion system (centered around convert function), informing the user that it cannot exactly convert a value of some type to match arguments of a function being called.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Int(1.2) # root cause\nappend!([1,2,3], 1.2) # same as above but shows the root cause deeper in the stack trace","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"In this case the function being Int and the value a floating point. The second example shows InexactError may be caused deeper inside an inconspicuous function call, where we want to extend an array by another value, which is unfortunately incompatible.","category":"page"},{"location":"lecture_01/lab/#ArgumentError","page":"Lab","title":"ArgumentError","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"As opposed to the previous two errors, ArgumentError can contain user specified error message and thus can serve multiple purposes. It is however recommended to throw this type of error, when the parameters to a function call do not match a valid signature, e.g. when factorial were given negative or non-integer argument (note that this is being handled in Julia by multiple dispatch and specific DomainError).","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"This example shows a concatenation of two 2d arrays of incompatible sizes 3x3 and 2x2.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"hcat(ones(3,3), zeros(2,2))","category":"page"},{"location":"lecture_01/lab/#KeyError","page":"Lab","title":"KeyError","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"This error is specific to hash table based objects such as the Dict type and tells the user that and indexing operation into such structure tried to access or delete a non-existent element.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"d = Dict(:a => [1,2,3], :b => [1,23])\nd[:c]","category":"page"},{"location":"lecture_01/lab/#TypeError","page":"Lab","title":"TypeError","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Type assertion failure, or calling an intrinsic function (inside LLVM, where code is strictly typed) with incorrect argument type. In practice this error comes up most often when comparing value of a type against the Bool type as seen in the example bellow.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"if 1 end # calls internally typeassert(1, Bool)\ntypeassert(1, Bool)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"In order to compare inside conditional statements such as if-elseif-else or the ternary operator x ? a : b the condition has to be always of Bool type, thus the example above can be fixed by the comparison operator: if 1 == 1 end (in reality either the left or the right side of the expression contains an expression or a variable to compare against).","category":"page"},{"location":"lecture_01/lab/#UndefVarError","page":"Lab","title":"UndefVarError","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"While this error is quite self-explanatory, the exact causes are often quite puzzling for the user. The reason behind the confusion is to do with code scoping, which comes into play for example when trying to access a local variable from outside of a given function or just updating a global variable from within a simple loop. ","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"In the first example we show the former case, where variable is declared from within a function and accessed from outside afterwards.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"function plusone(x)\n uno = 1\n return x + uno\nend\nuno # defined only within plusone","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Unless there is variable I_am_not_defined in the global scope, the following should throw an error.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"I_am_not_defined","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Often these kind of errors arise as a result of bad code practices, such as long running sessions of Julia having long forgotten global variables, that do not exist upon new execution (this one in particular has been addressed by the authors of the reactive Julia notebooks Pluto.jl).","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"For more details on code scoping we recommend particular places in the bachelor course lectures here and there.","category":"page"},{"location":"lecture_01/lab/#ErrorException-and-error-function","page":"Lab","title":"ErrorException & error function","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"ErrorException is the most generic error, which can be thrown/raised just by calling the error function with a chosen string message. As a result developers may be inclined to misuse this for any kind of unexpected behavior a user can run into, often providing out-of-context/uninformative messages.","category":"page"},{"location":"lecture_01/demo/#Extensibility-of-the-language","page":"Examples","title":"Extensibility of the language","text":"","category":"section"},{"location":"lecture_01/demo/#DifferentialEquations","page":"Examples","title":"DifferentialEquations","text":"","category":"section"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"A package for solving differential equations, similar to odesolve in Matlab.","category":"page"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"Example:","category":"page"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"using DifferentialEquations\nfunction lotka_volterra(du,u,p,t)\n x, y = u\n α, β, δ, γ = p\n du[1] = dx = α*x - β*x*y\n du[2] = dy = -δ*y + γ*x*y\nend\nu0 = [1.0,1.0]\ntspan = (0.0,10.0)\np = [1.5,1.0,3.0,1.0]\nprob = ODEProblem(lotka_volterra,u0,tspan,p)\n\nsol = solve(prob)\nusing Plots\nplot(sol)","category":"page"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"(Image: )","category":"page"},{"location":"lecture_01/demo/#Measurements","page":"Examples","title":"Measurements","text":"","category":"section"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"A package defining \"numbers with precision\" and complete algebra on these numbers:","category":"page"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"using Measurements\n\na = 4.5 ± 0.1\nb = 3.8 ± 0.4\n\n2a + b\nsin(a)/cos(a) - tan(a)","category":"page"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"It also defines recipes for Plots.jl how to plot such numbers.","category":"page"},{"location":"lecture_01/demo/#Starting-ODE-from-an-interval","page":"Examples","title":"Starting ODE from an interval","text":"","category":"section"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"using Measurements\nu0 = [1.0±0.1,1.0±0.01]\n\nprob = ODEProblem(lotka_volterra,u0,tspan,p)\nsol = solve(prob)\nplot(sol,denseplot=false)","category":"page"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"(Image: )","category":"page"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"all algebraic operations are defined, \npasses all grid refinement techniques\nplot uses the correct plotting for intervals","category":"page"},{"location":"lecture_01/demo/#Integration-with-other-toolkits","page":"Examples","title":"Integration with other toolkits","text":"","category":"section"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"Flux: toolkit for modelling Neural Networks. Neural network is a function.","category":"page"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"integration with Measurements,\nIntegration with ODE (think of NN as part of the ODE)","category":"page"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"Turing: Probabilistic modelling toolkit","category":"page"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"integration with FLux (NN)\ninteration with ODE\nusing arbitrary bijective transformations, Bijectors.jl","category":"page"},{"location":"lecture_02/lab/#lab02","page":"Lab","title":"Lab 2: Predator-Prey Agents","text":"","category":"section"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"In the next labs you will implement your own predator-prey model. The model will contain wolves, sheep, and - to feed your sheep - some grass. The final simulation will be turn-based and the agents will be able to eat each other, reproduce, and die in every iteration. At every iteration of the simulation each agent will step forward in time via the agent_step! function. The steps for the agent_step! methods of animals and plants are written below in pseudocode.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"# for animals:\nagent_step!(animal, world)\n decrement energy by 1\n find & eat food (with probability pf)\n die if no more energy\n reproduce (with probability pr)\n\n# for plants:\nagent_step!(plant, world)\n grow if not at maximum size","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"The world in which the agents live will be the simplest possible world with zero dimensions (i.e. a Dict of ID=>Agent). Running and plotting your final result could look something like the plot below.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"(Image: img)","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"We will start implementing the basic functionality for each Agent like eat!ing, reproduce!ing, and a very simplistic World for your agents to live in. In the next lab you will refine both the type hierarchy of your Agents, as well as the design of the World in order to leverage the power of Julia's type system and compiler.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"We start with a very basic type hierarchy:","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"abstract type Agent end\nabstract type Animal <: Agent end\nabstract type Plant <: Agent end","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"We will implement the World for our Agents later, but it will essentially be implemented by a Dict which maps unique IDs to an Agent. Hence, every agent will need an ID.","category":"page"},{"location":"lecture_02/lab/#The-Grass-Agent","page":"Lab","title":"The Grass Agent","text":"","category":"section"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Let's start by implementing some Grass which will later be able to grow during each iteration of our simulation.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Define a mutable struct called Grass which is a subtype of Plant has the fields id (the unique identifier of this Agent - every agent needs one!), size (the current size of the Grass), and max_size. All fields should be integers.\nDefine a constructor for Grass which, given only an ID and a maximum size m, will create an instance of Grass that has a randomly initialized size in the range 1m. It should also be possible to create Grass, just with an ID and a default max_size of 10.\nImplement Base.show(io::IO, g::Grass) to get custom printing of your Grass such that the Grass is displayed with its size in percent of its max_size.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Hint: You can implement a custom show method for a new type MyType like this:","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"struct MyType\n x::Bool\nend\nBase.show(io::IO, a::MyType) = print(io, \"MyType $(a.x)\")","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Since Julia 1.8 we can also declare some fields of mutable structs as const, which can be used both to prevent us from mutating immutable fields (such as the ID) but can also be used by the compiler in certain cases.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"mutable struct Grass <: Plant\n const id::Int\n size::Int\n const max_size::Int\nend\n\nGrass(id,m=10) = Grass(id, rand(1:m), m)\n\nfunction Base.show(io::IO, g::Grass)\n x = g.size/g.max_size * 100\n # hint: to type the leaf in the julia REPL you can do:\n # \\:herb:\n print(io,\"🌿 #$(g.id) $(round(Int,x))% grown\")\nend","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Creating a few Grass agents can then look like this:","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Grass(1,5)\ng = Grass(2)\ng.id = 5","category":"page"},{"location":"lecture_02/lab/#Sheep-and-Wolf-Agents","page":"Lab","title":"Sheep and Wolf Agents","text":"","category":"section"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Animals are slightly different from plants. They will have an energy E, which will be increase (or decrease) if the agent eats (or reproduces) by a certain amount Delta E. Later we will also need a probability to find food p_f and a probability to reproduce p_r.c","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Define two mutable structs Sheep and Wolf that are subtypes of Animal and have the fields id, energy, Δenergy, reprprob, and foodprob.\nDefine constructors with the following default values:\nFor 🐑: E=4, Delta E=02, p_r=08, and p_f=06.\nFor 🐺: E=10, Delta E=8, p_r=01, and p_f=02.\nOverload Base.show to get pretty printing for your two new animals.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Solution for Sheep","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"mutable struct Sheep <: Animal\n const id::Int\n const energy::Float64\n Δenergy::Float64\n const reprprob::Float64\n const foodprob::Float64\nend\n\nSheep(id, e=4.0, Δe=0.2, pr=0.8, pf=0.6) = Sheep(id,e,Δe,pr,pf)\n\nfunction Base.show(io::IO, s::Sheep)\n e = s.energy\n d = s.Δenergy\n pr = s.reprprob\n pf = s.foodprob\n print(io,\"🐑 #$(s.id) E=$e ΔE=$d pr=$pr pf=$pf\")\nend","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Solution for Wolf:","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"mutable struct Wolf <: Animal\n const id::Int\n energy::Float64\n const Δenergy::Float64\n const reprprob::Float64\n const foodprob::Float64\nend\n\nWolf(id, e=10.0, Δe=8.0, pr=0.1, pf=0.2) = Wolf(id,e,Δe,pr,pf)\n\nfunction Base.show(io::IO, w::Wolf)\n e = w.energy\n d = w.Δenergy\n pr = w.reprprob\n pf = w.foodprob\n print(io,\"🐺 #$(w.id) E=$e ΔE=$d pr=$pr pf=$pf\")\nend","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Sheep(4)\nWolf(5)","category":"page"},{"location":"lecture_02/lab/#The-World","page":"Lab","title":"The World","text":"","category":"section"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Before our agents can eat or reproduce we need to build them a World. The simplest (and as you will later see, somewhat suboptimal) world is essentially a Dict from IDs to agents. Later we will also need the maximum ID, lets define a world with two fields:","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"mutable struct World{A<:Agent}\n agents::Dict{Int,A}\n max_id::Int\nend","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Implement a constructor for the World which accepts a vector of Agents.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"function World(agents::Vector{<:Agent})\n max_id = maximum(a.id for a in agents)\n World(Dict(a.id=>a for a in agents), max_id)\nend\n\n# optional: overload Base.show\nfunction Base.show(io::IO, w::World)\n println(io, typeof(w))\n for (_,a) in w.agents\n println(io,\" $a\")\n end\nend","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_02/lab/#Sheep-eats-Grass","page":"Lab","title":"Sheep eats Grass","text":"","category":"section"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"We can implement the behaviour of our various agents with respect to each other by leveraging Julia's multiple dispatch.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Implement a function eat!(::Sheep, ::Grass, ::World) which increases the sheep's energy by Delta E multiplied by the size of the grass.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"After the sheep's energy is updated the grass is eaten and its size counter has to be set to zero.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Note that you do not yet need the world in this function. It is needed later for the case of wolves eating sheep.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"function eat!(sheep::Sheep, grass::Grass, w::World)\n sheep.energy += grass.size * sheep.Δenergy\n grass.size = 0\nend\nnothing # hide","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Below you can see how a fully grown grass is eaten by a sheep. The sheep's energy changes size of the grass is set to zero.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"grass = Grass(1)\nsheep = Sheep(2)\nworld = World([grass, sheep])\neat!(sheep,grass,world);\nworld","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Note that the order of the arguments has a meaning here. Calling eat!(grass,sheep,world) results in a MethodError which is great, because Grass cannot eat Sheep.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"eat!(grass,sheep,world);","category":"page"},{"location":"lecture_02/lab/#Wolf-eats-Sheep","page":"Lab","title":"Wolf eats Sheep","text":"","category":"section"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"The eat! method for wolves increases the wolf's energy by sheep.energy * wolf.Δenergy and kills the sheep (i.e. removes the sheep from the world). There are other situationsin which agents die , so it makes sense to implement another function kill_agent!(::Animal,::World).","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Hint: You can use delete! to remove agents from the dictionary in your world.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"function eat!(wolf::Wolf, sheep::Sheep, w::World)\n wolf.energy += sheep.energy * wolf.Δenergy\n kill_agent!(sheep,w)\nend\n\nkill_agent!(a::Agent, w::World) = delete!(w.agents, a.id)\nnothing # hide","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"With a correct eat! method you should get results like this:","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"grass = Grass(1);\nsheep = Sheep(2);\nwolf = Wolf(3);\nworld = World([grass, sheep, wolf])\neat!(wolf,sheep,world);\nworld","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"The sheep is removed from the world and the wolf's energy increased by Delta E.","category":"page"},{"location":"lecture_02/lab/#Reproduction","page":"Lab","title":"Reproduction","text":"","category":"section"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Currently our animals can only eat. In our simulation we also want them to reproduce. We will do this by adding a reproduce! method to Animal.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Write a function reproduce! that takes an Animal and a World. Reproducing will cost an animal half of its energy and then add an almost identical copy of the given animal to the world. The only thing that is different from parent to child is the ID. You can simply increase the max_id of the world by one and use that as the new ID for the child.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"function reproduce!(a::Animal, w::World)\n a.energy = a.energy/2\n new_id = w.max_id + 1\n â = deepcopy(a)\n â.id = new_id\n w.agents[â.id] = â\n w.max_id = new_id\nend","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"You can avoid mutating the id field (which could be considered bad practice) by reconstructing the child from scratch:","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"function reproduce!(a::A, w::World) where A<:Animal\n a.energy = a.energy/2\n a_vals = [getproperty(a,n) for n in fieldnames(A) if n!=:id]\n new_id = w.max_id + 1\n â = A(new_id, a_vals...)\n w.agents[â.id] = â\n w.max_id = new_id\nend\nnothing # hide","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"s1, s2 = Sheep(1), Sheep(2)\nw = World([s1, s2])\nreproduce!(s1, w);\nw","category":"page"},{"location":"lecture_07/lecture/#macro_lecture","page":"Lecture","title":"Macros","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"What is macro? In its essence, macro is a function, which ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"takes as an input an expression (parsed input)\nmodify the expressions in argument\ninsert the modified expression at the same place as the one that is parsed.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Macros are necessary because they execute after the code is parsed (2nd step in conversion of source code to binary as described in last lect, after Meta.parse) therefore, macros allow the programmer to generate and include fragments of customized code before the full program is compiled run. Since they are executed during parsing, they do not have access to the values of their arguments, but only to their syntax.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"To illustrate the difference, consider the following example:","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"A very convenient and highly recommended ways to write macros is to write functions modifying the Expression and then call that function in the macro. Let's demonstrate on an example, where every occurrence of sin is replaced by cos. We defined the function recursively traversing the AST and performing the substitution","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"replace_sin(x::Symbol) = x == :sin ? :cos : x\nreplace_sin(e::Expr) = Expr(e.head, map(replace_sin, e.args)...)\nreplace_sin(u) = u","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"and then we define the macro","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro replace_sin(ex)\n\treplace_sin(esc(ex))\nend\n\n@replace_sin(cosp1(x) = 1 + sin(x))\ncosp1(1) == 1 + cos(1)","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"notice the following","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"the definition of the macro is similar to the definition of the function with the exception that instead of the keyword function we use keyword macro\nwhen calling the macro, we signal to the compiler our intention by prepending the name of the macro with @. \nthe macro receives the expression(s) as the argument instead of the evaluated argument and also returns an expression that is placed on the position where the macro has been called\nwhen you are using macro, you should be as a user aware that the code you are entering can be arbitrarily modified and you can receive something completely different. This meanst that @ should also serve as a warning that you are leaving Julia's syntax. In practice, it make sense to make things akin to how they are done in Julia or to write Domain Specific Language with syntax familiar in that domain.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Inspecting the lowered code","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Meta.@lower @replace_sin( 1 + sin(x))","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"We observe that there is no trace of macro in lowered code (compare to Meta.@lower 1 + cos(x), which demonstrates that the macro has been expanded after the code has been parsed but before it has been lowered. In this sense macros are indispensible, as you cannot replace them simply by the combination of Meta.parse end eval. You might object that in the above example it is possible, which is true, but only because the effect of the macro is in the global scope.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"ex = Meta.parse(\"cosp1(x) = 1 + sin(x)\")\nex = replace_sin(ex)\neval(ex)","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"The following example cannot be achieved by the same trick, as the output of the macro modifies just the body of the function","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"function cosp2(x)\n\t@replace_sin 2 + sin(x)\nend\ncosp2(1) ≈ (2 + cos(1))","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"This is not possible","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"function parse_eval_cosp2(x)\n\tex = Meta.parse(\"2 + sin(x)\")\n\tex = replace_sin(ex)\n\teval(ex)\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"as can be seen from","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_lowered cosp2(1)\nCodeInfo(\n1 ─ %1 = Main.cos(x)\n│ %2 = 2 + %1\n└── return %2\n)\n\njulia> @code_lowered parse_eval_cosp2(1)\nCodeInfo(\n1 ─ %1 = Base.getproperty(Main.Meta, :parse)\n│ ex = (%1)(\"2 + sin(x)\")\n│ ex = Main.replace_sin(ex)\n│ %4 = Main.eval(ex)\n└── return %4\n)","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\nScope of evaleval function is always evaluated in the global scope of the Module in which the macro is called (note that there is that by default you operate in the Main module). Moreover, eval takes effect after the function has been has been executed. This can be demonstrated as add1(x) = x + 1\nfunction redefine_add(x)\n eval(:(add1(x) = x - 1))\n add1(x)\nend\njulia> redefine_add(1)\n2\n\njulia> redefine_add(1)\n0\n","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Macros are quite tricky to debug. Macro @macroexpand allows to observe the expansion of macros. Observe the effect as","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"@macroexpand @replace_sin(cosp1(x) = 1 + sin(x))","category":"page"},{"location":"lecture_07/lecture/#What-goes-under-the-hood-of-macro-expansion?","page":"Lecture","title":"What goes under the hood of macro expansion?","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Let's consider that the compiler is compiling","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"function cosp2(x)\n\t@replace_sin 2 + sin(x)\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"First, Julia parses the code into the AST as","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"ex = Meta.parse(\"\"\"\n function cosp2(x)\n\t @replace_sin 2 + sin(x)\nend\n\"\"\") |> Base.remove_linenums!\ndump(ex)","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"We observe that there is a macrocall in the AST, which means that Julia will expand the macro and put it in place","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"ex.args[2].args[1].head \t# the location of the macrocall\nex.args[2].args[1].args[1] # which macro to call\nex.args[2].args[1].args[2] # line number\nex.args[2].args[1].args[3]\t# on which expression","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"We can manullay run replace_sin and insert it back on the relevant sub-part of the sub-tree","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"ex.args[2].args[1] = replace_sin(ex.args[2].args[1].args[3])\nex |> dump","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"now, ex contains the expanded macro and we can see that it correctly defines the function","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"eval(ex)","category":"page"},{"location":"lecture_07/lecture/#Calling-macros","page":"Lecture","title":"Calling macros","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Macros can be called without parentheses","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro showarg(ex)\n\tprintln(\"single argument version\")\n\t@show ex\n\tex\nend\n@showarg(1 + 1)\n@showarg 1 + 1","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Macros use the very same multiple dispatch as functions, which allows to specialize macro calls","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro showarg(x1, x2::Symbol)\n\tprintln(\"two argument version, second is Symbol\")\n\t@show x1\n\t@show x2\n\tx1\nend\nmacro showarg(x1, x2::Expr)\n\tprintln(\"two argument version, second is Expr\")\n\t@show x1\n\t@show x2\n\tx1\nend\n@showarg(1 + 1, x)\n@showarg(1 + 1, 1 + 3)\n@showarg 1 + 1, 1 + 3\n@showarg 1 + 1 1 + 3","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"(the @showarg(1 + 1, :x) raises an error, since :(:x) is of Type QuoteNode). ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Observe that macro dispatch is based on the types of AST that are handed to the macro, not the types that the AST evaluates to at runtime.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"List of all defined versions of macro","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"methods(var\"@showarg\")","category":"page"},{"location":"lecture_07/lecture/#lec7_quotation","page":"Lecture","title":"Notes on quotation","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"In the previous lecture we have seen that we can quote a block of code, which tells the compiler to treat the input as a data and parse it. We have talked about three ways of quoting code.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":":(quoted code)\nMeta.parse(input_string)\nquote ... end","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"The truth is that Julia does not do full quotation, but a quasiquotation as it allows you to interpolate expressions inside the quoted code using $ symbol similar to the string. This is handy, as sometimes, when we want to insert into the quoted code an result of some computation / preprocessing. Observe the following difference in returned code","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"a = 5\n:(x = a)\n:(x = $(a))\nlet y = :x\n :(1 + y), :(1 + $y)\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"In contrast to the behavior of :() (or quote ... end, true quotation would not perform interpolation where unary $ occurs. Instead, we would capture the syntax that describes interpolation and produce something like the following:","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"(\n :(1 + x), # Quasiquotation\n Expr(:call, :+, 1, Expr(:$, :x)), # True quotation\n)","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"for (v, f) in [(:sin, :foo_sin)]\n\tquote\n\t\t$(f)(x) = $(v)(x)\n\tend |> Base.remove_linenums! |> dump\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"When we need true quoting, i.e. we need something to stay quoted, we can use QuoteNode as","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro true_quote(e)\n QuoteNode(e)\nend\n\nlet y = :x\n (\n @true_quote(1 + $y),\n :(1 + $y),\n )\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"At first glance, QuoteNode wrapper seems to be useless. But QuoteNode has clear value when it's used inside a macro to indicate that something should stay quoted even after the macro finishes its work. Also notice that the expression received by macro are quoted, not quasiquoted, since in the latter case $y would be replaced. ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"We can demonstrate it by defining a new macro no_quote which will just return the expression as is ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro no_quote(ex)\n ex\nend\n\nlet y = :x\n @no_quote(1 + $y)\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"The error code snippet errors telling us that the expression \"$\" is outside of a quote block. This is because the macro @no_quote has returned a block with $ occuring outside of quote or string definition.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\nSome macros like @eval (recall last example)for f in [:setindex!, :getindex, :size, :length]\n @eval $(f)(A::MyMatrix, args...) = $(f)(A.x, args...)\nendor @benchmark support interpolation of values. This interpolation needs to be handled by the logic of the macro and is not automatically handled by Julia language.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Macros do not know about runtime values, they only know about syntax trees. When a macro receives an expression with a x in it, it can't interpolate the value of x into the syntax tree because it reads the syntax tree before x ever has a value! ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Instead, when a macro is given an expression with $ in it, it assumes you're going to give your own meaning to x. In the case of BenchmarkTools.jl they return code that has to wait until runtime to receive the value of x and then splice that value into an expression which is evaluated and benchmarked. Nowhere in the actual body of the macro do they have access to the value of x though.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\nWhy $ for interpolation?The $ string for interpolation was used as it identifies the interpolation inside the string and inside the command. For examplea = 5\ns = \"a = $(a)\"\ntypoef(s)\nprintln(s)\nfilename = \"/tmp/test_of_interpolation\"\nrun(`touch $(filename)`)","category":"page"},{"location":"lecture_07/lecture/#lec7_hygiene","page":"Lecture","title":"Macro hygiene","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Macro hygiene is a term coined in 1986 addressing the following problem: if you're automatically generating code, it's possible that you will introduce variable names in your generated code that will clash with existing variable names in the scope in which a macro is called. These clashes might cause your generated code to read from or write to variables that you should not be interacting with. A macro is hygienic when it does not interact with existing variables, which means that when macro is evaluated, it should not have any effect on the surrounding code. ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"By default, all macros in Julia are hygienic which means that variables introduced in the macro have automatically generated names, where Julia ensures they will not collide with user's variable. These variables are created by gensym function / macro. ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\ngensymgensym([tag]) Generates a symbol which will not conflict with other variable names.julia> gensym(\"hello\")\nSymbol(\"##hello#257\")","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Let's demonstrate it on our own version of an macro @elapsed which will return the time that was needed to evaluate the block of code.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro tooclean_elapsed(ex)\n\tquote\n\t\ttstart = time()\n\t\t$(ex)\n\t\ttime() - tstart\n\tend\nend\n\nfib(n) = n <= 1 ? n : fib(n-1) + fib(n - 2)\nlet \n\ttstart = \"should not change the value and type\"\n\tt = @tooclean_elapsed r = fib(10)\n\tprintln(\"the evaluation of fib took \", t, \"s and result is \", r)\n\t@show tstart\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"We see that variable r has not been assigned during the evaluation of macro. We have also used let block in orders not to define any variables in the global scope. The problem with the above is that it cannot be nested. Why is that? Let's observe how the macro was expanded","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"julia> Base.remove_linenums!(@macroexpand @tooclean_elapsed r = fib(10))\nquote\n var\"#12#tstart\" = Main.time()\n var\"#13#r\" = Main.fib(10)\n Main.time() - var\"#12#tstart\"\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"We see that tstart in the macro definition was replaced by var\"#12#tstart\", which is a name generated by Julia's gensym to prevent conflict. The same happens to r, which was replaced by var\"#13#r\". This names are the result of Julia's hygiene-enforcing pass, which is intended to prevent us from overwriting existing variables during macro expansion. This pass usually makes our macros safer, but it is also a source of confusion because it introduces a gap between the expressions we generate and the expressions that end up in the resulting source code. Notice that in the case of tstart, we actually wanted to replace tstart with a unique name, such that if we by a bad luck define tstart in our code, it would not be affected, as we can see in this example.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"let \n\ttstart = \"should not change the value and type \"\n\tt = @tooclean_elapsed r = fib(10)\n\tprintln(tstart, \" \", typeof(tstart))\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"But in the second case, we would actually very much like the variable r to retain its name, such that we can accesss the results (and also, ex can access and change other local variables). Julia offer a way to escape from the hygienic mode, which means that the variables will be used and passed as-is. Notice the effect if we escape just the expression ex ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro justright_elapsed(ex)\n\tquote\n\t\ttstart = time()\n\t\t$(esc(ex))\n\t\ttime() - tstart\n\tend\nend\n\nlet \n\ttstart = \"should not change the value and type \"\n\tt = @justright_elapsed r = fib(10)\n\tprintln(\"the evaluation of fib took \", t, \"s and result is \", r)\n\tprintln(tstart, \" \", typeof(tstart))\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"which now works as intended. We can inspect the output again using @macroexpand","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"julia> Base.remove_linenums!(@macroexpand @justright_elapsed r = fib(10))\nquote\n var\"#19#tstart\" = Main.time()\n r = fib(10)\n Main.time() - var\"#19#tstart\"\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"and compare it to Base.remove_linenums!(@macroexpand @justright_elapsed r = fib(10)). We see that the expression ex has its symbols intact. To use the escaping / hygience correctly, you need to have a good understanding how the macro evaluation works and what is needed. Let's now try the third version of the macro, where we escape everything as","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro toodirty_elapsed(ex)\n\tex = quote\n\t\ttstart = time()\n\t\t$(ex)\n\t\ttime() - tstart\n\tend\n\tesc(ex)\nend\n\nlet \n\ttstart = \"should not change the value and type \"\n\tt = @toodirty_elapsed r = fib(10)\n\tprintln(\"the evaluation of fib took \", t, \"s and result is \", r)\n\tprintln(tstart, \" \", typeof(tstart))\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Using @macroexpand we observe that @toodirty_elapsed does not have any trace of hygiene.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"julia> Base.remove_linenums!(@macroexpand @toodirty_elapsed r = fib(10))\nquote\n tstart = time()\n r = fib(10)\n time() - tstart\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"From the above we can also see that hygiene-pass occurs after the macro has been applied but before the code is lowered. esc is inserted to AST as a special node Expr(:escape,...), which can be seen from the follows.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"julia> esc(:x)\n:($(Expr(:escape, :x)))","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"The definition in essentials.jl:480 is pretty simple as esc(@nospecialize(e)) = Expr(:escape, e), but it does not tell anything about the actual implementation, which is hidden probably in the macro-expanding logic.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"With that in mind, we can now understand our original example with @replace_sin. Recall that we have defined it as ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro replace_sin(ex)\n\treplace_sin(esc(ex))\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"where the escaping replace_sin(esc(ex)) in communicates to compiler that ex should be used as without hygienating the ex. Indeed, if we lower it","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"function cosp2(x)\n\t@replace_sin 2 + sin(x)\nend\n\njulia> @code_lowered(cosp2(1.0))\nCodeInfo(\n1 ─ %1 = Main.cos(x)\n│ %2 = 2 + %1\n└── return %2\n)","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"we see it works as intended. Whereas if we use hygienic version","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro hygienic_replace_sin(ex)\n\treplace_sin(ex)\nend\n\nfunction hcosp2(x)\n\t@hygienic_replace_sin 2 + sin(x)\nend\n\njulia> @code_lowered(hcosp2(1.0))\nCodeInfo(\n1 ─ %1 = Main.cos(Main.x)\n│ %2 = 2 + %1\n└── return %2\n)","category":"page"},{"location":"lecture_07/lecture/#Why-hygienating-the-function-calls?","page":"Lecture","title":"Why hygienating the function calls?","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"function foo(x)\n\tcos(x) = exp(x)\n\t@replace_sin 1 + sin(x)\nend\n\nfoo(1.0) ≈ 1 + exp(1.0)\n\nfunction foo2(x)\n\tcos(x) = exp(x)\n\t@hygienic_replace_sin 1 + sin(x)\nend\n\nx = 1.0\nfoo2(1.0) ≈ 1 + cos(1.0)","category":"page"},{"location":"lecture_07/lecture/#Can-I-do-the-hygiene-by-myself?","page":"Lecture","title":"Can I do the hygiene by myself?","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Yes, it is by some considered to be much simpler (and safer) then to understand, how macro hygiene works.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro manual_elapsed(ex)\n x = gensym()\n esc(quote\n \t\t$(x) = time()\n \t$(ex)\n \ttime() - $(x)\n end\n )\nend\n\nlet \n\tt = @manual_elapsed r = fib(10)\n\tprintln(\"the evaluation of fib took \", t, \"s and result is \", r)\nend\n","category":"page"},{"location":"lecture_07/lecture/#How-macros-compose?","page":"Lecture","title":"How macros compose?","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro m1(ex)\n\tprintln(\"m1: \")\n\tdump(ex)\n\tex\nend\n\nmacro m2(ex)\n\tprintln(\"m2: \")\n\tdump(ex)\n\tesc(ex)\nend\n\n@m1 @m2 1 + sin(1)","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"which means that macros are expanded in the order from the outer most to inner most, which is exactly the other way around than functions.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"@macroexpand @m1 @m2 1 + sin(1)","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"also notice that the escaping is only partial (running @macroexpand @m2 @m1 1 + sin(1) would not change the results).","category":"page"},{"location":"lecture_07/lecture/#Write-@exfiltrate-macro","page":"Lecture","title":"Write @exfiltrate macro","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Since Julia's debugger is a complicated story, people have been looking for tools, which would simplify the debugging. One of them is a macro @exfiltrate, which copies all variables in a given scope to a safe place, from where they can be collected later on. This helps you in evaluating the function. F","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Whyle a full implementation is provided in package Infiltrator.jl, we can implement such functionality by outselves.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"We collect names and values of variables in a given scope using the macro Base.@locals\nWe store variables in some global variable in some module, such that we have one place from which we can retrieve them and we are certain that this storage would not interact with any existing code.\nIf the @exfiltrate should be easy, ideally called without parameters, it has to be implemented as a macro to supply the relevant variables to be stored.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"module Exfiltrator\n\nconst environment = Dict{Symbol, Any}()\n\nfunction copy_variables!(d::Dict)\n\tforeach(k -> delete!(environment, k), keys(environment))\n\tfor (k, v) in d\n\t\tenvironment[k] = v\n\tend\nend\n\nmacro exfiltrate()\n\tv = gensym(:vars)\n\tquote\n\t\t$(v) = $(esc((Expr(:locals))))\n\t\tcopy_variables!($(v))\n\tend\nend\n\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Test it to ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"using Main.Exfiltrator: @exfiltrate\nlet \n\tx,y,z = 1,\"hello\", (a = \"1\", b = \"b\")\n\t@exfiltrate\nend\n\nExfiltrator.environment\n\nfunction inside_function()\n\ta,b,c = 1,2,3\n\t@exfiltrate\nend\n\ninside_function()\n\nExfiltrator.environment\n\nfunction a()\n\ta = 1\n\t@exfiltrate\nend\n\nfunction b()\n\tb = 1\n\ta()\nend\nfunction c()\n\tc = 1\n\tb()\nend\n\nc()\nExfiltrator.environment","category":"page"},{"location":"lecture_07/lecture/#Domain-Specific-Languages-(DSL)","page":"Lecture","title":"Domain Specific Languages (DSL)","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Macros are convenient for writing domain specific languages, which are languages designed for specific domain. This allows them to simplify notation and / or make the notation familiar for people working in the field. For example in Turing.jl, the model of coinflips can be specified as ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"@model function coinflip(y)\n\n # Our prior belief about the probability of heads in a coin.\n p ~ Beta(1, 1)\n\n # The number of observations.\n N = length(y)\n for n in 1:N\n # Heads or tails of a coin are drawn from a Bernoulli distribution.\n y[n] ~ Bernoulli(p)\n end\nend;","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"which resembles, but not copy Julia's syntax due to the use of ~. A similar DSLs can be seen in ModelingToolkit.jl for differential equations, in Soss.jl again for expressing probability problems, in Metatheory.jl / SymbolicUtils.jl for defining rules on elements of algebras, or JuMP.jl for specific mathematical programs.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"One of the reasons for popularity of DSLs is that macro system is very helpful in their implementation, but it also contraints the DSL, as it has to be parseable by Julia's parser. This is a tremendous helps, because one does not have to care about how to parse numbers, strings, parenthesess, functions, etc. (recall the last lecture about replacing occurences of i variable).","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Let's jump into the first example adapted from John Myles White's howto. We would like to write a macro, which allows us to define graph in Graphs.jl just by defining edges.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"@graph begin \n\t1 -> 2\n\t2 -> 3\n\t3 -> 1\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"The above should expand to","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"using Graphs\ng = DiGraph(3)\nadd_edge!(g, 1,2)\nadd_edge!(g, 2,3)\nadd_edge!(g, 3,1)\ng","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Let's start with easy and observe, how ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"ex = Meta.parse(\"\"\"\nbegin \n\t1 -> 2\n\t2 -> 3\n\t3 -> 1\nend\n\"\"\")\nex = Base.remove_linenums!(ex)","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"is parsed to ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"quote\n 1->begin\n 2\n end\n 2->begin\n 3\n end\n 3->begin\n 1\n end\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"We see that ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"the sequence of statements is parsed to block (we know that from last lecture).\n-> is parsed to ->, i.e. ex.args[1].head == :-> with parameters being the first vertex ex.args[1].args[1] == 1 and the second vertex is quoted to ex.args[1].args[2].head == :block. ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"The main job will be done in the function parse_edge, which will parse one edge. It will check that the node defines edge (otherwise, it will return nothing, which will be filtered out)","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"function parse_edge(ex)\n\t#checking the syntax\n\t!hasproperty(ex, :head) && return(nothing)\n\t!hasproperty(ex, :args) && return(nothing)\n\tex.head != :-> && return(nothing)\n\tlength(ex.args) != 2 && return(nothing)\n\t!hasproperty(ex.args[2], :head) && return(nothing)\n\tex.args[2].head != :block && length(ex.args[2].args) == 1 && return(nothing)\n\n\t#ready to go\n\tsrc = ex.args[1]\n\t@assert src isa Integer\n\tdst = ex.args[2].args[1]\n\t@assert dst isa Integer\n\t:(add_edge!(g, $(src), $(dst)))\nend\n\nfunction parse_graph(ex)\n\t@assert ex.head == :block\n\tex = Base.remove_linenums!(ex)\n\tedges = filter(!isnothing, parse_edge.(ex.args))\n\tn = maximum(e -> maximum(e.args[3:4]), edges)\n\tquote\n g = Graphs.DiGraph($(n))\n $(edges...)\n g\n end\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Once we have the first version, let's make everything hygienic","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"function parse_edge(g, ex::Expr)\n\t#checking the syntax\n\tex.head != :-> && return(nothing)\n\tlength(ex.args) != 2 && return(nothing)\n\t!hasproperty(ex.args[2], :head) && return(nothing)\n\tex.args[2].head != :block && length(ex.args[2].args) == 1 && return(nothing)\n\n\t#ready to go\n\tsrc = ex.args[1]\n\t@assert src isa Integer\n\tdst = ex.args[2].args[1]\n\t@assert dst isa Integer\n\t:(add_edge!($(g), $(src), $(dst)))\nend\nparse_edge(g, ex) = nothing\n\nfunction parse_graph(ex)\n\t@assert ex.head == :block\n\tg = gensym(:graph)\n\tex = Base.remove_linenums!(ex)\n\tedges = filter(!isnothing, parse_edge.(g, ex.args))\n\tn = maximum(e -> maximum(e.args[3:4]), edges)\n\tquote\n $(g) = Graphs.DiGraph($(n))\n $(edges...)\n $(g)\n end\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"and we are ready to go","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro graph(ex)\n\tparse_graph(ex)\nend\n\n@graph begin\n\t1 -> 2\n\t2 -> 3\n\t3 -> 1\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"and we can check the output with @macroexpand.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"julia> @macroexpand @graph begin\n 1 -> 2\n 2 -> 3\n 3 -> 1\n end\nquote\n #= REPL[173]:8 =#\n var\"#27###graph#273\" = (Main.Graphs).DiGraph(3)\n #= REPL[173]:9 =#\n Main.add_edge!(var\"#27###graph#273\", 1, 2)\n Main.add_edge!(var\"#27###graph#273\", 2, 3)\n Main.add_edge!(var\"#27###graph#273\", 3, 1)\n #= REPL[173]:10 =#\n var\"#27###graph#273\"\nend","category":"page"},{"location":"lecture_07/lecture/#non-standard-string-literals","page":"Lecture","title":"non-standard string literals","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Julia allows to customize parsing of strings. For example we can define regexp matcher as r\"^\\s*(?:#|$)\", i.e. using the usual string notation prepended by the string r.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"You can define these \"parsers\" by yourself using the macro definition with suffix _str","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro debug_str(p)\n\t@show p\n p\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"by invoking it","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"debug\"hello\"","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"we see that the string macro receives string as an argument. ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Why are they useful? Sometimes, we want to use syntax which is not compatible with Julia's parser. For example IntervalArithmetics.jl allows to define an interval open only from one side, for example [a, b), which is something that Julia's parser would not like much. String macro solves this problem by letting you to write the parser by your own.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"struct Interval{T}\n\tleft::T\n\tright::T\n\tleft_open::Bool\n\tright_open::Bool\nend\n\nfunction Interval(s::String)\n\ts[1] == '(' || s[1] == '[' || error(\"left nterval can be only [,(\")\n\ts[end] == ')' || s[end] == ']' || error(\"left nterval can be only ],)\")\n \tleft_open = s[1] == '(' ? true : false\n \tright_open = s[end] == ')' ? true : false\n \tss = parse.(Float64, split(s[2:end-1],\",\"))\n \tlength(ss) != 2 && error(\"interval should have two numbers separated by ','\")\n \tInterval(ss..., left_open, right_open)\nend\n\nfunction Base.show(io::IO, r::Interval)\n\tlb = r.left_open ? \"(\" : \"[\"\n\trb = r.right_open ? \")\" : \"]\"\n\tprint(io, lb,r.left,\",\",r.right,rb)\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"We can check it does the job by trying Interval(\"[1,2)\"). Finally, we define a string macro as ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro int_str(s)\n\tInterval(s)\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"which allows us to define interval as int\"[1,2)\".","category":"page"},{"location":"lecture_07/lecture/#Sources","page":"Lecture","title":"Sources","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Great discussion on evaluation of macros.","category":"page"},{"location":"","page":"Home","title":"Home","text":"\"Scientific\n\"Scientific","category":"page"},{"location":"","page":"Home","title":"Home","text":"","category":"page"},{"location":"","page":"Home","title":"Home","text":"using Plots\nENV[\"GKSwstype\"] = \"100\"\ngr()","category":"page"},{"location":"","page":"Home","title":"Home","text":"Scientific Programming requires the highest performance but we also want to write very high level code to enable rapid prototyping and avoid error prone, low level implementations.","category":"page"},{"location":"","page":"Home","title":"Home","text":"The Julia programming language is designed with exactly those requirements of scientific computing in mind. In this course we will show you how to make use of the tools and advantages that jit-compiled Julia provides over dynamic, high-level languages like Python or lower level languages like C++.","category":"page"},{"location":"","page":"Home","title":"Home","text":"
\n \n
\n Learn the power of abstraction.\n Example: The essence of forward mode automatic differentiation.\n
\n
","category":"page"},{"location":"","page":"Home","title":"Home","text":"Before joining the course, consider reading the following two blog posts to figure out if Julia is a language in which you want to invest your time.","category":"page"},{"location":"","page":"Home","title":"Home","text":"What is great about Julia.\nWhat is bad about Julia.","category":"page"},{"location":"#What-will-you-learn?","page":"Home","title":"What will you learn?","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"First and foremost you will learn how to think julia - meaning how write fast, extensible, reusable, and easy-to-read code using things like optional typing, multiple dispatch, and functional programming concepts. The later part of the course will teach you how to use more advanced concepts like language introspection, metaprogramming, and symbolic computing. Amonst others you will implement your own automatic differetiation (the backbone of modern machine learning) package based on these advanced techniques that can transform intermediate representations of Julia code.","category":"page"},{"location":"#Organization","page":"Home","title":"Organization","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"This course webpage contains all information about the course that you need, including lecture notes, lab instructions, and homeworks. The official format of the course is 2+2 (2h lectures/2h labs per week) for 4 credits.","category":"page"},{"location":"","page":"Home","title":"Home","text":"The official course code is: B0M36SPJ and the timetable for the winter semester 2022 can be found here.","category":"page"},{"location":"","page":"Home","title":"Home","text":"The course will be graded based on points from your homework (max. 20 points) and points from a final project (max. 30 points).","category":"page"},{"location":"","page":"Home","title":"Home","text":"Below is a table that shows which lectures have homeworks (and their points).","category":"page"},{"location":"","page":"Home","title":"Home","text":"Homework 1 2 3 4 5 6 7 8 9 10 11 12 13\nPoints 2 2 2 2 2 2 2 2 - 2 - 2 -","category":"page"},{"location":"","page":"Home","title":"Home","text":"Hint: The first few homeworks are easier. Use them to fill up your points.","category":"page"},{"location":"#final_project","page":"Home","title":"Final project","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"The final project will be individually agreed on for each student. Ideally you can use this project to solve a problem you have e.g. in your thesis, but don't worry - if you cannot come up with an own project idea, we will suggest one to you. More info and project suggestion can be found here.","category":"page"},{"location":"#Grading","page":"Home","title":"Grading","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"Your points from the homeworks and the final project are summed and graded by the standard grading scale below.","category":"page"},{"location":"","page":"Home","title":"Home","text":"Grade A B C D E F\nPoints 45-50 40-44 35-39 30-34 25-29 0-25","category":"page"},{"location":"#emails","page":"Home","title":"Teachers","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"– E-mail Room Role\nTomáš Pevný pevnak@protonmail.ch KN:E-406 Lecturer\nVašek Šmídl smidlva1@fjfi.cvut.cz KN:E-333 Lecturer\nMatěj Zorek zorekmat@fel.cvut.cz KN:E-333 Lab Instructor\nNiklas Heim heimnikl@fel.cvut.cz KN:E-333 Lab Instructor","category":"page"},{"location":"#Prerequisites","page":"Home","title":"Prerequisites","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"There are no hard requirements to take the course, but if you are not at all familiar with Julia we recommend you to take Julia for Optimization and Learning before enrolling in this course. The Functional Programming course also contains some helpful concepts for this course. And knowledge about computer hardware, namely basics of how CPU works, how it interacts with memory through caches, and basics of multi-threadding certainly helps.","category":"page"},{"location":"#References","page":"Home","title":"References","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"Official documentation\nModern Julia Workflows\nWorkflow tips, and what is new in v1.9\nThink Julia: How to Think Like a Computer Scientist\nFrom Zero to Julia!\nWikiBooks\nJustin Krumbiel's excellent introduction to the package manager.\njuliadatascience.io contains an excellent introduction to plotting with Makie.\nThe art of multiple dispatch\nMIT Course: Julia Computation\nTim Holy's Advanced Scientific Computing","category":"page"},{"location":"lecture_06/hw/#Homework-6:-Find-variables","page":"Homework","title":"Homework 6: Find variables","text":"","category":"section"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"Following the lab exercises, you may think that metaprogramming is a fun little exercise. Let's challenge this notion in this homework, where YOU are being trusted with catching all the edge cases in an AST.","category":"page"},{"location":"lecture_06/hw/#How-to-submit?","page":"Homework","title":"How to submit?","text":"","category":"section"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"Put the code of the compulsory task inside hw.jl. Zip only this file (not its parent folder) and upload it to BRUTE. Your file should not use any 3rd party dependency.","category":"page"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"
\n
Homework (2 points)
\n
","category":"page"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"Your task is to find all single letter variables in an expression, i.e. for example when given expression","category":"page"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"x + 2*y*z - c*x","category":"page"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"return an array of unique alphabetically sorted symbols representing variables in an expression.","category":"page"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"[:c, :x, :y, :z]","category":"page"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"Implement this in a function called find_variables. Note that there may be some edge cases that you may have to handle in a special way, such as ","category":"page"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"variable assignments r = x*x should return the variable on the left as well (r in this case)\nignoring symbols representing single letter function calls such as f(x)","category":"page"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"
\n","category":"page"},{"location":"lecture_06/hw/#Voluntary-exercise","page":"Homework","title":"Voluntary exercise","text":"","category":"section"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"
\n
Voluntary exercise
\n
","category":"page"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"Create a function that replaces each of +, -, * and / with the respective checked operation, which checks for overflow. E.g. + should be replaced by Base.checked_add.","category":"page"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"
\n","category":"page"},{"location":"lecture_05/hw/#Homework-5:-Root-finding-of-polynomials","page":"Homework","title":"Homework 5: Root finding of polynomials","text":"","category":"section"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"This homework should test your ability to use the knowledge of benchmarking, profiling and others to improve an existing implementation of root finding methods for polynomials. The provided code is of questionable quality. In spite of the artificial nature, it should simulate a situation in which you may find yourself quite often, as it represents some intermediate step of going from a simple script to something, that starts to resemble a package.","category":"page"},{"location":"lecture_05/hw/#How-to-submit?","page":"Homework","title":"How to submit?","text":"","category":"section"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"Put the modified root_finding.jl code inside hw.jl. Zip only this file (not its parent folder) and upload it to BRUTE. Your file should not use any dependency other than those already present in the root_finding.jl.","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"
\n
Homework (2 points)
\n
","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"Use profiler on the find_root function to find a piece of unnecessary code, that takes more time than the computation itself. The finding of roots with the polynomial ","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"p(x) = (x - 3)(x - 2)(x - 1)x(x + 1)(x + 2)(x + 3) = x^7 - 14x^5 + 49x^3 - 36x","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"should not take more than 50μs when running with the following parameters","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"atol = 1e-12\nmaxiter = 100\nstepsize = 0.95\n\nx₀ = find_root(p, Bisection(), -5.0, 5.0, maxiter, stepsize, atol)\nx₀ = find_root(p, Newton(), -5.0, 5.0, maxiter, stepsize, atol)\nx₀ = find_root(p, Secant(), -5.0, 5.0, maxiter, stepsize, atol)","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"Remove obvious type instabilities in both find_root and step! functions. Each variable with \"inferred\" type ::Any in @code_warntype will be penalized.","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"HINTS:","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"running the function repeatedly 1000x helps in the profiler sampling\nfocus on parts of the code that may have been used just for debugging purposes","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"
\n","category":"page"},{"location":"lecture_05/hw/#Voluntary-exercise","page":"Homework","title":"Voluntary exercise","text":"","category":"section"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"
\n
Voluntary exercise
\n
","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"Use Plots.jl to plot the polynomial p on the interval -5 5 and visualize the progress/convergence of each method, with a dotted vertical line and a dot on the x-axis for each subsequent root approximation x̃.","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"HINTS:","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"plotting scalar function f - plot(r, f), where r is a range of x values at which we evaluate f\nupdating an existing plot - either plot!(plt, ...) or plot!(...), in the former case the plot lives in variable plt whereas in the latter we modify some implicit global variable\nplotting dots - for example with scatter/scatter!\nplot([(1.0,2.0), (1.0,3.0)], ls=:dot) will create a dotted line from position (x=1.0,y=2.0) to (x=1.0,y=3.0)","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"
","category":"page"}] +[{"location":"lecture_08/hw/#hw08","page":"Homework","title":"Homework 08","text":"","category":"section"},{"location":"lecture_08/hw/","page":"Homework","title":"Homework","text":"In this homework you will write an additional rule for our scalar reverse AD from the lab. For this homework, please write all your code in one file hw.jl which you have to zip and upload to BRUTE as usual. The solution to the lab is below.","category":"page"},{"location":"lecture_08/hw/","page":"Homework","title":"Homework","text":"mutable struct TrackedReal{T<:Real}\n data::T\n grad::Union{Nothing,T}\n children::Dict\n # this field is only need for printing the graph. you can safely remove it.\n name::String\nend\n\ntrack(x::Real,name=\"\") = TrackedReal(x,nothing,Dict(),name)\n\nfunction Base.show(io::IO, x::TrackedReal)\n t = isempty(x.name) ? \"(tracked)\" : \"(tracked $(x.name))\"\n print(io, \"$(x.data) $t\")\nend\n\nfunction accum!(x::TrackedReal)\n if isnothing(x.grad)\n x.grad = sum(w*accum!(v) for (v,w) in x.children)\n end\n x.grad\nend\n\nfunction Base.:*(a::TrackedReal, b::TrackedReal)\n z = track(a.data * b.data, \"*\")\n a.children[z] = b.data # dz/da=b\n b.children[z] = a.data # dz/db=a\n z\nend\n\nfunction Base.:+(a::TrackedReal{T}, b::TrackedReal{T}) where T\n z = track(a.data + b.data, \"+\")\n a.children[z] = one(T)\n b.children[z] = one(T)\n z\nend\n\nfunction Base.sin(x::TrackedReal)\n z = track(sin(x.data), \"sin\")\n x.children[z] = cos(x.data)\n z\nend\n\nfunction gradient(f, args::Real...)\n ts = track.(args)\n y = f(ts...)\n y.grad = 1.0\n accum!.(ts)\nend","category":"page"},{"location":"lecture_08/hw/","page":"Homework","title":"Homework","text":"We will use it to compute the derivative of the Babylonian square root.","category":"page"},{"location":"lecture_08/hw/","page":"Homework","title":"Homework","text":"babysqrt(x, t=(1+x)/2, n=10) = n==0 ? t : babysqrt(x, (t+x/t)/2, n-1)\nnothing # hide","category":"page"},{"location":"lecture_08/hw/","page":"Homework","title":"Homework","text":"In order to differentiate through babysqrt you will need a reverse rule for / for Base.:/(TrackedReal,TrackedReal) as well as the cases where you divide with constants in volved (e.g. Base.:/(TrackedReal,Real)).","category":"page"},{"location":"lecture_08/hw/","page":"Homework","title":"Homework","text":"
\n
Homework (2 points)
\n
","category":"page"},{"location":"lecture_08/hw/","page":"Homework","title":"Homework","text":"Write the reverse rules for / and the missing rules for + such that you can differentiate through division and addition with and without constants.","category":"page"},{"location":"lecture_08/hw/","page":"Homework","title":"Homework","text":"
","category":"page"},{"location":"lecture_08/hw/","page":"Homework","title":"Homework","text":"function Base.:/(a::TrackedReal, b::TrackedReal)\n z = track(a.data / b.data)\n a.children[z] = 1/b.data\n b.children[z] = -a.data / b.data^2\n z\nend\nfunction Base.:/(a::TrackedReal, b::Real)\n z = track(a.data/b)\n a.children[z] = 1/b\n z\nend\n\nfunction Base.:+(a::Real, b::TrackedReal{T}) where T\n z = track(a + b.data, \"+\")\n b.children[z] = one(T)\n z\nend\nBase.:+(a::TrackedReal,b::Real) = b+a","category":"page"},{"location":"lecture_08/hw/","page":"Homework","title":"Homework","text":"You can verify your solution with the gradient function.","category":"page"},{"location":"lecture_08/hw/","page":"Homework","title":"Homework","text":"gradient(babysqrt, 2.0)\n1/(2babysqrt(2.0))","category":"page"},{"location":"lecture_04/lab/#Lab-04:-Packaging","page":"Lab","title":"Lab 04: Packaging","text":"","category":"section"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"projdir = dirname(Base.active_project())\ninclude(joinpath(projdir,\"src\",\"lecture_03\",\"Lab03Ecosystem.jl\"))\n\nfunction find_food(a::Animal, w::World)\n as = filter(x -> eats(a,x), w.agents |> values |> collect)\n isempty(as) ? nothing : sample(as)\nend\neats(::Animal{Sheep},g::Plant{Grass}) = g.size > 0\neats(::Animal{Wolf},::Animal{Sheep}) = true\neats(::Agent,::Agent) = false","category":"page"},{"location":"lecture_04/lab/#Warmup-Stepping-through-time","page":"Lab","title":"Warmup - Stepping through time","text":"","category":"section"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"We now have all necessary functions in place to make agents perform one step of our simulation. At the beginning of each step an animal looses energy. Afterwards it tries to find some food, which it will subsequently eat. If the animal then has less than zero energy it dies and is removed from the world. If it has positive energy it will try to reproduce.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"Plants have a simpler life. They simply grow if they have not reached their maximal size.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"Implement a method agent_step!(::Animal,::World) which performs the following steps:\nDecrement E of agent by 1.0.\nWith p_f, try to find some food and eat it.\nIf E0, the animal dies.\nWith p_r, try to reproduce.\nImplement a method agent_step!(::Plant,::World) which performs the following steps:\nIf the size of the plant is smaller than max_size, increment the plant's size by one.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"function agent_step!(p::Plant, w::World)\n if p.size < p.max_size\n p.size += 1\n end\nend\n\nfunction agent_step!(a::Animal, w::World)\n a.energy -= 1\n if rand() <= a.foodprob\n dinner = find_food(a,w)\n eat!(a, dinner, w)\n end\n if a.energy < 0\n kill_agent!(a,w)\n return\n end\n if rand() <= a.reprprob\n reproduce!(a,w)\n end\nend\n\nnothing # hide","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"An agent_step! of a sheep in a world with a single grass should make it consume the grass, let it reproduce, and eventually die if there is no more food and its energy is at zero:","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"sheep = Sheep(1,2.0,2.0,1.0,1.0,male);\ngrass = Grass(2,2,2);\nworld = World([sheep, grass])\nagent_step!(sheep, world); world\n# NOTE: The second agent step leads to an error.\n# Can you figure out what is the problem here?\nagent_step!(sheep, world); world","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"Finally, lets implement a function world_step! which performs one agent_step! for each agent. Note that simply iterating over all agents could lead to problems because we are mutating the agent dictionary. One solution for this is to iterate over a copy of all agent IDs that are present when starting to iterate over agents. Additionally, it could happen that an agent is killed by another one before we apply agent_step! to it. To solve this you can check if a given ID is currently present in the World.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"# make it possible to eat nothing\neat!(::Animal, ::Nothing, ::World) = nothing\n\nfunction world_step!(world::World)\n # make sure that we only iterate over IDs that already exist in the\n # current timestep this lets us safely add agents\n ids = copy(keys(world.agents))\n\n for id in ids\n # agents can be killed by other agents, so make sure that we are\n # not stepping dead agents forward\n !haskey(world.agents,id) && continue\n\n a = world.agents[id]\n agent_step!(a,world)\n end\nend","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"w = World([Sheep(1), Sheep(2), Wolf(3)])\nworld_step!(w); w\nworld_step!(w); w\nworld_step!(w); w","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"Finally, lets run a few simulation steps and plot the solution","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"n_grass = 1_000\nn_sheep = 40\nn_wolves = 4\n\ngs = [Grass(id) for id in 1:n_grass]\nss = [Sheep(id) for id in (n_grass+1):(n_grass+n_sheep)]\nws = [Wolf(id) for id in (n_grass+n_sheep+1):(n_grass+n_sheep+n_wolves)]\nw = World(vcat(gs,ss,ws))\n\ncounts = Dict(n=>[c] for (n,c) in agent_count(w))\nfor _ in 1:100\n world_step!(w)\n for (n,c) in agent_count(w)\n push!(counts[n],c)\n end\nend\n\nusing Plots\nplt = plot()\nfor (n,c) in counts\n plot!(plt, c, label=string(n), lw=2)\nend\nplt","category":"page"},{"location":"lecture_04/lab/#Package:-Ecosystem.jl","page":"Lab","title":"Package: Ecosystem.jl","text":"","category":"section"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"In the main section of this lab you will create your own Ecosystem.jl package to organize and test (!) the code that we have written so far.","category":"page"},{"location":"lecture_04/lab/#PkgTemplates.jl","page":"Lab","title":"PkgTemplates.jl","text":"","category":"section"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"The simplest way to create a new package in Julia is to use PkgTemplates.jl. ]add PkgTemplates to your global julia env and create a new package by running:","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"using PkgTemplates\nTemplate(interactive=true)(\"Ecosystem\")","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"to interactively specify various options for your new package or use the following snippet to generate it programmatically:","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"using PkgTemplates\n\n# define the package template\ntemplate = Template(;\n user = \"GithubUserName\", # github user name\n authors = [\"Author1\", \"Author2\"], # list of authors\n dir = \"/path/to/folder/\", # dir in which the package will be created\n julia = v\"1.8\", # compat version of Julia\n plugins = [\n !CompatHelper, # disable CompatHelper\n !TagBot, # disable TagBot\n Readme(; inline_badges = true), # added readme file with badges\n Tests(; project = true), # added Project.toml file for unit tests\n Git(; manifest = false), # add manifest.toml to .gitignore\n License(; name = \"MIT\") # addedMIT licence\n ],\n)\n\n# execute the package template (this creates all files/folders)\ntemplate(\"Ecosystem\")","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"This should have created a new folder Ecosystem which looks like below.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":".\n├── LICENSE\n├── Project.toml\n├── README.md\n├── src\n│ └── Ecosystem.jl\n└── test\n ├── Manifest.toml\n ├── Project.toml\n └── runtests.jl","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"If you ]activate /path/to/Ecosystem you should be able to run ]test to run the autogenerated test (which is not doing anything) and get the following output:","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"(Ecosystem) pkg> test\n Testing Ecosystem\n Status `/private/var/folders/6h/l9_skfms2v3dt8z3zfnd2jr00000gn/T/jl_zd5Uai/Project.toml`\n [e77cd98c] Ecosystem v0.1.0 `~/repos/Ecosystem`\n [8dfed614] Test `@stdlib/Test`\n Status `/private/var/folders/6h/l9_skfms2v3dt8z3zfnd2jr00000gn/T/jl_zd5Uai/Manifest.toml`\n [e77cd98c] Ecosystem v0.1.0 `~/repos/Ecosystem`\n [2a0f44e3] Base64 `@stdlib/Base64`\n [b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`\n [56ddb016] Logging `@stdlib/Logging`\n [d6f4376e] Markdown `@stdlib/Markdown`\n [9a3f8284] Random `@stdlib/Random`\n [ea8e919c] SHA v0.7.0 `@stdlib/SHA`\n [9e88b42a] Serialization `@stdlib/Serialization`\n [8dfed614] Test `@stdlib/Test`\n Testing Running tests...\nTest Summary: |Time\nEcosystem.jl | None 0.0s\n Testing Ecosystem tests passed ","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"warning: Warning\nFrom now on make sure that you always have the Ecosystem enviroment enabled. Otherwise you will not end up with the correct dependencies in your packages","category":"page"},{"location":"lecture_04/lab/#Adding-content-to-Ecosystem.jl","page":"Lab","title":"Adding content to Ecosystem.jl","text":"","category":"section"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"Next, let's add the types and functions we have defined so far. You can use include(\"path/to/file.jl\") in the main module file at src/Ecosystem.jl to bring some structure in your code. An exemplary file structure could look like below.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":".\n├── LICENSE\n├── Manifest.toml\n├── Project.toml\n├── README.md\n├── src\n│ ├── Ecosystem.jl\n│ ├── animal.jl\n│ ├── plant.jl\n│ └── world.jl\n└── test\n └── runtests.jl","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"While you are adding functionality to your package you can make great use of Revise.jl. Loading Revise.jl before your Ecosystem.jl will automatically recompile (and invalidate old methods!) while you develop. You can install it in your global environment and and create a $HOME/.config/startup.jl which always loads Revise. It can look like this:","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"# try/catch block to make sure you can start julia if Revise should not be installed\ntry\n using Revise\ncatch e\n @warn(e.msg)\nend","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"warning: Warning\nAt some point along the way you should run into problems with the sample functions or when trying using StatsBase. This is normal, because you have not added the package to the Ecosystem environment yet. Adding it is as easy as ]add StatsBase. Your Ecosystem environment should now look like this:(Ecosystem) pkg> status\nProject Ecosystem v0.1.0\nStatus `~/repos/Ecosystem/Project.toml`\n [2913bbd2] StatsBase v0.33.21","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"In order to use your new types/functions like below","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"using Ecosystem\n\nSheep(2)","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"you have to export them from your module. Add exports for all important types and functions.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"# src/Ecosystem.jl\nmodule Ecosystem\n\nusing StatsBase\n\nexport World\nexport Species, PlantSpecies, AnimalSpecies, Grass, Sheep, Wolf\nexport Agent, Plant, Animal\nexport agent_step!, eat!, eats, find_food, reproduce!, world_step!, agent_count\n\n# ....\n\nend","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_04/lab/#Unit-tests","page":"Lab","title":"Unit tests","text":"","category":"section"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"Every package should have tests which verify the correctness of your implementation, such that you can make changes to your codebase and remain confident that you did not break anything.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"Julia's Test package provides you functionality to easily write unit tests.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"In the file test/runtests.jl, create a new @testset and write three @tests which check that the show methods we defined for Grass, Sheep, and Wolf work as expected.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"The function repr(x) == \"some string\" to check if the string representation we defined in the Base.show overload returns what you expect.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"# using Ecosystem\nusing Test\n\n@testset \"Base.show\" begin\n g = Grass(1,1,1)\n s = Animal{Sheep}(2,1,1,1,1,male)\n w = Animal{Wolf}(3,1,1,1,1,female)\n @test repr(g) == \"🌿 #1 100% grown\"\n @test repr(s) == \"🐑♂ #2 E=1.0 ΔE=1.0 pr=1.0 pf=1.0\"\n @test repr(w) == \"🐺♀ #3 E=1.0 ΔE=1.0 pr=1.0 pf=1.0\"\nend","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_04/lab/#Github-CI","page":"Lab","title":"Github CI","text":"","category":"section"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"If you want you can upload you package to Github and add the julia-runtest Github Action to automatically test your code for every new push you make to the repository.","category":"page"},{"location":"lecture_04/lab/","page":"Lab","title":"Lab","text":"
","category":"page"},{"location":"lecture_03/lab/#lab03","page":"Lab","title":"Lab 3: Predator-Prey Agents","text":"","category":"section"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"projdir = dirname(Base.active_project())\ninclude(joinpath(projdir,\"src\",\"lecture_02\",\"Lab02Ecosystem.jl\"))","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"In this lab we will look at two different ways of extending our agent simulation to take into account that animals can have two different sexes: female and male.","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"In the first part of the lab you will re-use the code from lab 2 and create a new type of sheep (⚥Sheep) which has an additional field sex. In the second part you will redesign the type hierarchy from scratch using parametric types to make this agent system much more flexible and julian.","category":"page"},{"location":"lecture_03/lab/#Part-I:-Female-and-Male-Sheep","page":"Lab","title":"Part I: Female & Male Sheep","text":"","category":"section"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"The code from lab 2 that you will need in the first part of this lab can be found here.","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"The goal of the first part of the lab is to demonstrate the forwarding method (which is close to how things are done in OOP) by implementing a sheep that can have two different sexes and can only reproduce with another sheep of opposite sex.","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"This new type of sheep needs an additonal field sex::Symbol which can be either :male or :female. In OOP we would simply inherit from Sheep and create a ⚥Sheep with an additional field. In Julia there is no inheritance - only subtyping of abstract types. As you cannot inherit from a concrete type in Julia, we will have to create a wrapper type and forward all necessary methods. This is typically a sign of unfortunate type tree design and should be avoided, but if you want to extend a code base by an unforeseen type this forwarding of methods is a nice work-around. Our ⚥Sheep type will simply contain a classic sheep and a sex field","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"struct ⚥Sheep <: Animal\n sheep::Sheep\n sex::Symbol\nend\n⚥Sheep(id, e=4.0, Δe=0.2, pr=0.8, pf=0.6, sex=rand(Bool) ? :female : :male) = ⚥Sheep(Sheep(id,e,Δe,pr,pf),sex)\nnothing # hide","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"sheep = ⚥Sheep(1)\nsheep.sheep\nsheep.sex","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Instead of littering the whole code with custom getters/setters Julia allows us to overload the sheep.field behaviour by implementing custom getproperty/setproperty! methods.","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Implement custom getproperty/setproperty! methods which allow to access the Sheep inside the ⚥Sheep as if we would not be wrapping it.","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"# NOTE: the @forward macro we will discuss in a later lecture is based on this\n\nfunction Base.getproperty(s::⚥Sheep, name::Symbol)\n if name in fieldnames(Sheep)\n getfield(s.sheep,name)\n else\n getfield(s,name)\n end\nend\n\nfunction Base.setproperty!(s::⚥Sheep, name::Symbol, x)\n if name in fieldnames(Sheep)\n setfield!(s.sheep,name,x)\n else\n setfield!(s,name,x)\n end\nend","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"You should be able to do the following with your overloads now","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"sheep = ⚥Sheep(1)\nsheep.id\nsheep.sex\nsheep.energy += 1\nsheep","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"In order to make the ⚥Sheep work with the rest of the code we only have to forward the eat! method","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"eat!(s::⚥Sheep, food, world) = eat!(s.sheep, food, world);\nsheep = ⚥Sheep(1);\ngrass = Grass(2);\nworld = World([sheep,grass])\neat!(sheep, grass, world)","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"and implement a custom reproduce! method with the behaviour that we want.","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"However, the extension of Sheep to ⚥Sheep is a very object-oriented approach. With a little bit of rethinking, we can build a much more elegant solution that makes use of Julia's powerful parametric types.","category":"page"},{"location":"lecture_03/lab/#Part-II:-A-new,-parametric-type-hierarchy","page":"Lab","title":"Part II: A new, parametric type hierarchy","text":"","category":"section"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"First, let us note that there are two fundamentally different types of agents in our world: animals and plants. All species such as grass, sheep, wolves, etc. can be categorized as one of those two. We can use Julia's powerful, parametric type system to define one large abstract type for all agents Agent{S}. The Agent will either be an Animal or a Plant with a type parameter S which will represent the specific animal/plant species we are dealing with.","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"This new type hiearchy can then look like this:","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"abstract type Species end\n\nabstract type PlantSpecies <: Species end\nabstract type Grass <: PlantSpecies end\n\nabstract type AnimalSpecies <: Species end\nabstract type Sheep <: AnimalSpecies end\nabstract type Wolf <: AnimalSpecies end\n\nabstract type Agent{S<:Species} end\n\n# instead of Symbols we can use an Enum for the sex field\n# using an Enum here makes things easier to extend in case you\n# need more than just binary sexes and is also more explicit than\n# just a boolean\n@enum Sex female male","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"mutable struct World{A<:Agent}\n agents::Dict{Int,A}\n max_id::Int\nend\n\nfunction World(agents::Vector{<:Agent})\n max_id = maximum(a.id for a in agents)\n World(Dict(a.id=>a for a in agents), max_id)\nend\n\n# optional: overload Base.show\nfunction Base.show(io::IO, w::World)\n println(io, typeof(w))\n for (_,a) in w.agents\n println(io,\" $a\")\n end\nend","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Now we can create a concrete type Animal with the two parametric types and the fields that we already know from lab 2.","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"mutable struct Animal{A<:AnimalSpecies} <: Agent{A}\n const id::Int\n energy::Float64\n const Δenergy::Float64\n const reprprob::Float64\n const foodprob::Float64\n const sex::Sex\nend","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"To create an instance of Animal we have to specify the parametric type while constructing it","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Animal{Wolf}(1,5,5,1,1,female)","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Note that we now automatically have animals of any species without additional work. Starting with the overload of the show method we can already see that we can abstract away a lot of repetitive work into the type system. We can implement one single show method for all animal species!","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Implement Base.show(io::IO, a::Animal) with a single method for all Animals. You can get the pretty (unicode) printing of the Species types with another overload like this: Base.show(io::IO, ::Type{Sheep}) = print(io,\"🐑\")","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"function Base.show(io::IO, a::Animal{A}) where {A<:AnimalSpecies}\n e = a.energy\n d = a.Δenergy\n pr = a.reprprob\n pf = a.foodprob\n s = a.sex == female ? \"♀\" : \"♂\"\n print(io, \"$A$s #$(a.id) E=$e ΔE=$d pr=$pr pf=$pf\")\nend\n\n# note that for new species/sexes we will only have to overload `show` on the\n# abstract species types like below!\nBase.show(io::IO, ::Type{Sheep}) = print(io,\"🐑\")\nBase.show(io::IO, ::Type{Wolf}) = print(io,\"🐺\")","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Unfortunately we have lost the convenience of creating plants and animals by simply calling their species constructor. For example, Sheep is just an abstract type that we cannot instantiate. However, we can manually define a new constructor that will give us this convenience back. This is done in exactly the same way as defining a constructor for a concrete type:","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Sheep(id,E,ΔE,pr,pf,s=rand(Sex)) = Animal{Sheep}(id,E,ΔE,pr,pf,s)","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Ok, so we have a constructor for Sheep now. But what about all the other billions of species that you want to define in your huge master thesis project of ecosystem simulations? Do you have to write them all by hand? Do not despair! Julia has you covered.","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Overload all AnimalSpecies types with a constructor. You already know how to write constructors for specific types such as Sheep. Can you manage to sneak in a type variable? Maybe with Type?","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"function (A::Type{<:AnimalSpecies})(id::Int,E::T,ΔE::T,pr::T,pf::T,s::Sex) where T\n Animal{A}(id,E,ΔE,pr,pf,s)\nend\n\n# get the per species defaults back\nrandsex() = rand(instances(Sex))\nSheep(id; E=4.0, ΔE=0.2, pr=0.8, pf=0.6, s=randsex()) = Sheep(id, E, ΔE, pr, pf, s)\nWolf(id; E=10.0, ΔE=8.0, pr=0.1, pf=0.2, s=randsex()) = Wolf(id, E, ΔE, pr, pf, s)\nnothing # hide","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"We have our convenient, high-level behaviour back!","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Sheep(1)\nWolf(2)","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Check the methods for eat! and kill_agent! which involve Animals and update their type signatures such that they work for the new type hiearchy.","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"function eat!(wolf::Animal{Wolf}, sheep::Animal{Sheep}, w::World)\n wolf.energy += sheep.energy * wolf.Δenergy\n kill_agent!(sheep,w)\nend\n\n# no change\n# eat!(::Animal, ::Nothing, ::World) = nothing\n\n# no change\n# kill_agent!(a::Agent, w::World) = delete!(w.agents, a.id)\n\neats(::Animal{Wolf},::Animal{Sheep}) = true\neats(::Agent,::Agent) = false\n# this one needs to wait until we have `Plant`s\n# eats(::Animal{Sheep},g::Plant{Grass}) = g.size > 0\n\nnothing # hide","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Finally, we can implement the new behaviour for reproduce! which we wanted. Build a function which first finds an animal species of opposite sex and then lets the two reproduce (same behaviour as before).","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"mates(a::Animal{A}, b::Animal{A}) where A<:AnimalSpecies = a.sex != b.sex\nmates(::Agent, ::Agent) = false\n\nfunction find_mate(a::Animal, w::World)\n ms = filter(x->mates(x,a), w.agents |> values |> collect)\n isempty(ms) ? nothing : rand(ms)\nend\n\nfunction reproduce!(a::Animal{A}, w::World) where {A}\n m = find_mate(a,w)\n if !isnothing(m)\n a.energy = a.energy / 2\n vals = [getproperty(a,n) for n in fieldnames(Animal) if n ∉ [:id, :sex]]\n new_id = w.max_id + 1\n ŝ = Animal{A}(new_id, vals..., randsex())\n w.agents[ŝ.id] = ŝ\n w.max_id = new_id\n end\nend\nnothing # hide","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"s1 = Sheep(1, s=female)\ns2 = Sheep(2, s=male)\nw = World([s1, s2])\nreproduce!(s1, w); w","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"Implement the type hiearchy we designed for Plants as well.","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"mutable struct Plant{P<:PlantSpecies} <: Agent{P}\n id::Int\n size::Int\n max_size::Int\nend\n\n# constructor for all Plant{<:PlantSpecies} callable as PlantSpecies(...)\n(A::Type{<:PlantSpecies})(id, s, m) = Plant{A}(id,s,m)\n(A::Type{<:PlantSpecies})(id, m) = (A::Type{<:PlantSpecies})(id,rand(1:m),m)\n\n# default specific for Grass\nGrass(id; max_size=10) = Grass(id, rand(1:max_size), max_size)\n\nfunction Base.show(io::IO, p::Plant{P}) where P\n x = p.size/p.max_size * 100\n print(io,\"$P #$(p.id) $(round(Int,x))% grown\")\nend\n\nBase.show(io::IO, ::Type{Grass}) = print(io,\"🌿\")\n\nfunction eat!(sheep::Animal{Sheep}, grass::Plant{Grass}, w::World)\n sheep.energy += grass.size * sheep.Δenergy\n grass.size = 0\nend\neats(::Animal{Sheep},g::Plant{Grass}) = g.size > 0\n\nnothing # hide","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_03/lab/","page":"Lab","title":"Lab","text":"g = Grass(2)\ns = Sheep(3)\nw = World([g,s])\neat!(s,g,w); w","category":"page"},{"location":"lecture_12/lecture/#lec12","page":"Lecture","title":"Uncertainty Propagation in Ordinary Differential Equations","text":"","category":"section"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Differential equations are commonly used in science to describe many aspects of the physical world, ranging from dynamical systems and curves in space to complex multi-physics phenomena. ","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"As an example, consider a simple non-linear ordinary differential equation:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"beginalign\ndotx=alpha x-beta xydoty=-delta y+gamma xy \nendalign","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Which describes behavior of a predator-pray models in continuous times:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"x is the population of prey (sheep),\ny is the population of predator (wolfes)\nderivatives represent instantaneous growth rates of the populations\nt is the time and alpha beta gamma delta are parameters.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Can be written in vector arguments mathbfx=xy:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"fracdmathbfxdt=f(mathbfxtheta)","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"with arbitrary function f with vector of parameters theta.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"The first steps we may want to do with an ODE is to see it's evolution in time. The most simple approach is to discretize the time axis into steps: t = t_1 t_2 t_3 ldots t_T and evaluate solution at these points.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Replacing derivatives by differences:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"dot x leftarrow fracx_t-x_t-1Delta t","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"we can derive a general scheme (Euler solution):","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"mathbfx_t = mathbfx_t-1 + Deltat f(mathbfx_ttheta)","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"which can be written genericaly in julia :","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"\nfunction f(x,θ)\n α,β,γ,δ = θ\n x1,x2=x\n dx1 = α*x1 - β*x1*x2\n dx2 = δ*x1*x2 - γ*x2\n [dx1,dx2]\nend\n\nfunction solve(f,x0::AbstractVector,θ,dt,N)\n X = hcat([zero(x0) for i=1:N]...)\n X[:,1]=x0\n for t=1:N-1\n X[:,t+1]=X[:,t]+dt*f(X[:,t],θ)\n end\n X\nend","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Is simple and working (with sufficienty small dt):","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"(Image: )","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"ODE of this kind is an example of a \"complex\" simulation code that we may want to use, interact with, modify or incorporate into a more complex scheme.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"we will test how to re-define the elementary operations using custom types, automatic differentiation and automatic code generation\nwe will redefine the plotting operation to display the new type correctly\nwe will use composition to incorporate the ODE into a more complex solver","category":"page"},{"location":"lecture_12/lecture/#Uncertainty-propagation","page":"Lecture","title":"Uncertainty propagation","text":"","category":"section"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Prediction of the ODE model is valid only if all parameters and all initial conditions are accurate. This is almost never the case. While the number of sheep can be known, the number of wolfes in a forest is more uncertain. The same model holds for predator-prey in insects where the number of individuals can be only estimated.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Uncertain initial conditions:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"number of predators and prey given by a probability distribution \ninterval 0812 corresponds to uniform distribution U(0812)\ngaussian N(musigma), with mean mu and standard deviation sigma e.g. N(101)\nmore complicated distributions are more realistic (the number of animals is not negative!)","category":"page"},{"location":"lecture_12/lecture/#Ensemble-approach","page":"Lecture","title":"Ensemble approach","text":"","category":"section"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"The most simple approach is to represent distribution by an empirical density = discrete samples.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"p(mathbfx)approx frac1Ksum_k=1^K delta(mathbfx-mathbfx^(k))","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"In the case of a Gaussian, we just sample:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"K = 10\nX0 = [x0 .+ 0.1*randn(2) for _=1:K] # samples of initial conditions\nXens=[X=solve(f,X0[i],θ0,dt,N) for i=1:K] # solve multiple times","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"(can be implemented more elegantly using multiple dispatch on Vector{Vector})","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"(Image: )","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"While it is very simple and universal, it may become hard to interpret. ","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"What is the probability that it will higher than x_max?\nImproving accuracy with higher number of samples (expensive!)","category":"page"},{"location":"lecture_12/lecture/#Propagating-a-Gaussian","page":"Lecture","title":"Propagating a Gaussian","text":"","category":"section"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Propagation of uncertainty has been studied in many areas of science. Relation between accuracy and computational speed is always a tradeoff.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"A common appoach to propagation of uncertainty is linearized Gaussian:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"variable x is represented by gaussian N(musigma)\ntransformation of addition: x+asim N(mu+asigma)\ntransformation of multiplication: a*xsim N(a*mua*sigma)\ngeneral transformation approximated:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"g(x)sim N(g(mu)g(mu)*sigma)","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"This can be efficienty implemented in Julia:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"struct GNum{T} where T<:Real\n μ::T\n σ::T\nend\nimport Base: +, *\n+(x::GaussNum{T},a::T) where T =GaussNum(x.μ+a,x.σ)\n+(a::T,x::GaussNum{T}) where T =GaussNum(x.μ+a,x.σ)\n*(x::GaussNum{T},a::T) where T =GaussNum(x.μ*a,a*x.σ)\n*(a::T,x::GaussNum{T}) where T =GaussNum(x.μ*a,a*x.σ)","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"For the ODE we need multiplication of two Gaussians. Using Taylor expansion and neglecting covariances:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"g(x_1x_2)=Nleft(g(mu_1mu_2) sqrtleft(fracdgdx_1(mu_1mu_2)sigma_1right)^2 + left(fracdgdx_2(mu_1mu_2)sigma_2right)^2right)","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"which trivially applies to sum: x_1+x_2=N(mu_1+mu_2 sqrtsigma_1^2 + sigma_2^2)","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"+(x1::GaussNum{T},x2::GaussNum{T}) where T =GaussNum(x1.μ+x2.μ,sqrt(x1.σ.^2 + x2.σ.^2))\n*(x1::GaussNum{T},x2::GaussNum{T}) where T =GaussNum(x1.μ*x2.μ, sqrt(x2.μ*x1.σ.^2 + x1.μ*x2.σ.^2))\n","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Following the principle of defining the necessary functions on the type, we can make it pass through the ODE:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"it is necessary to define new initialization (functions zero)\ndefine nice-looking constructor ()\n±(a::T,b::T) where T: can be automated (macro, generated functions)","category":"page"},{"location":"lecture_12/lecture/#Flexibility","page":"Lecture","title":"Flexibility","text":"","category":"section"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"The great advantage of the former model was the ability to run an arbitrary code with uncertainty at an arbitrary number.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"For example, we may know the initial conditions, but do not know the parameter value.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"GX=solve(f,[1.0±0.1,1.0±0.1],[0.1±0.1,0.2,0.3,0.2],0.1,1000)","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"(Image: )","category":"page"},{"location":"lecture_12/lecture/#Disadvantage","page":"Lecture","title":"Disadvantage","text":"","category":"section"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"The result does not correspond to the ensemble version above.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"we have ignored the covariances\nextension to version with covariances is possible by keeping track of the correlations (Measurements.jl), where other variables are stored in a dictionary:\ncorrelations found by language manipulations\nvery flexible and easy-to-use\ndiscovering the covariances requires to build the covariance from ids. (Expensive if done too often).","category":"page"},{"location":"lecture_12/lecture/#Vector-uncertainty","page":"Lecture","title":"Vector uncertainty","text":"","category":"section"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"The previous simple approach ignores the covariances between variables. Even if we tract covariances linearly in the same fashion (Measurementsjl), the approach will suffer from a loss of precision under non-linearity. ","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"(Image: )","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"The linearization-based approach propogates through the non-linearity only the mean and models its neighborhood by a plane.\nPropagating all samples is too expensive\nMethods based on quadrature or cubature rules are a compromise","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"The cubature approach is based on moment matching:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"mu_g = int g(x) p(x) dx","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"for which is g(mu) poor approximation, corresponding to:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"mu_g = g(mu) = int g(x) delta(x-mu) dx","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"For Gaussian distribution, we can use a smarter integration rule, called the Gauss-Hermite quadrature:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"mu_g = int g(x) p(x) dx approx sum_j=1^J w_j g(x_j)","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"where x_j are prescribed quadrature points (see e.g. (Image: online tables))","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"In multivariate setting, the same problem is typically solved with the aim to reduce the computational cost to linear complexity with dimension. Most often aimimg at O(2d) complexity where d is the dimension of vector x.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"One of the most popular approaches today is based on cubature rules approximating the Gaussian in radial-spherical coordinates.","category":"page"},{"location":"lecture_12/lecture/#Cubature-rules","page":"Lecture","title":"Cubature rules","text":"","category":"section"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Consider Gaussian distribution with mean mu and covariance matrix Sigma that is positive definite with square root sqrtSigma, such that sqrtSigma sqrtSigma^T=Sigma. The quadrature pints are:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"x_i = mu + sqrtSigma q_i","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"beginalign\nq_1=sqrtdbeginbmatrix1\n0\nvdots\nendbmatrix\n\nq_2=sqrtdbeginbmatrix0\n1\nvdots\nendbmatrix ldots \n\nq_d+1=sqrtdbeginbmatrix-1\n0\nvdots\nendbmatrix\nq_d+2=sqrtdbeginbmatrix0\n-1\nvdots\nendbmatrix ldots\nendalign","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"that can be composed into a matrix Q=q_1ldots q_2d that is constant:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Q = sqrtd I_d -I_d","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"(Image: )","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Those quadrature points are in integration weighted by:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"w_i = frac12d i=1ldots2d","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"where d is dimension of the vectors.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"The quadrature points are propogated through the non-linearity in parallel (x_i=g(x_i)) and the resulting Gaussian distribution is:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"beginalign\nx sim N(muSigma)\nmu = frac12dsum_j=1^2d x_i\nSigma = frac12dsum_j=1^2d (x_i-mu)^T (x_i-mu)\nendalign","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"It is easy to check that if the sigma-points are propagated through an identity, they preserve the mean and variance. ","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"beginalign\nmu = frac12dsum_j=1^2d (mu + sqrtSigmaq_i)\n = frac12d(2dmu + sqrtSigma sum_j=1^2d (q_i)\n = mu\nendalign\n","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"For our example:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"(Image: )","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"only 4 trajectories propagated deterministically\ncan not be implemented using a single number type\nthe number of points to store is proportional to the dimension\nmanipulation requires operations from linear algebra\nmoving to representations in vector form\nsimple for initial conditions,\nhow to extend to operate also on parameters?","category":"page"},{"location":"lecture_12/lecture/#Smarter-implementation","page":"Lecture","title":"Smarter implementation","text":"","category":"section"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Easiest solution is to put the corresponding parts of the problem together:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"ode function f, \nits state x0,\nand parameters θ","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"can be wrapped into an ODEProblem","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"struct ODEProblem{F,T,X<:AbstractVector,P<:AbstractVector}\n f::F\n tspan::T\n x0::X\n θ::P\nend","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"the solver can operate on the ODEProbelm type","category":"page"},{"location":"lecture_12/lecture/#Unceratinty-propagation-in-vectors","page":"Lecture","title":"Unceratinty propagation in vectors","text":"","category":"section"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Example: consider uncertainty in state x_1x_2 and the first parameter theta_1. ","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Quick and dirty: ","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"getuncertainty(o::ODEProblem) = [o.u0[1:2];o.θ[1]]\nsetuncertainty!(o::ODEProblem,x::AbstractVector) = o.u0[1:2]=x[1:2],o.θ[1]=x[3]","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"and write a general Cubature solver using multiple dispatch.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Practical issues:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"how to check bounds? (Asserts)\nwhat if we provide an incompatible ODEProblem\ndefine a type that specifies the type of uncertainty? ","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"struct GaussODEProblem\n mean::ODEProblem\n unc_in_u # any indexing type accepted by to_index()\n unc_in_θ\n sqΣ0\nend","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"We can dispatch the cubature solver on GaussODEProblem and the ordinary solve on GaussODEProblem.OP internally.","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"getmean(gop::GaussODEProblem) =[ gop.mean.x0[gop.unc_in_u];gop.mean.θ[gop.unc_in_θ]]\nsetmean!(gop::GaussODEProblem,x::AbstractVector) = begin \n gop.mean.x0[gop.unc_in_u]=x[1:length(gop.unc_in_u)]\n gop.mean.θ[gop.unc_in_θ]=x[length(gop.unc_in_u).+[1:length(gop.unc_in_θ)]] \nend","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"Constructor accepts an ODEProblem with uncertain numbers and converts it to GaussODEProblem:","category":"page"},{"location":"lecture_12/lecture/","page":"Lecture","title":"Lecture","text":"goes through ODEProblem x0 and θ fields and checks their types\nreplaces GaussNums in ODEProblem by ordinary numbers\nremembers indices of GaussNum in x0 and θ\ncopies standard deviations in GaussNum to sqΣ0","category":"page"},{"location":"lecture_04/hw/#hw4","page":"Homework","title":"Homework 4","text":"","category":"section"},{"location":"lecture_04/hw/","page":"Homework","title":"Homework","text":"In this homework you will have to write two additional @testsets for the Ecosystem. One testset should be contained in a file test/sheep.jl and verify that the function eat!(::Animal{Sheep}, ::Plant{Grass}, ::World) works correctly. Another testset should be in the file test/wolf.jl and veryfiy that the function eat!(::Animal{Wolf}, ::Animal{Sheep}, ::World) works correctly.","category":"page"},{"location":"lecture_04/hw/#How-to-submit?","page":"Homework","title":"How to submit?","text":"","category":"section"},{"location":"lecture_04/hw/","page":"Homework","title":"Homework","text":"Zip the whole package folder Ecosystem.jl and upload it to BRUTE. The package has to include at least the following files:","category":"page"},{"location":"lecture_04/hw/","page":"Homework","title":"Homework","text":"├── src\n│ └── Ecosystem.jl\n└── test\n ├── sheep.jl # contains only a single @testset\n ├── wolf.jl # contains only a single @testset\n └── runtests.jl","category":"page"},{"location":"lecture_04/hw/","page":"Homework","title":"Homework","text":"Thet test/runtests.jl file can look like this:","category":"page"},{"location":"lecture_04/hw/","page":"Homework","title":"Homework","text":"using Test\nusing Ecosystem\n\ninclude(\"sheep.jl\")\ninclude(\"wolf.jl\")\n# ...","category":"page"},{"location":"lecture_04/hw/#Test-Sheep","page":"Homework","title":"Test Sheep","text":"","category":"section"},{"location":"lecture_04/hw/","page":"Homework","title":"Homework","text":"
\n
Homework:
\n
","category":"page"},{"location":"lecture_04/hw/","page":"Homework","title":"Homework","text":"Create a Sheep with food probability p_f=1\nCreate fully grown Grass and a World with the two agents.\nExecute eat!(::Animal{Sheep}, ::Plant{Grass}, ::World)\n@test that the size of the Grass now has size == 0","category":"page"},{"location":"lecture_04/hw/","page":"Homework","title":"Homework","text":"
","category":"page"},{"location":"lecture_04/hw/#Test-Wolf","page":"Homework","title":"Test Wolf","text":"","category":"section"},{"location":"lecture_04/hw/","page":"Homework","title":"Homework","text":"
\n
Homework:
\n
","category":"page"},{"location":"lecture_04/hw/","page":"Homework","title":"Homework","text":"Create a Wolf with food probability p_f=1\nCreate a Sheep and a World with the two agents.\nExecute eat!(::Animal{Wolf}, ::Animal{Sheep}, ::World)\n@test that the World only has one agent left in the agents dictionary","category":"page"},{"location":"lecture_04/hw/","page":"Homework","title":"Homework","text":"
","category":"page"},{"location":"lecture_02/lecture/#type_lecture","page":"Lecture","title":"Motivation","text":"","category":"section"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Before going into the details of Julia's type system, we will spend a few minutes motivating the roles of a type system, which are:","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Structuring code\nCommunicating to the compiler how a type will be used","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"The first aspect is important for the convenience of the programmer and enables abstractions in the language, the latter aspect is important for the speed of the generated code. Writing efficient Julia code is best viewed as a dialogue between the programmer and the compiler. [1] ","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Type systems according to Wikipedia:","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"In computer science and computer programming, a data type or simply type is an attribute of data which tells the compiler or interpreter how the programmer intends to use the data.\nA type system is a logical system comprising a set of rules that assigns a property called a type to the various constructs of a computer program, such as variables, expressions, functions or modules. These types formalize and enforce the otherwise implicit categories the programmer uses for algebraic data types, data structures, or other components.","category":"page"},{"location":"lecture_02/lecture/#Structuring-the-code-/-enforcing-the-categories","page":"Lecture","title":"Structuring the code / enforcing the categories","text":"","category":"section"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"The role of structuring the code and imposing semantic restriction means that the type system allows you to logically divide your program, and to prevent certain types of errors. Consider for example two types, Wolf and Sheep which share the same definition but the types have different names.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"struct Wolf\n name::String\n energy::Int\nend\n\nstruct Sheep\n name::String\n energy::Int\nend","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"This allows us to define functions applicable only to the corresponding type","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"howl(wolf::Wolf) = println(wolf.name, \" has howled.\")\nbaa(sheep::Sheep) = println(sheep.name, \" has baaed.\")\nnothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Therefore the compiler (or interpreter) enforces that a wolf can only howl and never baa and vice versa a sheep can only baa. In this sense, it ensures that howl(sheep) and baa(wolf) never happen.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"baa(Sheep(\"Karl\",3))\nbaa(Wolf(\"Karl\",3))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Notice the type of error of the latter call baa(Wolf(\"Karl\",3)). Julia raises MethodError which states that it has failed to find a function baa for the type Wolf (but there is a function baa for type Sheep).","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"For comparison, consider an alternative definition which does not have specified types","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"bark(animal) = println(animal.name, \" has howled.\")\nbaa(animal) = println(animal.name, \" has baaed.\")\nnothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"in which case the burden of ensuring that a wolf will never baa rests upon the programmer which inevitably leads to errors (note that severely constrained type systems are difficult to use).","category":"page"},{"location":"lecture_02/lecture/#Intention-of-use-and-restrictions-on-compilers","page":"Lecture","title":"Intention of use and restrictions on compilers","text":"","category":"section"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Types play an important role in generating efficient code by a compiler, because they tells the compiler which operations are permitted, prohibited, and can indicate invariants of type (e.g. constant size of an array). If compiler knows that something is invariant (constant), it can expoit such information. As an example, consider the following two alternatives to represent a set of animals:","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"a = [Wolf(\"1\", 1), Wolf(\"2\", 2), Sheep(\"3\", 3)]\nb = (Wolf(\"1\", 1), Wolf(\"2\", 2), Sheep(\"3\", 3))\nnothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"where a is an array which can contain arbitrary types and have arbitrary length whereas b is a Tuple which has fixed length in which the first two items are of type Wolf and the third item is of type Sheep. Moreover, consider a function which calculates the energy of all animals as","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"energy(animals) = mapreduce(x -> x.energy, +, animals)\nnothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"A good compiler makes use of the information provided by the type system to generate efficient code which we can verify by inspecting the compiled code using @code_native macro","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"@code_native debuginfo=:none energy(a)\n@code_native debuginfo=:none energy(b)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"one observes the second version produces more optimal code. Why is that?","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"In the first representation, a, the animals are stored in an Array{Any} which can have arbitrary size and can contain arbitrary animals. This means that the compiler has to compile energy(a) such that it works on such arrays.\nIn the second representation, b, the animals are stored in a Tuple, which specializes for lengths and types of items. This means that the compiler knows the number of animals and the type of each animal on each position within the tuple, which allows it to specialize.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"This difference will indeed have an impact on the time of code execution. On my i5-8279U CPU, the difference (as measured by BenchmarkTools) is","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"using BenchmarkTools\n@btime energy($(a))\n@btime energy($(b))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":" 70.2 ns (0 allocations: 0 bytes)\n 2.62 ns (0 allocations: 0 bytes)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Which nicely demonstrates that the choice of types affects performance. Does it mean that we should always use Tuples instead of Arrays? Surely not, it is just that each is better for different use-cases. Using Tuples means that the compiler will compile a special function for each length of tuple and each combination of types of items it contains, which is clearly wasteful.","category":"page"},{"location":"lecture_02/lecture/#type_system","page":"Lecture","title":"Julia's type system","text":"","category":"section"},{"location":"lecture_02/lecture/#Julia-is-dynamicaly-typed","page":"Lecture","title":"Julia is dynamicaly typed","text":"","category":"section"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Julia's type system is dynamic, which means that all types are resolved during runtime. But, if the compiler can infer types of all variables of the called function, it can specialize the function for that given type of variables which leads to efficient code. Consider a modified example where we represent two wolfpacks:","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"wolfpack_a = [Wolf(\"1\", 1), Wolf(\"2\", 2), Wolf(\"3\", 3)]\nwolfpack_b = Any[Wolf(\"1\", 1), Wolf(\"2\", 2), Wolf(\"3\", 3)]\nnothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"wolfpack_a carries a type Vector{Wolf} while wolfpack_b has the type Vector{Any}. This means that in the first case, the compiler knows that all items are of the type Wolfand it can specialize functions using this information. In case of wolfpack_b, it does not know which animal it will encounter (although all are of the same type), and therefore it needs to dynamically resolve the type of each item upon its use. This ultimately leads to less performant code.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"@benchmark energy($(wolfpack_a))\n@benchmark energy($(wolfpack_b))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":" 3.7 ns (0 allocations: 0 bytes)\n 69.4 ns (0 allocations: 0 bytes)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"To conclude, julia is indeed a dynamically typed language, but if the compiler can infer all types in a called function in advance, it does not have to perform the type resolution during execution, which produces performant code. This means and in hot (performance critical) parts of the code, you should be type stable, in other parts, it is not such big deal.","category":"page"},{"location":"lecture_02/lecture/#Classes-of-types","page":"Lecture","title":"Classes of types","text":"","category":"section"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Julia divides types into three classes: primitive, composite, and abstract.","category":"page"},{"location":"lecture_02/lecture/#Primitive-types","page":"Lecture","title":"Primitive types","text":"","category":"section"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Citing the documentation: A primitive type is a concrete type whose data consists of plain old bits. Classic examples of primitive types are integers and floating-point values. Unlike most languages, Julia lets you declare your own primitive types, rather than providing only a fixed set of built-in ones. In fact, the standard primitive types are all defined in the language itself.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"The definition of primitive types look as follows","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"primitive type Float16 <: AbstractFloat 16 end\nprimitive type Float32 <: AbstractFloat 32 end\nprimitive type Float64 <: AbstractFloat 64 end","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"and they are mainly used to jump-start julia's type system. It is rarely needed to define a special primitive type, as it makes sense only if you define special functions operating on its bits. This is almost excusively used for exposing special operations provided by the underlying CPU / LLVM compiler. For example + for Int32 is different from + for Float32 as they call a different intrinsic operations. You can inspect this jump-starting of the type system yourself by looking at Julia's source.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia> @which +(1,2)\n+(x::T, y::T) where T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8} in Base at int.jl:87","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"At int.jl:87","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"(+)(x::T, y::T) where {T<:BitInteger} = add_int(x, y)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"we see that + of integers is calling the function add_int(x, y), which is defined in the core part of the compiler in Intrinsics.cpp (yes, in C++), exposed in Core.Intrinsics","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"From Julia docs: Core is the module that contains all identifiers considered \"built in\" to the language, i.e. part of the core language and not libraries. Every module implicitly specifies using Core, since you can't do anything without those definitions.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Primitive types are rarely used, and they will not be used in this course. We mention them for the sake of completeness and refer the reader to the official Documentation (and source code of Julia).","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"An example of use of primitive type is a definition of one-hot vector in the library PrimitiveOneHot as ","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"primitive type OneHot{K} <: AbstractOneHotArray{1} 32 end","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"where K is the dimension of the one-hot vector. ","category":"page"},{"location":"lecture_02/lecture/#Abstract-types","page":"Lecture","title":"Abstract types","text":"","category":"section"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"An abstract type can be viewed as a set of concrete types. For example, an AbstractFloat represents the set of concrete types (BigFloat,Float64,Float32,Float16). This is used mainly to define general methods for sets of types for which we expect the same behavior (recall the Julia design motivation: if it quacks like a duck, waddles like a duck and looks like a duck, chances are it's a duck). Abstract types are defined with abstract type TypeName end. For example the following set of abstract types defines part of julia's number system.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"abstract type Number end\nabstract type Real <: Number end\nabstract type Complex <: Number end\nabstract type AbstractFloat <: Real end\nabstract type Integer <: Real end\nabstract type Signed <: Integer end\nabstract type Unsigned <: Integer end","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"where <: means \"is a subtype of\" and it is used in declarations where the right-hand is an immediate sypertype of a given type (Integer has the immediate supertype Real.) If the supertype is not supplied, it is considered to be Any, therefore in the above defition Number has the supertype Any. ","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"We can list childrens of an abstract type using function subtypes ``julia using InteractiveUtils: subtypes # hide subtypes(AbstractFloat)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"and we can also list the immediate `supertype` or climb the ladder all the way to `Any` using `supertypes`\n``julia\nusing InteractiveUtils: supertypes # hide\nsupertypes(AbstractFloat)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"supertype and subtypes print only types defined in Modules that are currently loaded to your workspace. For example with Julia without any Modules, subtypes(Number) returns [Complex, Real], whereas if I load Mods package implementing numbers defined over finite field, the same call returns [Complex, Real, AbstractMod].","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"It is relatively simple to print a complete type hierarchy of ","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"using AbstractTrees\nfunction AbstractTrees.children(t::Type)\n t === Function ? Vector{Type}() : filter!(x -> x !== Any,subtypes(t))\nend\nAbstractTrees.printnode(io::IO,t::Type) = print(io,t)\nprint_tree(Number)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"The main role of abstract types allows is in function definitions. They allow to define functions that can be used on variables with types with a given abstract type as a supertype. For example we can define a sgn function for all real numbers as","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"sgn(x::Real) = x > 0 ? 1 : x < 0 ? -1 : 0\nnothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"and we know it would be correct for all real numbers. This means that if anyone creates a new subtype of Real, the above function can be used. This also means that it is expected that comparison operations are defined for any real number. Also notice that Complex numbers are excluded, since they do not have a total order.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"For unsigned numbers, the sgn can be simplified, as it is sufficient to verify if they are different (greater) than zero, therefore the function can read","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"sgn(x::Unsigned) = x > 0 ? 1 : 0\nnothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"and again, it applies to all numbers derived from Unsigned. Recall that Unsigned <: Integer <: Real, how does Julia decide, which version of the function sgn to use for UInt8(0)? It chooses the most specific version, and thus for sgn(UInt8(0)) it will use sgn(x::Unsinged). If the compiler cannot decide, typically it encounters an ambiguity, it throws an error and recommends which function you should define to resolve it.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"The above behavior allows to define default \"fallback\" implementations and while allowing to specialize for sub-types. A great example is matrix multiplication, which has a generic (and slow) implementation with many specializations, which can take advantage of structure (sparse, banded), or use optimized implementations (e.g. blas implementation for dense matrices with eltype Float32 and Float64).","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Again, Julia does not make a difference between abstract types defined in Base libraries shipped with the language and those defined by you (the user). All are treated the same.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"From Julia documentation: Abstract types cannot be instantiated, which means that we cannot create a variable that would have an abstract type (try typeof(Number(1f0))). Also, abstract types cannot have any fields, therefore there is no composition (there are lengthy discussions of why this is so, one of the most definite arguments of creators is that abstract types with fields frequently lead to children types not using some fields (consider circle vs. ellipse)).","category":"page"},{"location":"lecture_02/lecture/#composite_types","page":"Lecture","title":"Composite types","text":"","category":"section"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Composite types are similar to struct in C (they even have the same memory layout) as they logically join together other types. It is not a great idea to think about them as objects (in OOP sense), because objects tie together data and functions on owned data. Contrary in Julia (as in C), functions operate on data of structures, but are not tied to them and they are defined outside them. Composite types are workhorses of Julia's type system, as user-defined types are mostly composite (or abstract).","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Composite types are defined using struct TypeName [fields] end. To define a position of an animal on the Euclidean plane as a type, we would write","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"struct PositionF64\n x::Float64\n y::Float64\nend","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"which defines a structure with two fields x and y of type Float64. Julia's compiler creates a default constructor, where both (but generally all) arguments are converted using (convert(Float64, x), convert(Float64, y) to the correct type. This means that we can construct a PositionF64 with numbers of different type that are convertable to Float64, e.g. PositionF64(1,1//2) but we cannot construct PositionF64 where the fields would be of different type (e.g. Int, Float32, etc.) or they are not trivially convertable (e.g. String).","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Fields in composite types do not have to have a specified type. We can define a VaguePosition without specifying the type","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"struct VaguePosition\n x\n y\nend","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"This works as the definition above except that the arguments are not converted to Float64 now. One can store different values in x and y, for example String (e.g. VaguePosition(\"Hello\",\"world\")). Although the above definition might be convenient, it limits the compiler's ability to specialize, as the type VaguePosition does not carry information about type of x and y, which has a negative impact on the performance. For example","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"using BenchmarkTools\nmove(a,b) = typeof(a)(a.x+b.x, a.y+b.y)\nx = [PositionF64(rand(), rand()) for _ in 1:100]\ny = [VaguePosition(rand(), rand()) for _ in 1:100]\n@benchmark reduce(move, $(x))\n@benchmark reduce(move, $(y))\nnothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Giving fields of a composite type an abstract type does not really solve the problem of the compiler not knowing the type. In this example, it still does not know, if it should use instructions for Float64 or Int8.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"``julia struct LessVaguePosition x::Real y::Real end","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"z = [LessVaguePosition(rand(), rand()) for _ in 1:100]; @benchmark reduce(move, z) nothing #hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nFrom the perspective of generating optimal code, both definitions are equally uninformative to the compiler as it cannot assume anything about the code. However, the `LessVaguePosition` will ensure that the position will contain only numbers, hence catching trivial errors like instantiating `VaguePosition` with non-numeric types for which arithmetic operators will not be defined (recall the discussion on the beginning of the lecture).\n\nAll structs defined above are immutable (as we have seen above in the case of `Tuple`), which means that one cannot change a field (unless the struct wraps a container, like and array, which allows that). For example this raises an error\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"a = LessVaguePosition(1,2) a.x = 2","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nIf one needs to make a struct mutable, use the keyword `mutable` before the keyword `struct` as\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia mutable struct MutablePosition x::Float64 y::Float64 end","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nIn mutable structures, we can change the values of fields.\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"a = MutablePosition(1e0, 2e0) a.x = 2; a","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nNote, that the memory layout of mutable structures is different, as fields now contain references to memory locations, where the actual values are stored (such structures cannot be allocated on stack, which increases the pressure on Garbage Collector).\n\nThe difference can be seen from ","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"a, b = PositionF64(1,2), PositionF64(1,2) @codenative debuginfo=:none move(a,b) a, b = MutablePosition(1,2), MutablePosition(1,2) @codenative debuginfo=:none move(a,b)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Why there is just one addition?\n\nAlso, the mutability is costly.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia x = [PositionF43(rand(), rand()) for _ in 1:100]; z = [MutablePosition(rand(), rand()) for _ in 1:100]; @benchmark reduce(move, x) @benchmark reduce(move, z)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n### Parametric types\nSo far, we had to trade-off flexibility for generality in type definitions. Can we have both? The answer is affirmative. The way to achieve this **flexibility** in definitions of the type while being able to generate optimal code is to **parametrize** the type definition. This is achieved by replacing types with a parameter (typically a single uppercase character) and decorating in definition by specifying different type in curly brackets. For example\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia struct PositionT{T} x::T y::T end u = [PositionT(rand(), rand()) for _ in 1:100] u = [PositionT(rand(Float32), rand(Float32)) for _ in 1:100]","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"@benchmark reduce(move, u) nothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nNotice that the compiler can take advantage of specializing for different types (which does not have an effect here as in modern processors addition of `Float` and `Int` takes the same time).\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia v = [PositionT(rand(1:100), rand(1:100)) for _ in 1:100] @benchmark reduce(move, v) nothing #hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nThe above definition suffers the same problem as `VaguePosition`, which is that it allows us to instantiate the `PositionT` with non-numeric types, e.g. `String`. We solve this by restricting the types `T` to be children of some supertype, in this case `Real`\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia struct Position{T<:Real} x::T y::T end","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nwhich will throw an error if we try to initialize it with `Position(\"1.0\", \"2.0\")`. Notice the flexibility we have achieved. We can use `Position` to store (and later compute) not only over `Float32` / `Float64` but any real numbers defined by other packages, for example with `Posit`s.\n``julia\nusing SoftPosit\nPosition(Posit8(3), Posit8(1))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"also notice that trying to construct the Position with different type of real numbers will fail, example Position(1f0,1e0)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Naturally, fields in structures can be of different types, as is in the below pointless example.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"``julia struct PositionXY{X<:Real, Y<:Real} x::X y::Y end","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nThe type can be parametrized by a concrete types. This is usefuyl to communicate the compiler some useful informations, for example size of arrays. \n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia struct PositionZ{T<:Real,Z} x::T y::T end","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"PositionZ{Int64,1}(1,2)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n\n### Abstract parametric types\nLike Composite types, Abstract types can also have parameters. These parameters define types that are common for all child types. A very good example is Julia's definition of arrays of arbitrary dimension `N` and type `T` of its items as","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia abstract type AbstractArray{T,N} end","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Different `T` and `N` give rise to different variants of `AbstractArrays`,\ntherefore `AbstractArray{Float32,2}` is different from `AbstractArray{Float64,2}`\nand from `AbstractArray{Float64,1}.` Note that these are still `Abstract` types,\nwhich means you cannot instantiate them. Their purpose is\n* to allow to define operations for broad class of concrete types\n* to inform the compiler about constant values, which can be used\nNotice in the above example that parameters of types do not have to be types, but can also be values of primitive types, as in the above example of `AbstractArray` `N` is the number of dimensions which is an integer value.\n\nFor convenience, it is common to give some important partially instantiated Abstract types an **alias**, for example `AbstractVector` as","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia const AbstractVector{T} = AbstractArray{T,1}","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"is defined in `array.jl:23` (in Julia 1.6.2), which allows us to define for example general prescription for the `dot` product of two abstract vectors as\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia function dot(a::AbstractVector, b::AbstractVector) @assert length(a) == length(b) mapreduce(*, +, a, b) end nothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nYou can verify that the above general function can be compiled to performant code if\nspecialized for particular arguments.\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia; ansicolor=true using InteractiveUtils: @codenative @codenative debuginfo=:none mapreduce(*,+, [1,2,3], [1,2,3])","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n## More on the use of types in function definitions\n### Terminology\nA *function* refers to a set of \"methods\" for a different combination of type parameters (the term function can be therefore considered as refering to a mere **name**). *Methods* define different behavior for different types of arguments for a given function. For example\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia move(a::Position, b::Position) = Position(a.x + b.x, a.y + b.y) move(a::Vector{<:Position}, b::Vector{<:Position}) = move.(a,b) nothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n`move` refers to a function with methods `move(a::Position, b::Position)` and `move(a::Vector{<:Position}, b::Vector{<:Position})`. When different behavior on different types is defined by a programmer, as shown above, it is also called *implementation specialization*. There is another type of specialization, called *compiler specialization*, which occurs when the compiler generates different functions for you from a single method. For example for\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"move(Position(1,1), Position(2,2)) move(Position(1.0,1.0), Position(2.0,2.0))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nthe compiler generates two methods, one for `Position{Int64}` and the other for `Position{Float64}`. Notice that inside generated functions, the compiler needs to use different intrinsic operations, which can be viewed from\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia; ansicolor=true @code_native debuginfo=:none move(Position(1,1), Position(2,2))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nand\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia; ansicolor=true @code_native debuginfo=:none move(Position(1.0,1.0), Position(2.0,2.0))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nNotice that `move` works on `Posits` defined in 3rd party libas well","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"move(Position(Posit8(1),Posit8(1)), Position(Posit8(2),Posit8(2)))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n## Intermezzo: How does the Julia compiler work?\nLet's walk through an example. Consider the following definitions\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia move(a::Position, by::Position) = Position(a.x + by.x, a.y + by.y) move(a::T, by::T) where {T<:Position} = Position(a.x + by.x, a.y + by.y) move(a::Position{Float64}, by::Position{Float64}) = Position(a.x + by.x, a.y + by.y) move(a::Vector{<:Position}, by::Vector{<:Position}) = move.(a, by) move(a::Vector{<:Position}, by::Position) = move.(a, by) nothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nand a function call\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia a = Position(1.0, 1.0) by = Position(2.0, 2.0) move(a, by)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n1. The compiler knows that you call the function `move`.\n2. The compiler infers the type of the arguments. You can view the result with\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"(typeof(a),typeof(by))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n3. The compiler identifies all `move`-methods with arguments of type `(Position{Float64}, Position{Float64})`:\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"wc = Base.getworldcounter() m = Base.method_instances(move, (typeof(a), typeof(by)), wc) m = first(m)","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n4a. If the method has been specialized (compiled), then the arguments are prepared and the method is invoked. The compiled specialization can be seen from\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"m.cache","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n4b. If the method has not been specialized (compiled), the method is compiled for the given type of arguments and continues as in step 4a.\nA compiled function is therefore a \"blob\" of **native code** living in a particular memory location. When Julia calls a function, it needs to pick the right block corresponding to a function with particular type of parameters.\n\nIf the compiler cannot narrow the types of arguments to concrete types, it has to perform the above procedure inside the called function, which has negative effects on performance, as the type resulution and identification of the methods can be slow, especially for methods with many arguments (e.g. 30ns for a method with one argument,\n100 ns for method with two arguements). **You always want to avoid run-time resolution inside the performant loop!!!**\nRecall the above example\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia wolfpacka = [Wolf(\"1\", 1), Wolf(\"2\", 2), Wolf(\"3\", 3)] @benchmark energy($(Expr(:incomplete, \"incomplete: premature end of input\"))a))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"BenchmarkTools.Trial: 10000 samples with 991 evaluations. Range (min … max): 40.195 ns … 66.641 ns ┊ GC (min … max): 0.00% … 0.00% Time (median): 40.742 ns ┊ GC (median): 0.00% Time (mean ± σ): 40.824 ns ± 1.025 ns ┊ GC (mean ± σ): 0.00% ± 0.00%","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"▂▃ ▃▅▆▅▆█▅▅▃▂▂ ▂ ▇██████████████▇▇▅▅▁▅▄▁▅▁▄▄▃▄▅▄▅▃▅▃▅▁▃▁▄▄▃▁▁▅▃▃▄▃▄▃▄▆▆▇▇▇▇█ █ 40.2 ns Histogram: log(frequency) by time 43.7 ns <","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Memory estimate: 0 bytes, allocs estimate: 0.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nand\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia wolfpackb = Any[Wolf(\"1\", 1), Wolf(\"2\", 2), Wolf(\"3\", 3)] @benchmark energy($(Expr(:incomplete, \"incomplete: premature end of input\"))b))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"BenchmarkTools.Trial: 10000 samples with 800 evaluations. Range (min … max): 156.406 ns … 212.344 ns ┊ GC (min … max): 0.00% … 0.00% Time (median): 157.136 ns ┊ GC (median): 0.00% Time (mean ± σ): 158.114 ns ± 4.023 ns ┊ GC (mean ± σ): 0.00% ± 0.00%","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"▅█▆▅▄▂ ▃▂▁ ▂ ██████▆▇██████▇▆▇█▇▆▆▅▅▅▅▅▃▄▄▅▄▄▄▄▅▁▃▄▄▃▃▄▃▃▃▄▄▄▅▅▅▅▁▅▄▃▅▄▄▅▅ █ 156 ns Histogram: log(frequency) by time 183 ns <","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Memory estimate: 0 bytes, allocs estimate: 0.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nAn interesting intermediate between fully abstract and fully concrete type happens, when the compiler knows that arguments have abstract type, which is composed of a small number of concrete types. This case called Union-Splitting, which happens when there is just a little bit of uncertainty. Julia will do something like","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia argtypes = typeof(args) push!(executionstack, args) if T == Tuple{Int, Bool} @goto compiledblob1234 else # the only other option is Tuple{Float64, Bool} @goto compiledblob_1236 end","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"For example\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia const WolfOrSheep = Union{Wolf, Sheep} wolfpackc = WolfOrSheep[Wolf(\"1\", 1), Wolf(\"2\", 2), Wolf(\"3\", 3)] @benchmark energy($(Expr(:incomplete, \"incomplete: premature end of input\"))c))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"BenchmarkTools.Trial: 10000 samples with 991 evaluations. Range (min … max): 43.600 ns … 73.494 ns ┊ GC (min … max): 0.00% … 0.00% Time (median): 44.106 ns ┊ GC (median): 0.00% Time (mean ± σ): 44.279 ns ± 0.931 ns ┊ GC (mean ± σ): 0.00% ± 0.00%","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":" █ ▁ ▃","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"▂▂▂▆▃██▅▃▄▄█▅█▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▁▂▂▂▂▂▂▂▁▂▂▂▂▂▂▂▂▂▂▂▂▂ ▃ 43.6 ns Histogram: frequency by time 47.4 ns <","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"Memory estimate: 0 bytes, allocs estimate: 0.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nThanks to union splitting, Julia is able to have performant operations on arrays with undefined / missing values for example\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"[1, 2, 3, missing] |> typeof","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n### More on matching methods and arguments\nIn the above process, the step, where Julia looks for a method instance with corresponding parameters can be very confusing. The rest of this lecture will focus on this. For those who want to have a formal background, we recommend [talk of Francesco Zappa Nardelli](https://www.youtube.com/watch?v=Y95fAipREHQ) and / or the one of [Jan Vitek](https://www.youtube.com/watch?v=LT4AP7CUMAw).\n\nWhen Julia needs to specialize a method instance, it needs to find it among multiple definitions. A single function can have many method instances, see for example `methods(+)` which lists all method instances of the `+`-function. How does Julia select the proper one?\n1. It finds all methods where the type of arguments match or are subtypes of restrictions on arguments in the method definition.\n2a. If there are multiple matches, the compiler selects the most specific definition.\n\n2b. If the compiler cannot decide, which method instance to choose, it throws an error.\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"confusedmove(a::Position{Float64}, by) = Position(a.x + by.x, a.y + by.y) confusedmove(a, by::Position{Float64}) = Position(a.x + by.x, a.y + by.y) confused_move(Position(1.0,2.0), Position(1.0,2.0))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n2c. If it cannot find a suitable method, it throws an error.\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"move(Position(1,2), VaguePosition(\"hello\",\"world\"))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nSome examples: Consider following definitions\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia move(a::Position, by::Position) = Position(a.x + by.x, a.y + by.y) move(a::T, by::T) where {T<:Position} = T(a.x + by.x, a.y + by.y) move(a::Position{Float64}, by::Position{Float64}) = Position(a.x + by.x, a.y + by.y) move(a::Vector{<:Position}, by::Vector{<:Position}) = move.(a, by) move(a::Vector{T}, by::Vector{T}) where {T<:Position} = move.(a, by) move(a::Vector{<:Position}, by::Position) = move.(a, by) nothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nWhich method will compiler select for\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"move(Position(1.0,2.0), Position(1.0,2.0))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nThe first three methods match the types of argumens, but the compiler will select the third one, since it is the most specific.\n\nWhich method will compiler select for\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"move(Position(1,2), Position(1,2))","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nAgain, the first and second method definitions match the argument, but the second is the most specific.\n\nWhich method will the compiler select for\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"move([Position(1,2)], [Position(1,2)])","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nAgain, the fourth and fifth method definitions match the argument, but the fifth is the most specific.\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"move([Position(1,2), Position(1.0,2.0)], [Position(1,2), Position(1.0,2.0)])","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n### Frequent problems\n1. Why does the following fail?\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"foo(a::Vector{Real}) = println(\"Vector{Real}\") foo([1.0,2,3])","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nJulia's type system is **invariant**, which means that `Vector{Real}` is different from `Vector{Float64}` and from `Vector{Float32}`, even though `Float64` and `Float32` are sub-types of `Real`. Therefore `typeof([1.0,2,3])` isa `Vector{Float64}` which is not subtype of `Vector{Real}.` For **covariant** languages, this would be true. For more information on variance in computer languages, [see here](https://en.wikipedia.org/wiki/Covariance_and_contravariance_(computer_science)). If the above definition of `foo` should be applicable to all vectors which has elements of subtype of `Real` we have define it as\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia foo(a::Vector{T}) where {T<:Real} = println(\"Vector{T} where {T<:Real}\") nothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nor equivalently but more tersely as\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia foo(a::Vector{<:Real}) = println(\"Vector{T} where {T<:Real}\") nothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n2. Diagonal rule says that a repeated type in a method signature has to be a concrete type (this is to avoid ambinguity if the repeated type is used inside function definition to define a new variable to change type of variables). Consider for example the function below\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia move(a::T, b::T) where {T<:Position} = T(a.x + by.x, a.y + by.y) nothing # hide","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\nwe cannot call it with `move(Position(1.0,2.0), Position(1,2))`, since in this case `Position(1.0,2.0)` is of type `Position{Float64}` while `Position(1,2)` is of type `Position{Int64}`.\n3. When debugging why arguments do not match a particular method definition, it is useful to use `typeof`, `isa`, and `<:` commands. For example\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"typeof(Position(1.0,2.0)) typeof(Position(1,2)) Position(1,2) isa Position{Float64} Position(1,2) isa Position{Real} Position(1,2) isa Position{<:Real} typeof(Position(1,2)) <: Position{<:Float64} typeof(Position(1,2)) <: Position{<:Real}","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"\n### A bizzare definition which you can encounter\nThe following definition of a one-hot matrix is taken from [Flux.jl](https://github.com/FluxML/Flux.jl/blob/1a0b51938b9a3d679c6950eece214cd18108395f/src/onehot.jl#L10-L12)\n","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"julia struct OneHotArray{T<:Integer, L, N, var\"N+1\", I<:Union{T,AbstractArray{T, N}}} <: AbstractArray{Bool, var\"N+1\"} indices::I end ```","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"The parameters of the type carry information about the type used to encode the position of one in each column in T, the dimension of one-hot vectors in L, the dimension of the storage of indices in N (which is zero for OneHotVector and one for OneHotMatrix), number of dimensions of the OneHotArray in var\"N+1\" and the type of underlying storage of indicies I.","category":"page"},{"location":"lecture_02/lecture/","page":"Lecture","title":"Lecture","text":"[1]: Type Stability in Julia, Pelenitsyn et al., 2021](https://arxiv.org/pdf/2109.01950.pdf)","category":"page"},{"location":"installation/#install","page":"Installation","title":"Installation","text":"","category":"section"},{"location":"installation/","page":"Installation","title":"Installation","text":"In order to participate in the course, everyone should install a recent version of Julia together with some text editor of choice. Furthermore during the course we will introduce some best practices of creating/testing and distributing your own Julia code, for which we will require a GitHub account.","category":"page"},{"location":"installation/","page":"Installation","title":"Installation","text":"We recommend to install Julia via juliaup. We are using the latest, stable version of Julia (which at the time of this writing is v1.9). Once you have installed juliaup you can get any Julia version you want via:","category":"page"},{"location":"installation/","page":"Installation","title":"Installation","text":"juliaup add $JULIA_VERSION\n\n# or more concretely:\njuliaup add 1.9\n\n# but please, just use the latest, stable version","category":"page"},{"location":"installation/","page":"Installation","title":"Installation","text":"Now you should be able to start Julia an be greated with the following:","category":"page"},{"location":"installation/","page":"Installation","title":"Installation","text":"$ julia\n _\n _ _ _(_)_ | Documentation: https://docs.julialang.org\n (_) | (_) (_) |\n _ _ _| |_ __ _ | Type \"?\" for help, \"]?\" for Pkg help.\n | | | | | | |/ _` | |\n | | |_| | | | (_| | | Version 1.9.2 (2023-07-05)\n _/ |\\__'_|_|_|\\__'_| | Official https://julialang.org/ release\n|__/ |\n\njulia>","category":"page"},{"location":"installation/#Julia-IDE","page":"Installation","title":"Julia IDE","text":"","category":"section"},{"location":"installation/","page":"Installation","title":"Installation","text":"There is no one way to install/develop and run Julia, which may be strange users coming from MATLAB, but for users of general purpose languages such as Python, C++ this is quite common. Most of the Julia programmers to date are using","category":"page"},{"location":"installation/","page":"Installation","title":"Installation","text":"Visual Studio Code,\nand the corresponding Julia extension.","category":"page"},{"location":"installation/","page":"Installation","title":"Installation","text":"This setup is described in a comprehensive step-by-step guide in our bachelor course Julia for Optimization & Learning.","category":"page"},{"location":"installation/","page":"Installation","title":"Installation","text":"Note that this setup is not a strict requirement for the lectures/labs and any other text editor with the option to send code to the terminal such as Vim (+Tmux), Emacs, or Sublime Text will suffice.","category":"page"},{"location":"installation/#GitHub-registration-and-Git-setup","page":"Installation","title":"GitHub registration & Git setup","text":"","category":"section"},{"location":"installation/","page":"Installation","title":"Installation","text":"As one of the goals of the course is writing code that can be distributed to others, we recommend a GitHub account, which you can create here (unless you already have one). In order to interact with GitHub repositories, we will be using git. For installation instruction (Windows only) see the section in the bachelor course.","category":"page"},{"location":"lecture_01/motivation/#Introduction-to-Scientific-Programming","page":"Motivation","title":"Introduction to Scientific Programming","text":"","category":"section"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"note: Loose definition of Scientific Programming\nScientific programming languages are designed and optimized for implementing mathematical formulas and for computing with matrices.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Examples of Scientific programming languages include ALGOL, APL, Fortran, J, Julia, Maple, MATLAB and R.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Key requirements for a Scientific programming language:","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Fast execution of the code (complex algorithms).\nEase of code reuse / code restructuring.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"
\n \n
\n Julia set.\n Stolen from\n Colorschemes.jl.\n
\n
","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"In contrast, to general-purpose language Julia has:","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"less concern with standalone executable/libraby compilation \nless concern with Application binary interface (ABI)\nless concern with business models (library + header files)\nless concern with public/private separation","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"tip: Example of a scientific task\nIn many applications, we encounter the task of optimization a function given by a routine (e.g. engineering, finance, etc.)using Optim\n\nP(x,y) = x^2 - 3x*y + 5y^2 - 7y + 3 # user defined function\n\nz₀ = [ 0.0\n 0.0 ] # starting point \n\noptimize(z -> P(z...), z₀, ConjugateGradient())\noptimize(z -> P(z...), z₀, Newton())\noptimize(z -> P(z...), z₀, Newton();autodiff = :forward)\n","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Very simple for a user, very complicated for a programmer. The program should:","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"pick the right optimization method (easy by config-like approach)\ncompute gradient (Hessian) of a user function","category":"page"},{"location":"lecture_01/motivation/#Classical-approach:-create-a-*fast*-library-and-flexible-calling-enviroment","page":"Motivation","title":"Classical approach: create a fast library and flexible calling enviroment","text":"","category":"section"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Crucial algorithms (sort, least squares...) are relatively small and well defined. Application of these algorithms to real-world problem is typically not well defined and requires more code. Iterative development. ","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Think of a problem of repeated execution of similar jobs with different options. Different level ","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"binary executable with command-line switches\nbinary executable with configuration file\nscripting language/environment (Read-Eval-Print Loop)","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"It is not a strict boundary, increasing expresivity of the configuration file will create a new scripting language.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Ending up in the 2 language problem. ","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Low-level programming = computer centric\nclose to the hardware\nallows excellent optimization for fast execution\nHigh-level programming = user centric\nrunning code with many different modifications as easily as possible\nallowing high level of abstraction","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"In scientific programming, the most well known scripting languages are: Python, Matlab, R","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"If you care about standard \"configurations\" they are just perfect. (PyTorch, BLAS)\nYou hit a problem with more complex experiments, such a modifying the internal algorithms.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"The scripting language typically makes decisions (if) at runtime. Becomes slow.","category":"page"},{"location":"lecture_01/motivation/#Examples","page":"Motivation","title":"Examples","text":"","category":"section"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Basic Linear Algebra Subroutines (BLAS)–MKL, OpenBlas–-with bindings (Matlab, NumPy)\nMatlab and Mex (C with pointer arithmetics)\nPython with transcription to C (Cython)","category":"page"},{"location":"lecture_01/motivation/#Convergence-efforts","page":"Motivation","title":"Convergence efforts","text":"","category":"section"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Just-in-time compilation (understands high level and converts to low-level)\nautomatic typing (auto in C++) (extends low-level with high-level concepts)","category":"page"},{"location":"lecture_01/motivation/#Julia-approach:-fresh-thinking","page":"Motivation","title":"Julia approach: fresh thinking","text":"","category":"section"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"(Image: )","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"A dance between specialization and abstraction. ","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Specialization allows for custom treatment. The right algorithm for the right circumstance is obtained by Multiple dispatch,\nAbstraction recognizes what remains the same after differences are stripped away. Abstractions in mathematics are captured as code through generic programming.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Why a new language?","category":"page"},{"location":"lecture_01/motivation/#Challenge","page":"Motivation","title":"Challenge","text":"","category":"section"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Translate high-level thinking with as much abstraction as possible into specific fast machine code.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Not so easy!","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"theorem: Indexing array x in Matlab:\nx = [1,2,3]\ny=x(4/2)\ny=x(5/2)In the first case it works, in the second throws an error.type instability \nfunction inde(x,n,m)=x(n/m) can never be fast.\nPoor language design choice!","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Simple solution","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Solved by different floating and integer division operation /,÷\nNot so simple with complex objects, e.g. triangular matrices","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Julia was designed as a high-level language that allows very high level abstract concepts but propagates as much information about the specifics as possible to help the compiler to generate as fast code as possible. Taking lessons from the inability to achieve fast code compilation (mostly from python).","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"(Image: )","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"julia is faster than C?","category":"page"},{"location":"lecture_01/motivation/#Julia-way","page":"Motivation","title":"Julia way","text":"","category":"section"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Design principle: abstraction should have zero runtime cost","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"flexible type system with strong typing (abstract types)\nmultiple dispatch\nsingle language from high to low levels (as much as possible) optimize execution as much as you can during compile time\nfunctions as symbolic abstraction layers","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"(Image: )","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"AST = Abstract Syntax Tree\nIR = Intermediate Representation","category":"page"},{"location":"lecture_01/motivation/#Teaser-example","page":"Motivation","title":"Teaser example","text":"","category":"section"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Function recursion with arbitrary number of arguments:","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"fsum(x) = x\nfsum(x,p...) = x+fsum(p...)","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Defines essentially a sum of inputs. Nice generic and abstract concept.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Possible in many languages:","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Matlab via nargin, varargin using construction if nargin==1, out=varargin{1}, else out=fsum(varargin{2:end}), end","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Julia solves this if at compile time. ","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"The generated code can be inspected by macro @code_llvm?","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"fsum(1,2,3)\n@code_llvm fsum(1,2,3)\n@code_llvm fsum(1.0,2.0,3.0)\nfz()=fsum(1,2,3)\n@code_llvm fz()","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Note that each call of fsum generates a new and different function.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Functions can act either as regular functions or like templates in C++. Compiler decides.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"This example is relatively simple, many other JIT languages can optimize such code. Julia allows taking this approach further.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Generality of the code:","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"fsum('c',1)\nfsum([1,2],[3,4],[5,6])","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Relies on multiple dispatch of the + function.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"More involved example:","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"using Zygote\n\nf(x)=3x+1 # user defined function\n@code_llvm f'(10)","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"The simplification was not achieved by the compiler alone.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Julia provides tools for AST and IR code manipulation\nautomatic differentiation via IR manipulation is implemented in Zygote.jl\nin a similar way, debugger is implemented in Debugger.jl\nvery simple to design domain specific language\nusing Turing\nusing StatsPlots\n\n@model function gdemo(x, y)\n s² ~ InverseGamma(2, 3)\n m ~ Normal(0, sqrt(s²))\n x ~ Normal(m, sqrt(s²))\n y ~ Normal(m, sqrt(s²))\nend","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Such tools allow building a very convenient user experience on abstract level, and reaching very efficient code.","category":"page"},{"location":"lecture_01/motivation/#Reproducibile-research","page":"Motivation","title":"Reproducibile research","text":"","category":"section"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Think about a code that was written some time ago. To run it, you often need to be able to have the same version of the language it was written for. ","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"Standard way language freezes syntax and guarantees some back-ward compatibility (Matlab), which prevents future improvements\nJulia approach allows easy recreation of the environment in which the code was developed. Every project (e.g. directory) can have its own environment","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"tip: Environment\nIs an independent set of packages that can be local to an individual project or shared and selected by name.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"tip: Package\nA package is a source tree with a standard layout providing functionality that can be reused by other Julia projects.","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"This allows Julia to be a rapidly evolving ecosystem with frequent changes due to:","category":"page"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"built-in package manager\nswitching between multiple versions of packages","category":"page"},{"location":"lecture_01/motivation/#Package-manager","page":"Motivation","title":"Package manager","text":"","category":"section"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"implemented by Pkg.jl\nsource tree have their structure defined by a convention\nhave its own mode in REPL\nallows adding packages for using (add) or development (dev)\nsupporting functions for creation (generate) and activation (activate) and many others","category":"page"},{"location":"lecture_01/motivation/#Julia-from-user's-point-of-view","page":"Motivation","title":"Julia from user's point of view","text":"","category":"section"},{"location":"lecture_01/motivation/","page":"Motivation","title":"Motivation","text":"compilation of everything to as specialized as possible\nvery fast code\nslow interaction (caching...)\ngenerating libraries is harder \nthink of fsum, \neverything is \".h\" (Eigen library)\ndebugging is different to matlab/python\nextensibility, Multiple dispatch = multi-functions\nallows great extensibility and code composition\nnot (yet) mainstream thinking\nJulia is not Object-oriented\nJulia is (not pure) functional language","category":"page"},{"location":"lecture_12/hw/#hw12","page":"Homework","title":"Homework 12 - The Runge-Kutta ODE Solver","text":"","category":"section"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"There exist many different ODE solvers. To demonstrate how we can get significantly better results with a simple update to Euler, you will implement the second order Runge-Kutta method RK2:","category":"page"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"beginalign*\ntilde x_n+1 = x_n + hf(x_n t_n)\n x_n+1 = x_n + frach2(f(x_nt_n)+f(tilde x_n+1t_n+1))\nendalign*","category":"page"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"RK2 is a 2nd order method. It uses not only f (the slope at a given point), but also f (the derivative of the slope). With some clever manipulations you can arrive at the equations above with make use of f without needing an explicit expression for it (if you want to know how, see here). Essentially, RK2 computes an initial guess tilde x_n+1 to then average the slopes at the current point x_n and at the guess tilde x_n+1 which is illustarted below. (Image: rk2)","category":"page"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"The code from the lab that you will need for this homework is given below. As always, put all your code in a file called hw.jl, zip it, and upload it to BRUTE.","category":"page"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"struct ODEProblem{F,T<:Tuple{Number,Number},U<:AbstractVector,P<:AbstractVector}\n f::F\n tspan::T\n u0::U\n θ::P\nend\n\n\nabstract type ODESolver end\n\nstruct Euler{T} <: ODESolver\n dt::T\nend\n\nfunction (solver::Euler)(prob::ODEProblem, u, t)\n f, θ, dt = prob.f, prob.θ, solver.dt\n (u + dt*f(u,θ), t+dt)\nend\n\n\nfunction solve(prob::ODEProblem, solver::ODESolver)\n t = prob.tspan[1]; u = prob.u0\n us = [u]; ts = [t]\n while t < prob.tspan[2]\n (u,t) = solver(prob, u, t)\n push!(us,u)\n push!(ts,t)\n end\n ts, reduce(hcat,us)\nend\n\n\n# Define & Solve ODE\n\nfunction lotkavolterra(x,θ)\n α, β, γ, δ = θ\n x₁, x₂ = x\n\n dx₁ = α*x₁ - β*x₁*x₂\n dx₂ = δ*x₁*x₂ - γ*x₂\n\n [dx₁, dx₂]\nend","category":"page"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"
\n
Homework (2 points)
\n
","category":"page"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"Implement the 2nd order Runge-Kutta solver according to the equations given above by overloading the call method of a new type RK2.","category":"page"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"(solver::RK2)(prob::ODEProblem, u, t)","category":"page"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"
","category":"page"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"struct RK2{T} <: ODESolver\n dt::T\nend\nfunction (solver::RK2)(prob::ODEProblem, u, t)\n f, θ, dt = prob.f, prob.θ, solver.dt\n du = f(u,θ)\n uh = u + du*dt\n u + dt/2*(du + f(uh,θ)), t+dt\nend","category":"page"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"You should be able to use it exactly like our Euler solver before:","category":"page"},{"location":"lecture_12/hw/","page":"Homework","title":"Homework","text":"using Plots\nusing JLD2\n\n# Define ODE\nfunction lotkavolterra(x,θ)\n α, β, γ, δ = θ\n x₁, x₂ = x\n\n dx₁ = α*x₁ - β*x₁*x₂\n dx₂ = δ*x₁*x₂ - γ*x₂\n\n [dx₁, dx₂]\nend\n\nθ = [0.1,0.2,0.3,0.2]\nu0 = [1.0,1.0]\ntspan = (0.,100.)\nprob = ODEProblem(lotkavolterra,tspan,u0,θ)\n\n# load correct data\ntrue_data = load(\"lotkadata.jld2\")\n\n# create plot\np1 = plot(true_data[\"t\"], true_data[\"u\"][1,:], lw=4, ls=:dash, alpha=0.7,\n color=:gray, label=\"x Truth\")\nplot!(p1, true_data[\"t\"], true_data[\"u\"][2,:], lw=4, ls=:dash, alpha=0.7,\n color=:gray, label=\"y Truth\")\n\n# Euler solve\n(t,X) = solve(prob, Euler(0.2))\nplot!(p1,t,X[1,:], color=3, lw=3, alpha=0.8, label=\"x Euler\", ls=:dot)\nplot!(p1,t,X[2,:], color=4, lw=3, alpha=0.8, label=\"y Euler\", ls=:dot)\n\n# RK2 solve\n(t,X) = solve(prob, RK2(0.2))\nplot!(p1,t,X[1,:], color=1, lw=3, alpha=0.8, label=\"x RK2\")\nplot!(p1,t,X[2,:], color=2, lw=3, alpha=0.8, label=\"y RK2\")","category":"page"},{"location":"lecture_06/lab/#introspection_lab","page":"Lab","title":"Lab 06: Code introspection and metaprogramming","text":"","category":"section"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"In this lab we are first going to inspect some tooling to help you understand what Julia does under the hood such as:","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"looking at the code at different levels\nunderstanding what method is being called\nshowing different levels of code optimization","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Secondly we will start playing with the metaprogramming side of Julia, mainly covering:","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"how to view abstract syntax tree (AST) of Julia code\nhow to manipulate AST","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"These topics will be extended in the next lecture/lab, where we are going use metaprogramming to manipulate code with macros.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"We will be again a little getting ahead of ourselves as we are going to use quite a few macros, which will be properly explained in the next lecture as well, however for now the important thing to know is that a macro is just a special function, that accepts as an argument Julia code, which it can modify.","category":"page"},{"location":"lecture_06/lab/#Quick-reminder-of-introspection-tooling","page":"Lab","title":"Quick reminder of introspection tooling","text":"","category":"section"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Let's start with the topic of code inspection, e.g. we may ask the following: What happens when Julia evaluates [i for i in 1:10]?","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"parsing ","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"using InteractiveUtils #hide\n:([i for i in 1:10]) |> dump","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"lowering","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Meta.@lower [i for i in 1:10]","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"typing","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"f() = [i for i in 1:10]\n@code_typed f()","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"LLVM code generation","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"@code_llvm f()","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"native code generation","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"@code_native f()","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Let's see how these tools can help us understand some of Julia's internals on examples from previous labs and lectures.","category":"page"},{"location":"lecture_06/lab/#Understanding-runtime-dispatch-and-type-instabilities","page":"Lab","title":"Understanding runtime dispatch and type instabilities","text":"","category":"section"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"We will start with a question: Can we spot internally some difference between type stable/unstable code?","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Inspect the following two functions using @code_lowered, @code_typed, @code_llvm and @code_native.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"x = rand(10^5)\nfunction explicit_len(x)\n length(x)\nend\n\nfunction implicit_len()\n length(x)\nend\nnothing #hide","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"For now do not try to understand the details, but focus on the overall differences such as length of the code.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"info: Redirecting `stdout`\nIf the output of the method introspection tools is too long you can use a general way of redirecting standard output stdout to a fileopen(\"./llvm_fun.ll\", \"w\") do file\n original_stdout = stdout\n redirect_stdout(file)\n @code_llvm fun()\n redirect_stdout(original_stdout)\nendIn case of @code_llvm and @code_native there are special options, that allow this out of the box, see help ? for underlying code_llvm and code_native. If you don't mind adding dependencies there is also the @capture_out from Suppressor.jl","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"@code_warntype explicit_sum(x)\n@code_warntype implicit_sum()\n\n@code_typed explicit_sum(x)\n@code_typed implicit_sum()\n\n@code_llvm explicit_sum(x)\n@code_llvm implicit_sum()\n\n@code_native explicit_sum(x)\n@code_native implicit_sum()","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"In this case we see that the generated code for such a simple operation is much longer in the type unstable case resulting in longer run times. However in the next example we will see that having longer code is not always a bad thing.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_06/lab/#Loop-unrolling","page":"Lab","title":"Loop unrolling","text":"","category":"section"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"In some cases the compiler uses loop unrolling[1] optimization to speed up loops at the expense of binary size. The result of such optimization is removal of the loop control instructions and rewriting the loop into a repeated sequence of independent statements.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"[1]: https://en.wikipedia.org/wiki/Loop_unrolling","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Inspect under what conditions does the compiler unroll the for loop in the polynomial function from the last lab.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"function polynomial(a, x)\n accumulator = a[end] * one(x)\n for i in length(a)-1:-1:1\n accumulator = accumulator * x + a[i]\n end\n accumulator \nend\nnothing #hide","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Compare the speed of execution with and without loop unrolling.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"these kind of optimization are lower level than intermediate language\nloop unrolling is possible when compiler knows the length of the input","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"using Test #hide\nusing BenchmarkTools\na = Tuple(ones(20)) # tuple has known size\nac = collect(a)\nx = 2.0\n\n@code_lowered polynomial(a,x) # cannot be seen here as optimizations are not applied\n@code_typed polynomial(a,x) # loop unrolling is not part of type inference optimization\nnothing #hide","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"@code_llvm polynomial(a,x)\n@code_llvm polynomial(ac,x)","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"More than 2x speedup","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"@btime polynomial($a,$x)\n@btime polynomial($ac,$x)","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_06/lab/#Recursion-inlining-depth","page":"Lab","title":"Recursion inlining depth","text":"","category":"section"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Inlining[2] is another compiler optimization that allows us to speed up the code by avoiding function calls. Where applicable compiler can replace f(args) directly with the function body of f, thus removing the need to modify stack to transfer the control flow to a different place. This is yet another optimization that may improve speed at the expense of binary size.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"[2]: https://en.wikipedia.org/wiki/Inline_expansion","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Rewrite the polynomial function from the last lab using recursion and find the length of the coefficients, at which inlining of the recursive calls stops occurring.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"function polynomial(a, x)\n accumulator = a[end] * one(x)\n for i in length(a)-1:-1:1\n accumulator = accumulator * x + a[i]\n end\n accumulator \nend","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"info: Splatting/slurping operator `...`\nThe operator ... serves two purposes inside function calls [3][4]:combines multiple arguments into onefunction printargs(args...)\n println(typeof(args))\n for (i, arg) in enumerate(args)\n println(\"Arg #$i = $arg\")\n end\nend\nprintargs(1, 2, 3)splits one argument into many different argumentsfunction threeargs(a, b, c)\n println(\"a = $a::$(typeof(a))\")\n println(\"b = $b::$(typeof(b))\")\n println(\"c = $c::$(typeof(c))\")\nend\nthreeargs([1,2,3]...) # or with a variable threeargs(x...)[3]: https://docs.julialang.org/en/v1/manual/faq/#What-does-the-...-operator-do?[4]: https://docs.julialang.org/en/v1/manual/functions/#Varargs-Functions","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"define two methods _polynomial!(ac, x, a...) and _polynomial!(ac, x, a) for the case of ≥2 coefficients and the last coefficient\nuse splatting together with range indexing a[1:end-1]...\nthe correctness can be checked using the built-in evalpoly\nrecall that these kind of optimization are possible just around the type inference stage\nuse container of known length to store the coefficients","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"_polynomial!(ac, x, a...) = _polynomial!(x * ac + a[end], x, a[1:end-1]...)\n_polynomial!(ac, x, a) = x * ac + a\npolynomial(a, x) = _polynomial!(a[end] * one(x), x, a[1:end-1]...)\n\n# the coefficients have to be a tuple\na = Tuple(ones(Int, 21)) # everything less than 22 gets inlined\nx = 2\npolynomial(a,x) == evalpoly(x,a) # compare with built-in function\n\n# @code_llvm polynomial(a,x) # seen here too, but code_typed is a better option\n@code_lowered polynomial(a,x) # cannot be seen here as optimizations are not applied\nnothing #hide","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"@code_typed polynomial(a,x)","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_06/lab/#AST-manipulation:-The-first-steps-to-metaprogramming","page":"Lab","title":"AST manipulation: The first steps to metaprogramming","text":"","category":"section"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Julia is so called homoiconic language, as it allows the language to reason about its code. This capability is inspired by years of development in other languages such as Lisp, Clojure or Prolog.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"There are two easy ways to extract/construct the code structure [5]","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"parsing code stored in string with internal Meta.parse","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"code_parse = Meta.parse(\"x = 2\") # for single line expressions (additional spaces are ignored)\ncode_parse_block = Meta.parse(\"\"\"\nbegin\n x = 2\n y = 3\n x + y\nend\n\"\"\") # for multiline expressions","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"constructing an expression using quote ... end or simple :() syntax","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"code_expr = :(x = 2) # for single line expressions (additional spaces are ignored)\ncode_expr_block = quote\n x = 2\n y = 3\n x + y \nend # for multiline expressions","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Results can be stored into some variables, which we can inspect further.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"typeof(code_parse)\ndump(code_parse)","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"typeof(code_parse_block)\ndump(code_parse_block)","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"The type of both multiline and single line expression is Expr with fields head and args. Notice that Expr type is recursive in the args, which can store other expressions resulting in a tree structure - abstract syntax tree (AST) - that can be visualized for example with the combination of GraphRecipes and Plots packages. ","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"using GraphRecipes #hide\nusing Plots #hide\nplot(code_expr_block, fontsize=12, shorten=0.01, axis_buffer=0.15, nodeshape=:rect)","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"This recursive structure has some major performance drawbacks, because the args field is of type Any and therefore modifications of this expression level AST won't be type stable. Building blocks of expressions are Symbols and literal values (numbers).","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"A possible nuisance of working with multiline expressions is the presence of LineNumber nodes, which can be removed with Base.remove_linenums! function.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Base.remove_linenums!(code_parse_block)","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Parsed expressions can be evaluate using eval function. ","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"eval(code_parse) # evaluation of :(x = 2) \nx # should be defined","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Before doing anything more fancy let's start with some simple manipulation of ASTs.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Define a variable code to be as the result of parsing the string \"j = i^2\". \nCopy code into a variable code2. Modify this to replace the power 2 with a power 3. Make sure that the original code variable is not also modified. \nCopy code2 to a variable code3. Replace i with i + 1 in code3.\nDefine a variable i with the value 4. Evaluate the different code expressions using the eval function and check the value of the variable j.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"code = Meta.parse(\"j = i^2\")\ncode2 = copy(code)\ncode2.args[2].args[3] = 3\ncode3 = copy(code2)\ncode3.args[2].args[2] = :(i + 1)\ni = 4\neval(code), eval(code2), eval(code3)","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Following up on the more general substitution of variables in an expression from the lecture, let's see how the situation becomes more complicated, when we are dealing with strings instead of a parsed AST.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"using Test #hide\nreplace_i(s::Symbol) = s == :i ? :k : s\nreplace_i(e::Expr) = Expr(e.head, map(replace_i, e.args)...)\nreplace_i(u) = u\nnothing #hide","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Given a function replace_i, which replaces variables i for k in an expression like the following","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"ex = :(i + i*i + y*i - sin(z))\n@test replace_i(ex) == :(k + k*k + y*k - sin(z))","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"write a different function sreplace_i(s), which does the same thing but instead of a parsed expression (AST) it manipulates a string, such as","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"s = string(ex)","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Use Meta.parse in combination with replace_i ONLY for checking of correctness.\nYou can use the replace function in combination with regular expressions.\nThink of some corner cases, that the method may not handle properly.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"The naive solution","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"sreplace_i(s) = replace(s, 'i' => 'k')\n@test Meta.parse(sreplace_i(s)) == replace_i(Meta.parse(s))","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"does not work in this simple case, because it will replace \"i\" inside the sin(z) expression. We can play with regular expressions to obtain something, that is more robust","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"sreplace_i(s) = replace(s, r\"([^\\w]|\\b)i(?=[^\\w]|\\z)\" => s\"\\1k\")\n@test Meta.parse(sreplace_i(s)) == replace_i(Meta.parse(s))","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"however the code may now be harder to read. Thus it is preferable to use the parsed AST when manipulating Julia's code.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"If the exercises so far did not feel very useful let's focus on one, that is similar to a part of the IntervalArithmetics.jl pkg.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Write function wrap!(ex::Expr) which wraps literal values (numbers) with a call to f(). You can test it on the following example","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"f = x -> convert(Float64, x)\nex = :(x*x + 2*y*x + y*y) # original expression\nrex = :(x*x + f(2)*y*x + y*y) # result expression\nnothing #hide","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"use recursion and multiple dispatch\ndispatch on ::Number to detect numbers in an expression\nfor testing purposes, create a copy of ex before mutating","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"function wrap!(ex::Expr)\n args = ex.args\n \n for i in 1:length(args)\n args[i] = wrap!(args[i])\n end\n\n return ex\nend\n\nwrap!(ex::Number) = Expr(:call, :f, ex)\nwrap!(ex) = ex\n\next, x, y = copy(ex), 2, 3\n@test wrap!(ex) == :(x*x + f(2)*y*x + y*y)\neval(ext)\neval(ex)","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"This kind of manipulation is at the core of some pkgs, such as aforementioned IntervalArithmetics.jl where every number is replaced with a narrow interval in order to find some bounds on the result of a computation.","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"","category":"page"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"[5]: Once you understand the recursive structure of expressions, the AST can be constructed manually like any other type.","category":"page"},{"location":"lecture_06/lab/#Resources","page":"Lab","title":"Resources","text":"","category":"section"},{"location":"lecture_06/lab/","page":"Lab","title":"Lab","text":"Julia's manual on metaprogramming\nDavid P. Sanders' workshop @ JuliaCon 2021 \nSteven Johnson's keynote talk @ JuliaCon 2019\nAndy Ferris's workshop @ JuliaCon 2018\nFrom Macros to DSL by John Myles White \nNotes on JuliaCompilerPlugin","category":"page"},{"location":"how_to_submit_hw/#homeworks","page":"Homework submission","title":"Homework submission","text":"","category":"section"},{"location":"how_to_submit_hw/","page":"Homework submission","title":"Homework submission","text":"This document should describe the homework submission procedure.","category":"page"},{"location":"lecture_07/macros/#Macros","page":"Macros","title":"Macros","text":"","category":"section"},{"location":"lecture_01/basics/#Syntax","page":"Basics","title":"Syntax","text":"","category":"section"},{"location":"lecture_01/basics/#Elementary-syntax:-Matlab-heritage","page":"Basics","title":"Elementary syntax: Matlab heritage","text":"","category":"section"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Very much like matlab:","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"indexing from 1\narray as first-class A=[1 2 3]","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Cheat sheet: https://cheatsheets.quantecon.org/","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Introduction: https://juliadocs.github.io/Julia-Cheat-Sheet/","category":"page"},{"location":"lecture_01/basics/#Arrays-are-first-class-citizens","page":"Basics","title":"Arrays are first-class citizens","text":"","category":"section"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Many design choices were motivated considering matrix arguments:","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"x *= 2 is implemented as x = x*2 causing new allocation (vectors).","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"The reason is consistency with matrix operations: A *= B works as A = A*B.","category":"page"},{"location":"lecture_01/basics/#Broadcasting-operator","page":"Basics","title":"Broadcasting operator","text":"","category":"section"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Julia generalizes matlabs .+ operation to general use for any function. ","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"a = [1 2 3]\nsin.(a)\nf(x)=x^2+3x+8\nf.(a)","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Solves the problem of inplace multiplication","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"x .*= 2 ","category":"page"},{"location":"lecture_01/basics/#Functional-roots-of-Julia","page":"Basics","title":"Functional roots of Julia","text":"","category":"section"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Function is a first-class citizen.","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Repetition of functional programming:","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"function mymap(f::Function,a::AbstractArray)\n b = similar(a)\n for i in eachindex(a)\n b[i]=f(a[i])\n end\n b\nend","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Allows for anonymous functions:","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"mymap(x->x^2+2,[1.0,2.0])","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Function properties:","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Arguments are passed by reference (change of mutable inputs inside the function is visible outside)\nConvention: function changing inputs have a name ending by \"!\" symbol\nreturn value \nthe last line of the function declaration, \nreturn keyword\nzero cost abstraction","category":"page"},{"location":"lecture_01/basics/#Different-style-of-writing-code","page":"Basics","title":"Different style of writing code","text":"","category":"section"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Definitions of multiple small functions and their composition","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"fsum(x) = x\nfsum(x,p...) = x+fsum(p[1],p[2:end]...)","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"a single methods may not be sufficient to understand the full algorithm. In procedural language, you may write:","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"function out=fsum(x,varargin)\n if nargin==2 # TODO: better treatment\n out=x\n else\n out = fsum(varargin{1},varargin{2:end})\n end","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"The need to build intuition for function composition.","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Dispatch is easier to optimize by the compiler.","category":"page"},{"location":"lecture_01/basics/#Operators-are-functions","page":"Basics","title":"Operators are functions","text":"","category":"section"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"operator function name\n[A B C ...] hcat\n[A; B; C; ...] vcat\n[A B; C D; ...] hvcat\nA' adjoint\nA[i] getindex\nA[i] = x setindex!\nA.n getproperty\nA.n = x setproperty!","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"struct Foo end\n\nBase.getproperty(a::Foo, x::Symbol) = x == :a ? 5 : error(\"does not have property $(x)\")","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"Can be redefined and overloaded for different input types. The getproperty method can define access to the memory structure.","category":"page"},{"location":"lecture_01/basics/#Broadcasting-revisited","page":"Basics","title":"Broadcasting revisited","text":"","category":"section"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"The a.+b syntax is a syntactic sugar for broadcast(+,a,b).","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"The special meaning of the dot is that they will be fused into a single call:","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"f.(g.(x .+ 1)) is treated by Julia as broadcast(x -> f(g(x + 1)), x). \nAn assignment y .= f.(g.(x .+ 1)) is treated as in-place operation broadcast!(x -> f(g(x + 1)), y, x).","category":"page"},{"location":"lecture_01/basics/","page":"Basics","title":"Basics","text":"The same logic works for lists, tuples, etc.","category":"page"},{"location":"lecture_03/lecture/#Design-patterns:-good-practices-and-structured-thinking","page":"Lecture","title":"Design patterns: good practices and structured thinking","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Every software developer has a desire to write better code. A desire to improve system performance. A desire to design software that is easy to maintain, easy to understand and explain.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Design patterns are recommendations and good practices accumulating knowledge of experienced programmers.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"The highest level of experience contains the design guiding principles:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"SOLID: Single Responsibility, Open/Closed, Liskov Substitution, Interface\nSegregation, Dependency Inversion\nDRY: Don't Repeat Yourself\nKISS: Keep It Simple, Stupid!\nPOLA: Principle of Least Astonishment\nYAGNI: You Aren't Gonna Need It (overengineering)\nPOLP: Principle of Least Privilege ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"While these high-level concepts are intuitive, they are too general to give specific answers.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"More detailed patterns arise for programming paradigms (declarative, imperative) with specific instances of functional or object-oriented programming.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"The concept of design patterns originates in the OOP paradigm. OOP defines a strict way how to write software. Sometimes it is not clear how to squeeze real world problems into those rules. Cookbook for many practical situations","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Gamma, E., Johnson, R., Helm, R., Johnson, R. E., & Vlissides, J. (1995). Design patterns: elements of reusable object-oriented software. Pearson Deutschland GmbH.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Defining 23 design patterns in three categories. Became extremely popular.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"(Image: ) (C) Scott Wlaschin","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Is julia OOP or FP? It is different from both, based on:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"types system (polymorphic)\nmultiple dispatch (extending single dispatch of OOP)\nfunctions as first class \ndecoupling of data and functions\nmacros","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Any guidelines to solve real-world problems?","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Hands-On Design Patterns and Best Practices with Julia Proven solutions to common problems in software design for Julia 1.x Tom Kwong, CFA","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Fundamental tradeoff: rules vs. freedom","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"freedom: in the C language it is possible to access assembler instructions, use pointer aritmetics:\nit is possible to write extremely efficient code\nit is easy to segfault, leak memory, etc.\nrules: in strict languages (strict OOP, strict functional programing) you lose freedom for certain guarantees:\ne.g. strict functional programing guarantees that the program provably terminates\noperations that are simple e.g. in pointer arithmetics may become clumsy and inefficient in those strict rules.\nthe compiler can validate the rules and complain if the code does not comply with them. ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Julia is again a dance between freedom and strict rules. It is more inclined to freedom. Provides few simple concepts that allow to construct design patterns common in other languages.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"the language does not enforce too many formalisms (via keywords (interface, trait, etc.) but they can be \nthe compiler cannot check for correctness of these \"patterns\"\nthe user has a lot of freedom (and responsibility)\nlots of features can be added by Julia packages (with various level of comfort)\nmacros","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Read: ","category":"page"},{"location":"lecture_03/lecture/#Design-Patterns-of-OOP-from-the-Julia-viewpoint","page":"Lecture","title":"Design Patterns of OOP from the Julia viewpoint","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"OOP is currently very popular concept (C++, Java, Python). It has strenghts and weaknesses. The Julia authors tried to keep the strength and overcome weaknesses. ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Key features of OOP:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Encapsulation \nInheritance \nPolymorphism ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Classical OOP languages define classes that bind processing functions to the data. Virtual methods are defined only for the attached methods of the classes.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"tip: Encapsulation\nRefers to bundling of data with the methods that operate on that data, or the restricting of direct access to some of an object's components. Encapsulation is used to hide the values or state of a structured data object inside a class, preventing direct access to them by clients in a way that could expose hidden implementation details or violate state invariance maintained by the methods. ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"tip: Making Julia to mimic OOP\nThere are many discussions how to make Julia to behave like an OOP. The best implementation to our knowledge is ObjectOriented","category":"page"},{"location":"lecture_03/lecture/#Encapsulation-Advantage:-Consistency-and-Validity","page":"Lecture","title":"Encapsulation Advantage: Consistency and Validity","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"With fields of data structure freely accessible, the information may become inconsistent.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"mutable struct Grass <: Plant\n id::Int\n size::Int\n max_size::Int\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"What if I create Grass with larger size than max_size?","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"grass = Grass(1,50,5)","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Freedom over Rules. Maybe I would prefer to introduce some rules.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Some encapsulation may be handy keeping it consistent. Julia has inner constructor.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"mutable struct Grass2 <: Plant\n id::Int\n size::Int\n max_size::Int\n Grass2(id,sz,msz) = sz > msz ? error(\"size can not be greater that max_size\") : new(id,sz,msz)\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"When defined, Julia does not provide the default outer constructor. ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"But fields are still accessible:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"grass.size = 10000","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Recall that grass.size=1000 is a syntax of setproperty!(grass,:size,1000), which can be redefined:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"function Base.setproperty!(obj::Grass, sym::Symbol, val)\n if sym==:size\n @assert val<=obj.max_size \"size have to be lower than max_size!\"\n end\n setfield!(obj,sym,val)\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Function setfield! can not be overloaded.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Julia has partial encapsulation via a mechanism for consistency checks. ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"warn: Array in unmutable struct can be mutated\nThe mutability applies to the structure and not to encapsulated structures.struct Foo\n x::Float64\n y::Vector{Float64}\n z::Dict{Int,Int}\nendIn the structure Foo, x cannot be mutated, but fields of y and key-value pairs of z can be mutated, because they are mutable containers. But I cannot replace y with a different Vector.","category":"page"},{"location":"lecture_03/lecture/#Encapsulation-Disadvantage:-the-Expression-Problem","page":"Lecture","title":"Encapsulation Disadvantage: the Expression Problem","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Encapsulation limits the operations I can do with an object. Sometimes too much. Consider a matrix of methods/types(data-structures)","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Consider an existing matrix of data and functions:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"data \\ methods find_food eat! grow! \nWolf \nSheep \nGrass ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"You have a good reason not to modify the original source (maintenance).","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Imagine we want to extend the world to use new animals and new methods for all animals.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Object-oriented programming ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"classes are primary objects (hierarchy)\ndefine animals as classes ( inheriting from abstract class)\nadding a new animal is easy\nadding a new method for all animals is hard (without modifying the original code)","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Functional programming ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"functions are primary\ndefine operations find_food, eat!\nadding a new operation is easy\nadding new data structure to existing operations is hard","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Solutions:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"multiple-dispatch = julia\nopen classes (monkey patching) = add methods to classes on the fly\nvisitor pattern = partial fix for OOP [extended visitor pattern using dynamic_cast]","category":"page"},{"location":"lecture_03/lecture/#Morale:","page":"Lecture","title":"Morale:","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Julia does not enforces creation getters/setters by default (setproperty is mapped to setfield)\nit provides tools to enforce access restriction if the user wants it.\ncan be used to imitate objects: ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"https://stackoverflow.com/questions/39133424/how-to-create-a-single-dispatch-object-oriented-class-in-julia-that-behaves-l/39150509#39150509","category":"page"},{"location":"lecture_03/lecture/#Polymorphism:","page":"Lecture","title":"Polymorphism:","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"tip: Polymorphism in OOP\nPolymorphism is the method in an object-oriented programming language that performs different things as per the object’s class, which calls it. With Polymorphism, a message is sent to multiple class objects, and every object responds appropriately according to the properties of the class. ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Example animals of different classes make different sounds. In Python:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"\nclass Sheep:\n def __init__(self, energy, Denergy):\n self.energy = energy\n self.Denergy = Denergy\n\n def make_sound(self):\n print(\"Baa\")\n\nsheep.make_sound()\nwolf.make_sound()","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Will make distinct sounds (baa, Howl). ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Can we achieve this in Julia?","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"make_sound(s::Sheep)=println(\"Baa\")\nmake_sound(w::Wolf)=println(\"Howl\")","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"tip: Implementation of virtual methods\nVirtual methods in OOP are typically implemented using Virtual Method Table, one for each class. (Image: )Julia has a single method table. Dispatch can be either static or dynamic (slow).","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Freedom vs. Rules. ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Duck typing is a type of polymorphism without static types\nmore programming freedom, less formal guarantees\njulia does not check if make_sound exists for all animals. May result in MethodError. Responsibility of a programmer.\ndefine make_sound(A::AbstractAnimal)","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"So far, the polymorphism coincides for OOP and julia becuase the method had only one argument => single argument dispatch.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Multiple dispatch is an extension of the classical first-argument-polymorphism of OOP, to all-argument polymorphism.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"tip: Challenge for OOP\nHow to code polymorphic behavior of interaction between two agents, e.g. an agent eating another agent in OOP?Complicated.... You need a \"design pattern\" for it.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"class Sheep(Animal):\n energy: float = 4.0\n denergy: float = 0.2\n reprprob: float = 0.5\n foodprob: float = 0.9\n\n # hard, if not impossible to add behaviour for a new type of food\n def eat(self, a: Agent, w: World):\n if isinstance(a, Grass)\n self.energy += a.size * self.denergy\n a.size = 0\n else:\n raise ValueError(f\"Sheep cannot eat {type(a).__name__}.\")","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Consider an extension to:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Flower : easy\nPoisonousGrass: harder","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Simple in Julia:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"eat!(w1::Sheep, a::Grass, w::World)=\neat!(w1::Sheep, a::Flower, w::World)=\neat!(w1::Sheep, a::PoisonousGrass, w::World)=","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Boiler-plate code can be automated by macros / meta programming.","category":"page"},{"location":"lecture_03/lecture/#Inheritance","page":"Lecture","title":"Inheritance","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"tip: Inheritance\nIs the mechanism of basing one object or class upon another object (prototype-based inheritance) or class (class-based inheritance), retaining similar implementation. Deriving new classes (sub classes) from existing ones such as super class or base class and then forming them into a hierarchy of classes. In most class-based object-oriented languages, an object created through inheritance, a \"child object\", acquires all the properties and behaviors of the \"parent object\" , with the exception of: constructors, destructor, overloaded operators.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Most commonly, the sub-class inherits methods and the data.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"For example, in python we can design a sheep with additional field. Think of a situation that we want to refine the reproduction procedure for sheeps by considering differences for male and female. We do not have information about gender in the original implementation. ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"In OOP, we can use inheritance.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"class Sheep:\n def __init__(self, energy, Denergy):\n self.energy = energy\n self.Denergy = Denergy\n\n def make_sound(self):\n print(\"Baa\")\n\nclass SheepWithGender(Sheep):\n def __init__(self, energy, Denergy,gender):\n super().__init__(energy, Denergy)\n self.gender = gender\n # make_sound is inherited \n\n# Can you do this in Julia?!","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Simple answer: NO, not exactly","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Sheep has fields, is a concrete type, we cannot extend it.\nwith modification of the original code, we can define AbstractSheep with subtypes Sheep and SheepWithGender.\nBut methods for AbstractAnimal works for sheeps! Is this inheritance?","category":"page"},{"location":"lecture_03/lecture/#Inheritance-vs.-Subtyping","page":"Lecture","title":"Inheritance vs. Subtyping","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Subtle difference:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"subtyping = equality of interface \ninheritance = reuse of implementation ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"In practice, subtyping reuse methods, not data fields.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"We have seen this in Julia, using type hierarchy: ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"agent_step!(a::Animal, w::World)\nall animals subtype of Animal \"inherit\" this method.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"The type hierarchy is only one way of subtyping. Julia allows many variations, e.g. concatenating different parts of hierarchies via the Union{} type:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"fancy_method(O::Union{Sheep,Grass})=println(\"Fancy\")","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Is this a good idea? It can be done completely Ad-hoc! Freedom over Rules.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"There are very good use-cases:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Missing values:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"x::AbstractVector{<:Union{<:Number, Missing}}","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"theorem: SubTyping issues\nWith parametric types, unions and other construction, subtype resolution may become a complicated problem. Julia can even crash. (Jan Vitek's Keynote at JuliaCon 2021)[https://www.youtube.com/watch?v=LT4AP7CUMAw]","category":"page"},{"location":"lecture_03/lecture/#Sharing-of-data-field-via-composition","page":"Lecture","title":"Sharing of data field via composition","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Composition is also recommended in OOP: (Composition over ingeritance)[https://en.wikipedia.org/wiki/Compositionoverinheritance]","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"struct ⚥Sheep <: Animal\n sheep::Sheep\n sex::Symbol\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"If we want our new ⚥Sheep to behave like the original Sheep, we need to forward the corresponding methods.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"eat!(a::⚥Sheep, b::Grass, w::World)=eat!(a.sheep, b, w)","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"and all other methods. Routine work. Boring! The whole process can be automated using macros @forward from Lazy.jl.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Why so complicated? Wasn't the original inheritance tree structure better?","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"multiple inheritance:\nyou just compose two different \"trees\".\ncommon example with ArmoredVehicle = Vehicle + Weapon\nDo you think there is only one sensible inheritance tree?","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"tip: Animal World\nThink of an inheritance tree of a full scope Animal world.Idea #1: Split animals by biological taxonomy (Image: )Hold on. Sharks and dolphins can swim very well!\nBoth bats and birds fly similarly!Idea #2: Split by the way they move!Idea #3: Split by way of ...","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"In fact, we do not have a tree, but more like a matrix/tensor:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":" swims flies walks\nbirds penguin eagle kiwi\nmammal dolphin bat sheep,wolf\ninsect backswimmer fly beetle","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Single type hierarchy will not work. Other approaches:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"interfaces\nparametric types","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Analyze what features of animals are common and compose the animal:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"abstract type HeatType end\nabstract type MovementType end\nabstract type ChildCare end\n\n\nmutable struct Animal{H<:HeatType,M<:MovementType,C<:ChildCare} \n id::Int\n ...\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Now, we can define methods dispatching on parameters of the main type.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Composition is simpler in such a general case. Composition over inheritance. ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"A simple example of parametric approach will be demonstarted in the lab.","category":"page"},{"location":"lecture_03/lecture/#Interfaces:-inheritance/subtyping-without-a-hierarchy-tree","page":"Lecture","title":"Interfaces: inheritance/subtyping without a hierarchy tree","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"In OOP languages such as Java, interfaces have a dedicated keyword such that compiler can check correctes of the interface implementation. ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"In Julia, interfaces can be achived by defining ordinary functions. Not so strict validation by the compiler as in other languages. Freedom...","category":"page"},{"location":"lecture_03/lecture/#Example:-Iterators","page":"Lecture","title":"Example: Iterators","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Many fundamental objects can be iterated: Arrays, Tuples, Data collections...","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"They do not have any common \"predecessor\". They are almost \"primitive\" types.\nthey share just the property of being iterable\nwe do not want to modify them in any way","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Example: of interface Iterators defined by \"duck typing\" via two functions.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Required methods Brief description\niterate(iter) Returns either a tuple of the first item and initial state or nothing if empty\niterate(iter, state) Returns either a tuple of the next item and next state or nothing if no items remain","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Defining these two methods for any object/collection C will make the following work:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"for o in C\n # do something\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"The compiler will not check if both functions exist.\nIf one is missing, it will complain about it when it needs it\nThe error message may be less informative than in the case of formal definition","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Note:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"even iterators may have different features: they can be finite or infinite\nfor finite iterators we can define useful functions (collect)\nhow to pass this information in an extensible way?","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Poor solution: if statements.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"function collect(iter)\n if iter isa Tuple...\n\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"The compiler can do that for us.","category":"page"},{"location":"lecture_03/lecture/#Traits:-cherry-picking-subtyping","page":"Lecture","title":"Traits: cherry picking subtyping","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Trait mechanism in Julia is build using the existing tools: Type System and Multiple Dispatch.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Traits have a few key parts:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Trait types: the different traits a type can have.\nTrait function: what traits a type has.\nTrait dispatch: using the traits.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"From iterators:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"# trait types:\n\nabstract type IteratorSize end\nstruct SizeUnknown <: IteratorSize end\nstruct HasLength <: IteratorSize end\nstruct IsInfinite <: IteratorSize end\n\n# Trait function: Input is a Type, output is a Type\nIteratorSize(::Type{<:Tuple}) = HasLength()\nIteratorSize(::Type) = HasLength() # HasLength is the default\n\n# ...\n\n# Trait dispatch\nBitArray(itr) = gen_bitarray(IteratorSize(itr), itr)\ngen_bitarray(isz::IteratorSize, itr) = gen_bitarray_from_itr(itr)\ngen_bitarray(::IsInfinite, itr) = throw(ArgumentError(\"infinite-size iterable used in BitArray constructor\"))\n","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"What is needed to define for a new type that I want to iterate over? ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Do you still miss inheritance in the OOP style?","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Many packages automating this with more structure:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"https://github.com/andyferris/Traitor.jl\nhttps://github.com/mauro3/SimpleTraits.jl\nhttps://github.com/tk3369/BinaryTraits.jl","category":"page"},{"location":"lecture_03/lecture/#Functional-tools:-Partial-evaluation","page":"Lecture","title":"Functional tools: Partial evaluation","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"It is common to create a new function which \"just\" specify some parameters.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"_prod(x) = reduce(*,x)\n_sum(x) = reduce(+,x)","category":"page"},{"location":"lecture_03/lecture/#Functional-tools:-Closures","page":"Lecture","title":"Functional tools: Closures","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"tip: Closure (lexical closure, function closure)\nA technique for implementing lexically scoped name binding in a language with first-class functions. Operationally, a closure is a record storing a function together with an environment.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"originates in functional programming\nnow widespread in many common languages, Python, Matlab, etc..\nmemory management relies on garbage collector in general (can be optimized by compiler)","category":"page"},{"location":"lecture_03/lecture/#Example","page":"Lecture","title":"Example","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"function adder(x)\n return y->x+y\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"creates a function that \"closes\" the argument x. Try: f=adder(5); f(3).","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"x = 30;\nfunction adder()\n return y->x+y\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"creates a function that \"closes\" variable x.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"f = adder(10)\nf(1)\ng = adder()\ng(1)\n","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Such function can be passed as an argument: together with the closed data.","category":"page"},{"location":"lecture_03/lecture/#Implementation-of-closures-in-julia:-documentation","page":"Lecture","title":"Implementation of closures in julia: documentation","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Closure is a record storing a function together with an environment. The environment is a mapping associating each free variable of the function (variables that are used locally, but defined in an enclosing scope) with the value or reference to which the name was bound when the closure was created.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"function adder(x)\n return y->x+y\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"is lowered to (roughly):","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"struct ##1{T}\n x::T\nend\n\n(_::##1)(y) = _.x + y\n\nfunction adder(x)\n return ##1(x)\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Note that the structure ##1 is not directly accessible. Try f.x and g.x.","category":"page"},{"location":"lecture_03/lecture/#Functor-Function-like-structure","page":"Lecture","title":"Functor = Function-like structure","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Each structure can have a method that is invoked when called as a function.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"(_::Sheep)()= println(\"🐑\")","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"You can think of it as sheep.default_method().","category":"page"},{"location":"lecture_03/lecture/#Coding-style","page":"Lecture","title":"Coding style","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"From Flux.jl:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"function train!(loss, ps, data, opt; cb = () -> ())\n ps = Params(ps)\n cb = runall(cb)\n @progress for d in data\n gs = gradient(ps) do\n loss(batchmemaybe(d)...)\n end\n update!(opt, ps, gs)\n cb()\n end\nend","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Is this confusing? What can cb() do and what it can not?","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Note that function train! does not have many local variables. The important ones are arguments, i.e. exist in the scope from which the function was invoked.","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"loss(x,y)=mse(model(x),y)\ncb() = @info \"training\" loss(x,y)\ntrain!(loss, ps, data, opt; cb=cb)","category":"page"},{"location":"lecture_03/lecture/#Usage","page":"Lecture","title":"Usage","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Usage of closures:","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"callbacks: the function can also modify the enclosed variable.\nabstraction: partial evaluation ","category":"page"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"theorem: Beware: Performance of captured variables\nInference of types may be difficult in closures: https://github.com/JuliaLang/julia/issues/15276 ","category":"page"},{"location":"lecture_03/lecture/#Aditional-materials","page":"Lecture","title":"Aditional materials","text":"","category":"section"},{"location":"lecture_03/lecture/","page":"Lecture","title":"Lecture","text":"Functional desighn pattersn","category":"page"},{"location":"lecture_09/lecture/#Manipulating-Intermediate-Represenation-(IR)","page":"Lecture","title":"Manipulating Intermediate Represenation (IR)","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"using InteractiveUtils: @code_typed, @code_lowered, code_lowered","category":"page"},{"location":"lecture_09/lecture/#Generated-functions","page":"Lecture","title":"Generated functions","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Sometimes it is convenient to generate function once types of arguments are known. For example if we have function foo(args...), we can generate different body for different length of Tuple and types in args. Do we really need such thing, or it is just wish of curious programmer? Not really, as ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"we can deal with variability of args using normal control-flow logic if length(args) == 1 elseif ...\nwe can (automatically) generate (a possibly very large) set of functions foo specialized for each length of args (or combination of types of args) and let multiple dispatch to deal with this\nwe cannot deal with this situation with macros, because macros do not see types, only parsed AST, which is in this case always the same.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Generated functions allow to specialize the code for a given type of argumnets. They are like macros in the sense that they return expressions and not results. But unlike macros, the input is not expression or value of arguments, but their types (the arguments are of type Type). They are also called when compiler needs (which means at least once for each combination of arguments, but possibly more times due to code invalidation).","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Let's look at an example","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"@generated function genplus(x, y)\n println(\"generating genplus(x, y)\")\n @show (x, y, typeof(x), typeof(y))\n quote \n println(\"executing generated genplus(x, y)\")\n @show (x, y, typeof(x), typeof(y))\n x + y\n end\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"and observe the output","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"julia> genplus(1.0, 1.0) == 1.0 + 1.0\ngenerating genplus(x, y)\n(x, y, typeof(x), typeof(y)) = (Float64, Float64, DataType, DataType)\nexecuting generated genplus(x, y)\n(x, y, typeof(x), typeof(y)) = (1.0, 1.0, Float64, Float64)\ntrue\n\njulia> genplus(1.0, 1.0) == 1.0 + 1.0\nexecuting generated genplus(x, y)\n(x, y, typeof(x), typeof(y)) = (1.0, 1.0, Float64, Float64)\ntrue\n\njulia> genplus(1, 1) == 1 + 1\ngenerating genplus(x, y)\n(x, y, typeof(x), typeof(y)) = (Int64, Int64, DataType, DataType)\nexecuting generated genplus(x, y)\n(x, y, typeof(x), typeof(y)) = (1, 1, Int64, Int64)\ntrue","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"which shows that the body of genplus is called for each combination of types of parameters, but the generated code is called whenever genplus is called.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Generated functions has to be pure in the sense that they are not allowed to have side effects, for example modifying some global variables. Note that printing is not allowed in pure functions, as it modifies the global buffer. From the above example this rule does not seems to be enforced, but not obeying it can lead to unexpected errors mostly caused by not knowing when and how many times the functions will be called.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Finally, generated functions cannot call functions that has been defined after their definition.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"@generated function genplus(x, y)\n foo()\n :(x + y)\nend\n\nfoo() = println(\"foo\")\ngenplus(1,1)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Here, the applicable method is foo.","category":"page"},{"location":"lecture_09/lecture/#An-example-that-explains-everything.","page":"Lecture","title":"An example that explains everything.","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Consider a version of map applicable to NamedTuples with permuted names. Recall the behavior of normal map, which works if the names are in the same order.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"x = (a = 1, b = 2, c = 3)\ny = (a = 4, b = 5, c = 6)\nmap(+, x, y)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The same does not work with permuted names:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"x = (a = 1, b = 2, c = 3)\ny = (c = 6, b = 5, a = 4)\nmap(+, x, y)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"How to fix this? The usual approach would be to iterate over the keys in named tuples:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"function permuted_map(f, x::NamedTuple{KX}, y::NamedTuple{KY}) where {KX, KY}\n ks = tuple(intersect(KX,KY)...)\n NamedTuple{ks}(map(k -> f(x[k], y[k]), ks))\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"But, can we do better? Recall that in NamedTuples, we exactly know the position of the arguments, hence we should be able to directly match the corresponding arguments without using get. ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Since creation (and debugging) of generated functions is difficult, we start with a single-argument unrolled map.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"@generated function unrolled_map(f, x::NamedTuple{KX}) where {KX} \n vals = [:(f(getfield(x, $(QuoteNode(k))))) for k in KX]\n :(($(vals...),))\nend\nunrolled_map(e->e+1, x)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"We see that inserting a Symbol specifying the field in the NamedTuple is a bit tricky. It needs to be quoted, since $() which is needed to substitute k for its value \"peels\" one layer of the quoting. Compare this to","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"vals = [:(f(getfield(x, $(k)))) for k in KX]","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Since getting the field is awkward, we write syntactic sugar for that","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"_get(name, k) = :(getfield($(name), $(QuoteNode(k))))","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"with that, we proceed to a nicer two argument function which we have desired:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"@generated function unrolled_map(f, x::NamedTuple{KX}, y::NamedTuple{KY}) where {KX, KY} \n ks = tuple(intersect(KX,KY)...)\n _get(name, k) = :(getfield($(name), $(QuoteNode(k))))\n vals = [:(f($(_get(:x, k)), $(_get(:y, k)))) for k in ks]\n :(NamedTuple{$(ks)}(($(vals...),)))\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"We can check that the unrolled_map unrolls the map and generates just needed operations","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"@code_typed unrolled_map(+, x, y)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"and compare this to the code generated by the non-generated version permuted_map:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"@code_typed permuted_map(+, x, y)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"which is not shown here for the sake of conciseness.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"For fun, we can create a version which replaces the Symbol arguments directly by position numbers","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"@generated function unrolled_map(f, x::NamedTuple{KX}, y::NamedTuple{KY}) where {KX, KY} \n ks = tuple(intersect(KX,KY)...)\n _get(name, k, KS) = :(getfield($(name), $(findfirst(k .== KS))))\n vals = [:(f($(_get(:x, k, KX)), $(_get(:y, k, KY)))) for k in KX]\n :(NamedTuple{$(KX)}(($(vals...),)))\nend","category":"page"},{"location":"lecture_09/lecture/#Optionally-generated-functions","page":"Lecture","title":"Optionally generated functions","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Macro @generated is expanded to","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"julia> @macroexpand @generated function gentest(x)\n return :(x + x)\n end\n\n:(function gentest(x)\n if $(Expr(:generated))\n return $(Expr(:copyast, :($(QuoteNode(:(x + x))))))\n else\n $(Expr(:meta, :generated_only))\n return\n end\n end)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"which is a function with an if-condition, where the first branch $(Expr(:generated)) generates the expression :(x + x) and returns it. The other spits out an error saying that the function has only a generated version. This suggests the possibility (and reality) that one can implement two versions of the same function; A generated and a normal version. It is left up to the compiler to decide which one to use. It is entirely up to the author to ensure that both versions are the same. Which version will the compiler take? The last comment on 23168 (as of time of writing) states:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"\"Currently the @generated branch is always used. In the future, which branch is used will mostly depend on whether the JIT compiler is enabled and available, and if it's not available, then it will depend on how much we were able to compile before the compiler was taken away. So I think it will mostly be a concern for those that might need static compilation and JIT-less deployment.\"","category":"page"},{"location":"lecture_09/lecture/#Contextual-dispatch-/-overdubbing","page":"Lecture","title":"Contextual dispatch / overdubbing","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Imagine that under some circumstances (context), you would like to use alternative implementations of some functions. One of the most cited motivations for this is automatic differentiation, where you would like to take the code as-is and calculate gradients with respect to some variables. Other use cases of this approach are mentioned in Cassette.jl:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"\"Downstream applications for Cassette include dynamic code analysis (e.g. profiling, record and replay style debugging, etc.), JIT compilation to new hardware/software backends, automatic differentiation, interval constraint programming, automatic parallelization/rescheduling, automatic memoization, lightweight multistage programming, graph extraction, and more.\"","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"In theory, we can do all the above by directly modifying the code or introducing new types, but that may require a lot of coding and changing of foreign libraries.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The technique we desire is called contextual dispatch, which means that under some context, we invoke a different function. The library Casette.jl provides a high-level API for overdubbing, but it is interesting to see, how it works, as it shows, how we can \"interact\" with the lowered code before the code is typed.","category":"page"},{"location":"lecture_09/lecture/#Insertion-of-code","page":"Lecture","title":"Insertion of code","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Imagine that julia has compiled some function. For example ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"foo(x,y) = x * y + sin(x)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"and observe its lowered SSA format","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_lowered foo(1.0, 1.0)\nCodeInfo(\n1 ─ %1 = x * y\n│ %2 = Main.sin(x)\n│ %3 = %1 + %2\n└── return %3\n)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The lowered form is convenient, because on the left hand, there is always one variable and the right-hand side is simplified to have (mostly) a single call / expression. Moreover, in the lowered form, all control flow operations like if, for, while and exceptions are converted to Goto and GotoIfNot, which simplifies their handling. ","category":"page"},{"location":"lecture_09/lecture/#Codeinfo","page":"Lecture","title":"Codeinfo","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"We can access the lowered form by","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"ci = @code_lowered foo(1.0, 1.0)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"which returns an object of type CodeInfo containing many fields docs. To make the investigation slightly more interesting, we modify the function a bit to have local variables:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"function foo(x,y) \n z = x * y \n z + sin(x)\nend\n\nci = @code_lowered foo(1.0, 1.0)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The most important (and interesting) field is code:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"ci.code","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"It contains expressions corresponding to each line of the lowered form. You are free to access them (and modify them with care). Variables identified with underscore Int, for example _2, are slotted variables which are variables which have a name in the code, defined via input arguments or through an explicit assignment :(=). The names of slotted variables are stored in ci.slotnames and they are of type ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"typeof(ci.code[1].args[2].args[2])\nci.slotnames[ci.code[1].args[2].args[2].id]\nci.slotnames[ci.code[1].args[2].args[3].id]\nci.slotnames[ci.code[1].args[1].id]","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The remaining variables are identified by an integer with prefix %, where the number corresponds to the line (index in ci.code), in which the variable was created. For example the fourth line :(%2 + %3) adds the results of the second line :(_4) containing variable z and the third line :(Main.sin(_2)). The type of each slot variable is stored in slottypes, which provides some information about how the variable is used (see docs). Note that if you modify / introduce slot variables, the length of slotnames and slottypes has to match and it has to be equal to the maximum number of slotted variables.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"CodeInfo also contains information about the source code. Each item of ci.code has an identifier in ci.codelocs which is an index into ci.linetable containing Core.LineInfoNode identifying lines in the source code (or in the REPL). Notice that ci.linetable is generally shorter then ci.codelocs, as one line of source code can be translated to multiple lines in lowered code. ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The important feature of the lowered form is that we can freely edit (create new) CodeInfo and that generated functions can return a CodeInfo object instead of the AST. However, you need to explicitly write a return statement (see issue 25678).","category":"page"},{"location":"lecture_09/lecture/#Strategy-for-overdubbing","page":"Lecture","title":"Strategy for overdubbing","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"In overdubbing, our intention is to recursively dive into called function definitions and modify / change their code. In our example below, with which we will demonstrate the manual implementation (for educational purposes), our goal is to enclose each function call with statements that log the exection time. This means we would like to implement a simplified recording profiler. This functionality cannot be implemented by a macros, since macros do not allow us to dive into function definitions. For example, in our function foo, we would would not be able to dive into the definition of sin (not that this is a terribly good idea, but the point should be clear).","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The overdubbing pattern works as follows.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"We define a @generated function overdub(f, args...) which takes as a first argument a function f and then its arguments.\nIn the function overdub we retrieve the CodeInfo for f(args...), which is possible as we know types of the arguments at this time.\nWe modify the the CodeInfo of f(args...) according to our liking. Importantly, we replace all function calls some_fun(some_args...) with overdub(some_fun, some_args...) which establishes the recursive pattern.\nModify the arguments of the CodeInfo of f(args...) to match overdub(f, args..).\nReturn the modified CodeInfo.","category":"page"},{"location":"lecture_09/lecture/#The-profiler","page":"Lecture","title":"The profiler","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The implementation of the simplified logging profiler is straightforward and looks as follows.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"module LoggingProfiler\nstruct Calls\n stamps::Vector{Float64} # contains the time stamps\n event::Vector{Symbol} # name of the function that is being recorded\n startstop::Vector{Symbol} # if the time stamp corresponds to start or to stop\n i::Ref{Int}\nend\n\nfunction Calls(n::Int)\n Calls(Vector{Float64}(undef, n+1), Vector{Symbol}(undef, n+1), Vector{Symbol}(undef, n+1), Ref{Int}(0))\nend\n\nfunction Base.show(io::IO, calls::Calls)\n offset = 0\n if calls.i[] >= length(calls.stamps)\n @warn \"The recording buffer was too small, consider increasing it\"\n end\n for i in 1:min(calls.i[], length(calls.stamps))\n offset -= calls.startstop[i] == :stop\n foreach(_ -> print(io, \" \"), 1:max(offset, 0))\n rel_time = calls.stamps[i] - calls.stamps[1]\n println(io, calls.event[i], \": \", rel_time)\n offset += calls.startstop[i] == :start\n end\nend\n\nglobal const to = Calls(100)\n\n\"\"\"\n record_start(ev::Symbol)\n\n record the start of the event, the time stamp is recorded after all counters are \n appropriately increased\n\"\"\"\nrecord_start(ev::Symbol) = record_start(to, ev)\nfunction record_start(calls, ev::Symbol)\n n = calls.i[] = calls.i[] + 1\n n > length(calls.stamps) && return \n calls.event[n] = ev\n calls.startstop[n] = :start\n calls.stamps[n] = time_ns()\nend\n\n\"\"\"\n record_end(ev::Symbol)\n\n record the end of the event, the time stamp is recorded before all counters are \n appropriately increased\n\"\"\"\nrecord_end(ev::Symbol) = record_end(to, ev::Symbol)\nfunction record_end(calls, ev::Symbol)\n t = time_ns()\n n = calls.i[] = calls.i[] + 1\n n > length(calls.stamps) && return \n calls.event[n] = ev\n calls.startstop[n] = :stop\n calls.stamps[n] = t\nend\n\nreset!() = to.i[] = 0\n\nfunction Base.resize!(calls::Calls, n::Integer)\n resize!(calls.stamps, n)\n resize!(calls.event, n)\n resize!(calls.startstop, n)\nend\n\n\nexportname(ex::GlobalRef) = QuoteNode(ex.name)\nexportname(ex::Symbol) = QuoteNode(ex)\nexportname(ex::Expr) = exportname(ex.args[1])\nexportname(i::Int) = QuoteNode(Symbol(\"Int(\",i,\")\"))\n\nfunction overdubbable(ex::Expr)\n ex.head != :call && return(false)\n a = ex.args[1]\n a != GlobalRef && return(true)\n a.mod != Core\nend \noverdubbable(ex) = false \n\n\nfunction timable(ex::Expr) \n ex.head != :call && return(false)\n length(ex.args) < 2 && return(false)\n ex.args[1] isa Core.GlobalRef && return(true)\n ex.args[1] isa Symbol && return(true)\n return(false)\nend\ntimable(ex) = false\n\nexport timable, exportname, overdubbable\nend","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The important functions are report_start and report_end which mark the beggining and end of the executed function. They differ mainly when time is recorded (on the end or on the start of the function call). The profiler has a fixed capacity to prevent garbage collection, which might be increased.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Let's now describe the individual parts of overdub before presenting it in its entirety. At first, we retrieve the codeinfo ci of the overdubbed function. For now, we will just assume we obtain it for example by","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"ci = @code_lowered foo(1.0, 1.0)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"we initialize the new CodeInfo object by emptying some dummy function as ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"dummy() = return\nnew_ci = code_lowered(dummy, Tuple{})[1]\nempty!(new_ci.code)\nempty!(new_ci.slotnames)\nempty!(new_ci.linetable)\nempty!(new_ci.codelocs)\nnew_ci","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Then, we need to copy the slot variables from the ci codeinfo of foo to the new codeinfo. Additionally, we have to add the arguments of overdub(f, args...) since the compiler sees overdub(f, args...) and not foo(x,y):","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"new_ci.slotnames = vcat([Symbol(\"#self#\"), :f, :args], ci.slotnames[2:end])\nnew_ci.slotflags = vcat([0x00, 0x00, 0x00], ci.slotflags[2:end])","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Above, we also filled the slotflags. Authors admit that names :f and :args in the above should be replaced by a gensymed name, but they do not anticipate this code to be used outside of this educative example where name-clashes might occur. We also copy information about the lines from the source code:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"foreach(s -> push!(new_ci.linetable, s), ci.linetable)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The most difficult part when rewriting CodeInfo objects is working with indexes, as the line numbers and left hand side variables are strictly ordered one by one and we need to properly change the indexes to reflect changes we made. We will therefore keep three lists","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"maps = (\n ssa = Dict{Int, Int}(),\n slots = Dict{Int, Any}(),\n goto = Dict{Int,Int}(),\n)\nnothing # hide","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"where ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"slots maps slot variables in ci to those in new_ci\nssa maps indexes of left-hand side assignments in ci to new_ci\ngoto maps lines to which GotoNode and GotoIfNot point to variables in ci to new_ci (in our profiler example, we need to ensure to jump on the beggining of logging of executions)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Mapping of slots can be initialized in advance, as it is a static shift by 2 :","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"maps.slots[1] = Core.SlotNumber(1)\nforeach(i -> maps.slots[i] = Core.SlotNumber(i + 2), 2:length(ci.slotnames)) ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"and we can check the correctness by","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"@assert all(ci.slotnames[i] == new_ci.slotnames[maps.slots[i].id] for i in 1:length(ci.slotnames)) #test that ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Equipped with that, we start rewriting the code of foo(x, y). We start by a small preample, where we assign values of args... to x, and y. For the sake of simplicity, we map the slotnames to either Core.SlotNumber or to Core.SSAValues which simplifies the rewriting logic a bit.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"newci_no = 0\nargs = (Float64, Float64)\nfor i in 1:length(args)\n newci_no +=1\n push!(new_ci.code, Expr(:call, Base.getindex, Core.SlotNumber(3), i))\n maps.slots[i+1] = Core.SSAValue(newci_no)\n push!(new_ci.codelocs, ci.codelocs[1])\nend","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Now we come to the pinnacle of rewriting the body of foo(x,y) while inserting calls to the profiler:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"for (ci_no, ex) in enumerate(ci.code)\n if timable(ex)\n fname = exportname(ex)\n push!(new_ci.code, Expr(:call, GlobalRef(LoggingProfiler, :record_start), fname))\n push!(new_ci.codelocs, ci.codelocs[ci_no])\n newci_no += 1\n maps.goto[ci_no] = newci_no\n ex = overdubbable(ex) ? Expr(:call, GlobalRef(Main, :overdub), ex.args...) : ex\n push!(new_ci.code, ex)\n push!(new_ci.codelocs, ci.codelocs[ci_no])\n newci_no += 1\n maps.ssa[ci_no] = newci_no\n push!(new_ci.code, Expr(:call, GlobalRef(LoggingProfiler, :record_end), fname))\n push!(new_ci.codelocs, ci.codelocs[ci_no])\n newci_no += 1\n else\n push!(new_ci.code, ex)\n push!(new_ci.codelocs, ci.codelocs[ci_no])\n newci_no += 1\n maps.ssa[ci_no] = newci_no\n end\nend","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"which yields","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"julia> new_ci.code\n15-element Vector{Any}:\n :((getindex)(_3, 1))\n :((getindex)(_3, 2))\n :(_4 = _2 * _3)\n :(_4)\n :(Main.LoggingProfiler.record_start(:sin))\n :(Main.overdub(Main.sin, _2))\n :(Main.LoggingProfiler.record_end(:sin))\n :(Main.LoggingProfiler.record_start(:+))\n :(Main.overdub(Main.:+, %2, %3))\n :(Main.LoggingProfiler.record_end(:+))\n :(return %4)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The important parts are:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Depending on the type of expressions (controlled by timable) we decide, if a function's execution time should be recorded.\nfname = exportname(ex) obtains the name of the profiled function call.\npush!(new_ci.code, Expr(:call, GlobalRef(LoggingProfiler, :record_start), fname)) records the start of the exection.\nmaps.goto[ci_ssa_no] = ssa_no updates the map from the code line number in ci to the one in new_ci.\nmaps.ssa[ci_ssa_no] = ssa_no updates the map from the SSA line number in ci to new_ci.\nex = overdubbable(ex) ? Expr(:call, GlobalRef(Main, :overdub), ex.args...) : ex modifies the function call (expression in general) to recurse the overdubbing.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Finally, we need to change the names of slot variables (Core.SlotNumber) and variables indexed by the SSA (Core.SSAValue).","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"for i in length(args)+1:length(new_ci.code)\n new_ci.code[i] = remap(new_ci.code[i], maps)\nend","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"where remap is defined by the following block of code","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"remap(ex::Expr, maps) = Expr(ex.head, remap(ex.args, maps)...)\nremap(args::AbstractArray, maps) = map(a -> remap(a, maps), args)\nremap(c::Core.GotoNode, maps) = Core.GotoNode(maps.goto[c.label])\nremap(c::Core.GotoIfNot, maps) = Core.GotoIfNot(remap(c.cond, maps), maps.goto[c.dest])\nremap(r::Core.ReturnNode, maps) = Core.ReturnNode(remap(r.val, maps))\nremap(a::Core.SlotNumber, maps) = maps.slots[a.id]\nremap(a::Core.SSAValue, maps) = Core.SSAValue(maps.ssa[a.id])\nremap(a::Core.NewvarNode, maps) = Core.NewvarNode(maps.slots[a.slot.id])\nremap(a::GlobalRef, maps) = a\nremap(a::QuoteNode, maps) = a\nremap(ex, maps) = ex","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"warn: Warn\nRetrieving the code properlyConsider the following function:function test(x::T) where T<:Union{Float64, Float32}\n x < T(pi)\nend\n\njulia> ci = @code_lowered test(1.0)\nCodeInfo(\n1 ─ %1 = ($(Expr(:static_parameter, 1)))(Main.pi)\n│ %2 = x < %1\n└── return %2\n)the Expr(:static_parameter, 1) in the first line of code obtains the type parameter T of the function test. Since this information is not accessible in the CodeInfo, it might render our tooling useless. The needed hook is Base.Meta.partially_inline! which partially inlines this into the CodeInfo object. The code to retrieve the CodeInfo adapted from IRTools is a little involved:function retrieve_code_info(sigtypes, world = Base.get_world_counter())\n S = Tuple{map(s -> Core.Compiler.has_free_typevars(s) ? typeof(s.parameters[1]) : s, sigtypes)...}\n _methods = Base._methods_by_ftype(S, -1, world)\n if isempty(_methods) \n @info(\"method $(sigtypes) does not exist\")\n return(nothing)\n end\n type_signature, raw_static_params, method = _methods[1]\n mi = Core.Compiler.specialize_method(method, type_signature, raw_static_params, false)\n ci = Base.isgenerated(mi) ? Core.Compiler.get_staged(mi) : Base.uncompressed_ast(method)\n Base.Meta.partially_inline!(ci.code, [], method.sig, Any[raw_static_params...], 0, 0, :propagate)\n ci\nendbutjulia> ci = retrieve_code_info((typeof(test), Float64))\nCodeInfo(\n @ REPL[5]:2 within `test'\n1 ─ %1 = ($(QuoteNode(Float64)))(Main.pi)\n│ %2 = x < %1\n└── return %2\n)it performs the needed inlining of Float64.","category":"page"},{"location":"lecture_09/lecture/#Implementing-the-profiler-with-IRTools","page":"Lecture","title":"Implementing the profiler with IRTools","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The above implementation of the profiler has shown, that rewriting IR manually is doable, but requires a lot of careful book-keeping. IRTools.jl makes our life much simpler, as they take away all the needed book-keeping and let us focus on what is important.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"using IRTools\nfunction foo(x, y)\n z = x * y\n z + sin(y)\nend;\nir = @code_ir foo(1.0, 1.0)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"We can see that at first sight, the representation of the lowered code in IRTools is similar to that of CodeInfo. Some notable differences:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"SlotNumber are converted to SSAValues\nSSA form is divided into blocks by GotoNode and GotoIfNot in the parsed CodeInfo\nSSAValues do not need to be ordered. The reordering is deffered to the moment when one converts IRTools.Inner.IR back to the CodeInfo.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Let's now use the IRTools to insert the timing statements into the code for foo:","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"using IRTools: xcall, insert!, insertafter!\n\nir = @code_ir foo(1.0, 1.0)\nfor (v, ex) in ir\n if timable(ex.expr)\n fname = exportname(ex.expr)\n insert!(ir, v, xcall(LoggingProfiler, :record_start, fname))\n insertafter!(ir, v, xcall(LoggingProfiler, :record_end, fname))\n end\nend\n\njulia> ir\n1: (%1, %2, %3)\n %7 = Main.LoggingProfiler.record_start(:*)\n %4 = %2 * %3\n %8 = Main.LoggingProfiler.record_end(:*)\n %9 = Main.LoggingProfiler.record_start(:sin)\n %5 = Main.sin(%3)\n %10 = Main.LoggingProfiler.record_end(:sin)\n %11 = Main.LoggingProfiler.record_start(:+)\n %6 = %4 + %5\n %12 = Main.LoggingProfiler.record_end(:+)\n return %6","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Observe that the statements are on the right places but they are not ordered. We can turn the ir object into an anonymous function","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"f = IRTools.func(ir)\nLoggingProfiler.reset!()\nf(nothing, 1.0, 1.0)\nLoggingProfiler.to","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"where we can observe that our profiler is working as it should. But this is not yet our final goal. Originally, our goal was to recursivelly dive into the nested functions. IRTools offers a macro @dynamo, which is similar to @generated but simplifies our job by allowing to return the IRTools.Inner.IR object and it also taking care of properly renaming the arguments. With that we write","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"using IRTools: @dynamo\nprofile_fun(f::Core.IntrinsicFunction, args...) = f(args...)\nprofile_fun(f::Core.Builtin, args...) = f(args...)\n\n@dynamo function profile_fun(f, args...)\n ir = IRTools.Inner.IR(f, args...)\n for (v, ex) in ir\n if timable(ex.expr)\n fname = exportname(ex.expr)\n insert!(ir, v, xcall(LoggingProfiler, :record_start, fname))\n insertafter!(ir, v, xcall(LoggingProfiler, :record_end, fname))\n end\n end\n for (x, st) in ir\n recursable(st.expr) || continue\n ir[x] = xcall(profile_fun, st.expr.args...)\n end\n return ir\nend","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"where the first pass is as it was above and the ir[x] = xcall(profile_fun, st.expr.args...) ensures that the profiler will recursively call itself. recursable is a filter defined as below, which is used to prevent profiling itself (and possibly other things).","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"recursable(gr::GlobalRef) = gr.name ∉ [:profile_fun, :record_start, :record_end]\nrecursable(ex::Expr) = ex.head == :call && recursable(ex.args[1])\nrecursable(ex) = false","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Additionally, the first two definitions of profile_fun for Core.IntrinsicFunction and for Core.Builtin prevent trying to dive into functions which do not have a Julia IR. And that's all. The full code is ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"using IRTools\nusing IRTools: var, xcall, insert!, insertafter!, func, recurse!, @dynamo\ninclude(\"loggingprofiler.jl\")\nLoggingProfiler.resize!(LoggingProfiler.to, 10000)\n\nfunction timable(ex::Expr) \n ex.head != :call && return(false)\n length(ex.args) < 2 && return(false)\n ex.args[1] isa Core.GlobalRef && return(true)\n ex.args[1] isa Symbol && return(true)\n return(false)\nend\ntimable(ex) = false\n\nfunction recursable_fun(ex::GlobalRef)\n ex.name ∈ (:profile_fun, :record_start, :record_end) && return(false)\n iswhite(recursable_list, ex) && return(true)\n isblack(recursable_list, ex) && return(false)\n return(isempty(recursable_list) ? true : false)\nend\n\nrecursable_fun(ex::IRTools.Inner.Variable) = true\n\nfunction recursable(ex::Expr) \n ex.head != :call && return(false)\n isempty(ex.args) && return(false)\n recursable(ex.args[1])\nend\n\nrecursable(ex) = false\n\nexportname(ex::GlobalRef) = QuoteNode(ex.name)\nexportname(ex::Symbol) = QuoteNode(ex)\nexportname(ex::Expr) = exportname(ex.args[1])\nexportname(i::Int) = QuoteNode(Symbol(\"Int(\",i,\")\"))\n\nprofile_fun(f::Core.IntrinsicFunction, args...) = f(args...)\nprofile_fun(f::Core.Builtin, args...) = f(args...)\n\n@dynamo function profile_fun(f, args...)\n ir = IRTools.Inner.IR(f, args...)\n for (v, ex) in ir\n if timable(ex.expr)\n fname = exportname(ex.expr)\n insert!(ir, v, xcall(LoggingProfiler, :record_start, fname))\n insertafter!(ir, v, xcall(LoggingProfiler, :record_end, fname))\n end\n end\n for (x, st) in ir\n recursable(st.expr) || continue\n ir[x] = xcall(profile_fun, st.expr.args...)\n end\n # recurse!(ir)\n return ir\nend\n\nmacro record(ex)\n esc(Expr(:call, :profile_fun, ex.args...))\nend\n\nLoggingProfiler.reset!()\n@record foo(1.0, 1.0)\nLoggingProfiler.to","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"where you should notice the long time the first execution of @record foo(1.0, 1.0) takes. This is caused by the compiler specializing for every function into which we dive into. The second execution of @record foo(1.0, 1.0) is fast. It is also interesting to observe how the time of the compilation is logged by the profiler. The output of the profiler to is not shown here due to the length of the output.","category":"page"},{"location":"lecture_09/lecture/#Petite-Zygote","page":"Lecture","title":"Petite Zygote","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"IRTools.jl were created for Zygote.jl –- Julia's source-to-source AD system currently powering Flux.jl. An interesting aspect of Zygote was to recognize that TensorFlow is in its nutshell a compiler, PyTorch is an interpreter. So the idea was to let Julia's compiler compile the gradient and perform optimizations that are normally performed with normal code. Recall that a lot of research went into how to generate efficient code and it is reasonable to use this research. Zygote.jl provides mainly reversediff, but there was an experimental support for forwarddiff.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"One of the questions when developing an AD engine is where and how to create a computation graph. Recall that in TensorFlow, you specify it through a domain specific language, in PyTorch it generated on the fly. Mike Innes' idea was use SSA form provided by the julia compiler. ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_lowered foo(1.0, 1.0)\nCodeInfo(\n1 ─ z = x * y\n│ %2 = z\n│ %3 = Main.sin(y)\n│ %4 = %2 + %3\n└── return %4\n)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"It is very easy to differentiate each line, as they correspond to single expressions (or function calls) and importantly, each variable is assigned exactly once. The strategy to use it for AD would as follows.","category":"page"},{"location":"lecture_09/lecture/#Strategy","page":"Lecture","title":"Strategy","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"We assume to have a set of AD rules (e.g. ChainRules), which for a given function returns its evaluation and pullback. If Zygote.jl is tasked with computing the gradient.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"If a rule exists for this function, directly return the rule.\nIf not, deconstruct the function into a sequence of functions using CodeInfo / IR representation\nReplace statements by calls to obtain the evaluation of the statements and the pullback.\nChain pullbacks in reverse order.\nReturn the function evaluation and the chained pullback.","category":"page"},{"location":"lecture_09/lecture/#Simplified-implementation","page":"Lecture","title":"Simplified implementation","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The following code is adapted from this example","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"using IRTools, ChainRules\nusing IRTools: @dynamo, IR, Pipe, finish, substitute, return!, block, blocks,\n returnvalue, arguments, isexpr, xcall, self, stmt\n\nstruct Pullback{S,T}\n data::T\nend\n\nPullback{S}(data) where S = Pullback{S,typeof(data)}(data)\n\nfunction primal(ir, T = Any)\n pr = Pipe(ir)\n calls = []\n ret = []\n for (v, st) in pr\n ex = st.expr\n if isexpr(ex, :call)\n t = insert!(pr, v, stmt(xcall(Main, :forward, ex.args...), line = st.line))\n pr[v] = xcall(:getindex, t, 1)\n J = push!(pr, xcall(:getindex, t, 2))\n push!(calls, v)\n push!(ret, J)\n end\n end\n pb = Expr(:call, Pullback{T}, xcall(:tuple, ret...))\n return!(pr, xcall(:tuple, returnvalue(block(ir, 1)), pb))\n return finish(pr), calls\nend\n\n@dynamo function forward(m...)\n ir = IR(m...)\n ir == nothing && return :(error(\"Non-differentiable function \", repr(args[1])))\n length(blocks(ir)) == 1 || error(\"control flow is not supported\")\n return primal(ir, Tuple{m...})[1]\nend\n","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"where ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"the generated function forward calls primal to perform AD manual chainrule\nactual chainrule is performed in the for loop\nevery function call is replaced xcall(Main, :forward, ex.args...), which is the recursion we have observed above. stmt allows to insert information about lines in the source code).\nthe output of the forward is the value of the function, and pullback, the function calculating gradient with respect to its inputs.\npr[v] = xcall(:getindex, t, 1) fixes the output of the overwritten function call to be the output of forward(...)\nthe next line logs the pullback \nExpr(:call, Pullback{T}, xcall(:tuple, ret...)) will serve to call generated function which will assemble the pullback in the right order","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Let's now observe how the the IR of foo is transformed","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"ir = IR(typeof(foo), Float64, Float64)\njulia> primal(ir)[1]\n1: (%1, %2, %3)\n %4 = Main.forward(Main.:*, %2, %3)\n %5 = Base.getindex(%4, 1)\n %6 = Base.getindex(%4, 2)\n %7 = Main.forward(Main.sin, %3)\n %8 = Base.getindex(%7, 1)\n %9 = Base.getindex(%7, 2)\n %10 = Main.forward(Main.:+, %5, %8)\n %11 = Base.getindex(%10, 1)\n %12 = Base.getindex(%10, 2)\n %13 = Base.tuple(%6, %9, %12)\n %14 = (Pullback{Any, T} where T)(%13)\n %15 = Base.tuple(%11, %14)\n return %15","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Every function call was transformed into the sequence of forward(...) and obtaining first and second item from the returned typle.\nLine %14 constructs the Pullback, which (as will be seen shortly below) will allow to generate the pullback for the generated function\nLine %15 generates the returned tuple, where the first item is the function value (computed at line %11) and pullback (constructed at libe %15).","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"We define few AD rules by specializing forward with calls from ChainRules","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"forward(::typeof(sin), x) = ChainRules.rrule(sin, x)\nforward(::typeof(*), x, y) = ChainRules.rrule(*, x, y)\nforward(::typeof(+), x, y) = ChainRules.rrule(+, x, y)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Zygote implements this inside the generated function, such that whatever is added to ChainRules is automatically reflected. The process is not as trivial (see has_chain_rule) and for the brevity is not shown here. ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"We now obtain the value and the pullback of function foo as ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"julia> v, pb = forward(foo, 1.0, 1.0);","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The pullback contains in data field with individual jacobians that have been collected in ret in primal function.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"pb.data[1]\npb.data[2]\npb.data[3]","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The function for which the Jacobian has been created is stored in type parameter S of the Pullback type. The pullback for foo is generated in another generated function, as Pullback struct is a functor. This is an interesting design pattern, which allows us to return closure from a generated function. ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Let's now investigate the code generating code for pullback.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"\n_sum() = 0\n_sum(x) = x\n_sum(x...) = xcall(:+, x...)\n\nfunction pullback(pr)\n ir = empty(pr)\n grads = Dict()\n grad(x) = _sum(get(grads, x, [])...)\n grad(x, x̄) = push!(get!(grads, x, []), x̄)\n grad(returnvalue(block(pr, 1)), IRTools.argument!(ir))\n data = push!(ir, xcall(:getfield, self, QuoteNode(:data)))\n _, pbs = primal(pr)\n pbs = Dict(pbs[i] => push!(ir, xcall(:getindex, data, i)) for i = 1:length(pbs))\n for v in reverse(keys(pr))\n ex = pr[v].expr\n isexpr(ex, :call) || continue\n Δs = push!(ir, Expr(:call, pbs[v], grad(v)))\n for (i, x) in enumerate(ex.args)\n grad(x, push!(ir, xcall(:getindex, Δs, i)))\n end\n end\n return!(ir, xcall(:tuple, [grad(x) for x in arguments(pr)]...))\nend\n\n@dynamo function (pb::Pullback{S})(Δ) where S\n return pullback(IR(S.parameters...))\nend","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"Let's walk how the reverse is constructed for pr = IR(typeof(foo), Float64, Float64)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"ir = empty(pr)\ngrads = Dict()\ngrad(x) = _sum(get(grads, x, [])...)\ngrad(x, x̄) = push!(get!(grads, x, []), x̄)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"construct the empty ir for the constructed pullback, defines Dict where individual contributors of the gradient with respect to certain variable will be stored, and two function for pushing statements to to grads. The next statement","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"grad(returnvalue(block(pr, 1)), IRTools.argument!(ir))","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"pushes to grads statement that the gradient of the output of the primal pr is provided as an argument of the pullback IRTools.argument!(ir). ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"data = push!(ir, xcall(:getfield, self, QuoteNode(:data)))\n_, pbs = primal(pr)\npbs = Dict(pbs[i] => push!(ir, xcall(:getindex, data, i)) for i = 1:length(pbs))","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"sets data to the data field of the Pullback structure containing pullback functions. Then it create a dictionary pbs, where the output of each call in the primal (identified by the line) is mapped to the corresponding pullback, which is now a line in the IR representation. The IR so far looks as ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"1: (%1)\n %2 = Base.getfield(IRTools.Inner.Self(), :data)\n %3 = Base.getindex(%2, 1)\n %4 = Base.getindex(%2, 2)\n %5 = Base.getindex(%2, 3)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"and pbs contains ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"julia> pbs\nDict{IRTools.Inner.Variable, IRTools.Inner.Variable} with 3 entries:\n %6 => %5\n %4 => %3\n %5 => %4","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"says that the pullback of a function producing variable at line %6 in the primal is stored at variable %5 in the contructed pullback. The real deal comes in the for loop ","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"for v in reverse(keys(pr))\n ex = pr[v].expr\n isexpr(ex, :call) || continue\n Δs = push!(ir, Expr(:call, pbs[v], grad(v)))\n for (i, x) in enumerate(ex.args)\n grad(x, push!(ir, xcall(:getindex, Δs, i)))\n end\nend","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"which iterates the primal pr in the reverse order and for every call, it inserts statement to calls the appropriate pullback Δs = push!(ir, Expr(:call, pbs[v], grad(v))) and adds gradients with respect to the inputs to values accumulating corresponding gradient in the loop for (i, x) in enumerate(ex.args) ... The last line","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"return!(ir, xcall(:tuple, [grad(x) for x in arguments(pr)]...))","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"puts statements accumulating gradients with respect to individual variables to the ir.","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The final generated IR code looks as","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"julia> pullback(IR(typeof(foo), Float64, Float64))\n1: (%1)\n %2 = Base.getfield(IRTools.Inner.Self(), :data)\n %3 = Base.getindex(%2, 1)\n %4 = Base.getindex(%2, 2)\n %5 = Base.getindex(%2, 3)\n %6 = (%5)(%1)\n %7 = Base.getindex(%6, 1)\n %8 = Base.getindex(%6, 2)\n %9 = Base.getindex(%6, 3)\n %10 = (%4)(%9)\n %11 = Base.getindex(%10, 1)\n %12 = Base.getindex(%10, 2)\n %13 = (%3)(%8)\n %14 = Base.getindex(%13, 1)\n %15 = Base.getindex(%13, 2)\n %16 = Base.getindex(%13, 3)\n %17 = %12 + %16\n %18 = Base.tuple(0, %15, %17)\n return %18","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"and it calculates the gradient with respect to the input as","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"julia> pb(1.0)\n(0, 1.0, 1.5403023058681398)","category":"page"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"where the first item is gradient with parameters of the function itself.","category":"page"},{"location":"lecture_09/lecture/#Conclusion","page":"Lecture","title":"Conclusion","text":"","category":"section"},{"location":"lecture_09/lecture/","page":"Lecture","title":"Lecture","text":"The above examples served to demonstrate that @generated functions offers extremely powerful paradigm, especially if coupled with manipulation of intermediate representation. Within few lines of code, we have implemented reasonably powerful profiler and reverse AD engine. Importantly, it has been done without a single-purpose engine or tooling. ","category":"page"},{"location":"lecture_10/hw/#hw09","page":"Homework","title":"Homework 9: Accelerating 1D convolution with threads","text":"","category":"section"},{"location":"lecture_10/hw/#How-to-submit","page":"Homework","title":"How to submit","text":"","category":"section"},{"location":"lecture_10/hw/","page":"Homework","title":"Homework","text":"Put all the code of inside hw.jl. Zip only this file (not its parent folder) and upload it to BRUTE. You should not not import anything but Base.Threads or just Threads. ","category":"page"},{"location":"lecture_10/hw/","page":"Homework","title":"Homework","text":"
\n
Homework (2 points)
\n
","category":"page"},{"location":"lecture_10/hw/","page":"Homework","title":"Homework","text":"Implement multithreaded discrete 1D convolution operator[1] without padding (output will be shorter). The required function signature: thread_conv1d(x, w), where x is the signal array and w the kernel. For testing correctness of the implementation you can use the following example of a step function and it's derivative realized by kernel [-1, 1]:","category":"page"},{"location":"lecture_10/hw/","page":"Homework","title":"Homework","text":"using Test\n@test all(thread_conv1d(vcat([0.0, 0.0, 1.0, 1.0, 0.0, 0.0]), [-1.0, 1.0]) .≈ [0.0, -1.0, 0.0, 1.0, 0.0])","category":"page"},{"location":"lecture_10/hw/","page":"Homework","title":"Homework","text":"[1]: Discrete convolution with finite support https://en.wikipedia.org/wiki/Convolution#Discrete_convolution","category":"page"},{"location":"lecture_10/hw/","page":"Homework","title":"Homework","text":"Your parallel implementation will be tested both in sequential and two threaded mode with the following inputs","category":"page"},{"location":"lecture_10/hw/","page":"Homework","title":"Homework","text":"using Random\nRandom.seed!(42)\nx = rand(10_000_000)\nw = [1.0, 2.0, 4.0, 2.0, 1.0]\n@btime thread_conv1d($x, $w);","category":"page"},{"location":"lecture_10/hw/","page":"Homework","title":"Homework","text":"On your local machine you should be able to achieve 0.6x reduction in execution time with two threads, however the automatic eval system is a noisy environment and therefore we require only 0.8x reduction therein. This being said, please reach out to us, if you encounter any issues.","category":"page"},{"location":"lecture_10/hw/","page":"Homework","title":"Homework","text":"HINTS:","category":"page"},{"location":"lecture_10/hw/","page":"Homework","title":"Homework","text":"start with single threaded implementation\ndon't forget to reverse the kernel\n@threads macro should be all you need\nfor testing purposes create a simple script, that you can run with julia -t 1 and julia -t 2","category":"page"},{"location":"lecture_10/hw/","page":"Homework","title":"Homework","text":"
\n","category":"page"},{"location":"lecture_07/hw/#hw07","page":"Homework","title":"Homework 7: Creating world in 3 days/steps","text":"","category":"section"},{"location":"lecture_07/hw/#How-to-submit","page":"Homework","title":"How to submit","text":"","category":"section"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"Put all the code of inside hw.jl. Zip only this file (not its parent folder) and upload it to BRUTE. You should assume that only","category":"page"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"using Ecosystem","category":"page"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"will be put before your file is executed, but do not include them in your solution. The version of Ecosystem pkg should be the same as in HW4.","category":"page"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"
\n
Homework (2 points)
\n
","category":"page"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"Create a macro @ecosystem that should be able to define a world given a list of statements @add # $species ${optional:sex}","category":"page"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"world = @ecosystem begin\n @add 10 Sheep female # adds 10 female sheep\n @add 2 Sheep male # adds 2 male sheep\n @add 100 Grass # adds 100 pieces of grass\n @add 3 Wolf # adds 5 wolf with random sex\nend","category":"page"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"@add should not be treated as a macro, but rather just as a syntax, that can be easily matched.","category":"page"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"As this is not a small task let's break it into 3 steps. (These intemediate steps will also be checked in BRUTE.)","category":"page"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"Define method default_config(::Type{T}) for each T in Grass, Wolf,..., which returns a named tuple of default parameters for that particular agent (you can choose the default values however you like).\nDefine method _add_agents(max_id, count::Int, species::Type{<:Species}) and _add_agents(max_id, count::Int, species::Type{<:AnimalSpecies}, sex::Sex) that return an array of count agents of species species with id going from max_id+1 to max_id+count. Default parameters should be constructed with default_config. Make sure you can handle even animals with random sex (@add 3 Wolf).\nDefine the underlying function _ecosystem(ex), which parses the block expression and creates a piece of code that constructs the world.","category":"page"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"You can test the macro (more precisely the _ecosystem function) with the following expression","category":"page"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"ex = :(begin\n @add 10 Sheep female\n @add 2 Sheep male\n @add 100 Grass\n @add 3 Wolf\nend)\ngenex = _ecosystem(ex)\nworld = eval(genex)","category":"page"},{"location":"lecture_07/hw/","page":"Homework","title":"Homework","text":"
\n","category":"page"},{"location":"lecture_10/lab/#parallel_lab","page":"Lab","title":"Lab 10: Parallel computing","text":"","category":"section"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"In this lab we are going to introduce tools that Julia's ecosystem offers for different ways of parallel computing. As an ilustration for how capable Julia was/is consider the fact that it has joined (alongside C,C++ and Fortran) the so-called \"PetaFlop club\"[1], a list of languages capable of running at over 1PFLOPS.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"[1]: Blogpost \"Julia Joins Petaflop Club\" https://juliacomputing.com/media/2017/09/julia-joins-petaflop-club/","category":"page"},{"location":"lecture_10/lab/#Introduction","page":"Lab","title":"Introduction","text":"","category":"section"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Nowadays there is no need to convince anyone about the advantages of having more cores available for your computation be it on a laptop, workstation or a cluster. The trend can be nicely illustrated in the figure bellow: (Image: 42-cpu-trend) Image source[2]","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"[2]: Performance metrics trend of CPUs in the last 42years: https://www.karlrupp.net/2018/02/42-years-of-microprocessor-trend-data/","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"However there are some shortcomings when going from sequential programming, that we have to note","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"We don't think in parallel\nWe learn to write and reason about programs serially\nThe desire for parallelism often comes after you've written your algorithm (and found it too slow!)\nHarder to reason and therefore harder to debug\nThe number of cores is increasing, thus knowing how the program scales is crucial (not just that it runs better)\nBenchmarking parallel code, that tries to exhaust the processor pool is much more affected by background processes","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"warning: Shortcomings of parallelism\nParallel computing brings its own set of problems and not an insignificant overhead with data manipulation and communication, therefore try always to optimize your serial code as much as you can before advancing to parallel acceleration.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"warning: Disclaimer\nWith the increasing complexity of computer HW some statements may become outdated. Moreover we won't cover as many tips that you may encounter on a parallel programming specific course, which will teach you more in the direction of how to think in parallel, whereas here we will focus on the tools that you can use to realize the knowledge gained therein.","category":"page"},{"location":"lecture_10/lab/#Process-based-parallelism","page":"Lab","title":"Process based parallelism","text":"","category":"section"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"As the name suggest process based parallelism is builds on the concept of running code on multiple processes, which can run even on multiple machines thus allowing to scale computing from a local machine to a whole network of machines - a major difference from the other parallel concept of threads. In Julia this concept is supported within standard library Distributed and the scaling to cluster can be realized by 3rd party library ClusterManagers.jl.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Let's start simply with knowing how to start up additional Julia processes. There are two ways:","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"by adding processes using cmd line argument -p ##","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"julia -p 4","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"by adding processes after startup using the addprocs(##) function from std library Distributed","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"julia> using Distributed\njulia> addprocs(4) # returns a list of ids of individual processes\n4-element Vector{Int64}:\n 2\n 3\n 4\n 5\njulia> nworkers() # returns number of workers\n4\njulia> nprocs() # returns number of processes `nworkers() + 1`\n5","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"The result shown in a process manager such as htop:","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":".../julia-1.6.2/bin/julia --project \n.../julia-1.6.2/bin/julia -Cnative -J/home/honza/Apps/julia-1.6.2/lib/julia/sys.so -g1 --bind-to 127.0.0.1 --worker\n.../julia-1.6.2/bin/julia -Cnative -J/home/honza/Apps/julia-1.6.2/lib/julia/sys.so -g1 --bind-to 127.0.0.1 --worker\n.../julia-1.6.2/bin/julia -Cnative -J/home/honza/Apps/julia-1.6.2/lib/julia/sys.so -g1 --bind-to 127.0.0.1 --worker\n.../julia-1.6.2/bin/julia -Cnative -J/home/honza/Apps/julia-1.6.2/lib/julia/sys.so -g1 --bind-to 127.0.0.1 --worker","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Both of these result in total of 5 running processes - 1 controller, 4 workers - with their respective ids accessible via myid() function call. Note that the controller process has always id 1 and other processes are assigned subsequent integers, see for yourself with @everywhere macro, which runs easily code on all or a subset of processes.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"@everywhere println(myid())\n@everywhere [2,3] println(myid()) # select a subset of workers","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"The same way that we have added processes we can also remove them","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"julia> workers() # returns array of worker ids\n4-element Vector{Int64}:\n 2\n 3\n 4\n 5\njulia> rmprocs(2) # kills worker with id 2\nTask (done) @0x00007ff2d66a5e40\njulia> workers()\n3-element Vector{Int64}:\n 3\n 4\n 5","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"As we have seen from the htop/top output, added processes start with specific cmd line arguments, however they are not shared with any aliases that we may have defined, e.g. julia ~ julia --project=.. Therefore in order to use an environment, we have to first activate it on all processes","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"@everywhere begin\n using Pkg; Pkg.activate(@__DIR__) # @__DIR__ equivalent to a call to pwd()\nend","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"or we can load files containing this line on all processes with cmdline option -L ###.jl together with -p ##.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"There are generally two ways of working with multiple processes","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"using low level functionality - we specify what/where is loaded, what/where is being run and when we fetch results\n@everywhere to run everywhere and wait for completion\n@spawnat and remotecall to run at specific process and return Future (a reference to a future result - remote reference)\nfetch - fetching remote reference\npmap - for easily mapping a function over a collection\nusing high level functionality - define only simple functions and apply them on collections\nDistributedArrays' with DArrays\nTransducers.jl pipelines\nDagger.jl out-of-core and parallel computing","category":"page"},{"location":"lecture_10/lab/#Sum-with-processes","page":"Lab","title":"Sum with processes","text":"","category":"section"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Writing your own sum of an array function is a good way to show all the potential problems, you may encounter with parallel programming. For comparison here is the naive version that uses zero for initialization and @inbounds for removing boundschecks.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"function naive_sum(a)\n r = zero(eltype(a))\n for aᵢ in a\n r += aᵢ\n end\n r\nend","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Its performance will serve us as a sequential baseline.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"julia> using BenchmarkTools\njulia> a = rand(10_000_000); # 10^7\njulia> sum(a) ≈ naive_sum(a)\ntrue\njulia> @btime sum($a)\n5.011 ms (0 allocations: 0 bytes)\njulia> @btime naive_sum($a)\n11.786 ms (0 allocations: 0 bytes)","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Note that the built-in sum exploits single core parallelism with Single instruction, multiple data (SIMD instructions) and is thus faster.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Write a distributed/multiprocessing version of sum function dist_sum(a, np=nworkers()) without the help of DistributedArrays. Measure the speed up when doubling the number of workers (up to the number of logical cores - see note on hyper threading).","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"map builtin sum over chunks of the array using pmap\nthere are built in partition iterators Iterators.partition(array, chunk_size)\nchunk_size should relate to the number of available workers\npmap has the option to pass the ids of workers as the second argument pmap(f, WorkerPool([2,4]), collection)\npmap collects the partial results to the controller where it can be collected with another sum","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"using Distributed\naddprocs(4)\n\n@everywhere begin\n using Pkg; Pkg.activate(@__DIR__)\nend\n\nfunction dist_sum(a, np=nworkers())\n chunk_size = div(length(a), np)\n sum(pmap(sum, WorkerPool(workers()[1:np]), Iterators.partition(a, chunk_size)))\nend\n\ndist_sum(a) ≈ sum(a)\n@btime dist_sum($a)\n\n@time dist_sum(a, 1) # 74ms \n@time dist_sum(a, 2) # 46ms\n@time dist_sum(a, 4) # 49ms\n@time dist_sum(a, 8) # 35ms","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"As you can see the built-in pmap already abstracts quite a lot from the process and all the data movement is handled internally, however in order to show off how we can abstract even more, let's use the DistributedArrays.jl pkg.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Write a distributed/multiprocessing version of sum function dist_sum_lib(a, np=nworkers()) with the help of DistributedArrays. Measure the speed up when doubling the number of workers (up to the number of logical cores - see note on hyper threading).","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"chunking and distributing the data can be handled for us using the distribute function on an array (creates a DArray)\ndistribute has an option to specify on which workers should an array be distributed to\nsum function has a method for DArray\nremember to run using DistributedArrays on every process","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Setting up.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"using Distributed\naddprocs(8)\n\n@everywhere begin\n using Pkg; Pkg.activate(@__DIR__)\nend\n\n@everywhere begin\n using DistributedArrays\nend ","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"And the actual computation.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"adist = distribute(a) # distribute array to workers |> typeof - DArray\n@time adist = distribute(a) # we should not disregard this time\n@btime sum($adist) # call the built-in function (dispatch on DArrray)\n\nfunction dist_sum_lib(a, np=nworkers())\n adist = distribute(a, procs = workers()[1:np])\n sum(adist)\nend\n\ndist_sum_lib(a) ≈ sum(a)\n@btime dist_sum_lib($a)\n\n@time dist_sum_lib(a, 1) # 80ms \n@time dist_sum_lib(a, 2) # 54ms\n@time dist_sum_lib(a, 4) # 48ms\n@time dist_sum_lib(a, 8) # 33ms","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"In both previous examples we have included the data transfer time from the controller process, in practice however distributed computing is used in situations where the data may be stored on individual local machines. As a general rule of thumb we should always send only instruction what to do and not the actual data to be processed. This will be more clearly demonstrated in the next more practical example.","category":"page"},{"location":"lecture_10/lab/#lab10_dist_file_p","page":"Lab","title":"Distributed file processing","text":"","category":"section"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Distributed is often used in processing of files, such as the commonly encountered mapreduce jobs with technologies like Hadoop, Spark, where the files live on a distributed file system and a typical job requires us to map over all the files and gather some statistics such as histograms, sums and others. We will simulate this situation with the Julia's pkg codebase, which on a typical user installation can contain up to hundreds of thousand of .jl files (depending on how extensively one uses Julia).","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Write a distributed pipeline for computing a histogram of symbols found in AST by parsing Julia source files in your .julia/packages/ directory. We have already implemented most of the code that you will need (available as source code here).","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\npkg_processing.jl

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"using Markdown #hide\ncode = Markdown.parse(\"\"\"```julia\\n$(readchomp(\"./pkg_processing.jl\"))\\n```\"\"\") #hide\ncode","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Your task is to write a function that does the map and reduce steps, that will create and gather the dictionaries from different workers. There are two ways to do a map","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"either over directories inside .julia/packages/ - call it distributed_histogram_pkgwise\nor over all files obtained by concatenation of filter_jl outputs (NOTE that this might not be possible if the listing itself is expensive - speed or memory requirements) - call it distributed_histogram_filewise","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Measure if the speed up scales linearly with the number of processes by restricting the number of workers inside a pmap.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"for each file path apply tokenize to extract symbols and follow it with the update of a local histogram\ntry writing sequential version first\neither load ./pkg_processing.jl on startup with -L and -p options or include(\"./pkg_processing.jl\") inside @everywhere\nuse pmap to easily iterate in parallel over a collection - the result should be an array of histogram, which has to be merged on the controller node (use builtin mergewith! function in conjunction with reduce)\npmap supports do syntax","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"pmap(collection) do item\n do_something(item)\nend","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"pkg directory can be obtained with joinpath(DEPOT_PATH[1], \"packages\")","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"BONUS: What is the most frequent symbol in your codebase?","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Let's implement first a sequential version as it is much easier to debug.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"include(\"./pkg_processing.jl\")\n\nusing ProgressMeter\nfunction sequential_histogram(path)\n h = Dict{Symbol, Int}()\n @showprogress for pkg_dir in sample_all_installed_pkgs(path)\n for jl_path in filter_jl(pkg_dir)\n syms = tokenize(jl_path)\n for s in syms\n v = get!(h, s, 0)\n h[s] += 1\n end\n end\n end\n h\nend\npath = joinpath(DEPOT_PATH[1], \"packages\") # usually the first entry\n@time h = sequential_histogram(path) # 87s","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"First we try to distribute over package folders. TODO add the ability to run it only on some workers","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"using Distributed\naddprocs(8)\n\n@everywhere begin\n using Pkg; Pkg.activate(@__DIR__)\n # we have to realize that the code that workers have access to functions we have defined\n include(\"./pkg_processing.jl\") \nend\n\n\"\"\"\n merge_with!(h1, h2)\n\nMerges count dictionary `h2` into `h1` by adding the counts. Equivalent to `Base.mergewith!(+)`.\n\"\"\"\nfunction merge_with!(h1, h2)\n for s in keys(h2)\n get!(h1, s, 0)\n h1[s] += h2[s]\n end\n h1\nend\n\nusing ProgressMeter\nfunction distributed_histogram_pkgwise(path, np=nworkers())\n r = @showprogress pmap(WorkerPool(workers()[1:np]), sample_all_installed_pkgs(path)) do pkg_dir\n h = Dict{Symbol, Int}()\n for jl_path in filter_jl(pkg_dir)\n syms = tokenize(jl_path)\n for s in syms\n v = get!(h, s, 0)\n h[s] += 1\n end\n end\n h\n end\n reduce(merge_with!, r)\nend\npath = joinpath(DEPOT_PATH[1], \"packages\")\n\n@time h = distributed_histogram_pkgwise(path, 2) # 41.5s\n@time h = distributed_histogram_pkgwise(path, 4) # 24.0s\n@time h = distributed_histogram_pkgwise(path, 8) # 24.0s","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Second we try to distribute over all files.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"function distributed_histogram_filewise(path, np=nworkers())\n jl_files = reduce(vcat, filter_jl(pkg_dir) for pkg_dir in sample_all_installed_pkgs(path))\n r = @showprogress pmap(WorkerPool(workers()[1:np]), jl_files) do jl_path\n h = Dict{Symbol, Int}()\n syms = tokenize(jl_path)\n for s in syms\n v = get!(h, s, 0)\n h[s] += 1\n end\n h\n end\n reduce(merge_with!, r)\nend\npath = joinpath(DEPOT_PATH[1], \"packages\")\n@time h = distributed_histogram_pkgwise(path, 2) # 46.9s\n@time h = distributed_histogram_pkgwise(path, 4) # 24.8s\n@time h = distributed_histogram_pkgwise(path, 8) # 20.4s","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Here we can see that we have improved the timings a bit by increasing granularity of tasks.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"BONUS: You can do some analysis with DataFrames","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"using DataFrames\ndf = DataFrame(:sym => collect(keys(h)), :count => collect(values(h)));\nsort!(df, :count, rev=true);\ndf[1:50,:]","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_10/lab/#lab10_thread","page":"Lab","title":"Threading","text":"","category":"section"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"The number of threads for a Julia process can be set up in an environmental variable JULIA_NUM_THREADS or directly on Julia startup with cmd line option -t ## or --threads ##. If both are specified the latter takes precedence.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"julia -t 8","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"In order to find out how many threads are currently available, there exist the nthreads function inside Base.Threads library. There is also an analog to the Distributed myid example, called threadid.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"julia> using Base.Threads\njulia> nthreads()\n8\njulia> threadid()\n1","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"As opposed to distributed/multiprocessing programming, threads have access to the whole memory of Julia's process, therefore we don't have to deal with separate environment manipulation, code loading and data transfers. However we have to be aware of the fact that memory can be modified from two different places and that there may be some performance penalties of accessing memory that is physically further from a given core (e.g. caches of different core or different NUMA[3] nodes). Another significant difference from distributed computing is that we cannot spawn additional threads on the fly in the same way that we have been able to do with addprocs function.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"[3]: NUMA - https://en.wikipedia.org/wiki/Non-uniform_memory_access","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"info: Hyper threads\nIn most of today's CPUs the number of threads is larger than the number of physical cores. These additional threads are usually called hyper threads[4] or when talking about cores - logical cores. The technology relies on the fact, that for a given \"instruction\" there may be underutilized parts of the CPU core's machinery (such as one of many arithmetic units) and if a suitable work/instruction comes in it can be run simultaneously. In practice this means that adding more threads than physical cores may not be accompanied with the expected speed up.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"[4]: Hyperthreading - https://en.wikipedia.org/wiki/Hyper-threading","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"The easiest (not always yielding the correct result) way how to turn a code into multi threaded code is putting the @threads macro in front of a for loop, which instructs Julia to run the body on separate threads.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"julia> A = Array{Union{Int,Missing}}(missing, nthreads());\njulia> for i in 1:nthreads()\n A[threadid()] = threadid()\nend\njulia> A # only the first element is filled\n8-element Vector{Union{Missing, Int64}}:\n 1\n missing\n missing\n missing\n missing\n missing\n missing\n missing","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"julia> A = Array{Union{Int,Missing}}(missing, nthreads());\njulia> @threads for i in 1:nthreads()\n A[threadid()] = threadid()\nend\njulia> A # the expected results\n8-element Vector{Union{Missing, Int64}}:\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8","category":"page"},{"location":"lecture_10/lab/#Multithreaded-sum","page":"Lab","title":"Multithreaded sum","text":"","category":"section"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Armed with this knowledge let's tackle the problem of the simple sum.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"function threaded_sum_naive(a)\n r = zero(eltype(a))\n @threads for i in eachindex(a)\n @inbounds r += a[i]\n end\n return r\nend","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Comparing this with the built-in sum we see not an insignificant discrepancy (one that cannot be explained by reordering of computation) and moreover the timings show us some ridiculous overhead.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"julia> using BenchmarkTools\njulia> a = rand(10_000_000); # 10^7\njulia> sum(a), threaded_sum_naive(a)\n(5.000577175855193e6, 625888.2270955174)\njulia> @btime sum($a)\n 4.861 ms (0 allocations: 0 bytes)\njulia> @btime threaded_sum_naive($a)\n 163.379 ms (20000042 allocations: 305.18 MiB)","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Recalling what has been said above we have to be aware of the fact that the data can be accessed from multiple threads at once, which if not taken into an account means that each thread reads possibly outdated value and overwrites it with its own updated state. ","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"There are two solutions which we will tackle in the next two exercises. ","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Implement threaded_sum_atom, which uses Atomic wrapper around the accumulator variable r in order to ensure correct locking of data access. ","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"use atomic_add! as a replacement of r += A[i]\n\"collect\" the result by dereferencing variable r with empty bracket operator []","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"info: Side note on dereferencing\nIn Julia we can create references to a data types, which are guarranteed to point to correct and allocated type in memory, as long as a reference exists the memory is not garbage collected. These are constructed with Ref(x), Ref(a, 7) or Ref{T}() for reference to variable x, 7th element of array a and an empty reference respectively. Dereferencing aka asking about the underlying value is done using empty bracket operator [].x = 1 # integer\nrx = Ref(x) # reference to that particular integer `x`\nx == rx[] # dereferencing yields the same valueThere also exist unsafe references/pointers Ptr, however we should not really come into a contact with those.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"BONUS: Try chunking the array and calling sum on individual chunks to obtain some real speedup.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"function threaded_sum_atom(a)\n r = Atomic{eltype(a)}(zero(eltype(a)))\n @threads for i in eachindex(a)\n @inbounds atomic_add!(r, a[i])\n end\n return r[]\nend\n\njulia> sum(a) ≈ threaded_sum_atom(a)\ntrue\njulia> @btime threaded_sum_atom($a)\n 661.502 ms (42 allocations: 3.66 KiB)","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"That's better but far from the performance we need. ","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"BONUS: There is a fancier and faster way to do this by chunking the array","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"function threaded_sum_fancy_atom(a)\n r = Atomic{eltype(a)}(zero(eltype(a)))\n len, rem = divrem(length(a), nthreads())\n @threads for t in 1:nthreads()\n rₜ = zero(eltype(a))\n @simd for i in (1:len) .+ (t-1)*len\n @inbounds rₜ += a[i]\n end\n atomic_add!(r, rₜ)\n end\n # catch up any stragglers\n result = r[]\n @simd for i in length(a)-rem+1:length(a)\n @inbounds result += a[i]\n end\n return result\nend\n\njulia> sum(a) ≈ threaded_sum_fancy_atom(a)\ntrue\njulia> @btime threaded_sum_fancy_atom($a)\n 2.983 ms (42 allocations: 3.67 KiB)","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Finally we have beaten the \"sequential\" sum. The quotes are intentional, because the Base's implementation of a sum uses Single instruction, multiple data (SIMD) instructions as well, which allow to process multiple elements at once.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Implement threaded_sum_buffer, which uses an array of length nthreads() (we will call this buffer) for local aggregation of results of individual threads. ","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"use threadid() to index the buffer array\nsum the buffer array to obtain final result","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"function threaded_sum_buffer(a)\n R = zeros(eltype(a), nthreads())\n @threads for i in eachindex(a)\n @inbounds R[threadid()] += a[i]\n end\n r = zero(eltype(a))\n # sum the partial results from each thread\n for i in eachindex(R)\n @inbounds r += R[i]\n end\n return r\nend\n\njulia> sum(a) ≈ threaded_sum_buffer(a)\ntrue\njulia> @btime threaded_sum_buffer($a)\n 2.750 ms (42 allocations: 3.78 KiB)","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Though this implementation is cleaner and faster, there is possible drawback with this implementation, as the buffer R lives in a continuous part of the memory and each thread that accesses it brings it to its caches as a whole, thus invalidating the values for the other threads, which it in the same way.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Seeing how multithreading works on a simple example, let's apply it on the \"more practical\" case of the Symbol histogram from exercise above.","category":"page"},{"location":"lecture_10/lab/#lab10_dist_file_t","page":"Lab","title":"Multithreaded file processing","text":"","category":"section"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Write a multithreaded analog of the file processing pipeline from exercise above. Again the task is to write the map and reduce steps, that will create and gather the dictionaries from different workers. There are two ways to map","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"either over directories inside .julia/packages/ - threaded_histogram_pkgwise\nor over all files obtained by concatenation of filter_jl outputs - threaded_histogram_filewise","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Compare the speedup with the version using process based parallelism.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"create a separate dictionary for each thread in order to avoid the need for atomic operations\n","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"BONUS: In each of the cases count how many files/pkgs each thread processed. Would the dynamic scheduler help us in this situation?","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Setup is now much simpler.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"using Base.Threads\ninclude(\"./pkg_processing.jl\") \npath = joinpath(DEPOT_PATH[1], \"packages\")","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Firstly the version with folder-wise parallelism.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"function threaded_histogram_pkgwise(path)\n ht = [Dict{Symbol, Int}() for _ in 1:nthreads()]\n @threads for pkg_dir in sample_all_installed_pkgs(path)\n h = ht[threadid()]\n for jl_path in filter_jl(pkg_dir)\n syms = tokenize(jl_path)\n for s in syms\n v = get!(h, s, 0)\n h[s] += 1\n end\n end\n end\n reduce(mergewith!(+), ht)\nend\n\njulia> @time h = threaded_histogram_pkgwise(path)\n 26.958786 seconds (81.69 M allocations: 10.384 GiB, 4.58% gc time)","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Secondly the version with file-wise parallelism.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"function threaded_histogram_filewise(path)\n jl_files = reduce(vcat, filter_jl(pkg_dir) for pkg_dir in sample_all_installed_pkgs(path))\n ht = [Dict{Symbol, Int}() for _ in 1:nthreads()]\n @threads for jl_path in jl_files\n h = ht[threadid()]\n syms = tokenize(jl_path)\n for s in syms\n v = get!(h, s, 0)\n h[s] += 1\n end\n end\n reduce(mergewith!(+), ht)\nend\n\njulia> @time h = threaded_histogram_filewise(path)\n 29.677184 seconds (81.66 M allocations: 10.411 GiB, 4.13% gc time)","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_10/lab/#Task-switching","page":"Lab","title":"Task switching","text":"","category":"section"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"There is a way how to run \"multiple\" things at once, which does not necessarily involve either threads or processes. In Julia this concept is called task switching or asynchronous programming, where we fire off our requests in a short time and let the cpu/os/network handle the distribution. As an example which we will try today is querying a web API, which has some variable latency. In the usuall sequantial fashion we can always post queries one at a time, however generally the APIs can handle multiple request at a time, therefore in order to better utilize them, we can call them asynchronously and fetch all results later, in some cases this will be faster.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"info: Burst requests\nIt is a good practice to check if an API supports some sort of batch request, because making a burst of single request might lead to a worse performance for others and a possible blocking of your IP/API key.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Consider following functions","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"function a()\n for i in 1:10\n sleep(1)\n end\nend\n\nfunction b()\n for i in 1:10\n @async sleep(1)\n end\nend\n\nfunction c()\n @sync for i in 1:10\n @async sleep(1)\n end\nend","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"How much time will the execution of each of them take?","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\nSolution

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"@time a() # 10s\n@time b() # ~0s\n@time c() # >~1s","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"Choose one of the free web APIs and query its endpoint using the HTTP.jl library. Implement both sequential and asynchronous version. Compare them on an burst of 10 requests.","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"use HTTP.request for GET requests on your chosen API, e.g. r = HTTP.request(\"GET\", \"https://catfact.ninja/fact\") for random cat fact\nconverting body of a response can be done simply by constructing a String out of it - String(r.body)\nin order to parse a json string use JSON.jl's parse function\nJulia offers asyncmap - asynchronous map","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"using HTTP, JSON\n\nfunction query_cat_fact()\n r = HTTP.request(\"GET\", \"https://catfact.ninja/fact\")\n j = String(r.body)\n d = JSON.parse(j)\n d[\"fact\"]\nend\n\n# without asyncmap\nfunction get_cat_facts_async(n)\n facts = Vector{String}(undef, n)\n @sync for i in 1:10\n @async facts[i] = query_cat_fact()\n end\n facts\nend\n\nget_cat_facts_async(n) = asyncmap(x -> query_cat_fact(), Base.OneTo(n))\nget_cat_facts(n) = map(x -> query_cat_fact(), Base.OneTo(n))\n\n@time get_cat_facts_async(10) # ~0.15s\n@time get_cat_facts(10) # ~1.1s","category":"page"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_10/lab/#Resources","page":"Lab","title":"Resources","text":"","category":"section"},{"location":"lecture_10/lab/","page":"Lab","title":"Lab","text":"parallel computing course by Julia Computing","category":"page"},{"location":"lecture_10/lecture/#Parallel-programming-with-Julia","page":"Lecture","title":"Parallel programming with Julia","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Julia offers different levels of parallel programming","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"distributed processing, where jobs are split among different Julia processes\nmulti-threadding, where jobs are split among multiple threads within the same processes\nSIMD instructions\nTask switching.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"In this lecture, we will focus mainly on the first two, since SIMD instructions are mainly used for low-level optimization (such as writing your own very performant BLAS library), and task switching is not a true paralelism, but allows to run a different task when one task is waiting for example for IO.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"The most important lesson is that before you jump into the parallelism, be certain you have made your sequential code as fast as possible.","category":"page"},{"location":"lecture_10/lecture/#Process-level-paralelism","page":"Lecture","title":"Process-level paralelism","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Process-level paralelism means we run several instances of Julia (in different processes) and they communicate between each other using inter-process communication (IPC). The implementation of IPC differs if parallel julia instances share the same machine, or they are on different machines spread over the network. By default, different processes do not share any libraries or any variables. They are loaded clean and it is up to the user to set-up all needed code and data.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Julia's default modus operandi is a single main instance controlling several workers. This main instance has myid() == 1, worker processes receive higher numbers. Julia can be started with multiple workers from the very beggining, using -p switch as ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"julia -p n","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"where n is the number of workers, or you can add workers after Julia has been started by","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using Distributed\naddprocs(n)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"You can also remove workers using rmprocs. When Julia is started with -p, Distributed library is loaded by default on main worker. Workers can be on the same physical machines, or on different machines. Julia offer integration via ClusterManagers.jl with most schedulling systems.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"If you want to evaluate piece of code on all workers including main process, a convenience macro @everywhere is offered.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere @show myid()","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"As we have mentioned, workers are loaded without libraries. We can see that by running","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere InteractiveUtils.varinfo()","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"which fails, but after loading InteractiveUtils everywhere","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using Statistics\n@everywhere begin \n\tusing InteractiveUtils\n\tprintln(InteractiveUtils.varinfo(;imported = true))\nend","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"we see that Statistics was loaded only on the main process. Thus, there is not magical sharing of data and code. With @everywhere macro we can define function and variables, and import libraries on workers as ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tfoo(x, y) = x * y + sin(y)\n\tfoo(x) = foo(x, myid())\n\tx = rand()\nend\n@everywhere @show foo(1.0)\n@everywhere @show x","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"The fact that x has different values on different workers and master demonstrates again the independency of processes. While we can set up everything using @everywhere macro, we can also put all the code for workers into a separate file, e.g. worker.jl and load it on all workers using -L worker.jl.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Julia's multi-processing model is based on message-passing paradigm, but the abstraction is more akin to procedure calls. This means that users are saved from prepending messages with headers and implementing logic deciding which function should be called for thich header. Instead, we can schedulle an execution of a function on a remote worker and return the control immeadiately to continue in our job. A low-level function providing this functionality is remotecall(fun, worker_id, args...). For example ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tfunction delayed_foo(x, y, n )\n\t\tsleep(n)\n\t\tprintln(\"woked up\")\n\t\tfoo(x, y)\n\tend\nend\nr = remotecall(delayed_foo, 2, 1, 1, 60)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"returns immediately, even though the function will take at least 60 seconds. r does not contain result of foo(1, 1), but a struct Future, which is a remote reference in Julia's terminology. It points to data located on some machine, indicates, if they are available and allows to fetch them from the remote worker. fetch is blocking, which means that the execution is blocked until data are available (if they are never available, the process can wait forever.) The presence of data can be checked using isready, which in case of Future returned from remote_call indicate that the computation has finished.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"isready(r)\nfetch(r) == foo(1, 1)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"An advantage of the remote reference is that it can be freely shared around processes and the result can be retrieved on different node then the one which issued the call.s","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"r = remotecall(delayed_foo, 2, 1, 1, 60)\nremotecall(r -> println(\"value: \",fetch(r), \" retrieved on \", myid()) , 3, r)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"An interesting feature of fetch is that it re-throw an exception raised on a different process.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tfunction exfoo()\n\t\tthrow(\"Exception from $(myid())\")\n\tend\nend\nr = @spawnat 2 exfoo()","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"where we have used @spawnat instead of remote_call. It is higher level alternative executing a closure around the expression (in this case exfoo()) on a specified worker, in this case 2. Coming back to the example, when we fetch the result r, the exception is throwed on the main process, not on the worker","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"fetch(r)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@spawnat can be executed with :any to signal that the user does not care, where the function will be executed and it will be left up to Julia.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"r = @spawnat :any foo(1,1)\nfetch(r)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Finally, if you would for some reason need to wait for the computed value, you can use ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"remotecall_fetch(foo, 2, 1, 1)","category":"page"},{"location":"lecture_10/lecture/#Running-example:-Julia-sets","page":"Lecture","title":"Running example: Julia sets","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Our example for explaining mechanisms of distributed computing will be Julia set fractals, as they can be easily paralelized. The example is adapted from Eric Aubanel. Some fractals (Julia set, Mandelbrot) are determined by properties of some complex-valued functions. Julia set counts, how many iteration is required for f(z) = z^2+c to be bigger than two in absolute value, f(z) =2. The number of iterations can then be mapped to the pixel's color, which creates a nice visualization we know.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"function juliaset_pixel(z₀, c)\n z = z₀\n for i in 1:255\n abs2(z)> 4.0 && return (i - 1)%UInt8\n z = z*z + c\n end\n return UInt8(255)\nend","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"A nice property of fractals like Julia set is that the computation can be easily paralelized, since the value of each pixel is independent from the remaining. In our experiments, the level of granulity will be one column, since calculation of single pixel is so fast, that thread or process switching will have much higher overhead.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"function juliaset_column!(img, c, n, j)\n x = -2.0 + (j-1)*4.0/(n-1)\n for i in 1:n\n y = -2.0 + (i-1)*4.0/(n-1)\n @inbounds img[i,j] = juliaset_pixel(x+im*y, c)\n end\n nothing\nend","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"To calculate full image","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"function juliaset(x, y, n=1000)\n c = x + y*im\n img = Array{UInt8,2}(undef,n,n)\n for j in 1:n\n juliaset_column!(img, c, n, j)\n end\n return img\nend","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"and run it and view it","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using Plots\nfrac = juliaset(-0.79, 0.15)\nplot(heatmap(1:size(frac,1),1:size(frac,2), frac, color=:Spectral))","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"or with GLMakie","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using GLMakie\nfrac = juliaset(-0.79, 0.15)\nheatmap(frac)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"To observe the execution length, we will use BenchmarkTools.jl ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using BenchmarkTools\njulia> @btime juliaset(-0.79, 0.15);\n 39.822 ms (2 allocations: 976.70 KiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Let's now try to speed-up the computation using more processes. We first make functions available to workers","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tfunction juliaset_pixel(z₀, c)\n\t z = z₀\n\t for i in 1:255\n\t abs2(z)> 4.0 && return (i - 1)%UInt8\n\t z = z*z + c\n\t end\n\t return UInt8(255)\n\tend\n\n\tfunction juliaset_column!(img, c, n, colj, j)\n\t x = -2.0 + (j-1)*4.0/(n-1)\n\t for i in 1:n\n\t y = -2.0 + (i-1)*4.0/(n-1)\n\t @inbounds img[i,colj] = juliaset_pixel(x+im*y, c)\n\t end\n\t nothing\n\tend\nend","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"For the actual parallelisation, we split the computation of the whole image into bands, such that each worker computes a smaller portion.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tfunction juliaset_columns(c, n, columns)\n\t img = Array{UInt8,2}(undef, n, length(columns))\n\t for (colj, j) in enumerate(columns)\n\t juliaset_column!(img, c, n, colj, j)\n\t end\n\t img\n\tend\nend\n\nfunction juliaset_spawn(x, y, n = 1000)\n c = x + y*im\n columns = Iterators.partition(1:n, div(n, nworkers()))\n r_bands = [@spawnat w juliaset_columns(c, n, cols) for (w, cols) in enumerate(columns)]\n slices = map(fetch, r_bands)\n reduce(hcat, slices)\nend","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"we observe some speed-up over the serial version, but not linear in terms of number of workers","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"julia> @btime juliaset(-0.79, 0.15);\n 38.699 ms (2 allocations: 976.70 KiB)\n\njulia> @btime juliaset_spawn(-0.79, 0.15);\n 21.521 ms (480 allocations: 1.93 MiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"In the above example, we spawn one function on each worker and collect the results. In essence, we are performing map over bands. Julia offers for this usecase a parallel version of map pmap. With that, our example can look like","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"function juliaset_pmap(x, y, n = 1000, np = nworkers())\n c = x + y*im\n columns = Iterators.partition(1:n, div(n, np))\n slices = pmap(cols -> juliaset_columns(c, n, cols), columns)\n reduce(hcat, slices)\nend\n\njulia> @btime juliaset_pmap(-0.79, 0.15);\n 17.597 ms (451 allocations: 1.93 MiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"which has slightly better timing then the version based on @spawnat and fetch (as explained below in section about Threads, the parallel computation of Julia set suffers from each pixel taking different time to compute, which can be relieved by dividing the work into more parts:","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"julia> @btime juliaset_pmap(-0.79, 0.15, 1000, 16);\n 12.686 ms (1439 allocations: 1.96 MiB)","category":"page"},{"location":"lecture_10/lecture/#Shared-memory","page":"Lecture","title":"Shared memory","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"When main and all workers are located on the same process, and the OS supports sharing memory between processes (by sharing memory pages), we can use SharedArrays to avoid sending the matrix with results.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin\n\tusing SharedArrays\n\tfunction juliaset_shared(x, y, n=1000)\n\t c = x + y*im\n\t img = SharedArray(Array{UInt8,2}(undef,n,n))\n\t @sync @distributed for j in 1:n\n\t juliaset_column!(img, c, n, j, j)\n\t end\n\t return img\n\tend \nend\n\njulia> @btime juliaset_shared(-0.79, 0.15);\n 19.088 ms (963 allocations: 1017.92 KiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"The allocation of the Shared Array mich be costly, let's try to put the allocation outside of the loop","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"img = SharedArray(Array{UInt8,2}(undef,1000,1000))\nfunction juliaset_shared!(img, x, y, n=1000)\n c = x + y*im\n @sync @distributed for j in 1:n\n juliaset_column!(img, c, n, j, j)\n end\n return img\nend \n\njulia> @btime juliaset_shared!(img, -0.79, 0.15);\n 17.399 ms (614 allocations: 27.61 KiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"but both versions are not akin. It seems like the alocation of SharedArray costs approximately 2ms.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@distributed for (Distributed.pfor) does not allows to supply, as it splits the for cycle to nworkers() processes. Above we have seen that more splits is better","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tfunction juliaset_columns!(img, c, n, columns)\n\t for (colj, j) in enumerate(columns)\n\t juliaset_column!(img, c, n, colj, j)\n\t end\n\tend\nend\n\nimg = SharedArray(Array{UInt8,2}(undef,1000,1000))\nfunction juliaset_shared!(img, x, y, n=1000, np = nworkers())\n c = x + y*im\n columns = Iterators.partition(1:n, div(n, np))\n pmap(cols -> juliaset_columns!(img, c, n, cols), columns)\n return img\nend \n\njulia> @btime juliaset_shared!(img, -0.79, 0.15, 1000, 16);\n 11.760 ms (1710 allocations: 85.98 KiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Which is almost 1ms faster than without used of pre-allocated SharedArray. Notice the speedup is now 38.699 / 11.76 = 3.29×","category":"page"},{"location":"lecture_10/lecture/#Synchronization-/-Communication-primitives","page":"Lecture","title":"Synchronization / Communication primitives","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"The orchestration of a complicated computation might be difficult with relatively low-level remote calls. A producer / consumer paradigm is a synchronization paradigm that uses queues. Consumer fetches work intructions from the queue and pushes results to different queue. Julia supports this paradigm with Channel and RemoteChannel primitives. Importantly, putting to and taking from queue is an atomic operation, hence we do not have take care of race conditions. The code for the worker might look like","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tfunction juliaset_channel_worker(instructions, results)\n\t\twhile isready(instructions)\n\t\t\tc, n, cols = take!(instructions)\n\t\t\tput!(results, (cols, juliaset_columns(c, n, cols)))\n\t\tend\n\tend\nend","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"The code for the main will look like","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"function juliaset_channels(x, y, n = 1000, np = nworkers())\n\tc = x + y*im\n\tcolumns = Iterators.partition(1:n, div(n, np))\n\tinstructions = RemoteChannel(() -> Channel(np))\n\tforeach(cols -> put!(instructions, (c, n, cols)), columns)\n\tresults = RemoteChannel(()->Channel(np))\n\trfuns = [@spawnat i juliaset_channel_worker(instructions, results) for i in workers()]\n\n\timg = Array{UInt8,2}(undef, n, n)\n\tfor i in 1:np\n\t\tcols, impart = take!(results)\n\t\timg[:,cols] .= impart;\n\tend\n\timg\nend\n\njulia> @btime juliaset_channels(-0.79, 0.15);","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"The execution time is much higher then what we have observed in the previous cases and changing the number of workers does not help much. What went wrong? The reason is that setting up the infrastructure around remote channels is a costly process. Consider the following alternative, where (i) we let workers to run endlessly and (ii) the channel infrastructure is set-up once and wrapped into an anonymous function","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tfunction juliaset_channel_worker(instructions, results)\n\t\twhile true\n\t\t c, n, cols = take!(instructions)\n\t\t put!(results, (cols, juliaset_columns(c, n, cols)))\n\t\tend\n\tend\nend\n\nfunction juliaset_init(x, y, n = 1000, np = nworkers())\n c = x + y*im\n columns = Iterators.partition(1:n, div(n, np))\n T = Tuple{ComplexF64,Int64,UnitRange{Int64}}\n instructions = RemoteChannel(() -> Channel{T}(np))\n T = Tuple{UnitRange{Int64},Array{UInt8,2}}\n results = RemoteChannel(()->Channel{T}(np))\n foreach(p -> remote_do(juliaset_channel_worker, p, instructions, results), workers())\n function compute()\n img = Array{UInt8,2}(undef, n, n)\n foreach(cols -> put!(instructions, (c, n, cols)), columns)\n for i in 1:np\n cols, impart = take!(results)\n img[:,cols] .= impart;\n end\n img\n end \nend\n\nt = juliaset_init(-0.79, 0.15)\njulia> @btime t();\n 17.697 ms (776 allocations: 1.94 MiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"with which we obtain the comparable speed to the pmap approach.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\nremote_do vs remote_callInstead of @spawnat (remote_call) we can also use remote_do as foreach(p -> remote_do(juliaset_channel_worker, p, instructions, results), workers), which executes the function juliaset_channel_worker at worker p with parameters instructions and results but does not return Future handle to receive the future results.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\nChannel and RemoteChannelAbstractChannel has to implement the interface put!, take!, fetch, isready and wait, i.e. it should behave like a queue. Channel is an implementation if an AbstractChannel that facilitates a communication within a single process (for the purpose of multi-threadding and task switching). Channel can be easily created by Channel{T}(capacity), which can be infinite. The storage of a channel can be seen in data field, but a direct access will of course break all guarantees like atomicity of take! and put!. For communication between proccesses, the <:AbstractChannel has to be wrapped in RemoteChannel. The constructor for RemoteChannel(f::Function, pid::Integer=myid()) has a first argument a function (without arguments) which constructs the Channel (or something like that) on the remote machine identified by pid and returns the RemoteChannel. The storage thus resides on the machine specified by pid and the handle provided by the RemoteChannel can be freely passed to any process. (For curious, ProcessGroup Distributed.PGRP contains an information about channels on machines.) ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"In the above example, juliaset_channel_worker defined as ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"function juliaset_channel_worker(instructions, results)\n\twhile true\n\t c, n, cols = take!(instructions)\n\t put!(results, (cols, juliaset_columns(c, n, cols)))\n\tend\nend","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"runs forever due to the while true loop. ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Julia does not provide by default any facility to kill the remote execution except sending ctrl-c to the remote worker as interrupt(pids::Integer...). To stop the computation, we usually extend the type accepted by the instructions channel to accept some stopping token (e.g. :stop) and stop.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tfunction juliaset_channel_worker(instructions, results)\n\t\twhile true\n\t\t\ti = take!(instructions)\n\t\t\ti === :stop && break\n\t\t\tc, n, cols = i\n\t\t\tput!(results, (cols, juliaset_columns(c, n, cols)))\n\t\tend\n\t\tprintln(\"worker $(myid()) stopped\")\n\t\tput!(results, :stop)\n\tend\nend\n\nfunction juliaset_init(x, y, n = 1000, np = nworkers())\n c = x + y*im\n columns = Iterators.partition(1:n, div(n, np))\n instructions = RemoteChannel(() -> Channel(np))\n results = RemoteChannel(()->Channel(np))\n foreach(p -> remote_do(juliaset_channel_worker, p, instructions, results), workers())\n function compute()\n img = Array{UInt8,2}(undef, n, n)\n foreach(cols -> put!(instructions, (c, n, cols)), columns)\n for i in 1:np\n cols, impart = take!(results)\n img[:,cols] .= impart;\n end\n img\n end \nend\n\nt = juliaset_init(-0.79, 0.15)\nt()\nforeach(i -> put!(t.instructions, :stop), workers())","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"In the above example we paid the price of introducing type instability into the channels, which now contain types Any instead of carefully constructed tuples. But the impact on the overall running time is negligible","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"t = juliaset_init(-0.79, 0.15)\njulia> @btime t()\n 17.551 ms (774 allocations: 1.94 MiB)\nforeach(i -> put!(t.instructions, :stop), workers())","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"In some use-cases, the alternative can be to put all jobs to the RemoteChannel before workers are started, and then stop the workers when the remote channel is empty as ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tfunction juliaset_channel_worker(instructions, results)\n\t\twhile !isready(instructions)\n\t\t c, n, cols = take!(instructions)\n\t\t put!(results, (cols, juliaset_columns(c, n, cols)))\n\t\tend\n\tend\nend","category":"page"},{"location":"lecture_10/lecture/#Sending-data","page":"Lecture","title":"Sending data","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Sending parameters of functions and receiving results from a remotely called functions migh incur a significant cost. ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Try to minimize the data movement as much as possible. A prototypical example is","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"A = rand(1000,1000);\nBref = @spawnat :any A^2;","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"and","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Bref = @spawnat :any rand(1000,1000)^2;","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"It is not only volume of data (in terms of the number of bytes), but also a complexity of objects that are being sent. Serialization can be very time consuming, an efficient converstion to something simple might be worth","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using BenchmarkTools\n@everywhere begin \n\tusing Random\n\tv = [randstring(rand(1:20)) for i in 1:1000];\n\tp = [i => v[i] for i in 1:1000]\n\td = Dict(p)\n\n\tsend_vec() = v\n\tsend_dict() = d\n\tsend_pairs() = p\n\tcustom_serialization() = (length.(v), join(v, \"\"))\nend\n\n@btime remotecall_fetch(send_vec, 2);\n@btime remotecall_fetch(send_dict, 2);\n@btime remotecall_fetch(send_pairs, 2);\n@btime remotecall_fetch(custom_serialization, 2);","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Some type of objects cannot be properly serialized and deserialized","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"a = IdDict(\n\t:a => rand(1,1),\n\t)\nb = remotecall_fetch(identity, 2, a)\na[:a] === a[:a]\na[:a] === b[:a]","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"If you need to send the data to worker, i.e. you want to define (overwrite) a global variable there","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tg = rand()\n\tshow_secret() = println(\"secret of \", myid(), \" is \", g)\nend\n@everywhere show_secret()\n\nfor i in workers()\n\tremotecall_fetch(g -> eval(:(g = $(g))), i, g)\nend\n@everywhere show_secret()","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"which is implemented in the ParallelDataTransfer.jl with other variants, but in general, this construct should be avoided.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Alternatively, you can overwrite a global variable","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tg = rand()\n\tshow_secret() = println(\"secret of \", myid(), \" is \", g)\n\tfunction set_g(x) \n\t\tglobal g\n\t\tg = x\n\t\tnothing\n\tend\nend\n\n@everywhere show_secret()\nremote_do(set_g, 2, 2)\n@everywhere show_secret()","category":"page"},{"location":"lecture_10/lecture/#Practical-advices","page":"Lecture","title":"Practical advices","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Recall that (i) workers are started as clean processes and (ii) they might not share the same environment with the main process. The latter is due to the possibility of remote machines to have a different directory structure. ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@everywhere begin \n\tusing Pkg\n\tprintln(Pkg.project().path)\nend","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Our advices earned by practice are:","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"to have shared directory (shared home) with code and to share the location of packages\nto place all code for workers to one file, let's call it worker.jl (author of this includes the code for master as well).\nput to the beggining of worker.jl code activating specified environment as (or specify environmnet for all workers in environment variable as export JULIA_PROJECT=\"$PWD\")","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using Pkg\nPkg.activate(@__DIR__)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"and optionally","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Pkg.resolve()\nPkg.instantiate()","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"run julia as","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"julia -p ?? -L worker.jl main.jl","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"where main.jl is the script to be executed on the main node. Or","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"julia -p ?? -L worker.jl -e \"main()\"","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"where main() is the function defined in worker.jl to be executed on the main node.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"A complete example can be seen in juliaset_p.jl.","category":"page"},{"location":"lecture_10/lecture/#Multi-threadding","page":"Lecture","title":"Multi-threadding","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"So far, we have been able to decrese the computation from 39ms to something like 13ms. Can we improve? Let's now turn our attention to multi-threadding, where we will not pay the penalty for IPC. Moreover, the computation of Julia set is multi-thread friendly, as all the memory can be pre-allocatted. We slightly modify our code to accept different methods distributing the work among slices in the pre-allocated matrix. To start Julia with support of multi-threadding, run it with julia -t n, where n is the number of threads. It is reccomended to set n to number of physical cores, since in hyper-threadding two threads shares arithmetic units of a single core, and in applications for which Julia was built, they are usually saturated.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using BenchmarkTools\nfunction juliaset_pixel(z₀, c)\n z = z₀\n for i in 1:255\n abs2(z)> 4.0 && return (i - 1)%UInt8\n z = z*z + c\n end\n return UInt8(255)\nend\n\nfunction juliaset_column!(img, c, n, j)\n x = -2.0 + (j-1)*4.0/(n-1)\n for i in 1:n\n y = -2.0 + (i-1)*4.0/(n-1)\n @inbounds img[i,j] = juliaset_pixel(x+im*y, c)\n end\n nothing\nend\n\nfunction juliaset(x, y, n=1000)\n c = x + y*im\n img = Array{UInt8,2}(undef,n,n)\n for j in 1:n\n juliaset_column!(img, c, n, j)\n end\n return img\nend\n\njulia> @btime juliaset(-0.79, 0.15, 1000);\n 38.932 ms (2 allocations: 976.67 KiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Let's now try to speed-up the calculation using multi-threadding. Julia v0.5 has introduced multi-threadding with static-scheduller with a simple syntax: just prepend the for-loop with a Threads.@threads macro. With that, the first multi-threaded version will looks like","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"function juliaset_static(x, y, n=1000)\n c = x + y*im\n img = Array{UInt8,2}(undef,n,n)\n Threads.@threads :static for j in 1:n\n juliaset_column!(img, c, n, j)\n end\n return img\nend","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"with benchmark","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"julia> \t@btime juliaset_static(-0.79, 0.15, 1000);\n 15.751 ms (27 allocations: 978.75 KiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Although we have used four-threads, and the communication overhead should be next to zero, the speed improvement is 24. Why is that? ","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"To understand bettern what is going on, we have improved the profiler we have been developing last week. The logging profiler logs time of entering and exitting every function call of every thread, which is useful to understand, what is going on. The api is not yet polished, but it will do its job. Importantly, to prevent excessive logging, we ask to log only some functions.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using LoggingProfiler\nfunction juliaset_static(x, y, n=1000)\n c = x + y*im\n img = Array{UInt8,2}(undef,n,n)\n Threads.@threads :dynamic for j in 1:n\n LoggingProfiler.@recordfun juliaset_column!(img, c, n, j)\n end\n return img\nend\n\nLoggingProfiler.initbuffer!(1000)\njuliaset_static(-0.79, 0.15, 1000);\nLoggingProfiler.recorded()\nLoggingProfiler.adjustbuffer!()\njuliaset_static(-0.79, 0.15, 1000)\nLoggingProfiler.export2svg(\"/tmp/profile.svg\")\nLoggingProfiler.export2luxor(\"profile.png\")","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"(Image: profile.png) From the visualization of the profiler we can see not all threads were working the same time. Thread 1 and 4 were working less that Thread 2 and 3. The reason is that the static scheduller partition the total number of columns (1000) into equal parts, where the total number of parts is equal to the number of threads, and assign each to a single thread. In our case, we will have four parts each of size 250. Since execution time of computing value of each pixel is not the same, threads with a lot zero iterations will finish considerably faster. This is the incarnation of one of the biggest problems in multi-threadding / schedulling. A contemprary approach is to switch to dynamic schedulling, which divides the problem into smaller parts, and when a thread is finished with one part, it assigned new not-yet computed part.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"From 1.5, one can specify the scheduller for Threads.@thread [scheduller] for construct to be either :static and / or :dynamic. The :dynamic is compatible with the partr dynamic scheduller. From 1.8, :dynamic is default, but the range is dividided into nthreads() parts, which is the reason why we do not see an improvement.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Dynamic scheduller is also supported using by Threads.@spawn macro. The prototypical approach used for invocation is the fork-join model, where one recursivelly partitions the problems and wait in each thread for the other","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"function juliaset_recspawn!(img, c, n, lo=1, hi=n, ntasks=128)\n if hi - lo > n/ntasks-1\n mid = (lo+hi)>>>1\n finish = Threads.@spawn juliaset_recspawn!(img, c, n, lo, mid, ntasks)\n juliaset_recspawn!(img, c, n, mid+1, hi, ntasks)\n wait(finish)\n return\n end\n for j in lo:hi\n juliaset_column!(img, c, n, j)\n end\n nothing\nend","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Measuring the time we observe four-times speedup, which corresponds to the number of threads.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"function juliaset_forkjoin(x, y, n=1000, ntasks = 16)\n c = x + y*im\n img = Array{UInt8,2}(undef,n,n)\n juliaset_recspawn!(img, c, n, 1, n, ntasks)\n return img\nend\n\njulia> @btime juliaset_forkjoin(-0.79, 0.15);\n 10.326 ms (142 allocations: 986.83 KiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"This is so far our fastest construction with speedup 38.932 / 10.326 = 3.77×.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Unfortunatelly, the LoggingProfiler does not handle task migration at the moment, which means that we cannot visualize the results. Due to task switching overhead, increasing the granularity might not pay off.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"4 tasks: 16.262 ms (21 allocations: 978.05 KiB)\n8 tasks: 10.660 ms (45 allocations: 979.80 KiB)\n16 tasks: 10.326 ms (142 allocations: 986.83 KiB)\n32 tasks: 10.786 ms (238 allocations: 993.83 KiB)\n64 tasks: 10.211 ms (624 allocations: 1021.89 KiB)\n128 tasks: 10.224 ms (1391 allocations: 1.05 MiB)\n256 tasks: 10.617 ms (2927 allocations: 1.16 MiB)\n512 tasks: 11.012 ms (5999 allocations: 1.38 MiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using FLoops, FoldsThreads\nfunction juliaset_folds(x, y, n=1000, basesize = 2)\n c = x + y*im\n img = Array{UInt8,2}(undef,n,n)\n @floop ThreadedEx(basesize = basesize) for j in 1:n\n juliaset_column!(img, c, n, j)\n end\n return img\nend\n\njulia> @btime juliaset_folds(-0.79, 0.15, 1000);\n 10.253 ms (3960 allocations: 1.24 MiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"where basesize is the size of the smallest part allocated to a single thread, in this case 2 columns.","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"julia> @btime juliaset_folds(-0.79, 0.15, 1000);\n 10.575 ms (52 allocations: 980.12 KiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"function juliaset_folds(x, y, n=1000, basesize = 2)\n c = x + y*im\n img = Array{UInt8,2}(undef,n,n)\n @floop DepthFirstEx(basesize = basesize) for j in 1:n\n juliaset_column!(img, c, n, j)\n end\n return img\nend\njulia> @btime juliaset_folds(-0.79, 0.15, 1000);\n 10.421 ms (3582 allocations: 1.20 MiB)","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"We can identify the best smallest size of the work basesize and measure its influence on the time","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"map(2 .^ (0:7)) do bs \n\tt = @belapsed juliaset_folds(-0.79, 0.15, 1000, $(bs));\n\t(;basesize = bs, time = t)\nend |> DataFrame","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":" Row │ basesize time\n │ Int64 Float64\n─────┼─────────────────────\n 1 │ 1 0.0106803\n 2 │ 2 0.010267\n 3 │ 4 0.0103081\n 4 │ 8 0.0101652\n 5 │ 16 0.0100204\n 6 │ 32 0.0100097\n 7 │ 64 0.0103293\n 8 │ 128 0.0105411","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"We observe that the minimum is for basesize = 32, for which we got 3.8932× speedup. ","category":"page"},{"location":"lecture_10/lecture/#Garbage-collector-is-single-threadded","page":"Lecture","title":"Garbage collector is single-threadded","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Keep reminded that while threads are very easy very convenient to use, there are use-cases where you might be better off with proccess, even though there will be some communication overhead. One such case happens when you need to allocate and free a lot of memory. This is because Julia's garbage collector is single-threadded (in 1.10 it is now partially multi-threaded). Imagine a task of making histogram of bytes in a directory. For a fair comparison, we will use Transducers, since they offer thread and process based paralelism","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using Transducers\n@everywhere begin \n\tfunction histfile(filename)\n\t\th = Dict{UInt8,Int}()\n\t\tforeach(open(read, filename, \"r\")) do b \n\t\t\th[b] = get(h, b, 0) + 1\n\t\tend\n\t\th\n\tend\nend\n\nfiles = filter(isfile, readdir(\"/Users/tomas.pevny/Downloads/\", join = true))\n@elapsed foldxd(mergewith(+), files |> Map(histfile))\n150.863183701","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"and using the multi-threaded version of map","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"@elapsed foldxt(mergewith(+), files |> Map(histfile))\n205.309952618","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"we see that the threadding is actually worse than process based paralelism despite us paying the price for serialization and deserialization of Dict. Needless to say that changing Dict to Vector as","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"using Transducers\n@everywhere begin \n\tfunction histfile(filename)\n\t\th = Dict{UInt8,Int}()\n\t\tforeach(open(read, filename, \"r\")) do b \n\t\t\th[b] = get(h, b, 0) + 1\n\t\tend\n\t\th\n\tend\nend\nfiles = filter(isfile, readdir(\"/Users/tomas.pevny/Downloads/\", join = true))\n@elapsed foldxd(mergewith(+), files |> Map(histfile))\n86.44577969\n@elapsed foldxt(mergewith(+), files |> Map(histfile))\n105.32969331","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"is much better.","category":"page"},{"location":"lecture_10/lecture/#Locks-/-lock-free-multi-threadding","page":"Lecture","title":"Locks / lock-free multi-threadding","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"Avoid locks.","category":"page"},{"location":"lecture_10/lecture/#Take-away-message","page":"Lecture","title":"Take away message","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"When deciding, what kind of paralelism to employ, consider following","category":"page"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"for tightly coupled computation over shared data, multi-threadding is more suitable due to non-existing sharing of data between processes\nbut if the computation requires frequent allocation and freeing of memery, or IO, separate processes are multi-suitable, since garbage collectors are independent between processes\nMaking all cores busy while achieving an ideally linear speedup is difficult and needs a lot of experience and knowledge. Tooling and profilers supporting debugging of parallel processes is not much developped.\nTransducers thrives for (almost) the same code to support thread- and process-based paralelism.","category":"page"},{"location":"lecture_10/lecture/#Materials","page":"Lecture","title":"Materials","text":"","category":"section"},{"location":"lecture_10/lecture/","page":"Lecture","title":"Lecture","text":"http://cecileane.github.io/computingtools/pages/notes1209.html\nhttps://lucris.lub.lu.se/ws/portalfiles/portal/61129522/julia_parallel.pdf\nhttp://igoro.com/archive/gallery-of-processor-cache-effects/\nhttps://www.csd.uwo.ca/~mmorenom/cs2101amoreno/ParallelcomputingwithJulia.pdf\nComplexity of thread schedulling https://www.youtube.com/watch?v=YdiZa0Y3F3c\nTapIR –- Teaching paralelism to Julia compiler https://www.youtube.com/watch?v=-JyK5Xpk7jE\nThreads: https://juliahighperformance.com/code/Chapter09.html\nProcesses: https://juliahighperformance.com/code/Chapter10.html\nAlan Adelman uses FLoops in https://www.youtube.com/watch?v=dczkYlOM2sg\nExamples: ?Heat equation? from [https://hpc.llnl.gov/training/tutorials/](introduction-parallel-computing-tutorial#Examples(https://hpc.llnl.gov/training/tutorials/)","category":"page"},{"location":"lecture_05/lab/#perf_lab","page":"Lab","title":"Lab 05: Practical performance debugging tools","text":"","category":"section"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Performance is crucial in scientific computing. There is a big difference if your experiments run one minute or one hour. We have already developed quite a bit of code, both in and outside packages, on which we are going to present some of the tooling that Julia provides for finding performance bottlenecks. Performance of your code or more precisely the speed of execution is of course relative (preference, expectation, existing code) and it's hard to find the exact threshold when we should start to care about it. When starting out with Julia, we recommend not to get bogged down by the performance side of things straightaway, but just design the code in the way that feels natural to you. As opposed to other languages Julia offers you to write the things \"like you are used to\" (depending on your background), e.g. for cycles are as fast as in C; vectorization of mathematical operators works the same or even better than in MATLAB, NumPy. ","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Once you have tested the functionality, you can start exploring the performance of your code by different means:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"manual code inspection - identifying performance gotchas (tedious, requires skill)\nautomatic code inspection - Jet.jl (probably not as powerful as in statically typed languages)\nbenchmarking - measuring variability in execution time, comparing with some baseline (only a statistic, non-specific)\nprofiling - measuring the execution time at \"each line of code\" (no easy way to handle advanced parallelism, ...)\nallocation tracking - similar to profiling but specifically looking at allocations (one sided statistic)","category":"page"},{"location":"lecture_05/lab/#Checking-type-stability","page":"Lab","title":"Checking type stability","text":"","category":"section"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Recall that type stable function is written in a way, that allows Julia's compiler to infer all the types of all the variables and produce an efficient native code implementation without the need of boxing some variables in a structure whose types is known only during runtime. Probably unbeknown to you we have already seen an example of type unstable function (at least in some situations) in the first lab, where we have defined the polynomial function:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"function polynomial(a, x)\n accumulator = 0\n for i in length(a):-1:1\n accumulator += x^(i-1) * a[i] # ! 1-based indexing for arrays\n end\n return accumulator\nend","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"The exact form of compiled code and also the type stability depends on the arguments of the function. Let's explore the following two examples of calling the function:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Integer number valued arguments","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"a = [-19, 7, -4, 6]\nx = 3\npolynomial(a, x)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Float number valued arguments","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"xf = 3.0\npolynomial(a, xf)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"The result they produce is the \"same\" numerically, however it differs in the output type. Though you have probably not noticed it, there should be a difference in runtime (assuming that you have run it once more after its compilation). It is probably a surprise to no one, that one of the methods that has been compiled is type unstable. This can be check with the @code_warntype macro:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"using InteractiveUtils #hide\n@code_warntype polynomial(a, x) # type stable\n@code_warntype polynomial(a, xf) # type unstable","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"We are getting a little ahead of ourselves in this lab, as understanding of these expressions is part of the future lecture. Anyway the output basically shows what the compiler thinks of each variable in the code, albeit for us in less readable form than the original code. The more red the color is of the type info the less sure the inferred type is. Our main focus should be on the return type of the function which is just at the start of the code with the keyword Body. In the first case the return type is an Int64, whereas in the second example the compiler is unsure whether the type is Float64 or Int64, marked as the Union type of the two. Fortunately for us this type instability can be fixed with a single line edit, but we will see later that it is not always the case.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"note: Type stability\nHaving a variable represented as Union of multiple types in a functions is a lesser evil than having Any, as we can at least enumerate statically the available options of functions to which to dynamically dispatch and in some cases there may be a low penalty.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Create a new function polynomial_stable, which is type stable and measure the difference in evaluation time. ","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"HINTS: ","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Ask for help on the one and zero keyword, which are often as a shorthand for these kind of functions.\nrun the function with the argument once before running @time or use @btime if you have BenchmarkTools readily available in your environment\nTo see some measurable difference with this simple function, a longer vector of coefficients may be needed.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"function polynomial_stable(a, x)\n accumulator = zero(x)\n for i in length(a):-1:1\n accumulator += x^(i-1) * a[i]\n end\n accumulator\nend","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"@code_warntype polynomial_stable(a, x) # type stable\n@code_warntype polynomial_stable(a, xf) # type stable","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"polynomial(a, xf) #hide\npolynomial_stable(a, xf) #hide\n@time polynomial(a, xf)\n@time polynomial_stable(a, xf)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Only really visible when evaluating multiple times.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"julia> using BenchmarkTools\n\njulia> @btime polynomial($a, $xf)\n 31.806 ns (0 allocations: 0 bytes)\n128.0\n\njulia> @btime polynomial_stable($a, $xf)\n 28.522 ns (0 allocations: 0 bytes)\n128.0","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Difference only a few nanoseconds.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Note: Recalling homework from lab 1. Adding zero also extends this function to the case of x being a matrix, see ? menu.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Code stability issues are something unique to Julia, as its JIT compilation allows it to produce code that contains boxed variables, whose type can be inferred during runtime. This is one of the reasons why interpreted languages are slow to run but fast to type. Julia's way of solving it is based around compiling functions for specific arguments, however in order for this to work without the interpreter, the compiler has to be able to infer the types.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"There are other problems (such as unnecessary allocations), that you can learn to spot in your code, however the code stability issues are by far the most commonly encountered problems among beginner users of Julia wanting to squeeze more out of it.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"note: Advanced tooling\nSometimes @code_warntype shows that the function's return type is unstable without any hints to the possible problem, fortunately for such cases a more advanced tools such as Cthuhlu.jl or JET.jl have been developed.","category":"page"},{"location":"lecture_05/lab/#Benchmarking-with-BenchmarkTools","page":"Lab","title":"Benchmarking with BenchmarkTools","text":"","category":"section"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"In the last exercise we have encountered the problem of timing of code to see, if we have made any progress in speeding it up. Throughout the course we will advertise the use of the BenchmarkTools package, which provides an easy way to test your code multiple times. In this lab we will focus on some advanced usage tips and gotchas that you may encounter while using it. ","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"There are few concepts to know in order to understand how the pkg works","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"evaluation - a single execution of a benchmark expression (default 1)\nsample - a single time/memory measurement obtained by running multiple evaluations (default 1e5)\ntrial - experiment in which multiple samples are gathered ","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"The result of a benchmark is a trial in which we collect multiple samples of time/memory measurements, which in turn may be composed of multiple executions of the code in question. This layering of repetition is required to allow for benchmarking code at different runtime magnitudes. Imagine having to benchmark operations which are faster than the act of measuring itself - clock initialization, dispatch of an operation and subsequent time subtraction.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"The number of samples/evaluations can be set manually, however most of the time won't need to know about them, due to an existence of a tuning method tune!, which tries to run the code once to estimate the correct ration of evaluation/samples. ","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"The most commonly used interface of Benchmarkools is the @btime macro, which returns an output similar to the regular @time macro however now aggregated over samples by taking their minimum (a robust estimator for the location parameter of the time distribution, should not be considered an outlier - usually the noise from other processes/tasks puts the results to the other tail of the distribution and some miraculous noisy speedups are uncommon. In order to see the underlying sampling better there is also the @benchmark macro, which runs in the same way as @btime, but prints more detailed statistics which are also returned in the Trial type instead of the actual code output.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"julia> @btime sum($(rand(1000)))\n 174.274 ns (0 allocations: 0 bytes)\n504.16236531044757\n\njulia> @benchmark sum($(rand(1000)))\nBenchmarkTools.Trial: 10000 samples with 723 evaluations.\n Range (min … max): 174.274 ns … 364.856 ns ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 174.503 ns ┊ GC (median): 0.00%\n Time (mean ± σ): 176.592 ns ± 7.361 ns ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n █▃ ▃▃ ▁\n █████████▇█▇█▇▇▇▇▇▆▆▇▆▆▆▆▆▆▅▆▆▅▅▅▆▆▆▆▅▅▅▅▅▅▅▅▆▅▅▅▄▄▅▅▄▄▅▃▅▅▄▅ █\n 174 ns Histogram: log(frequency) by time 206 ns <\n\n Memory estimate: 0 bytes, allocs estimate: 0.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"danger: Interpolation ~ `$` in BenchmarkTools\nIn the previous example we have used the interpolation signs $ to indicate that the code inside should be evaluated once and stored into a local variable. This allows us to focus only on the benchmarking of code itself instead of the input generation. A more subtle way where this is crops up is the case of using previously defined global variable, where instead of data generation we would measure also the type inference at each evaluation, which is usually not what we want. The following list will help you decide when to use interpolation.@btime sum($(rand(1000))) # rand(1000) is stored as local variable, which is used in each evaluation\n@btime sum(rand(1000)) # rand(1000) is called in each evaluation\nA = rand(1000)\n@btime sum($A) # global variable A is inferred and stored as local, which is used in each evaluation\n@btime sum(A) # global variable A has to be inferred in each evaluation","category":"page"},{"location":"lecture_05/lab/#Profiling","page":"Lab","title":"Profiling","text":"","category":"section"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Profiling in Julia is part of the standard library in the Profile module. It implements a fairly simple sampling based profiler, which in a nutshell asks at regular intervals, where the code execution is currently at. As a result we get an array of stacktraces (= chain of function calls), which allow us to make sense of where the execution spent the most time. The number of samples, that can be stored and the period in seconds can be checked after loading Profile into the session with the init() function.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"using Profile\nProfile.init()","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"The same function, but with keyword arguments, can be used to change these settings, however these settings are system dependent. For example on Windows, there is a known issue that does not allow to sample faster than at 0.003s and even on Linux based system this may not do much. There are some further caveat specific to Julia:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"When running profile from REPL, it is usually dominated by the interactive part which spawns the task and waits for it's completion.\nCode has to be run before profiling in order to filter out all the type inference and interpretation stuff. (Unless compilation is what we want to profile.)\nWhen the execution time is short, the sampling may be insufficient -> run multiple times.","category":"page"},{"location":"lecture_05/lab/#Polynomial-with-scalars","page":"Lab","title":"Polynomial with scalars","text":"","category":"section"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Let's look at our favorite polynomial function or rather it's type stable variant polynomial_stable under the profiling lens.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"# clear the last trace (does not have to be run on fresh start)\nProfile.clear()\n\n@profile polynomial_stable(a, xf)\n\n# text based output of the profiler\n# not shown here because it is not incredibly informative\nProfile.print()","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Unless the machine that you run the code on is really slow, the resulting output contains nothing or only some internals of Julia's interactive REPL. This is due to the fact that our polynomial function take only few nanoseconds to run. When we want to run profiling on something, that takes only a few nanoseconds, we have to repeatedly execute the function.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"function run_polynomial_stable(a, x, n) \n for _ in 1:n\n polynomial_stable(a, x)\n end\nend\n\na = rand(-10:10, 10) # using longer polynomial\n\nrun_polynomial_stable(a, xf, 10) #hide\nProfile.clear()\n@profile run_polynomial_stable(a, xf, Int(1e5))\nProfile.print()","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"In order to get more of a visual feel for profiling, there are packages that allow you to generate interactive plots or graphs. In this lab we will use ProfileSVG.jl, which does not require any fancy IDE or GUI libraries.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"@profview run_polynomial_stable(a, xf, Int(1e5))","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"(Image: poly_stable)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Let's compare this with the type unstable situation.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"First let's define the function that allows us to run the polynomial multiple times.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"function run_polynomial(a, x, n) \n for _ in 1:n\n polynomial(a, x)\n end\nend","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"@profview run_polynomial(a, xf, Int(1e5)) # clears the profile for us","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"(Image: poly_unstable)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Other options for viewing profiler outputs","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"ProfileView - close cousin of ProfileSVG, spawns GTK window with interactive FlameGraph\nVSCode - always imported @profview macro, flamegraphs (js extension required), filtering, one click access to source code \nPProf - serializes the profiler output to protobuffer and loads it in pprof web app, graph visualization of stacktraces","category":"page"},{"location":"lecture_05/lab/#horner","page":"Lab","title":"Applying fixes","text":"","category":"section"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"We have noticed that no matter if the function is type stable or unstable the majority of the computation falls onto the power function ^ and there is a way to solve this using a clever technique called Horner schema[1], which uses distributive and associative rules to convert the sum of powers into an incremental multiplication of partial results.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Rewrite the polynomial function using the Horner schema/method[1]. Moreover include the type stability fixes from polynomial_stable You should get more than 3x speedup when measured against the old implementation (measure polynomial against polynomial_stable.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"BONUS: Profile the new method and compare the differences in traces.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"[1]: Explanation of the Horner schema can be found on https://en.wikipedia.org/wiki/Horner%27s_method.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"function polynomial(a, x)\n accumulator = a[end] * one(x)\n for i in length(a)-1:-1:1\n accumulator = accumulator * x + a[i]\n end\n accumulator \nend","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Speed up:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"49ns -> 8ns ~ 6x on integer valued input \n59ns -> 8ns ~ 7x on real valued input","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"julia> @btime polynomial($a, $x)\n 8.008 ns (0 allocations: 0 bytes)\n97818\n\njulia> @btime polynomial_stable($a, $x)\n 49.173 ns (0 allocations: 0 bytes)\n97818\n\njulia> @btime polynomial($a, $xf)\n 8.008 ns (0 allocations: 0 bytes)\n97818.0\n\njulia> @btime polynomial_stable($a, $xf)\n 58.773 ns (0 allocations: 0 bytes)\n97818.0","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"These numbers will be different on different HW.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"BONUS: The profile trace does not even contain the calling of mathematical operators and is mainly dominated by the iteration utilities. In this case we had to increase the number of runs to 1e6 to get some meaningful trace.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"@profview run_polynomial(a, xf, Int(1e6))","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"(Image: poly_horner)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"","category":"page"},{"location":"lecture_05/lab/#Where-to-find-source-code?","page":"Lab","title":"Where to find source code?","text":"","category":"section"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"As most of Julia is written in Julia itself it is sometimes helpful to look inside for some details or inspiration. The code of Base and stdlib pkgs is located just next to Julia's installation in the ./share/julia subdirectory","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"./julia-1.6.2/\n ├── bin\n ├── etc\n │ └── julia\n ├── include\n │ └── julia\n │ └── uv\n ├── lib\n │ └── julia\n ├── libexec\n └── share\n ├── appdata\n ├── applications\n ├── doc\n │ └── julia # offline documentation (https://docs.julialang.org/en/v1/)\n └── julia\n ├── base # base library\n ├── stdlib # standard library\n └── test","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Other packages installed through Pkg interface are located in the .julia/ directory which is located in your $HOMEDIR, i.e. /home/$(user)/.julia/ on Unix based systems and /Users/$(user)/.julia/ on Windows.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"~/.julia/\n ├── artifacts\n ├── compiled\n ├── config # startup.jl lives here\n ├── environments\n ├── logs\n ├── packages # packages are here\n └── registries","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"If you are using VSCode, the paths visible in the REPL can be clicked through to he actual source code. Moreover in that environment the documentation is usually available upon hovering over code.","category":"page"},{"location":"lecture_05/lab/#Setting-up-benchmarks-to-our-liking","page":"Lab","title":"Setting up benchmarks to our liking","text":"","category":"section"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"In order to control the number of samples/evaluation and the amount of time given to a given benchmark, we can simply append these as keyword arguments to @btime or @benchmark in the following way","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"julia> @benchmark sum($(rand(1000))) evals=100 samples=10 seconds=1\nBenchmarkTools.Trial: 10 samples with 100 evaluations.\n Range (min … max): 174.580 ns … 188.750 ns ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 175.420 ns ┊ GC (median): 0.00%\n Time (mean ± σ): 176.585 ns ± 4.293 ns ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n █ \n █▅▁█▁▅▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▅ ▁\n 175 ns Histogram: frequency by time 189 ns <\n\n Memory estimate: 0 bytes, allocs estimate: 0.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"which runs the code repeatedly for up to 1s, where each of the 10 samples in the trial is composed of 10 evaluations. Setting up these parameters ourselves creates a more controlled environment in which performance regressions can be more easily identified.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Another axis of customization is needed when we are benchmarking mutable operations such as sort!, which sorts an array in-place. One way of achieving a consistent benchmark is by omitting the interpolation such as","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"julia> @benchmark sort!(rand(1000))\nBenchmarkTools.Trial: 10000 samples with 1 evaluation.\n Range (min … max): 27.250 μs … 95.958 μs ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 29.875 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 30.340 μs ± 2.678 μs ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n ▃▇█▄▇▄ \n ▁▁▁▂▃▆█████████▆▅▃▄▃▃▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▂\n 27.2 μs Histogram: frequency by time 41.3 μs <\n\n Memory estimate: 7.94 KiB, allocs estimate: 1.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"however now we are again measuring the data generation as well. A better way of doing such timing is using the built in setup keyword, into which you can put a code that has to be run before each sample and which won't be measured.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"julia> @benchmark sort!(y) setup=(y=rand(1000))\nBenchmarkTools.Trial: 10000 samples with 7 evaluations.\n Range (min … max): 7.411 μs … 25.869 μs ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 7.696 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 7.729 μs ± 305.383 ns ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n ▂▄▅▆█▇▇▆▄▃ \n ▁▁▁▁▂▂▃▄▅▆████████████▆▅▃▂▂▂▁▁▁▁▁▁▁▁▁▂▂▁▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▃\n 7.41 μs Histogram: frequency by time 8.45 μs <\n\n Memory estimate: 0 bytes, allocs estimate: 0.","category":"page"},{"location":"lecture_05/lab/#Ecosystem-debugging","page":"Lab","title":"Ecosystem debugging","text":"","category":"section"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Let's now apply what we have learned so far on the much bigger codebase of our Ecosystem.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"include(\"ecosystems/lab04/Ecosystem.jl\")\n\nfunction make_counter()\n n = 0\n counter() = n += 1\nend\n\nfunction create_world()\n n_grass = 1_000\n n_sheep = 40\n n_wolves = 4\n\n nextid = make_counter()\n\n World(vcat(\n [Grass(nextid()) for _ in 1:n_grass],\n [Sheep(nextid()) for _ in 1:n_sheep],\n [Wolf(nextid()) for _ in 1:n_wolves],\n ))\nend\nworld = create_world();\nnothing # hide","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Use @profview and @code_warntype to find the type unstable and slow parts of our simulation.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Precompile everything by running one step of our simulation and run the profiler like this:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"world_step!(world)\n@profview for i=1:100 world_step!(world) end","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"You should get a flamegraph similar to the one below: (Image: lab04-ecosystem)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Red bars indicate type instabilities. The bars stacked on top of them are high, narrow and not filling the whole width, indicating that the problem is pretty serious. In our case the worst offender is the filter method inside find_food and find_mate functions. In both cases the bars on top of it are narrow and not the full with, meaning that not that much time has been really spend working, but instead inferring the types in the function itself during runtime.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"As a reminder, this is the find_food function:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"# original\nfunction find_food(a::Animal, w::World)\n as = filter(x -> eats(a,x), w.agents |> values |> collect)\n isempty(as) ? nothing : sample(as)\nend","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Just from looking at that piece of code its not obvious what is the problem, however the red color indicates that the code may be type unstable. Let's see if that is the case by evaluation the function with some isolated inputs.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"using InteractiveUtils # hide\nw = Wolf(4000)\nfind_food(w, world)\n@code_warntype find_food(w, world)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Indeed we see that the return type is not inferred precisely but ends up being just the Union{Nothing, Agent}, this is better than straight out Any, which is the union of all types but still, julia has to do dynamic dispatch here, which is slow.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"The underlying issue here is that we are working array of type Vector{Agent}, where Agent is abstract, which does not allow the compiler to specialize the code for the loop body.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_05/lab/#Different-Ecosystem.jl-versions","page":"Lab","title":"Different Ecosystem.jl versions","text":"","category":"section"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"In order to fix the type instability in the Vector{Agent} we somehow have to rethink our world such that we get a vector of a concrete type. Optimally we would have one vector for each type of agent that populates our world. Before we completely redesign how our world works we can try a simple hack that might already improve things. Instead of letting julia figure our which types of agents we have (which could be infinitely many), we can tell the compiler at least that we have only three of them: Wolf, Sheep, and Grass.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"We can do this with a tiny change in the constructor of our World:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"function World(agents::Vector{<:Agent})\n ids = [a.id for a in agents]\n length(unique(ids)) == length(agents) || error(\"Not all agents have unique IDs!\")\n\n # construct Dict{Int,Union{Animal{Wolf}, Animal{Sheep}, Plant{Grass}}}\n # instead of Dict{Int,Agent}\n types = unique(typeof.(agents))\n dict = Dict{Int,Union{types...}}(a.id => a for a in agents)\n\n World(dict, maximum(ids))\nend","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Run the benchmark script provided here to get timings for find_food and reproduce! for the original ecosystem.\nRun the same benchmark with the modified World constructor.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Which differences can you observe? Why is one version faster than the other?","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"It turns out that with this simple change we can already gain a little bit of speed:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":" find_food reproduce!\nAnimal{A} & Dict{Int,Agent} 43.917 μs 439.666 μs\nAnimal{A} & Dict{Int,Union{...}} 12.208 μs 340.041 μs","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"We are gaining performance here because for small Unions of types the julia compiler can precompile the multiple available code branches. If we have just a Dict of Agents this is not possible.","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"This however, does not yet fix our type instabilities completely. We are still working with Unions of types which we can see again using @code_warntype:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"include(\"ecosystems/animal_S_world_DictUnion/Ecosystem.jl\")\n\nfunction make_counter()\n n = 0\n counter() = n += 1\nend\n\nfunction create_world()\n n_grass = 1_000\n n_sheep = 40\n n_wolves = 4\n\n nextid = make_counter()\n\n World(vcat(\n [Grass(nextid()) for _ in 1:n_grass],\n [Sheep(nextid()) for _ in 1:n_sheep],\n [Wolf(nextid()) for _ in 1:n_wolves],\n ))\nend\nworld = create_world();","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"using InteractiveUtils # hide\nw = Wolf(4000)\nfind_food(w, world)\n@code_warntype find_food(w, world)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"Julia still has to perform runtime dispatch on the small Union of Agents that is in our dictionary. To avoid this we could create a world that - instead of one plain dictionary - works with a tuple of dictionaries with one entry for each type of agent. Our world would then look like this:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"# pseudocode:\nworld ≈ (\n :Grass => Dict{Int, Plant{Grass}}(...),\n :Sheep => Dict{Int, Animal{Sheep}}(...),\n :Wolf => Dict{Int, Animal{Wolf}}(...)\n)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"In order to make this work we have to touch our ecosystem code in a number of places, mostly related to find_food and reproduce!. You can find a working version of the ecosystem with a world based on NamedTuples here. With this slightly more involved update we can gain another bit of speed:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":" find_food reproduce!\nAnimal{A} & Dict{Int,Agent} 43.917 μs 439.666 μs\nAnimal{A} & Dict{Int,Union{...}} 12.208 μs 340.041 μs\nAnimal{A} & NamedTuple{Dict,...} 8.639 μs 273.103 μs","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"And type stable code!","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"include(\"ecosystems/animal_S_world_NamedTupleDict/Ecosystem.jl\")\n\nfunction make_counter()\n n = 0\n counter() = n += 1\nend\n\nfunction create_world()\n n_grass = 1_000\n n_sheep = 40\n n_wolves = 4\n\n nextid = make_counter()\n\n World(vcat(\n [Grass(nextid()) for _ in 1:n_grass],\n [Sheep(nextid()) for _ in 1:n_sheep],\n [Wolf(nextid()) for _ in 1:n_wolves],\n ))\nend\nworld = create_world();","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"using InteractiveUtils # hide\nw = Wolf(4000)\nfind_food(w, world)\n@code_warntype find_food(w, world)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"The last optimization we can do is to move the Sex of our animals from a field into a parametric type. Our world would then look like below:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"# pseudocode:\nworld ≈ (\n :Grass => Dict{Int, Plant{Grass}}(...),\n :SheepFemale => Dict{Int, Animal{Sheep,Female}}(...),\n :SheepMale => Dict{Int, Animal{Sheep,Male}}(...),\n :WolfFemale => Dict{Int, Animal{Wolf,Female}}(...)\n :WolfMale => Dict{Int, Animal{Wolf,Male}}(...)\n)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"This should give us a lot of speedup in the reproduce! function, because we will not have to filter for the correct sex anymore, but instead can just pick the NamedTuple that is associated with the correct type of mate. Unfortunately, changing the type signature of Animal essentially means that we have to touch every line of code of our original ecosystem. However, the gain we get for it is quite significant:","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":" find_food reproduce!\nAnimal{A} & Dict{Int,Agent} 43.917 μs 439.666 μs\nAnimal{A} & Dict{Int,Union{...}} 12.208 μs 340.041 μs\nAnimal{A} & NamedTuple{Dict,...} 8.639 μs 273.103 μs\nAnimal{A,S} & NamedTuple{Dict,...} 7.823 μs 77.646 ns\nAnimal{A,S} & Dict{Int,Union{...}} 13.416 μs 6.436 ms","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"The implementation of the new version with two parametric types can be found here. The completely blue (i.e. type stable) @profview of this version of the Ecosystem is quite satisfying to see","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"(Image: neweco)","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"The same is true for the output of @code_warntype","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"include(\"ecosystems/animal_ST_world_NamedTupleDict/Ecosystem.jl\")\n\nfunction make_counter()\n n = 0\n counter() = n += 1\nend\n\nfunction create_world()\n n_grass = 1_000\n n_sheep = 40\n n_wolves = 4\n\n nextid = make_counter()\n\n World(vcat(\n [Grass(nextid()) for _ in 1:n_grass],\n [Sheep(nextid()) for _ in 1:n_sheep],\n [Wolf(nextid()) for _ in 1:n_wolves],\n ))\nend\nworld = create_world();\nnothing # hide","category":"page"},{"location":"lecture_05/lab/","page":"Lab","title":"Lab","text":"using InteractiveUtils # hide\nw = Wolf(4000)\nfind_food(w, world)\n@code_warntype find_food(w, world)","category":"page"},{"location":"lecture_05/lab/#Useful-resources","page":"Lab","title":"Useful resources","text":"","category":"section"},{"location":"lecture_12/lab/#lab12","page":"Lab","title":"Lab 12 - Differential Equations","text":"","category":"section"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"In this lab you will implement a simple solver for ordinary differential equations (ODE) as well as a less verbose version of the GaussNums that were introduced in the lecture.","category":"page"},{"location":"lecture_12/lab/#Euler-ODE-Solver","page":"Lab","title":"Euler ODE Solver","text":"","category":"section"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"In this first part you will implement your own, simple, ODE framwork (feel free to make it a package;) in which you can easily specify different ODE solvers. The API is heavily inspired by DifferentialEquations.jl, so if you ever need to use it, you will already have a feeling for how it works.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Like in the lecture, we want to be able to specify an ODE like below.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"function lotkavolterra(x,θ)\n α, β, γ, δ = θ\n x₁, x₂ = x\n\n dx₁ = α*x₁ - β*x₁*x₂\n dx₂ = δ*x₁*x₂ - γ*x₂\n\n [dx₁, dx₂]\nend\nnothing # hide","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"In the lecture we then solved it with a solve function that received all necessary arguments to fully specify how the ODE should be solved. The number of necessary arguments to solve can quickly become very large, so we will introduce a new API for solve which will always take only two arguments: solve(::ODEProblem, ::ODESolver). The solve function will only do some book-keeping and call the solver until the ODE is solved for the full tspan.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"The ODEProblem will contain all necessary parameters to fully specify the ODE that should be solved. In our case that is the function f that defines the ODE itself, initial conditions u0, ODE parameters θ, and the time domain of the ODE tspan:","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"struct ODEProblem{F,T<:Tuple{Number,Number},U<:AbstractVector,P<:AbstractVector}\n f::F\n tspan::T\n u0::U\n θ::P\nend","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"The solvers will all be subtyping the abstract type ODESolver. The Euler solver from the lecture will need one field dt which specifies its time step:","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"abstract type ODESolver end\n\nstruct Euler{T} <: ODESolver\n dt::T\nend","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Overload the call-method of Euler","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"(solver::Euler)(prob::ODEProblem, u, t)","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"such that calling the solver with an ODEProblem will perform one step of the Euler solver and return updated ODE varialbes u1 and the corresponding timestep t1.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"function (solver::Euler)(prob::ODEProblem, u, t)\n f, θ, dt = prob.f, prob.θ, solver.dt\n (u + dt*f(u,θ), t+dt)\nend","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"# define ODEProblem\nθ = [0.1,0.2,0.3,0.2]\nu0 = [1.0,1.0]\ntspan = (0.,100.)\nprob = ODEProblem(lotkavolterra,tspan,u0,θ)\n\n# run one solver step\nsolver = Euler(0.2)\n(u1,t1) = solver(prob,u0,0.)","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Implement the function solve(::ODEProblem,::ODESolver) which calls the solver as many times as are necessary to solve the ODE for the full time domain. solve should return a vector of timesteps and a corresponding matrix of variables.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"function solve(prob::ODEProblem, solver::ODESolver)\n t = prob.tspan[1]; u = prob.u0\n us = [u]; ts = [t]\n while t < prob.tspan[2]\n (u,t) = solver(prob, u, t)\n push!(us,u)\n push!(ts,t)\n end\n ts, reduce(hcat,us)\nend\nnothing # hide","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"You can load the true solution and compare it in a plot like below. The file that contains the correct solution is located here: lotkadata.jld2.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"using JLD2\nusing Plots\n\ntrue_data = load(\"lotkadata.jld2\")\n\np1 = plot(true_data[\"t\"], true_data[\"u\"][1,:], lw=4, ls=:dash, alpha=0.7, color=:gray, label=\"x Truth\")\nplot!(p1, true_data[\"t\"], true_data[\"u\"][2,:], lw=4, ls=:dash, alpha=0.7, color=:gray, label=\"y Truth\")\n\n(t,X) = solve(prob, Euler(0.2))\n\nplot!(p1,t,X[1,:], color=1, lw=3, alpha=0.8, label=\"x Euler\")\nplot!(p1,t,X[2,:], color=2, lw=3, alpha=0.8, label=\"y Euler\")","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"As you can see in the plot above, the Euler method quickly becomes quite inaccurate because we make a step in the direction of the tangent which inevitably leads us away from the perfect solution as shown in the plot below. (Image: euler)","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"In the homework you will implement a Runge-Kutta solver to get a much better accuracy with the same step size.","category":"page"},{"location":"lecture_12/lab/#Automating-GaussNums","page":"Lab","title":"Automating GaussNums","text":"","category":"section"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Next you will implement your own uncertainty propagation. In the lecture you have already seen the new number type that we need for this:","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"struct GaussNum{T<:Real} <: Real\n μ::T\n σ::T\nend","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise (tiny)
\n
","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Overload the ± (type: \\pm) symbol to define GaussNums like this: 2.0 ± 1.0. Additionally, overload the show function such that GaussNums are printed with the ± as well.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"±(x,y) = GaussNum(x,y)\nBase.show(io::IO, x::GaussNum) = print(io, \"$(x.μ) ± $(x.σ)\")","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Recall, that for a function f(bm x) with N inputs, the uncertainty sigma_f is defined by","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"sigma_f = sqrtsum_i=1^N left( fracdfdx_isigma_i right)^2","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"To make GaussNums work for arithmetic operations we could manually implement all desired functions as we started doing in the lecture. With the autodiff package Zygote we can automate the generation of these functions. In the next two exercises you will implement a macro @register that takes a function and defines the corresponding uncertainty propagation rule according to the equation above.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Implement a helper function uncertain(f, args::GaussNum...) which takes a function f and its args and returns the resulting GaussNum with an uncertainty defined by the equation above.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Hint: You can compute the gradient of a function with Zygote, for example:","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"using Zygote;\nf(x,y) = x*y;\nZygote.gradient(f, 2., 3.)","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"function uncertain(f, args::GaussNum...)\n μs = (x.μ for x in args)\n dfs = Zygote.gradient(f, μs...)\n\n σ² = mapreduce(+, zip(dfs,args)) do (df,x)\n (df * x.σ)^2\n end\n\n GaussNum(f(μs...), sqrt(σ²))\nend\nnothing # hide","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Now you can propagate uncertainties through any function like this:","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"x1 = 2.0 ± 2.0\nx2 = 2.0 ± 2.0\nuncertain(*, x1, x2)","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"You can verify the correctness of your implementation by comparing to the manual implementation from the lecture.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"For convenience, implement the macro @register which will define the uncertainty propagation rule for a given function. E.g. for the function * the macro should generate code like below","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Base.:*(args::GaussNum...) = uncertain(*, args...)","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Hint: If you run into trouble with module names of functions you can make use of","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"getmodule(f) = first(methods(f)).module\ngetmodule(*)","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"function _register(func::Symbol)\n mod = getmodule(eval(func))\n :($(mod).$(func)(args::GaussNum...) = uncertain($func, args...))\nend\n\nfunction _register(funcs::Expr)\n Expr(:block, map(_register, funcs.args)...)\nend\n\nmacro register(funcs)\n _register(funcs)\nend\nnothing # hide","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Lets register some arithmetic functions and see if they work","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"@register *\nx1 * x2\n@register - +\nx1 + x2\nx1 - x2","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"To finalize the definition of our new GaussNum we can define conversion and promotion rules such that we do not have to define things like","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"+(x::GaussNum, y::Real) = ...\n+(x::Real, y::GaussNum) = ...","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Define convert and promote_rules such that you can perform arithmetic operations on GaussNums and other Reals.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Hint: When converting a normal number to a GaussNum you can set the standard deviation to zero.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Base.convert(::Type{T}, x::T) where T<:GaussNum = x\nBase.convert(::Type{GaussNum{T}}, x::Number) where T = GaussNum(x,zero(T))\nBase.promote_rule(::Type{GaussNum{T}}, ::Type{S}) where {T,S} = GaussNum{T}\nBase.promote_rule(::Type{GaussNum{T}}, ::Type{GaussNum{T}}) where T = GaussNum{T}","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"You can test if everything works by adding/multiplying floats to GuassNums.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"1.0±1.0 + 2.0\n[1.0±0.001, 2.0]","category":"page"},{"location":"lecture_12/lab/#Propagating-Uncertainties-through-ODEs","page":"Lab","title":"Propagating Uncertainties through ODEs","text":"","category":"section"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"With our newly defined GaussNum we can easily propagate uncertainties through our ODE solvers without changing a single line of their code. Try it!","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"θ = [0.1±0.001, 0.2, 0.3, 0.2]\nu0 = [1.0±0.1, 1.0±0.1]\ntspan = (0.,100.)\ndt = 0.1\nprob = ODEProblem(lotkavolterra,tspan,u0,θ)\n\nt, X = solve(prob, Euler(0.1))","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Create a plot that takes a Vector{<:GaussNum} and plots the mean surrounded by the uncertainty.","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"mu(x::GaussNum) = x.μ\nsig(x::GaussNum) = x.σ\n\nfunction uncertainplot(t, x::Vector{<:GaussNum})\n p = plot(\n t,\n mu.(x) .+ sig.(x),\n xlabel = \"x\",\n ylabel = \"y\",\n fill = (mu.(x) .- sig.(x), :lightgray, 0.5),\n linecolor = nothing,\n primary = false, # no legend entry\n )\n \n # add the data to the plots\n plot!(p, t, mu.(X[1,:])) \n\n return p\nend","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"uncertainplot(t, X[1,:])","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"Unfortunately, with this approach, we would have to define things like uncertainplot! and kwargs to the function by hand. To make plotting GaussNums more pleasant we can make use of the @recipe macro from Plots.jl. It allows to define plot recipes for custom types (without having to depend on Plots.jl). Additionally, it makes it easiert to support all the different ways of creating plots (e.g. via plot or plot!, and with support for all keyword args) without having to overload tons of functions manually. If you want to read more about plot recipies in the docs of RecipesBase.jl. An example of a recipe for vectors of GaussNums could look like this:","category":"page"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"@recipe function plot(ts::AbstractVector, xs::AbstractVector{<:GaussNum})\n # you can set a default value for an attribute with `-->`\n # and force an argument with `:=`\n μs = [x.μ for x in xs]\n σs = [x.σ for x in xs]\n @series begin\n :seriestype := :path\n # ignore series in legend and color cycling\n primary := false\n linecolor := nothing\n fillcolor := :lightgray\n fillalpha := 0.5\n fillrange := μs .- σs\n # ensure no markers are shown for the error band\n markershape := :none\n # return series data\n ts, μs .+ σs\n end\n ts, μs\nend\n\n# now we can easily plot multiple things on to of each other\np1 = plot(t, X[1,:], label=\"x\", lw=3)\nplot!(p1, t, X[2,:], label=\"y\", lw=3)","category":"page"},{"location":"lecture_12/lab/#References","page":"Lab","title":"References","text":"","category":"section"},{"location":"lecture_12/lab/","page":"Lab","title":"Lab","text":"MIT18-330S12: Chapter 5\nRK2 derivation","category":"page"},{"location":"lecture_06/lecture/#introspection","page":"Lecture","title":"Language introspection","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"What is metaprogramming? A high-level code that writes high-level code by Steven Johnson.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Why do we need metaprogramming? ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"In general, we do not need it, as we can do whatever we need without it, but it can help us to remove a boilerplate code. \nAs an example, consider a @show macro, which just prints the name of the variable (or the expression) and its evaluation. This means that instead of writing println(\"2+exp(4) = \", 2+exp(4)) we can just write @show 2+exp(4).\nWe have seen @time or @benchmark, which is difficult to implement using normal function, since when you pass 2+exp(4) as a function argument, it will be automatically evaluated. You need to pass it as an expression, that can be evaluated within the function.\n@chain macro from Chain.jl improves over native piping |>\nWe have seen @forward macro implementing encapsulation.\nMacros are used to insert compilation directives not accessible through the syntax, e.g. @inbounds.\nA chapter on its own is definition of Domain Specific Languages.","category":"page"},{"location":"lecture_06/lecture/#Translation-stages-from-source-code-to-machine-code","page":"Lecture","title":"Translation stages from source code to machine code","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Julia (as any modern compiler) uses several stages to convert source code to native code. ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Julia's steps consists from [1] \nparse the source code to abstract syntax tree (AST) –- parse function \nexpand macros –- macroexpand function\nsyntax desugaring\nstamentize controlflow\nresolve scopes \ngenerate intermediate representation (IR) (\"goto\" form) –- expand or code_lowered functions\ntop-level evaluation, method sorting –- methods\ntype inference\ninlining and high level optimization –- code_typed\nLLVM IR generation –- code_llvm\nLLVM optimizer, native code generation –- code_native\nsteps 3-6 are done in inseparable stage\nJulia's IR is in static single assignment form","category":"page"},{"location":"lecture_06/lecture/#Example:-Fibonacci-numbers","page":"Lecture","title":"Example: Fibonacci numbers","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Consider for example a function computing the Fibonacci numbers[1]","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"function nextfib(n)\n\ta, b = one(n), one(n)\n\twhile b < n\n\t\ta, b = b, a + b\n\tend\n\treturn b\nend","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"[1]: From StackOverflow ","category":"page"},{"location":"lecture_06/lecture/#Parsing","page":"Lecture","title":"Parsing","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"The first thing the compiler does is that it will parse the source code (represented as a string) to the abstract syntax tree (AST). We can inspect the results of this stage as ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> parsed_fib = Meta.parse(\n\"\"\"\n\tfunction nextfib(n)\n\t\ta, b = one(n), one(n)\n\t\twhile b < n\n\t\t\ta, b = b, a + b\n\t\tend\n\t\treturn b\n\tend\"\"\")\n:(function nextfib(n)\n #= none:1 =#\n #= none:2 =#\n (a, b) = (one(n), one(n))\n #= none:3 =#\n while b < n\n #= none:4 =#\n (a, b) = (b, a + b)\n end\n #= none:6 =#\n return b\n end)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"AST is a tree representation of the source code, where the parser has already identified individual code elements function call, argument blocks, etc. The parsed code is represented by Julia objects, therefore it can be read and modified by Julia from Julia at your wish (this is what is called homo-iconicity of a language the itself being derived from Greek words homo- meaning \"the same\" and icon meaning \"representation\"). Using TreeView","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"using TreeView, TikzPictures\ng = tikz_representation(walk_tree(parsed_fib))\nTikzPictures.save(SVG(\"parsed_fib.svg\"), g)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"(Image: parsed_fib.svg)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"We can see that the AST is indeed a tree, with function being the root node (caused by us parsing a function). Each inner node represents a function call with children of the inner node being its arguments. An interesting inner node is the Block representing a sequence of statements, where we can also see information about lines in the source code inserted as comments. Lisp-like S-Expression can be printed using Meta.show_sexpr(parsed_fib).","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"(:function, (:call, :nextfib, :n), (:block,\n :(#= none:1 =#),\n :(#= none:2 =#),\n (:(=), (:tuple, :a, :b), (:tuple, (:call, :one, :n), (:call, :one, :n))),\n :(#= none:3 =#),\n (:while, (:call, :<, :b, :n), (:block,\n :(#= none:4 =#),\n (:(=), (:tuple, :a, :b), (:tuple, :b, (:call, :+, :a, :b)))\n )),\n :(#= none:6 =#),\n (:return, :b)\n ))","category":"page"},{"location":"lecture_06/lecture/#Expanding-macros","page":"Lecture","title":"Expanding macros","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"If we insert a \"useless\" macro to nextfib, for example @show b, we see that the macro is not expanded and it is left there as-is.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> parsed_fib = Meta.parse(\n\"\"\"\n function nextfib(n)\n a, b = one(n), one(n)\n while b < n\n a, b = b, a + b\n end\n @show b\n return b\n end\"\"\")\n:(function nextfib(n)\n #= none:1 =#\n #= none:2 =#\n (a, b) = (one(n), one(n))\n #= none:3 =#\n while b < n\n #= none:4 =#\n (a, b) = (b, a + b)\n #= none:5 =#\n end\n #= none:6 =#\n #= none:6 =# @show b\n #= none:7 =#\n return b\n end)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"and we can ask for expansion of the macro","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> macroexpand(Main, parsed_fib)\n:(function nextfib(n)\n #= none:1 =#\n #= none:2 =#\n (a, b) = (one(n), one(n))\n #= none:3 =#\n while b < n\n #= none:4 =#\n (a, b) = (b, a + b)\n #= none:5 =#\n end\n #= none:6 =#\n begin\n Base.println(\"b = \", Base.repr(begin\n #= show.jl:1047 =#\n local var\"#62#value\" = b\n end))\n var\"#62#value\"\n end\n #= none:7 =#\n return b\n end)","category":"page"},{"location":"lecture_06/lecture/#Lowering","page":"Lecture","title":"Lowering","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"The next stage is lowering, where AST is converted to Static Single Assignment Form (SSA), in which \"each variable is assigned exactly once, and every variable is defined before it is used\". Loops and conditionals are transformed into gotos and labels using a single unless/goto construct (this is not exposed in user-level Julia).","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_lowered nextfib(3)\nCodeInfo(\n1 ─ %1 = Main.one(n)\n│ %2 = Main.one(n)\n│ a = %1\n└── b = %2\n2 ┄ %5 = b < n\n└── goto #4 if not %5\n3 ─ %7 = b\n│ %8 = a + b\n│ a = %7\n│ b = %8\n└── goto #2\n4 ─ return b\n)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"or alternatively lowered_fib = Meta.lower(@__MODULE__, parsed_fib). ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"We can see that ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"compiler has introduced a lot of variables \nwhile (and for) loops has been replaced by a goto, where goto can be conditional","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"For inserted debugging information, there is an option to pass keyword argument debuginfo=:source. ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_lowered debuginfo=:source nextfib(3)\nCodeInfo(\n @ none:2 within `nextfib'\n1 ─ %1 = Main.one(n)\n│ %2 = Main.one(n)\n│ a = %1\n└── b = %2\n @ none:3 within `nextfib'\n2 ┄ %5 = b < n\n└── goto #4 if not %5\n @ none:4 within `nextfib'\n3 ─ %7 = b\n│ %8 = a + b\n│ a = %7\n│ b = %8\n└── goto #2\n @ none:6 within `nextfib'\n4 ─ return b\n)","category":"page"},{"location":"lecture_06/lecture/#Code-Typing","page":"Lecture","title":"Code Typing","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Code typing is the process in which the compiler attaches types to variables and tries to infer types of objects returned from called functions. If the compiler fails to infer the returned type, it will give the variable type Any, in which case a dynamic dispatch will be used in subsequent operations with the variable. Inspecting typed code is therefore important for detecting type instabilities (the process can be difficult and error prone, fortunately, new tools like Jet.jl may simplify this task). The output of typing can be inspected using @code_typed macro. If you know the types of function arguments, aka function signature, you can call directly function InteractiveUtils.code_typed(nextfib, (typeof(3),)).","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_typed nextfib(3)\nCodeInfo(\n1 ─ nothing::Nothing\n2 ┄ %2 = φ (#1 => 1, #3 => %6)::Int64\n│ %3 = φ (#1 => 1, #3 => %2)::Int64\n│ %4 = Base.slt_int(%2, n)::Bool\n└── goto #4 if not %4\n3 ─ %6 = Base.add_int(%3, %2)::Int64\n└── goto #2\n4 ─ return %2\n) => Int64","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"We can see that ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"some calls have been inlined, e.g. one(n) was replaced by 1 and the type was inferred as Int. \nThe expression b < n has been replaced with its implementation in terms of the slt_int intrinsic (\"signed integer less than\") and the result of this has been annotated with return type Bool. \nThe expression a + b has been also replaced with its implementation in terms of the add_int intrinsic and its result type annotated as Int64. \nAnd the return type of the entire function body has been annotated as Int64.\nThe phi-instruction %2 = φ (#1 => 1, #3 => %6) is a selector function, which returns the value depending on from which branch do you come from. In this case, variable %2 will have value 1, if the control was transfered from block #1 and it will have value copied from variable %6 if the control was transferreed from block 3 see also. The φ stands from phony variable.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"When we have called @code_lower, the role of types of arguments was in selecting - via multiple dispatch - the appropriate function body among different methods. Contrary in @code_typed, the types of parameters determine the choice of inner methods that need to be called (again with multiple dispatch). This process can trigger other optimization, such as inlining, as seen in the case of one(n) being replaced with 1 directly, though here this replacement is hidden in the φ function. ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Note that the same view of the code is offered by the @code_warntype macro, which we have seen in the previous lecture. The main difference from @code_typed is that it highlights type instabilities with red color and shows only unoptimized view of the code. You can view the unoptimized code with a keyword argument optimize=false:","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_typed optimize=false nextfib(3)\nCodeInfo(\n1 ─ %1 = Main.one(n)::Core.Const(1)\n│ %2 = Main.one(n)::Core.Const(1)\n│ (a = %1)::Core.Const(1)\n└── (b = %2)::Core.Const(1)\n2 ┄ %5 = (b < n)::Bool\n└── goto #4 if not %5\n3 ─ %7 = b::Int64\n│ %8 = (a + b)::Int64\n│ (a = %7)::Int64\n│ (b = %8)::Int64\n└── goto #2\n4 ─ return b\n) => Int64","category":"page"},{"location":"lecture_06/lecture/#Lowering-to-LLVM-IR","page":"Lecture","title":"Lowering to LLVM IR","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Julia uses the LLVM compiler framework to generate machine code. LLVM stands for low-level virtual machine and it is basis of many modern compilers (see wiki). We can see the textual form of code lowered to LLVM IR by invoking ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_llvm debuginfo=:source nextfib(3)\n; @ REPL[10]:1 within `nextfib'\ndefine i64 @julia_nextfib_890(i64 signext %0) {\ntop:\n br label %L2\n\nL2: ; preds = %L2, %top\n %value_phi = phi i64 [ 1, %top ], [ %1, %L2 ]\n %value_phi1 = phi i64 [ 1, %top ], [ %value_phi, %L2 ]\n; @ REPL[10]:3 within `nextfib'\n; ┌ @ int.jl:83 within `<'\n %.not = icmp slt i64 %value_phi, %0\n; └\n; @ REPL[10]:4 within `nextfib'\n; ┌ @ int.jl:87 within `+'\n %1 = add i64 %value_phi1, %value_phi\n; └\n; @ REPL[10]:3 within `nextfib'\n br i1 %.not, label %L2, label %L8\n\nL8: ; preds = %L2\n; @ REPL[10]:6 within `nextfib'\n ret i64 %value_phi\n}","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"LLVM code can be tricky to understand first, but one gets used to it. Notice references to the source code, which help with orientation. From the code above, we may infer","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"code starts by jumping to label L2, from where it reads values of two variables to two \"registers\" value_phi and value_phi1 (variables in LLVM starts with %). \nBoth registers are treated as int64 and initialized by 1. \n[ 1, %top ], [ %value_phi, %L2 ] means that values are initialized as 1 if you come from the label top and as value value_phi if you come from %2. This is the LLVM's selector (phony φ).\nicmp slt i64 %value_phi, %0 compares the variable %value_phi to the content of variable %0. Notice the anotation that we are comparing Int64.\n%1 = add i64 %value_phi1, %value_phi adds two variables %value_phi1 and %value_phi. Note again than we are using Int64 addition. \nbr i1 %.not, label %L2, label %L8 implements a conditional jump depending on the content of %.not variable. \nret i64 %value_phi returns the value indicating it to be an Int64.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"It is not expected you will be directly operating on the LLVM code, though there are libraries which does that. For example Enzyme.jl performs automatic differentiation of LLVM code, which has the benefit of being able to take a gradient through setindex!.","category":"page"},{"location":"lecture_06/lecture/#Producing-the-native-vode","page":"Lecture","title":"Producing the native vode","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Native code The last stage is generation of the native code, which Julia executes. The native code depends on the target architecture (e.g. x86, ARM). As in previous cases there is a macro for viewing the compiled code @code_native","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_native debuginfo=:source nextfib(3)\n\t.section\t__TEXT,__text,regular,pure_instructions\n; ┌ @ REPL[10]:1 within `nextfib'\n\tmovl\t$1, %ecx\n\tmovl\t$1, %eax\n\tnopw\t(%rax,%rax)\nL16:\n\tmovq\t%rax, %rdx\n\tmovq\t%rcx, %rax\n; │ @ REPL[10]:4 within `nextfib'\n; │┌ @ int.jl:87 within `+'\n\taddq\t%rcx, %rdx\n\tmovq\t%rdx, %rcx\n; │└\n; │ @ REPL[10]:3 within `nextfib'\n; │┌ @ int.jl:83 within `<'\n\tcmpq\t%rdi, %rax\n; │└\n\tjl\tL16\n; │ @ REPL[10]:6 within `nextfib'\n\tretq\n\tnopw\t%cs:(%rax,%rax)\n; └","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"and the output is used mainly for debugging / inspection. ","category":"page"},{"location":"lecture_06/lecture/#Looking-around-the-language","page":"Lecture","title":"Looking around the language","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Language introspection is very convenient for investigating, how things are implemented and how they are optimized / compiled to the native code.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"note: Reminder `@which`\nThough we have already used it quite a few times, recall the very useful macro @which, which identifies the concrete function called in a function call. For example @which mapreduce(sin, +, [1,2,3,4]). Note again that the macro here is a convenience macro to obtain types of arguments from the expression. Under the hood, it calls InteractiveUtils.which(function_name, (Base.typesof)(args...)). Funny enough, you can call @which InteractiveUtils.which(+, (Base.typesof)(1,1)) to inspect, where which is defined.","category":"page"},{"location":"lecture_06/lecture/#Broadcasting","page":"Lecture","title":"Broadcasting","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Broadcasting is not a unique concept in programming languages (Python/Numpy, MATLAB), however its implementation in Julia allows to easily fuse operations. For example ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"x = randn(100)\nsin.(x) .+ 2 .* cos.(x) .+ x","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"is all computed in a single loop. We can inspect, how this is achieved in the lowered code:","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> Meta.@lower sin.(x) .+ 2 .* cos.(x) .+ x\n:($(Expr(:thunk, CodeInfo(\n @ none within `top-level scope'\n1 ─ %1 = Base.broadcasted(sin, x)\n│ %2 = Base.broadcasted(cos, x)\n│ %3 = Base.broadcasted(*, 2, %2)\n│ %4 = Base.broadcasted(+, %1, %3)\n│ %5 = Base.broadcasted(+, %4, x)\n│ %6 = Base.materialize(%5)\n└── return %6\n))))","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Notice that we have not used the usual @code_lowered macro, because the statement to be lowered is not a function call. In these cases, we have to use @code_lowered, which can handle more general program statements. On these cases, we cannot use @which either, as that applies to function calls only as well.","category":"page"},{"location":"lecture_06/lecture/#Generators","page":"Lecture","title":"Generators","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Meta.@lower [x for x in 1:4]\n:($(Expr(:thunk, CodeInfo(\n @ none within `top-level scope'\n1 ─ %1 = 1:4\n│ %2 = Base.Generator(Base.identity, %1)\n│ %3 = Base.collect(%2)\n└── return %3\n))))","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"from which we see that the Generator is implemented using the combination of a Base.collect, which is a function collecting items of a sequence and Base.Generator(f,x), which implements an iterator, which applies function f on elements of x over which is being iterated. So an almost magical generators have instantly lost their magic.","category":"page"},{"location":"lecture_06/lecture/#Closures","page":"Lecture","title":"Closures","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"adder(x) = y -> y + x\n\njulia> @code_lowered adder(5)\nCodeInfo(\n1 ─ %1 = Main.:(var\"#8#9\")\n│ %2 = Core.typeof(x)\n│ %3 = Core.apply_type(%1, %2)\n│ #8 = %new(%3, x)\n└── return #8\n)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Core.apply_type is a one way of constructing an object briefly mentioned in description of object allocation.","category":"page"},{"location":"lecture_06/lecture/#The-effect-of-type-instability","page":"Lecture","title":"The effect of type-instability","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"struct Wolf\n\tname::String\n\tenergy::Int\nend\n\nstruct Sheep\n\tname::String\n\tenergy::Int\nend\n\nsound(wolf::Wolf) = println(wolf.name, \" has howled.\")\nsound(sheep::Sheep) = println(sheep.name, \" has baaed.\")\nstable_pack = (Wolf(\"1\", 1), Wolf(\"2\", 2), Sheep(\"3\", 3))\nunstable_pack = [Wolf(\"1\", 1), Wolf(\"2\", 2), Sheep(\"3\", 3)]\n@code_typed map(sound, stable_pack)\n@code_typed map(sound, unstable_pack)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\nCthulhu.jlCthulhu.jl is a library (tool) which simplifies the above, where we want to iteratively dive into functions called in some piece of code (typically some function). Cthulhu is different from te normal debugger, since the debugger is executing the code, while Cthulhu is just lower_typing the code and presenting functions (with type of arguments inferred).","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"using Cthulhu\n@descend map(sound, unstable_pack)","category":"page"},{"location":"lecture_06/lecture/#General-notes-on-metaprogramming","page":"Lecture","title":"General notes on metaprogramming","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"According to an excellent talk by Steven Johnson, you should use metaprogramming sparingly, because on one hand it's very powerful, but on the other it is generally difficult to read and it can lead to unexpected errors. Julia allows you to interact with the compiler at two different levels.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"After the code is parsed to AST, you can modify it directly or through macros.\nWhen SSA form is being typed, you can create custom functions using the concept of generated functions or directly emit intermediate representation.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"More functionalities are coming through the JuliaCompilerPlugins project, but we will not talk about them (yet), as they are not mature yet. ","category":"page"},{"location":"lecture_06/lecture/#What-is-Quotation?","page":"Lecture","title":"What is Quotation?","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"When we are doing metaprogramming, we need to somehow tell the compiler that the next block of code is not a normal block of code to be executed, but that it should be interpreted as data and in any sense it should not be evaluated. Quotation refers to exactly this syntactic sugar. In Julia, quotation is achieved either through :(...) or quote ... end.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Notice the difference between","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"1 + 1 ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"and ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":":(1 + 1)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"The type returned by the quotation depends on what is quoted. Observe the returned type of the following quoted code","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":":(1) |> typeof\n:(:x) |> typeof\n:(1 + x) |> typeof\nquote\n 1 + x\n x + 1\nend |> typeof","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"All of these snippets are examples of the quoted code, but only :(1 + x) and the quote block produce objects of type Expr. An interesting return type is the QuoteNode, which allows to insert piece of code which should contain elements that should not be interpolated. Most of the time, quoting returns Expressions.","category":"page"},{"location":"lecture_06/lecture/#Expressions","page":"Lecture","title":"Expressions","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Abstract Syntax Tree, the output of Julia's parser, is expressed using Julia's own datastructures, which means that you can freely manipulate it (and constructed) from the language itself. This property is called homoiconicity. Julia's compiler allows you to intercept compilation just after it has parsed the source code. Before we will take advantage of this power, we should get familiar with the strucute of the AST.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"The best way to inspect the AST is through the combination ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Meta.parse, which parses the source code to AST, \ndump which print AST to terminal, \neval which evaluates the AST within the current module.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Let's start by investigating a very simple statement 1 + 1, whose AST can be constructed either by Meta.parse(\"1 + 1\") or :(1 + 1) or quote 1+1 end (the last one includes also the line information metadata).","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> p = :(1+1)\n:(1 + 1)\n\njulia> typeof(p)\nExpr\n\njulia> dump(p)\nExpr\n head: Symbol call\n args: Array{Any}((3,))\n 1: Symbol +\n 2: Int64 1\n 3: Int64 1","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"The parsed code p is of type Expr, which according to Julia's help[2] is a type representing compound expressions in parsed julia code (ASTs). Each expression consists: of a head Symbol identifying which kind of expression it is (e.g. a call, for loop, conditional statement, etc.), and subexpressions (e.g. the arguments of a call). The subexpressions are stored in a Vector{Any} field called args. If you recall the figure above, where AST was represented as a tree, head gives each node the name name args are either some parameters of the node, or they point to childs of that node. The interpretation of the node depends on the its type stored in head (note that the word type used here is not in the Julia sense).","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"[2]: Help: Core.Expr","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"info: `Symbol` type\nWhen manipulations of expressions, we encounter the term Symbol. Symbol is the smallest atom from which the program (in AST representation) is built. It is used to identify an element in the language, for example variable, keyword or function name. Symbol is not a string, since string represents itself, whereas Symbol can represent something else (a variable). An illustrative example[3] goes as follows.julia> eval(:foo)\nERROR: foo not defined\n\njulia> foo = \"hello\"\n\"hello\"\n\njulia> eval(:foo)\n\"hello\"\n\njulia> eval(\"foo\")\n\"foo\"which shows that what the symbol :foo evaluates to depends on what – if anything – the variable foo is bound to, whereas \"foo\" always just evaluates to \"foo\".Symbols can be constructed either by prepending any string with : or by calling Symbol(...), which concatenates the arguments and create the symbol out of it. All of the following are symbolsjulia> :+\n:+\n\njulia> :function\n:function\n\njulia> :call\n:call\n\njulia> :x\n:x\n\njulia> Symbol(:Very,\"_twisted_\",:symbol,\"_definition\")\n:Very_twisted_symbol_definition\n\njulia> Symbol(\"Symbol with blanks\")\nSymbol(\"Symbol with blanks\")Symbols therefore allows us to operate with a piece of code without evaluating it.In Julia, symbols are \"interned strings\", which means that compiler attaches each string a unique identifier (integer), such that it can quickly compare them. Compiler uses Symbols exclusively and the important feature is that they can be quickly compared. This is why people like to use them as keys in Dict.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"[3]: An example provided by Stefan Karpinski.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"info: `Expr`essions\nFrom Julia's help[2]:Expr(head::Symbol, args...)A type representing compound expressions in parsed julia code (ASTs). Each expression consists of a head Symbol identifying which kind of expression it is (e.g. a call, for loop, conditional statement, etc.), and subexpressions (e.g. the arguments of a call). The subexpressions are stored in a Vector{Any} field called args. The expression is simple yet very flexible. The head Symbol tells how the expression should be treated and arguments provide all needed parameters. Notice that the structure is also type-unstable. This is not a big deal, since the expression is used to generate code, hence it is not executed repeatedly.","category":"page"},{"location":"lecture_06/lecture/#Construct-code-from-scratch","page":"Lecture","title":"Construct code from scratch","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Since Expr is a Julia structure, we can construct it manually as we can construct any other structure","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> Expr(:call, :+, 1 , 1) |> dump\nExpr\n head: Symbol call\n args: Array{Any}((3,))\n 1: Symbol +\n 2: Int64 1\n 3: Int64 1","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"yielding to the same structure as we have created above. Expressions can be evaluated using eval, as has been said. to programmatically evaluate our expression, let's do ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"e = Expr(:call, :+, 1, 1)\neval(e)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"We are free to use variables (identified by symbols) inside the expression ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"e = Expr(:call, :+, :x, 5)\neval(e)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"but unless they are not defined within the scope, the expression cannot produce a meaningful result","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"x = 3\neval(e)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":":(1 + sin(x)) == Expr(:call, :+, 1, Expr(:call, :sin, :x))","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Since the expression is a Julia structure, we are free to manipulate it. Let's for example substitutue x in e = :(x + 5) with 2x.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"e = :(x + 5)\ne.args = map(e.args) do a \n\ta == :x ? Expr(:call, :*, 2, :x) : a \nend","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"or ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"e = :(x + 5)\ne.args = map(e.args) do a \n\ta == :x ? :(2*x) : a \nend","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"and verify that the results are correct.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> dump(e)\nExpr\n head: Symbol call\n args: Array{Any}((3,))\n 1: Symbol +\n 2: Expr\n head: Symbol call\n args: Array{Any}((3,))\n 1: Symbol *\n 2: Int64 2\n 3: Symbol x\n 3: Int64 5\n\njulia> eval(e)\n11","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"As already mentioned, the manipulation of Expression can be arbitrary. In the above example, we have been operating directly on the arguments. But what if x would be deeper in the expression, as for example in 2(3 + x) + 2(2 - x)? We can implement the substitution using multiple dispatch as we would do when implementing any other function in Julia.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"substitue_x(x::Symbol) = x == :x ? :(2*x) : x\nsubstitue_x(e::Expr) = Expr(e.head, map(substitue_x, e.args)...)\nsubstitue_x(u) = u","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"which works as promised.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> e = :(2(3 + 2x) + 2(2 - x))\n:(2 * (3 + x) + 2 * (2 - x))\njulia> f = substitue_x(e)\n:(2 * (3 + 2x) + 2 * (2 - 2x))","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"or we can replace the sin function","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"replace_sin(x::Symbol) = x == :sin ? :cos : x\nreplace_sin(e::Expr) = Expr(e.head, map(replace_sin, e.args)...)\nreplace_sin(u) = u","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"replace_sin(:(1 + sin(x)))","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Sometimes, we want to operate on a block of code as opposed to single line expressions. Recall that a block of code is defined-quoted with quote ... end. Let us see how replace_x can handle the following example:","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"e = quote \n\ta = x + 3\n\tb = 2 - x\n\t2a + 2b\nend","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> replace_x(e) |> Base.remove_linenums!\nquote\n a = 2x + 3\n b = 2 - 2x\n 2a + 2b\nend\n\njulia> replace_x(e) |> eval\n10","category":"page"},{"location":"lecture_06/lecture/#Brittleness-of-code-manipulation","page":"Lecture","title":"Brittleness of code manipulation","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"When we are manipulating the AST or creating new expressions from scratch, there is no syntactic validation performed by the parser. It is therefore very easy to create AST which does not make any sense and cannot be compiled. We have already seen that we can refer to variables that were not defined yet (this makes perfect sense). The same goes with functions (which also makes a lot of sense).","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"e = :(g() + 5)\neval(e)\ng() = 5\neval(e)","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"But we can also introduce keywords which the language does not know. For example ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"e = Expr(:my_keyword, 1, 2, 3)\n:($(Expr(:my_keyword, 1, 2, 3)))\n\njulia> e.head\n:my_keyword\n\njulia> e.args\n3-element Vector{Any}:\n 1\n 2\n 3\n\njulia> eval(e)\nERROR: syntax: invalid syntax (my_keyword 1 2 3)\nStacktrace:\n [1] top-level scope\n @ none:1\n [2] eval\n @ ./boot.jl:360 [inlined]\n [3] eval(x::Expr)\n @ Base.MainInclude ./client.jl:446\n [4] top-level scope\n @ REPL[8]:1","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"notice that error is not related to undefined variable / function, but the invalid syntax. This also demonstrates the role of head in Expr. More on Julia AST can be found in the developer documentation.","category":"page"},{"location":"lecture_06/lecture/#Alternative-way-to-look-at-code","page":"Lecture","title":"Alternative way to look at code","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Meta.parse(\"x[3]\") |> dump","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"We can see a new Symbol ref as a head and the position 3 of variable x.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Meta.parse(\"(1,2,3)\") |> dump","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Meta.parse(\"1/2/3\") |> dump","category":"page"},{"location":"lecture_06/lecture/#Code-generation","page":"Lecture","title":"Code generation","text":"","category":"section"},{"location":"lecture_06/lecture/#Using-metaprogramming-in-inheritance-by-encapsulation","page":"Lecture","title":"Using metaprogramming in inheritance by encapsulation","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Recall that Julia (at the moment) does not support inheritance, therefore the only way to adopt functionality of some object and extend it is through encapsulation. Assuming we have some object T, we wrap that object into a new structure. Let's work out a concrete example, where we define the our own matrix. ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"struct MyMatrix{T} <: AbstractMatrix{T}\n\tx::Matrix{T}\nend","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Now, to make it useful, we should define all the usual methods, like size, length, getindex, setindex!, etc. We can list methods defined with Matrix as an argument methodswith(Matrix) (recall this will load methods that are defined with currently loaded libraries). Now, we would like to overload them. To minimize the written code, we can write","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"import Base: setindex!, getindex, size, length\nfor f in [:setindex!, :getindex, :size, :length]\n\teval(:($(f)(A::MyMatrix, args...) = $(f)(A.x, args...)))\nend","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"which we can verify now that it works as expected ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"julia> a = MyMatrix([1 2 ; 3 4])\n2×2 MyMatrix{Int64}:\n 1 2\n 3 4\n\njulia> a[4]\n4\n\njulia> a[3] = 0\n0\n\njulia> a\n2×2 MyMatrix{Int64}:\n 1 0\n 3 4","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"In this way, Julia acts as its own preprocessor. The above look can be equally written as ","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"for f in [:setindex!, :getindex, :size, :length]\n println(\"$(f)(A::MyMatrix, args...) = $(f)(A.x, args...)\")\nend","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"for f in [:setindex!, :getindex, :size, :length]\n\ts = \"Base.$(f)(A::MyMatrix, args...) = $(f)(A.x, args...)\"\n\tprintln(s)\n\teval(Meta.parse(s))\nend","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"for f in [:setindex!, :getindex, :size, :length] \t@eval f(A::MyMatrix, args...) = f(A.x, args...) end","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Notice that we have just hand-implemented parts of @forward macro from MacroTools, which does exactly this.","category":"page"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"","category":"page"},{"location":"lecture_06/lecture/#Resources","page":"Lecture","title":"Resources","text":"","category":"section"},{"location":"lecture_06/lecture/","page":"Lecture","title":"Lecture","text":"Introduction to Julia by Jeff Bezanson on first JuliaCon\nJulia's manual on metaprogramming\nDavid P. Sanders' workshop @ JuliaCon 2021 \nSteven Johnson's keynote talk @ JuliaCon 2019\nJames Nash's Is Julia Aot or JIT @ JuliaCon 2017\nAndy Ferris's workshop @ JuliaCon 2018\nFrom Macros to DSL by John Myles White \nNotes on JuliaCompilerPlugin","category":"page"},{"location":"lecture_09/lab/#Lab-09-Generated-Functions-and-IR","page":"Lab","title":"Lab 09 - Generated Functions & IR","text":"","category":"section"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"In this lab you will practice two advanced meta programming techniques:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Generated functions can help you write specialized code for certain kinds of parametric types with more flexibility and/or less code.\nIRTools.jl is a package that simplifies the manipulation of lowered and typed Julia code","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"using BenchmarkTools","category":"page"},{"location":"lecture_09/lab/#@generated-Functions","page":"Lab","title":"@generated Functions","text":"","category":"section"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Remember the three most important things about generated functions:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"They return quoted expressions (like macros).\nYou have access to type information of your input variables.\nThey have to be pure","category":"page"},{"location":"lecture_09/lab/#A-faster-polynomial","page":"Lab","title":"A faster polynomial","text":"","category":"section"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Throughout this course we have come back to our polynomial function which evaluates a polynomial based on the Horner schema. Below you can find a version of the function that operates on a tuple of length N.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"function polynomial(x, p::NTuple{N}) where N\n acc = p[N]\n for i in N-1:-1:1\n acc = x*acc + p[i]\n end\n acc\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Julia has its own implementation of this function called evalpoly. If we compare the performance of our polynomial and Julia's evalpoly we can observe a pretty big difference:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"x = 2.0\np = ntuple(float,20);\n\n@btime polynomial($x,$p)\n@btime evalpoly($x,$p)","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Julia's implementation uses a generated function which specializes on different tuple lengths (i.e. it unrolls the loop) and eliminates the (small) overhead of looping over the tuple. This is possible, because the length of the tuple is known during compile time. You can check the difference between polynomial and evalpoly yourself via the introspectionwtools you know - e.g. @code_lowered.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Rewrite the polynomial function as a generated function with the signature","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"genpoly(x::Number, p::NTuple{N}) where N","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Hints:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Remember that you have to generate a quoted expression inside your generated function, so you will need things like :($expr1 + $expr2).\nYou can debug the expression you are generating by omitting the @generated macro from your function.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"@generated function genpoly(x, p::NTuple{N}) where N\n ex = :(p[$N])\n for i in N-1:-1:1\n ex = :(x*$ex + p[$i])\n end\n ex\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"You should get the same performance as evalpoly (and as @poly from Lab 7 with the added convenience of not having to spell out all the coefficients in your code like: p = @poly 1 2 3 ...).","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"@btime genpoly($x,$p)","category":"page"},{"location":"lecture_09/lab/#Fast,-Static-Matrices","page":"Lab","title":"Fast, Static Matrices","text":"","category":"section"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Another great example that makes heavy use of generated functions are static arrays. A static array is an array of fixed size which can be implemented via an NTuple. This means that it will be allocated on the stack, which can buy us a lot of performance for smaller static arrays. We define a StaticMatrix{T,C,R,L} where the paramteric types represent the matrix element type T (e.g. Float32), the number of rows R, the number of columns C, and the total length of the matrix L=C*R (which we need to set the size of the NTuple).","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"struct StaticMatrix{T,R,C,L} <: AbstractArray{T,2}\n data::NTuple{L,T}\nend\n\nfunction StaticMatrix(x::AbstractMatrix{T}) where T\n (R,C) = size(x)\n StaticMatrix{T,R,C,C*R}(x |> Tuple)\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"As a warm-up, overload the Base functions size, length, getindex(x::StaticMatrix,i::Int), and getindex(x::Solution,r::Int,c::Int).","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Base.size(x::StaticMatrix{T,R,C}) where {T,R,C} = (R,C)\nBase.length(x::StaticMatrix{T,R,C,L}) where {T,R,C,L} = L\nBase.getindex(x::StaticMatrix, i::Int) = x.data[i]\nBase.getindex(x::StaticMatrix{T,R,C}, r::Int, c::Int) where {T,R,C} = x.data[R*(c-1) + r]","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"You can check if everything works correctly by comparing to a normal Matrix:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"x = rand(2,3)\nx[1,2]\na = StaticMatrix(x)\na[1,2]","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Overload matrix multiplication between two static matrices","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Base.:*(x::StaticMatrix{T,K,M},y::StaticMatrix{T,M,N})","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"with a generated function that creates an expression without loops. Below you can see an example for an expression that would be generated from multiplying two 2times 2 matrices.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":":(StaticMatrix{T,2,2,4}((\n (x[1,1]*y[1,1] + x[1,2]*y[2,1]),\n (x[2,1]*y[1,1] + x[2,2]*y[2,1]),\n (x[1,1]*y[1,2] + x[1,2]*y[2,2]),\n (x[2,1]*y[1,2] + x[2,2]*y[2,2])\n)))","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Hints:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"You can get output like above by leaving out the @generated in front of your overload.\nIt might be helpful to implement matrix multiplication in a normal Julia function first.\nYou can construct an expression for a sum of multiple elements like below.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Expr(:call,:+,1,2,3)\nExpr(:call,:+,1,2,3) |> eval","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"@generated function Base.:*(x::StaticMatrix{T,K,M}, y::StaticMatrix{T,M,N}) where {T,K,M,N}\n zs = map(Iterators.product(1:K, 1:N) |> collect |> vec) do (k,n)\n Expr(:call, :+, [:(x[$k,$m] * y[$m,$n]) for m=1:M]...)\n end\n z = Expr(:tuple, zs...)\n :(StaticMatrix{$T,$K,$N,$(K*N)}($z))\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"You can check that your matrix multiplication works by multiplying two random matrices. Which one is faster?","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"a = rand(2,3)\nb = rand(3,4)\nc = StaticMatrix(a)\nd = StaticMatrix(b)\na*b\nc*d","category":"page"},{"location":"lecture_09/lab/#OptionalArgChecks.jl","page":"Lab","title":"OptionalArgChecks.jl","text":"","category":"section"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"The package OptionalArgChecks.jl makes is possible to add checks to a function which can then be removed by calling the function with the @skip macro. For example, we can check if the input to a function f is an even number","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"function f(x::Number)\n iseven(x) || error(\"Input has to be an even number!\")\n x\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"If you are doing more involved argument checking it can take quite some time to perform all your checks. However, if you want to be fast and are completely sure that you are always passing in the correct inputs to your function, you might want to remove them in some cases. Hence, we would like to transform the IR of the function above","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"using IRTools\nusing IRTools: @code_ir\n@code_ir f(1)","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"To some thing like this","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"transformed_f(x::Number) = x\n@code_ir transformed_f(1)","category":"page"},{"location":"lecture_09/lab/#Marking-Argument-Checks","page":"Lab","title":"Marking Argument Checks","text":"","category":"section"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"As a first step we will implement a macro that marks checks which we might want to remove later by surrounding it with :meta expressions. This will make it easy to detect which part of the code can be removed. A :meta expression can be created like this","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Expr(:meta, :mark_begin)\nExpr(:meta, :mark_end)","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"and they will not be evaluated but remain in your IR. To surround an expression with two meta expressions you can use a :block expression:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"ex = :(x+x)\nExpr(:block, :(print(x)), ex, :(print(x)))","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Define a macro @mark that takes an expression and surrounds it with two meta expressions marking the beginning and end of a check. Hints","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Defining a function _mark(ex::Expr) which manipulates your expressions can help a lot with debugging your macro.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"function _mark(ex::Expr)\n return Expr(\n :block,\n Expr(:meta, :mark_begin),\n esc(ex),\n Expr(:meta, :mark_end),\n )\nend\n\nmacro mark(ex)\n _mark(ex)\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"If you have defined a _mark function you can test that it works like this","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"_mark(:(println(x)))","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"The complete macro should work like below","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"function f(x::Number)\n @mark @show x\n x\nend;\n@code_ir f(2)\nf(2)","category":"page"},{"location":"lecture_09/lab/#Removing-Argument-Checks","page":"Lab","title":"Removing Argument Checks","text":"","category":"section"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Now comes tricky part for which we need IRTools.jl. We want to remove all lines that are between our two meta blocks. You can delete the line that corresponds to a certain variable with the delete! and the var functions. E.g. deleting the line that defines variable %4 works like this:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"using IRTools: delete!, var\n\nir = @code_ir f(2)\ndelete!(ir, var(4))","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Write a function skip(ir::IR) which deletes all lines between the meta expression :mark_begin and :mark_end.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Hints You can check whether a statement is one of our meta expressions like this:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"ismarkbegin(e::Expr) = Meta.isexpr(e,:meta) && e.args[1]===:mark_begin\nismarkbegin(Expr(:meta,:mark_begin))","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"ismarkend(e::Expr) = Meta.isexpr(e,:meta) && e.args[1]===:mark_end\n\nfunction skip(ir)\n delete_line = false\n for (x,st) in ir\n isbegin = ismarkbegin(st.expr)\n isend = ismarkend(st.expr)\n\n if isbegin\n delete_line = true\n end\n \n if delete_line\n delete!(ir,x)\n end\n\n if isend\n delete_line = false\n end\n end\n ir\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Your function should transform the IR of f like below.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"ir = @code_ir f(2)\nir = skip(ir)\nusing IRTools: func\nfunc(ir)(nothing, 2) # no output from @show!","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"However, if we have a slightly more complicated IR like below this version of our function will fail. It actually fails so badly that running func(ir)(nothing,2) after skip will cause the build of this page to crash, so we cannot show you the output here ;).","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"function g(x)\n @mark iseven(x) && println(\"even\")\n x\nend\n\nir = @code_ir g(2)\nir = skip(ir)","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"The crash is due to %4 not existing anymore. We can fix this by emptying the block in which we found the :mark_begin expression and branching to the block that contains :mark_end (unless they are in the same block already). If some (branching) code in between remained, it should then be removed by the compiler because it is never reached.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Use the functions IRTools.block, IRTools.branches, IRTools.empty!, and IRTools.branch! to modify skip such that it also empties the :mark_begin block, and adds a branch to the :mark_end block (unless they are the same block).","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Hints","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"block gets you the block of IR in which a given variable is if you call e.g. block(ir,var(4)).\nempty! removes all statements in a block.\nbranches returns all branches of a block.\nbranch!(a,b) creates a branch from the end of block a to the beginning block b","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"using IRTools: block, branch!, empty!, branches\nfunction skip(ir)\n delete_line = false\n orig = nothing\n for (x,st) in ir\n isbegin = ismarkbegin(st.expr)\n isend = ismarkend(st.expr)\n\n if isbegin\n delete_line = true\n end\n\n # this part is new\n if isbegin\n orig = block(ir,x)\n elseif isend\n dest = block(ir,x)\n if orig != dest\n empty!(branches(orig))\n branch!(orig,dest)\n end\n end\n \n if delete_line\n delete!(ir,x)\n end\n\n if isend\n delete_line = false\n end\n end\n ir\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"The result should construct valid IR for our g function.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"g(2)\nir = @code_ir g(2)\nir = skip(ir)\nfunc(ir)(nothing,2)","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"And it should not break when applying it to f.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"f(2)\nir = @code_ir f(2)\nir = skip(ir)\nfunc(ir)(nothing,2)","category":"page"},{"location":"lecture_09/lab/#Recursively-Removing-Argument-Checks","page":"Lab","title":"Recursively Removing Argument Checks","text":"","category":"section"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"The last step to finalize the skip function is to make it work recursively. In the current version we can handle functions that contain @mark statements, but we are not going any deeper than that. Nested functions will not be touched:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"foo(x) = bar(baz(x))\n\nfunction bar(x)\n @mark iseven(x) && println(\"The input is even.\")\n x\nend\n\nfunction baz(x)\n @mark x<0 && println(\"The input is negative.\")\n x\nend\n\nnothing # hide","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"ir = @code_ir foo(-2)\nir = skip(ir)\nfunc(ir)(nothing,-2)","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"For recursion we will use the macro IRTools.@dynamo which will make recursion of our skip function a lot easier. Additionally, it will save us from all the func(ir)(nothing, args...) statements. To use @dynamo we have to slightly modify how we call skip:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"@dynamo function skip(args...)\n ir = IR(args...)\n \n # same code as before that modifies `ir`\n # ...\n\n return ir\nend\n\n# now we can call `skip` like this\nskip(f,2)","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Now we can easily use skip in recursion, because we can just pass the arguments of an expression like this:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"using IRTools: xcall\n\nfor (x,st) in ir\n isexpr(st.expr,:call) || continue\n ir[x] = xcall(skip, st.expr.args...)\nend","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"The function xcall will create an expression that calls skip with the given arguments and returns Expr(:call, skip, args...). Note that you can modify expressions of a given variable in the IR via setindex!.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Modify skip such that it uses @dynamo and apply it recursively to all :call expressions that you ecounter while looping over the given IR. This will dive all the way down to Core.Builtins and Core.IntrinsicFunctions which you cannot maniuplate anymore (because they are written in C). You have to end the recursion at these places which can be done via multiple dispatch of skip on Builtins and IntrinsicFunctions.","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Once you are done with this you can also define a macro such that you can conveniently call @skip with an expression:","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"skip(f,2)\n@skip f(2)","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"using IRTools: @dynamo, xcall, IR\n\n# this is where we want to stop recursion\nskip(f::Core.IntrinsicFunction, args...) = f(args...)\nskip(f::Core.Builtin, args...) = f(args...)\n\n@dynamo function skip(args...)\n ir = IR(args...)\n delete_line = false\n orig = nothing\n for (x,st) in ir\n isbegin = ismarkbegin(st.expr)\n isend = ismarkend(st.expr)\n\n if isbegin\n delete_line = true\n end\n\n if isbegin\n orig = block(ir,x)\n elseif isend\n dest = block(ir,x)\n if orig != dest\n empty!(branches(orig))\n branch!(orig,dest)\n end\n end\n \n if delete_line\n delete!(ir,x)\n end\n\n if isend\n delete_line = false\n end\n\n # this part is new\n if haskey(ir,x) && Meta.isexpr(st.expr,:call)\n ir[x] = xcall(skip, st.expr.args...)\n end\n end\n return ir\nend\n\nmacro skip(ex)\n ex.head == :call || error(\"Input expression has to be a `:call`.\")\n return xcall(skip, ex.args...)\nend\nnothing # hide","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"@code_ir foo(2)\n@code_ir skip(foo,2)\nfoo(-2)\nskip(foo,-2)\n@skip foo(-2)","category":"page"},{"location":"lecture_09/lab/#References","page":"Lab","title":"References","text":"","category":"section"},{"location":"lecture_09/lab/","page":"Lab","title":"Lab","text":"Static matrices with @generated functions blog post\nOptionalArgChecks.jl\nIRTools Dynamo","category":"page"},{"location":"lecture_02/hw/#Homework-2:-Predator-Prey-Agents","page":"Homework","title":"Homework 2: Predator-Prey Agents","text":"","category":"section"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"In this lab you will continue working on your agent simulation. If you did not manage to finish the homework, do not worry, you can use this script which contains all the functionality we developed in the lab.","category":"page"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"projdir = dirname(Base.active_project())\ninclude(joinpath(projdir,\"src\",\"lecture_02\",\"Lab02Ecosystem.jl\"))","category":"page"},{"location":"lecture_02/hw/#How-to-submit?","page":"Homework","title":"How to submit?","text":"","category":"section"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"Put all your code (including your or the provided solution of lab 2) in a script named hw.jl. Zip only this file (not its parent folder) and upload it to BRUTE. Your file cannot contain any package dependencies. For example, having a using Plots in your code will cause the automatic evaluation to fail.","category":"page"},{"location":"lecture_02/hw/#Counting-Agents","page":"Homework","title":"Counting Agents","text":"","category":"section"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"To monitor the different populations in our world we need a function that counts each type of agent. For Animals we simply have to count how many of each type are currently in our World. In the case of Plants we will use the fraction of size(plant)/max_size(plant) as a measurement quantity.","category":"page"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"
\n
Compulsory Homework (2 points)
\n
","category":"page"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"Implement a function agent_count that can be called on a single Agent and returns a number between (01) (i.e. always 1 for animals; and size(plant)/max_size(plant) for plants).\nAdd a method for a vector of agents Vector{<:Agent} which sums all agent counts.\nAdd a method for a World which returns a dictionary that contains pairs of Symbols and the agent count like below:","category":"page"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"agent_count(p::Plant) = p.size / p.max_size\nagent_count(::Animal) = 1\nagent_count(as::Vector{<:Agent}) = sum(agent_count,as)\n\nfunction agent_count(w::World)\n function op(d::Dict,a::A) where A<:Agent\n n = nameof(A)\n if n in keys(d)\n d[n] += agent_count(a)\n else\n d[n] = agent_count(a)\n end\n return d\n end\n foldl(op, w.agents |> values |> collect, init=Dict{Symbol,Real}())\nend","category":"page"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"grass1 = Grass(1,5,5);\nagent_count(grass1)\n\ngrass2 = Grass(2,1,5);\nagent_count([grass1,grass2]) # one grass is fully grown; the other only 20% => 1.2\n\nsheep = Sheep(3,10.0,5.0,1.0,1.0);\nwolf = Wolf(4,20.0,10.0,1.0,1.0);\nworld = World([grass1, grass2, sheep, wolf]);\nagent_count(world)","category":"page"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"Hint: You can get the name of a type by using the nameof function:","category":"page"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"nameof(Grass)","category":"page"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"Use as much dispatch as you can! ;)","category":"page"},{"location":"lecture_02/hw/","page":"Homework","title":"Homework","text":"
","category":"page"},{"location":"lecture_01/hw/#Homework-1:-Extending-polynomial-the-other-way","page":"Homework","title":"Homework 1: Extending polynomial the other way","text":"","category":"section"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"
\n
Homework (2 points)
\n
","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"Extend the original polynomial function to the case where x is a square matrix. Create a function called circlemat, that returns nxn matrix A(n) with the following elements","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"leftA(n)right_ij = \nbegincases\n 1 textif (i = j-1 land j 1) lor (i = n land j=1) \n 1 textif (i = j+1 land j n) lor (i = 1 land j=n) \n 0 text otherwise\nendcases","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"and evaluate the polynomial","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"f(A) = I + A + A^2 + A^3","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":", at point A = A(10).","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"HINTS for matrix definition: You can try one of these options:","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"create matrix with all zeros with zeros(n,n), use two nested for loops going in ranges 1:n and if condition with logical or ||, and && \nemploy array comprehension with nested loops [expression for i in 1:n, j in 1:n] and ternary operator condition ? true branch : false","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"HINTS for polynomial extension:","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"extend the original example (one with for-loop) to initialize the accumulator variable with matrix of proper size (use size function to get the dimension), using argument typing for x is preferred to distinguish individual implementations <: AbstractMatrix","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"or","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"test later defined polynomial methods, that may work out of the box","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"
\n","category":"page"},{"location":"lecture_01/hw/#How-to-submit?","page":"Homework","title":"How to submit?","text":"","category":"section"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"Put all the code for the exercise above in a file called hw.jl and upload it to BRUTE. If you have any questions, write an email to one of the lab instructors of the course.","category":"page"},{"location":"lecture_01/hw/#Voluntary","page":"Homework","title":"Voluntary","text":"","category":"section"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"
\n
Exercise (voluntary)
\n
","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"Install GraphRecipes and Plots packages into the environment defined during the lecture and figure out, how to plot the graph defined by adjacency matrix A from the homework.","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"HINTS:","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"There is help command inside the the pkg mod of the REPL. Type ? add to find out how to install a package. Note that both pkgs are registered.\nFollow a guide in the Plots pkg's documentation, which is accessible through docs icon on top of the README in the GitHub repository. Direct link.","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"Activate the environment in pkg mode, if it is not currently active.","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"pkg> activate .","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"Installing pkgs is achieved using the add command. Running ] ? add returns a short piece of documentation for this command:","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"pkg> ? add\n[...]\n Examples\n\n pkg> add Example # most commonly used for registered pkgs (installs usually the latest release)\n pkg> add Example@0.5 # install with some specific version (realized through git tags)\n pkg> add Example#master # install from master branch directly\n pkg> add Example#c37b675 # install from specific git commit\n pkg> add https://github.com/JuliaLang/Example.jl#master # install from specific remote repository (when pkg is not registered)\n pkg> add git@github.com:JuliaLang/Example.jl.git # same as above but using the ssh protocol\n pkg> add Example=7876af07-990d-54b4-ab0e-23690620f79a # when there are multiple pkgs with the same name","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"As the both Plots and GraphRecipes are registered and we don't have any version requirements, we will use the first option.","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"pkg> add Plots\npkg> add GraphRecipes","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"This process downloads the pkgs and triggers some build steps, if for example some binary dependencies are needed. The process duration depends on the \"freshness\" of Julia installation and the size of each pkg. With Plots being quite dependency heavy, expect few minutes. After the installation is complete we can check the updated environment with the status command.","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"pkg> status","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"The plotting itself as easy as calling the graphplot function on our adjacency matrix.","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"using GraphRecipes, Plots\nA = [ 0 1 0 0 0 0 0 0 0 1; 1 0 1 0 0 0 0 0 0 0; 0 1 0 1 0 0 0 0 0 0; 0 0 1 0 1 0 0 0 0 0; 0 0 0 1 0 1 0 0 0 0; 0 0 0 0 1 0 1 0 0 0; 0 0 0 0 0 1 0 1 0 0; 0 0 0 0 0 0 1 0 1 0; 0 0 0 0 0 0 0 1 0 1; 1 0 0 0 0 0 0 0 1 0]# hide\ngraphplot(A)","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"graphplot(A) #hide","category":"page"},{"location":"lecture_01/hw/","page":"Homework","title":"Homework","text":"

","category":"page"},{"location":"lecture_04/lecture/#pkg_lecture","page":"Lecture","title":"Package development","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Organization of the code is more important with the increasing size of the project and the number of contributors and users. Moreover, it will become essential when different codebases are expected to be combined and reused. ","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Julia was designed from the beginning to encourage code reuse across different codebases as possible\nJulia ecosystem lives on a namespace. From then, it builds projects and environments.","category":"page"},{"location":"lecture_04/lecture/#Namespaces-and-modules","page":"Lecture","title":"Namespaces and modules","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Namespace logically separate fragments of source code so that they can be developed independently without affecting each other. If I define a function in one namespace, I will still be able to define another function in a different namespace even though both functions have the same name.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"prevents confusion when common words are used in different meaning:\nToo general name of functions \"create\", \"extend\", \"loss\", \nor data \"X\", \"y\" (especially in mathematics, think of π)\nmay not be an issue if used with different types\nModules is Julia syntax for a namespace","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Example:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"module MySpace\nfunction test1()\n println(\"test1\")\nend\nfunction test2()\n println(\"test2\")\nend\nexport test1\n#include(\"filename.jl\")\nend","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Function include copies content of the file to this location (will be part of the module).","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Creates functions:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"MySpace.test1\nMySpace.test2","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"For easier manipulation, these functions can be \"exported\" to be exposed to the outer world (another namespace).","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Keyword: using exposes the exported functions and structs:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"using .MySpace","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"The dot means that the module was defined in this scope.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Keyword: import imports function with availability to redefine it.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Combinations:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"usecase results\nusing MySpace MySpace.test1\n MySpace.test2\n test1\nusing MySpace: test1 test1\nimport MySpace MySpace.test1*\n MySpace.test2*\nimport MySpace: test1 test1*\nimport MySpace: test2 test2*","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"symbol \"*\" denotes functions that can be redefined","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":" using .MySpace: test1\n test1()=println(\"new test\")\n import .MySpace: test1\n test1()=println(\"new test\")","category":"page"},{"location":"lecture_04/lecture/#Conflicts:","page":"Lecture","title":"Conflicts:","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"When importing/using functions with name that is already imported/used from another module:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"the imported functions/structs are invalidated. \nboth functions has to be acessed by their full names.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Resolution:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"It may be easier to cherry pick only the functions we need (rather than importing all via using)\nrename some function using keyword as\nimport MySpace2: test1 as t1","category":"page"},{"location":"lecture_04/lecture/#Submodules","page":"Lecture","title":"Submodules","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Modules can be used or included within other modules:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"module A\n a=1;\nend\nmodule B\n module C\n c = 2\n end\n b = C.c # you can read from C (by reference)\n using ..A: a\n # a= b # but not write to A\nend;","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"REPL of Julia is a module called \"Main\". ","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"modules are not copied, but referenced, i.e. B.b===B.C.c\nincluding one module twice (from different packages) is not a problem\nJulia 1.9 has the ability to change the contextual module in the REPL: REPL.activate(TestPackage)","category":"page"},{"location":"lecture_04/lecture/#Revise.jl","page":"Lecture","title":"Revise.jl","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"The fact that Julia can redefine a function in a Module by importing it is used by package Revise.jl to synchronize REPL with a module or file.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"So far, we have worked in REPL. If you have a file that is loaded and you want to modify it, you would need to either:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"reload the whole file, or\ncopy the changes to REPL","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Revise.jl does the latter automatically.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Example demo:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"using Revise.jl\nincludet(\"example.jl\")","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Works with: ","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"any package loaded with import or using, \nscript loaded with includet, \nBase julia itself (with Revise.track(Base))\nstandard libraries (with, e.g., using Unicode; Revise.track(Unicode))","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Does not work with variables!","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"How it works: monitors source code for changes and then does:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"for def in setdiff(oldexprs, newexprs)\n # `def` is an expression that defines a method.\n # It was in `oldexprs`, but is no longer present in `newexprs`--delete the method.\n delete_methods_corresponding_to_defexpr(mod, def)\nend\nfor def in setdiff(newexprs, oldexprs)\n # `def` is an expression for a new or modified method. Instantiate it.\n Core.eval(mod, def)\nend","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"See Modern Julia Workflows for more hints","category":"page"},{"location":"lecture_04/lecture/#Namespaces-and-scoping","page":"Lecture","title":"Namespaces & scoping","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Every module introduces a new global scope. ","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Global scope\nNo variable or function is expected to exist outside of it\nEvery module is equal to a global scope (no single \"global\" exists)\nThe REPL has a global module called Main\nLocal scope\nVariables in Julia do not need to be explicitly declared, they are created by assignments: x=1. \nIn local scope, the compiler checks if variable x does not exist outside. We have seen:\nx=1\nf(y)=x+y\nThe rules for local scope determine how to treat assignment of x. If local x exists, it is used, if it does not:\nin hard scope: new local x is created\nin soft scope: checks if x exists outside (global)\nif not: new local x is created\nif yes: the split is REPL/non-interactive:\nREPL: global x is used (convenience, as of 1.6)\nnon-interactive: local x is created","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"keyword local and global can be used to specify which variable to use","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"From documentation:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Construct Scope type Allowed within\nmodule, baremodule global global\nstruct local (soft) global\nfor, while, try local (soft) global, local\nmacro local (hard) global\nfunctions, do blocks, let blocks, comprehensions, generators local (hard) global, local","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Question:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"x=1\nf()= x=3\nf()\n@show x;","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"x = 1\nfor _ = 1:1\n x=3\nend\n@show x;","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Notice that if does not introduce new scope","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"if true \n branch_taken = true \nelse\n branch_not_taken = true \nend","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"tip: do-block\nLet's assume a function, which takes as a first argument a function g(f::Function, args...) = println(\"f called on $(args) evaluates to \", f(args...))We can use g as g(+, 1, 2), or with a lambda function g(x -> x^2, 2). But sometimes, it might be useful to the lambda function to span multiple lines. This can be achieved by a do block as g(1,2,3) do a,b,c\n a*b + c\nend","category":"page"},{"location":"lecture_04/lecture/#Packages","page":"Lecture","title":"Packages","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Package is a source tree with a standard layout. It provides a module and thus can be loaded with include or using.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Minimimal package:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"PackageName/\n├── src/\n│ └── PackageName.jl\n├── Project.toml","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Contains:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Project.toml file describing basic properties:\nName, does not have to be unique (federated package sources)\nUUID, has to be unique (generated automatically)\noptionally [deps], [targets],...\nfile src/PackageName.jl that defines module PackageName which is executed when loaded.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Many other optional directories:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"directory tests/, (almost mandatory)\ndirectory docs/ (common)\ndirectory scripts/, examples/,... (optional)","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"tip: Type-Piracy\n\"Type piracy\" refers to the practice of extending or redefining methods in Base or other packages on types that you have not defined. In extreme cases, you can crash Julia (e.g. if your method extension or redefinition causes invalid input to be passed to a ccall). Type piracy can complicate reasoning about code, and may introduce incompatibilities that are hard to predict and diagnose.module A\nimport Base.*\n*(x::Symbol, y::Symbol) = Symbol(x,y)\nend","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"The package typically loads other modules that form package dependencies.","category":"page"},{"location":"lecture_04/lecture/#Project-environments","page":"Lecture","title":"Project environments","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Is a package that does not contain Name and UUID in Project.toml. It's used when you don't need to create a package for your work. It's created by activate some/path in REPL package mode. ","category":"page"},{"location":"lecture_04/lecture/#Project-Manifest","page":"Lecture","title":"Project Manifest","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Both package and environment can contain an additional file Manifest.toml. This file tracks full dependency tree of a project including versions of the packages on which it depends.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"for example:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"# This file is machine-generated - editing it directly is not advised\n\n[[AbstractFFTs]]\ndeps = [\"LinearAlgebra\"]\ngit-tree-sha1 = \"485ee0867925449198280d4af84bdb46a2a404d0\"\nuuid = \"621f4979-c628-5d54-868e-fcf4e3e8185c\"\nversion = \"1.0.1\"\n\n[[AbstractTrees]]\ngit-tree-sha1 = \"03e0550477d86222521d254b741d470ba17ea0b5\"\nuuid = \"1520ce14-60c1-5f80-bbc7-55ef81b5835c\"\nversion = \"0.3.4\"","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Content of files Project.toml and Manifest.toml are maintained by PackageManager.","category":"page"},{"location":"lecture_04/lecture/#Package-manager","page":"Lecture","title":"Package manager","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Handles both packages and projects:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"creating a project ]generate PkgName\nadding an existing project add PkgName or add https://github.com/JuliaLang/Example.jl\nNames are resolved by Registrators (public or private).\nremoving ]rm PkgName\nupdating ]update\ndeveloping ]dev http://... \nadd treats packages as being finished, version handling pkg manager. Precompiles!\ndev leaves all operations on the package to the user (git versioning, etc.). Always read content of files","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"By default these operations are related to environment .julia/environments/v1.9","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"E.g. running and updating will update packages in Manifest.toml in this directory. What if the update breaks functionality of some project package that uses special features?","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"There can and should be more than one environment!","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Project environments are based on files with installed packages.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"switching by ]activate Path - creates Project.toml if not existing\nfrom that moment, all package modifications will be relevant only to this project!\nwhen switching to a new project ]instantiate will prepare (download and precompile) the environment\ncreates Manifest.toml = list of all exact versions of all packages \nwhich Packages are visible is determined by LOAD_PATH\ntypically contaings default libraries and default environment\nit is different for REPL and Pkg.tests ! No default env. in tests. ","category":"page"},{"location":"lecture_04/lecture/#Package-hygiene-workflow","page":"Lecture","title":"Package hygiene - workflow","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"theorem: Potential danger\nPackage dependencies may not be compatible: package A requires C@<0.2\npackage B requires C@>0.3\nwhat should happen when ]add A and add B?","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"keep your \"@v#.#\" as clean as possible (recommended are only debugging/profiling packages)\nuse packages as much as you can, even for short work with scripts ]activate .\nadding a package existing elsewhere is cheap (global cache)\nif do you not wish to store any files just test random tricks of a cool package: ]activate --temp","category":"page"},{"location":"lecture_04/lecture/#Package-development-with-Revise","page":"Lecture","title":"Package development with Revise","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Developing a package with interactive test/development:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Create a package/module at one directory MainPackage\nCreate a script at another directory MainScript, and activate it ]activate .\ndev MainPackage in the MainScript environment\nRevise.jl will watch the MainPackage so it is always up to date\nin dev mode you have full control over commits etc.","category":"page"},{"location":"lecture_04/lecture/#Package-Extensions","page":"Lecture","title":"Package Extensions","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Some functionality of a package that depends on external packages may not be always needed. A typical example is plotting and visualization that may reguire heavy visualization packages. These are completely unnecessary e.g. in distributed server number crunching.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"The extension is a new module depending on: i) the base package, and ii) the conditioning package. It will not be compiled if the conditioning package is not loaded. Once the optional package is loaded, the extension will be automatically compiled and loaded.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"New feature since Julia 1.9:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"new directory in project tree: ext/ the extending module is stored here\nnew section in Project.toml called [extensions] listing extension names and their conditioning packages","category":"page"},{"location":"lecture_04/lecture/#Unit-testing,-/test","page":"Lecture","title":"Unit testing, /test","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Without explicit keywords for checking constructs (think missing functions in interfaces), the good quality of the code is guaranteed by detailed unit testing.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"each package should have directory /test\nfile /test/runtest.jl is run by the command ]test of the package manager\nthis file typically contains include of other tests\nno formal structure of tests is prescribed\ntest files are just ordinary julia scripts\nuser is free to choose what to test and how (freedom x formal rules)\ntesting functionality is supported by macros @test and @teststet\n@testset \"trigonometric identities\" begin\n θ = 2/3*π\n @test sin(-θ) ≈ -sin(θ)\n @test cos(-θ) ≈ cos(θ)\n @test sin(2θ) ≈ 2*sin(θ)*cos(θ)\n @test cos(2θ) ≈ cos(θ)^2 - sin(θ)^2\nend;","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Testset is a collection of tests that will be run and summarized in a common report.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Testsets can be nested: testsets in testsets\ntests can be in loops or functions\nfor i=1:10\n @test a[i]>0\nend\nUseful macro ≈ checks for equality with given tolerance\na=5+1e-8\n@test a≈5\n@test a≈5 atol=1e-10","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"@testset resets RNG to Random.GLOBAL_SEED before and after the test for repeatability \nThe same results of RNG are not guaranteed between Julia versions!\nTest coverage: package Coverage.jl\nCan be run automatically by continuous integration, e.g. GitHub actions\nintegration in VSCode test via package TestItems.jl ","category":"page"},{"location":"lecture_04/lecture/#Documentation-and-Style,-/docs","page":"Lecture","title":"Documentation & Style, /docs","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"A well written package is reusable if it is well documented. ","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"The simpliest kind of documentation is the docstring:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"\"Auxiliary function for printing a hello\"\nhello()=println(\"hello\")\n\n\"\"\"\nMore complex function that adds π to input:\n- x is the input argument (itemize)\n\nCan be written in latex: ``x \\leftarrow x + \\pi``\n\"\"\"\naddπ(x) = x+π","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Yieds:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"tip: Renders as\nMore complex function that adds π to input:x is the input argument (itemize)Can be written in latex: x leftarrow x + pi","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Structure of the document","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"PackageName/\n├── src/\n│ └── SourceFile.jl\n├── docs/\n│ ├── build/\n│ ├── src/\n│ └── make.jl\n...","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Where the line-by-line documentation is in the source files.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"/docs/src folder can contain more detailed information: introductory pages, howtos, tutorials, examples\nrunning make.jl controls which pages are generated in what form (html or latex) documentation in the /build directory\nautomated with GitHub actions","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Documentation is generated by the julia code.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"code in documentation can be evaluated","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"x=3\n@show x","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"documentation can be added by code:\nstruct MyType\n value::String\nend\n\nDocs.getdoc(t::MyType) = \"Documentation for MyType with value $(t.value)\"\n\nx = MyType(\"x\")\ny = MyType(\"y\")\nSee ?x and ?y. \nIt uses the same very standard building blocks: multiple dispatch.","category":"page"},{"location":"lecture_04/lecture/#Precompilation","page":"Lecture","title":"Precompilation","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"By default, every package is precompiled when loading and stored in compiled form in a cache.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"If it defines methods that extend previously defined (e.g. from Base), it may affect already loaded packages which need to be recompiled as well. May take time.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Julia has a tracking mechanism that stores information about the whole graph of dependencies. ","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Faster code can be achieved by the precompile directive:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"module FSum\n\nfsum(x) = x\nfsum(x,p...) = x+fsum(p...)\n\nprecompile(fsum,(Float64,Float64,Float64))\nend","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Can be investigated using MethodAnalysis.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"using MethodAnalysis\nmi =methodinstances(fsum)","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Useful packages:","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"PackageCompiler.jl has three main purposes:\nCreating custom sysimages for reduced latency when working locally with packages that has a high startup time.\nCreating \"apps\" which are a bundle of files including an executable that can be sent and run on other machines without Julia being installed on that machine.\nCreating a relocatable C library bundle form of Julia code.","category":"page"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"AutoSysimages.jl allows easy generation of precompiles images - reduces package loading","category":"page"},{"location":"lecture_04/lecture/#Additional-material","page":"Lecture","title":"Additional material","text":"","category":"section"},{"location":"lecture_04/lecture/","page":"Lecture","title":"Lecture","text":"Modern Julia Workflows","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"These notes are from poking around Core.Compiler to see, how they are different from working just with CodeInfo and IRTools.jl. Notes are mainly around IRCode. Why there is a Core.Compiler.IRCode when there was Core.CodeInfo? Seems to be historical reasons. At the beginning, Julia did not have any intermediate representation and code directly emitted LLVM. Then, it has received an CodeInfo as in intermediate representation. IRCode seems like an evolution of CodeInfo. Core.Compiler works mostly with IRCode, but the IRCode can be converted to the CodeInfo and the other way around. IRCode seems to be designed more for implementation of various optimisation phases. Personal experience tells me it is much nicer to work with even on the low level. ","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"Throughout the explanation, we assume that Core.Compiler was imported as CC to decrease the typing load.","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"Let's play with a simple silly function ","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"\nfunction foo(x,y) \n z = x * y \n z + sin(x)\nend","category":"page"},{"location":"lecture_09/ircode/#IRCode","page":"-","title":"IRCode","text":"","category":"section"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"We can obtain CC.IRCode","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"import Core.Compiler as CC\n(ir, rt) = only(Base.code_ircode(foo, (Float64, Float64), optimize_until = \"compact 1\"))","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"which returns Core.Compiler.IRCode in ir and return-type Float64 in rt. The output might look like ","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"julia> (ir, rt) = only(Base.code_ircode(foo, (Float64, Float64), optimize_until = \"compact 1\"))\n 1─ %1 = (_2 * _3)::Float64 \n │ %2 = Main.sin(_2)::Float64 \n │ %3 = (%1 + %2)::Float64 \n └── return %3 \n => Float64\n","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"Options of optimize_until are compact 1, compact 2, nothing. I do not see a difference between compact 2 and compact 2.","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"The IRCode structure is defined as","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"struct IRCode\n stmts::InstructionStream\n argtypes::Vector{Any}\n sptypes::Vector{VarState}\n linetable::Vector{LineInfoNode}\n cfg::CFG\n new_nodes::NewNodeStream\n meta::Vector{Expr}\nend","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"where","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"stmts is a stream of instruction (more in this below)\nargtypes holds types of arguments of the function whose IRCode we have obtained\nsptypes is a vector of VarState. It seems to be related to parameters of types\nlinetable is a table of unique lines in the source code from which statements \ncfg holds control flow graph, which contains building blocks and jumps between them\nnew_nodes is an infrastructure that can be used to insert new instructions to the existing IRCode . The idea behind is that since insertion requires a renumbering all statements, they are put in a separate queue. They are put to correct position with a correct SSANumber by calling compact!.\nmeta is something.","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"Before going further, let's take a look on InstructionStream defined as ","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"struct InstructionStream\n inst::Vector{Any}\n type::Vector{Any}\n info::Vector{CallInfo}\n line::Vector{Int32}\n flag::Vector{UInt8}\nend","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"where ","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"inst is a vector of instructions, stored as Expressions. The allowed fields in head are described here\ntype is the type of the value returned by the corresponding statement\nCallInfo is ???some info???\nline is an index into IRCode.linetable identifying from which line in source code the statement comes from\nflag are some flags providing additional information about the statement.","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"- `0x01 << 0` = statement is marked as `@inbounds`\n- `0x01 << 1` = statement is marked as `@inline`\n- `0x01 << 2` = statement is marked as `@noinline`\n- `0x01 << 3` = statement is within a block that leads to `throw` call\n- `0x01` << 4 = statement may be removed if its result is unused, in particular it is thus be both pure and effect free\n- `0x01 << 5-6 = `\n- `0x01 << 7 = ` has out-of-band info","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"For the above foo function, the InstructionStream looks like","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"julia> DataFrame(flag = ir.stmts.flag, info = ir.stmts.info, inst = ir.stmts.inst, line = ir.stmts.line, type = ir.stmts.type)\n4×5 DataFrame\n Row │ flag info inst line type\n │ UInt8 CallInfo Any Int32 Any\n─────┼────────────────────────────────────────────────────────────────────────\n 1 │ 112 MethodMatchInfo(MethodLookupResu… _2 * _3 1 Float64\n 2 │ 80 MethodMatchInfo(MethodLookupResu… Main.sin(_2) 2 Float64\n 3 │ 112 MethodMatchInfo(MethodLookupResu… %1 + %2 2 Float64\n 4 │ 0 NoCallInfo() return %3 2 Any","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"We can index into the statements as ir.stmts[1], which provides a \"view\" into the vector. To obtain the first instruction, we can do ir.stmts[1][:inst].","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"The IRCode is typed, but the fields can contain Any. It is up to the user to provide corrrect types of the output and there is no helper functions to perform typing. A workaround is shown in the Petite Diffractor project. Julia's sections of the manual https://docs.julialang.org/en/v1/devdocs/ssair/ and seems incredibly useful. The IR form they talk about seems to be Core.Compiler.IRCode. ","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"It seems to be that it is possible to insert IR instructions into the it structure by queuing that to the field stmts and then call compact!, which would perform the heavy machinery of relabeling everything.","category":"page"},{"location":"lecture_09/ircode/#Example-of-modifying-the-function-through-IRCode","page":"-","title":"Example of modifying the function through IRCode","text":"","category":"section"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"Below is an MWE that tries to modify the IRCode of a function and execute it. The goal is to change the function foo to fooled.","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"import Core.Compiler as CC\nusing Core: SSAValue, GlobalRef, ReturnNode\n\nfunction foo(x,y) \n z = x * y \n z + sin(x)\nend\n\nfunction fooled(x,y) \n z = x * y \n z + sin(x) + cos(y)\nend\n\n(ir, rt) = only(Base.code_ircode(foo, (Float64, Float64), optimize_until = \"compact 1\"));\nnr = CC.insert_node!(ir, 2, CC.NewInstruction(Expr(:call, Core.GlobalRef(Main, :cos), Core.Argument(3)), Float64))\nnr2 = CC.insert_node!(ir, 4, CC.NewInstruction(Expr(:call, GlobalRef(Main, :+), SSAValue(3), nr), Float64))\nCC.setindex!(ir.stmts[4], ReturnNode(nr2), :inst)\nir = CC.compact!(ir)\nirfooled = Core.OpaqueClosure(ir)\nirfooled(1.0, 2.0) == fooled(1.0, 2.0)","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"So what we did?","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"(ir, rt) = only(Base.code_ircode(foo, (Float64, Float64), optimize_until = \"compact 1\")) obtain the IRCode of the function foo when called with both arguments being Float64. rt contains the return type of the \nA new instruction cos is inserted to the ir by Core.Compiler.insert_node!, which takes as an argument an IRCode, position (2 in our case), and new instruction. The new instruction is created by NewInstruction accepting as an input expression Expr and a return type. Here, we force it to be Float64, but ideally it should be inferred. (This would be the next stage). Or, may-be, we can run it through type inference? . The new instruction is added to the ir.new_nodes instruction stream and obtain a new SSAValue returned in nr, which can be then used further.\nWe add one more instruction + that uses output of the instruction we add in step 2, nr and SSAValue from statement 3 of the original IR (at this moment, the IR is still numbered with respect to the old IR, the renumbering will happen later.) The output of this second instruction is returned in nr2.\nThen, we rewrite the return statement to return nr2 instead of SSAValue(3).\nir = CC.compact!(ir) is superimportant since it moves the newly added statements from ir.new_stmts to ir.stmts and importantly renumbers SSAValues. Even though the function is mutating, the mutation here is meant that the argument is changed, but the new correct IRCode is returned and therefore has to be reassigned.\nThe function is created through OpaqueClosure.\nThe last line certifies that the function do what it should do.","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"There is no infrastructure to make the above manipulation transparent, like is the case of @generated function and codeinfo. It is possible to hook through generated function by converting the IRCode to untyped CodeInfo, in which case you do not have to bother with typing.","category":"page"},{"location":"lecture_09/ircode/#How-to-obtain-code-info-the-proper-way?","page":"-","title":"How to obtain code info the proper way?","text":"","category":"section"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"This is the way code info is obtained in the diffractor.","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"mthds = Base._methods_by_ftype(sig, -1, world)\nmatch = only(mthds)\n\nmi = Core.Compiler.specialize_method(match)\nci = Core.Compiler.retrieve_code_info(mi, world)","category":"page"},{"location":"lecture_09/ircode/#CodeInfo","page":"-","title":"CodeInfo","text":"","category":"section"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"IRTools.jl are great for modifying CodeInfo. I have found two tools for modifying IRCode and I wonder if they have been abandoned because they were both dead ends or because of lack of human labor. I am also aware of Also, this is quite cool play with IRStuff.","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"Resources","category":"page"},{"location":"lecture_09/ircode/","page":"-","title":"-","text":"https://vchuravy.dev/talks/licm/\nCompilerPluginTools \nCodeInfoTools.jl.\nTKF's CodeInfo.jl is nice for visualization of the IRCode\nDiffractor is an awesome source of howto. For example function my_insert_node! in src/stage1/hacks.jl\nhttps://nbviewer.org/gist/tkf/d4734be24d2694a3afd669f8f50e6b0f/00_notebook.ipynb\nhttps://github.com/JuliaCompilerPlugins/Mixtape.jl","category":"page"},{"location":"lecture_07/lab/#macro_lab","page":"Lab","title":"Lab 07: Macros","text":"","category":"section"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"A little reminder from the lecture, a macro in its essence is a function, which ","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"takes as an input an expression (parsed input)\nmodifies the expressions in arguments\ninserts the modified expression at the same place as the one that is parsed.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"In this lab we are going to use what we have learned about manipulation of expressions and explore avenues of where macros can be useful","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"convenience (@repeat, @show)\nperformance critical code generation (@poly)\nalleviate tedious code generation (@species, @eats)\njust as a syntactic sugar (@ecosystem)","category":"page"},{"location":"lecture_07/lab/#Show-macro","page":"Lab","title":"Show macro","text":"","category":"section"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Let's start with dissecting \"simple\" @show macro, which allows us to demonstrate advanced concepts of macros and expression manipulation.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"x = 1\n@show x + 1\nlet y = x + 1 # creates a temporary local variable\n println(\"x + 1 = \", y)\n y # show macro also returns the result\nend\n\n# assignments should create the variable\n@show x = 3\nlet y = x = 2 \n println(\"x = 2 = \", y)\n y\nend\nx # should be equal to 2","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"The original Julia's implementation is not dissimilar to the following macro definition:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"macro myshow(ex)\n quote\n println($(QuoteNode(ex)), \" = \", repr(begin local value = $(esc(ex)) end))\n value\n end\nend","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Testing it gives us the expected behavior","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"@myshow xx = 1 + 1\nxx # should be defined","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"In this \"simple\" example, we had to use the following concepts mentioned already in the lecture:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"QuoteNode(ex) is used to wrap the expression inside another layer of quoting, such that when it is interpolated into :() it stays being a piece of code instead of the value it represents - TRUE QUOTING\nesc(ex) is used in case that the expression contains an assignment, that has to be evaluated in the top level module Main (we are escaping the local context) - ESCAPING\n$(QuoteNode(ex)) and $(esc(ex)) is used to evaluate an expression into another expression. INTERPOLATION\nlocal value = is used in order to return back the result after evaluation","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Lastly, let's mention that we can use @macroexpand to see how the code is manipulated in the @myshow macro","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"@macroexpand @show x + 1","category":"page"},{"location":"lecture_07/lab/#Repeat-macro","page":"Lab","title":"Repeat macro","text":"","category":"section"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"In the profiling/performance labs we have sometimes needed to run some code multiple times in order to gather some samples and we have tediously written out simple for loops inside functions such as this","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"function run_polynomial(n, a, x)\n for _ in 1:n\n polynomial(a, x)\n end\nend","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"We can remove this boilerplate code by creating a very simple macro that does this for us.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Define macro @repeat that takes two arguments, first one being the number of times a code is to be run and the other being the actual code.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"julia> @repeat 3 println(\"Hello!\")\nHello!\nHello!\nHello!","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Before defining the macro, it is recommended to write the code manipulation functionality into a helper function _repeat, which helps in organization and debugging of macros.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"_repeat(3, :(println(\"Hello!\"))) # testing \"macro\" without defining it","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"use $ interpolation into a for loop expression; for example given ex = :(1+x) we can interpolate it into another expression :($ex + y) -> :(1 + x + y)\nif unsure what gets interpolated use round brackets :($(ex) + y)\nmacro is a function that creates code that does what we want","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"BONUS: What happens if we call @repeat 3 x = 2? Is x defined?","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"macro repeat(n::Int, ex)\n return _repeat(n, ex)\nend\n\nfunction _repeat(n::Int, ex)\n :(for _ in 1:$n\n $ex\n end)\nend\n\n_repeat(3, :(println(\"Hello!\")))\n@repeat 3 println(\"Hello!\")","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Even if we had used escaping the expression x = 2 won't get evaluated properly due to the induced scope of the for loop. In order to resolve this we would have to specially match that kind of expression and generate a proper syntax withing the for loop global $ex. However we may just warn the user in the docstring that the usage is disallowed. ","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Note that this kind of repeat macro is also defined in the Flux.jl machine learning framework, wherein it's called @epochs and is used for creating training loop.","category":"page"},{"location":"lecture_07/lab/#lab07_polymacro","page":"Lab","title":"Polynomial macro","text":"","category":"section"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"This is probably the last time we are rewriting the polynomial function, though not quite in the same way. We have seen in the last lab, that some optimizations occur automatically, when the compiler can infer the length of the coefficient array, however with macros we can generate optimized code directly (not on the same level - we are essentially preparing already unrolled/inlined code).","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Ideally we would like to write some macro @poly that takes a polynomial in a mathematical notation and spits out an anonymous function for its evaluation, where the loop is unrolled. ","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Example usage:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"p = @poly x 3x^2+2x^1+10x^0 # the first argument being the independent variable to match\np(2) # return the value","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"However in order to make this happen, let's first consider much simpler case of creating the same but without the need for parsing the polynomial as a whole and employ the fact that macro can have multiple arguments separated by spaces.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"p = @poly 3 2 10\np(2)","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Create macro @poly that takes multiple arguments and creates an anonymous function that constructs the unrolled code. Instead of directly defining the macro inside the macro body, create helper function _poly with the same signature that can be reused outside of it.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Recall Horner's method polynomial evaluation from previous labs:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"function polynomial(a, x)\n accumulator = a[end] * one(x)\n for i in length(a)-1:-1:1\n accumulator = accumulator * x + a[i]\n #= accumulator = muladd(x, accumulator, a[i]) =# # equivalent\n end\n accumulator \nend","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"you can use muladd function as replacement for ac * x + a[i]\nthink of the accumulator variable as the mathematical expression that is incrementally built (try to write out the Horner's method[1] to see it)\nyou can nest expression arbitrarily\nthe order of coefficients has different order than in previous labs (going from high powers of x last to them being first)\nuse evalpoly to check the correctness","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"using Test\np = @poly 3 2 10\n@test p(2) == evalpoly(2, [10,2,3]) # reversed coefficients","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"[1]: Explanation of the Horner schema can be found on https://en.wikipedia.org/wiki/Horner%27s_method.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"using InteractiveUtils #hide\nmacro poly(a...)\n return _poly(a...)\nend\n\nfunction _poly(a...)\n N = length(a)\n ex = :($(a[1]))\n for i in 2:N\n ex = :(muladd(x, $ex, $(a[i]))) # equivalent of :(x * $ex + $(a[i]))\n end\n :(x -> $ex)\nend\n\np = @poly 3 2 10\np(2) == evalpoly(2, [10,2,3])\n@code_lowered p(2) # can show the generated code","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Moving on to the first/harder case, where we need to parse the mathematical expression.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Create macro @poly that takes two arguments first one being the independent variable and second one being the polynomial written in mathematical notation. As in the previous case this macro should define an anonymous function that constructs the unrolled code. ","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"julia> p = @poly x 3x^2+2x^1+10x^0 # the first argument being the independent variable to match","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"though in general we should be prepared for some edge cases, assume that we are really strict with the syntax allowed (e.g. we really require spelling out x^0, even though it is mathematically equivalent to 1)\nreuse the _poly function from the previous exercise\nuse the MacroTools.jl to match/capture a_*$v^(n_), where v is the symbol of independent variable, this is going to be useful in the following steps\nget maximal rank of the polynomial\nget coefficient for each power","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"note: `MacroTools.jl`\nThough not the most intuitive, MacroTools.jl pkg help us with writing custom macros. We will use two utilities@captureThis macro is used to match a pattern in a single expression and return values of particular spots. For examplejulia> using MacroTools\njulia> @capture(:[1, 2, 3, 4, 5, 6, 7], [1, a_, 3, b__, c_])\ntrue\n\njulia> a, b, c\n(2,[4,5,6],7)postwalk/prewalkIn order to extend @capture to more complicated expression trees, we can used either postwalk or prewalk to walk the AST and match expression along the way. For examplejulia> using MacroTools: prewalk, postwalk\njulia> ex = quote\n x = f(y, g(z))\n return h(x)\n end\n\njulia> postwalk(ex) do x\n @capture(x, fun_(arg_)) && println(\"Function: \", fun, \" with argument: \", arg)\n x\n end;\nFunction: g with argument: z\nFunction: h with argument: xNote that the x or the iteration is required, because by default postwalk/prewalk replaces currently read expression with the output of the body of do block.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"using MacroTools\nusing MacroTools: postwalk, prewalk\n\nmacro poly(v::Symbol, p::Expr)\n a = Tuple(reverse(_get_coeffs(v, p)))\n return _poly(a...)\nend\n\nfunction _max_rank(v, p)\n mr = 0\n postwalk(p) do x\n if @capture(x, a_*$v^(n_))\n mr = max(mr, n)\n end\n x\n end\n mr\nend\n\nfunction _get_coeffs(v, p)\n N = _max_rank(v, p) + 1\n coefficients = zeros(N)\n postwalk(p) do x\n if @capture(x, a_*$v^(n_))\n coefficients[n+1] = a\n end\n x\n end\n coefficients\nend","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Let's test it.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"p = @poly x 3x^2+2x^1+10x^0\np(2) == evalpoly(2, [10,2,3])\n@code_lowered p(2) # can show the generated code","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_07/lab/#Ecosystem-macros","page":"Lab","title":"Ecosystem macros","text":"","category":"section"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"There are at least two ways how we can make our life simpler when using our Ecosystem and EcosystemCore pkgs. Firstly, recall that in order to test our simulation we always had to write something like this:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"function create_world()\n n_grass = 500\n regrowth_time = 17.0\n\n n_sheep = 100\n Δenergy_sheep = 5.0\n sheep_reproduce = 0.5\n sheep_foodprob = 0.4\n\n n_wolves = 8\n Δenergy_wolf = 17.0\n wolf_reproduce = 0.03\n wolf_foodprob = 0.02\n\n gs = [Grass(id, regrowth_time) for id in 1:n_grass];\n ss = [Sheep(id, 2*Δenergy_sheep, Δenergy_sheep, sheep_reproduce, sheep_foodprob) for id in n_grass+1:n_grass+n_sheep];\n ws = [Wolf(id, 2*Δenergy_wolf, Δenergy_wolf, wolf_reproduce, wolf_foodprob) for id in n_grass+n_sheep+1:n_grass+n_sheep+n_wolves];\n World(vcat(gs, ss, ws))\nend\nworld = create_world();","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"which includes the tedious process of defining the agent counts, their parameters and last but not least the unique id manipulation. As part of the HW for this lecture you will be tasked to define a simple DSL, which can be used to define a world in a few lines.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Secondly, the definition of a new Animal or Plant, that did not have any special behavior currently requires quite a bit of repetitive code. For example defining a new plant type Broccoli goes as follows","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"abstract type Broccoli <: PlantSpecies end\nBase.show(io::IO,::Type{Broccoli}) = print(io,\"🥦\")\n\nEcosystemCore.eats(::Animal{Sheep},::Plant{Broccoli}) = true","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"and definition of a new animal like a Rabbit looks very similar","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"abstract type Rabbit <: AnimalSpecies end\nBase.show(io::IO,::Type{Rabbit}) = print(io,\"🐇\")\n\nEcosystemCore.eats(::Animal{Rabbit},p::Plant{Grass}) = size(p) > 0\nEcosystemCore.eats(::Animal{Rabbit},p::Plant{Broccoli}) = size(p) > 0","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"In order to make this code \"clearer\" (depends on your preference) we will create two macros, which can be called at one place to construct all the relations.","category":"page"},{"location":"lecture_07/lab/#New-Animal/Plant-definition","page":"Lab","title":"New Animal/Plant definition","text":"","category":"section"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Our goal is to be able to define new plants and animal species, while having a clear idea about their relations. For this we have proposed the following macros/syntax:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"@species Plant Broccoli 🥦\n@species Animal Rabbit 🐇\n@eats Rabbit [Grass => 0.5, Broccoli => 1.0, Mushroom => -1.0]","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Unfortunately the current version of Ecosystem and EcosystemCore, already contains some definitions of species such as Sheep, Wolf and Mushroom, which may collide with definitions during prototyping, therefore we have created a modified version of those pkgs, which will be provided in the lab.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"note: Testing relations\nWe can test the current definition with the following code that constructs \"eating matrix\"using Ecosystem\nusing Ecosystem.EcosystemCore\n\nfunction eating_matrix()\n _init(ps::Type{<:PlantSpecies}) = ps(1, 10.0)\n _init(as::Type{<:AnimalSpecies}) = as(1, 10.0, 1.0, 0.8, 0.7)\n function _check(s1, s2)\n try\n if s1 !== s2\n EcosystemCore.eats(_init(s1), _init(s2)) ? \"✅\" : \"❌\"\n else\n return \"❌\"\n end\n catch e\n if e isa MethodError\n return \"❔\"\n else\n throw(e)\n end\n end\n end\n\n animal_species = subtypes(AnimalSpecies)\n plant_species = subtypes(PlantSpecies)\n species = vcat(animal_species, plant_species)\n em = [_check(s, ss) for (s,ss) in Iterators.product(animal_species, species)]\n string.(hcat([\"🌍\", animal_species...], vcat(permutedims(species), em)))\nend\neating_matrix()\n 🌍 🐑 🐺 🌿 🍄\n 🐑 ❌ ❌ ✅ ✅\n 🐺 ✅ ❌ ❌ ❌","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Based on the following example syntax, ","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"@species Plant Broccoli 🥦\n@species Animal Rabbit 🐇","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"write macro @species inside Ecosystem pkg, which defines the abstract type, its show function and exports the type. For example @species Plant Broccoli 🥦 should generate code:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"abstract type Broccoli <: PlantSpecies end\nBase.show(io::IO,::Type{Broccoli}) = print(io,\"🥦\")\nexport Broccoli","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Define first helper function _species to inspect the macro's output. This is indispensable, as we are defining new types/constants and thus we may otherwise encounter errors during repeated evaluation (though only if the type signature changed).","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"_species(:Plant, :Broccoli, :🥦)\n_species(:Animal, :Rabbit, :🐇)","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"use QuoteNode in the show function just like in the @myshow example\nescaping esc is needed for the returned in order to evaluate in the top most module (Ecosystem/Main)\nideally these changes should be made inside the modified Ecosystem pkg provided in the lab (though not everything can be refreshed with Revise) - there is a file ecosystem_macros.jl just for this purpose\nmultiple function definitions can be included into a quote end block\ninterpolation works with any expression, e.g. $(typ == :Animal ? AnimalSpecies : PlantSpecies)","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"BONUS: Based on @species define also macros @animal and @plant with two arguments instead of three, where the species type is implicitly carried in the macro's name.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Macro @species","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"macro species(typ, name, icon)\n esc(_species(typ, name, icon))\nend\n\nfunction _species(typ, name, icon)\n quote\n abstract type $name <: $(typ == :Animal ? AnimalSpecies : PlantSpecies) end\n Base.show(io::IO, ::Type{$name}) = print(io, $(QuoteNode(icon)))\n export $name\n end\nend\n\n_species(:Plant, :Broccoli, :🥦)\n_species(:Animal, :Rabbit, :🐇)","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"And the bonus macros @plant and @animal","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"macro plant(name, icon)\n return :(@species Plant $name $icon)\nend\n\nmacro animal(name, icon)\n return :(@species Animal $name $icon)\nend","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"The next exercise applies macros to the agents eating behavior.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Define macro @eats inside Ecosystem pkg that assigns particular species their eating habits via eat! and eats functions. The macro should process the following example syntax","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"@eats Rabbit [Grass => 0.5, Broccoli => 1.0],","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"where Grass => 0.5 defines the behavior of the eat! function. The coefficient is used here as a multiplier for the energy balance, in other words the Rabbit should get only 0.5 of energy for a piece of Grass.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"ideally these changes should be made inside the modified Ecosystem pkg provided in the lab (though not everything can be refreshed with Revise) - there is a file ecosystem_macros.jl just for this purpose\nescaping esc is needed for the returned in order to evaluate in the top most module (Ecosystem/Main)\nyou can create an empty quote end block with code = Expr(:block) and push new expressions into its args incrementally\nuse dispatch to create specific code for the different combinations of agents eating other agents (there may be catch in that we have to first eval the symbols before calling in order to know if they are animals or plants)","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"note: Reminder of `EcosystemCore` `eat!` and `eats` functionality\nIn order to define that an Wolf eats Sheep, we have to define two methodsEcosystemCore.eats(::Animal{Wolf}, ::Animal{Sheep}) = true\n\nfunction EcosystemCore.eat!(ae::Animal{Wolf}, af::Animal{Sheep}, w::World)\n incr_energy!(ae, $(multiplier)*energy(af)*Δenergy(ae))\n kill_agent!(af, w)\nendIn order to define that an Sheep eats Grass, we have to define two methodsEcosystemCore.eats(::Animal{Sheep}, p::Plant{Grass}) = size(p)>0\n\nfunction EcosystemCore.eat!(a::Animal{Sheep}, p::Plant{Grass}, w::World)\n incr_energy!(a, $(multiplier)*size(p)*Δenergy(a))\n p.size = 0\nend","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"BONUS: You can try running the simulation with the newly added agents.","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"macro eats(species::Symbol, foodlist::Expr)\n return esc(_eats(species, foodlist))\nend\n\n\nfunction _generate_eat(eater::Type{<:AnimalSpecies}, food::Type{<:PlantSpecies}, multiplier)\n quote\n EcosystemCore.eats(::Animal{$(eater)}, p::Plant{$(food)}) = size(p)>0\n function EcosystemCore.eat!(a::Animal{$(eater)}, p::Plant{$(food)}, w::World)\n incr_energy!(a, $(multiplier)*size(p)*Δenergy(a))\n p.size = 0\n end\n end\nend\n\nfunction _generate_eat(eater::Type{<:AnimalSpecies}, food::Type{<:AnimalSpecies}, multiplier)\n quote\n EcosystemCore.eats(::Animal{$(eater)}, ::Animal{$(food)}) = true\n function EcosystemCore.eat!(ae::Animal{$(eater)}, af::Animal{$(food)}, w::World)\n incr_energy!(ae, $(multiplier)*energy(af)*Δenergy(ae))\n kill_agent!(af, w)\n end\n end\nend\n\n_parse_eats(ex) = Dict(arg.args[2] => arg.args[3] for arg in ex.args if arg.head == :call && arg.args[1] == :(=>))\n\nfunction _eats(species, foodlist)\n cfg = _parse_eats(foodlist)\n code = Expr(:block)\n for (k,v) in cfg\n push!(code.args, _generate_eat(eval(species), eval(k), v))\n end\n code\nend\n\nspecies = :Rabbit \nfoodlist = :([Grass => 0.5, Broccoli => 1.0])\n_eats(species, foodlist)","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"","category":"page"},{"location":"lecture_07/lab/#Resources","page":"Lab","title":"Resources","text":"","category":"section"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"macros in Julia documentation","category":"page"},{"location":"lecture_07/lab/#Type{T}-type-selectors","page":"Lab","title":"Type{T} type selectors","text":"","category":"section"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"We have used ::Type{T} signature[2] at few places in the Ecosystem family of packages (and it will be helpful in the HW as well), such as in the show methods","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Base.show(io::IO,::Type{World}) = print(io,\"🌍\")","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"This particular example defines a method where the second argument is the World type itself and not an instance of a World type. As a result we are able to dispatch on specific types as values. ","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"Furthermore we can use subtyping operator to match all types in a hierarchy, e.g. ::Type{<:AnimalSpecies} matches all animal species","category":"page"},{"location":"lecture_07/lab/","page":"Lab","title":"Lab","text":"[2]: https://docs.julialang.org/en/v1/manual/types/#man-typet-type","category":"page"},{"location":"projects/#projects","page":"Projects","title":"Projects","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"The goal of the project should be to create something, which is actually useful. Therefore we offer a lot of freedom in how the project will look like with the condition that you should spent around 60 hours on it (this number was derived as follows: each credit is worth 30 hours minus 13 lectures + labs minus 10 homeworks 2 hours each) and you should demonstrate some skills in solving the project. In general, we can distinguish three types of project depending on the beneficiary:","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"You benefit: Use / try to solve a well known problem using Julia language,\nOur group: work with your tutors on a topic researched in the AIC group, \nJulia community: choose an issue in a registered Julia project you like and fix it (documentation issues are possible but the resulting documentation should be very nice.).","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"The project should be of sufficient complexity that verify your skill of the language (to be agreed individually).","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"Below, we list some potential projects for inspiration.","category":"page"},{"location":"projects/#Implementing-new-things","page":"Projects","title":"Implementing new things","text":"","category":"section"},{"location":"projects/#Lenia-(Continuous-Game-of-Life)","page":"Projects","title":"Lenia (Continuous Game of Life)","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"Lenia is a continuous version of Conway's Game of Life. Implement a Julia version. For example, you could focus either on performance compared to the python version, or build nice visualizations with Makie.jl.","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"Nice tutorial from Conway to Lenia","category":"page"},{"location":"projects/#The-Equation-Learner-And-Its-Symbolic-Representation","page":"Projects","title":"The Equation Learner And Its Symbolic Representation","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"In many scientific and engineering one searches for interpretable (i.e. human-understandable) models instead of the black-box function approximators that neural networks provide. The equation learner (EQL) is one approach that can identify concise equations that describe a given dataset.","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"The EQL is essentially a neural network with different unary or binary activation functions at each indiviual unit. The network weights are regularized during training to obtain a sparse model which hopefully results in a model that represents a simple equation.","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"The goal of this project is to implement the EQL, and if there is enough time the improved equation learner (iEQL). The equation learners should be tested on a few toy problems (possibly inspired by the tasks in the papers). Finally, you will implement functionality that can transform the learned model into a symbolic, human readable, and exectuable Julia expression.","category":"page"},{"location":"projects/#Architecture-visualizer","page":"Projects","title":"Architecture visualizer","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"Create an extension of Flux / Lux and to visualize architecture of a neural network suitable for publication. Something akin PlotNeuralNet.","category":"page"},{"location":"projects/#Learning-Large-Language-Models-with-reduced-precition-(Mentor:-Tomas-Pevny)","page":"Projects","title":"Learning Large Language Models with reduced precition (Mentor: Tomas Pevny)","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"Large Language Models ((Chat) GPT, LLama, Falcon, Palm, ...) are huge. A recent trend is to perform optimization in reduced precision, for example in int8 instead of Float32. Such feature is currently missing in Julia ecosystem and this project should be about bringing this to the community (for an introduction, read these blogs LLM-int8 and emergent features, A gentle introduction to 8-bit Matrix Multiplication). The goal would be to implement this as an additional type of Number / Matrix and overload multiplication on CPU (and ideally on GPU) to make it transparent for neural networks? What I will learn? In this project, you will learn a lot about the (simplicity of) implementation of deep learning libraries and you will practice abstraction of Julia's types. You can furthermore learn about GPU Kernel programming and Transformers.jl library.","category":"page"},{"location":"projects/#Planning-algorithms-(Mentor:-Tomas-Pevny)","page":"Projects","title":"Planning algorithms (Mentor: Tomas Pevny)","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"Extend SymbolicPlanners.jl with the mm-ϵ variant of the bi-directional search MM: A bidirectional search algorithm that is guaranteed to meet in the middle. This pull request might be very helpful in understanding better the library.","category":"page"},{"location":"projects/#A-Rule-Learning-Algorithms-(Mentor:-Tomas-Pevny)","page":"Projects","title":"A Rule Learning Algorithms (Mentor: Tomas Pevny)","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"Rule-based models are simple and very interpretable models that have been around for a long time and are gaining popularity again. The goal of this project is to implement one of these algorithms","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"sequential covering","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"algorithm called RIPPER and evaluate it on a number of datasets.","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"Learning Certifiably Optimal Rule Lists for Categorical Data\nBoolean decision rules via column generation\nLearning Optimal Decision Trees with SAT\nA SAT-based approach to learn explainable decision sets","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"To increase the impact of the project, consider interfacing it with MLJ.jl","category":"page"},{"location":"projects/#Parallel-optimization-(Mentor:-Tomas-Pevny)","page":"Projects","title":"Parallel optimization (Mentor: Tomas Pevny)","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"Implement one of the following algorithms to train neural networks in parallel. Can be implemented in a separate package or consider extending FluxDistributed.jl. Do not forget to verify that the method actually works!!!","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"Hogwild!\nLocal sgd with periodic averaging: Tighter analysis and adaptive synchronization\nDistributed optimization for deep learning with gossip exchange","category":"page"},{"location":"projects/#Solve-issues-in-existing-projects:","page":"Projects","title":"Solve issues in existing projects:","text":"","category":"section"},{"location":"projects/#Create-Yao-backend-for-quantum-simulation-(Mentor:-Niklas-Heim)","page":"Projects","title":"Create Yao backend for quantum simulation (Mentor: Niklas Heim)","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"The recently published quantum programming library Qadence needs a Julia backend. The tricky quantum parts are already implemented in a library called Yao.jl. The goal of this project is to take the Qadence (Python) representation and translate it to Yao.jl (Julia). You will work with the Python/Julia interfacing library PythonCall.jl to realize this and benchmark the Julia backend in the end to assess if it is faster than the existing python implementation.","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"If this sounds interesting, talk to Niklas.","category":"page"},{"location":"projects/#Address-issues-in-markov-decision-processes-(Mentor:-Jan-Mrkos)","page":"Projects","title":"Address issues in markov decision processes (Mentor: Jan Mrkos)","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"Fix type stability issue in MCTS.jl, prepare benchmarks, and evaluate the impact of the changes. Details can be found in this issue. This project will require learnind a little bit about Markov Decision Processes if you don't know them already.","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"If it sounds interesting, get in touch with lecturer/lab assistant, who will connect you with Jan Mrkos.","category":"page"},{"location":"projects/#Extend-HMil-library-with-Retentative-networks-(mentor-Tomas-Pevny)","page":"Projects","title":"Extend HMil library with Retentative networks (mentor Tomas Pevny)","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"Retentative networks were recently proposed as a low-cost alternative to Transformer models without sacrificing performance (according to authors). By implementing Retentative Networks, te HMil library will be able to learn sequences (not just sets), which might nicely extend its applicability.","category":"page"},{"location":"projects/#Address-issues-in-HMil/JsonGrinder-library-(mentor-Simon-Mandlik)","page":"Projects","title":"Address issues in HMil/JsonGrinder library (mentor Simon Mandlik)","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"These are open source toolboxes that are used internally in Avast. Lots of general functionality is done, but some love is needed in polishing.","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"refactor the codebase using package extensions (e.g. for FillArrays)\nimprove compilation time (tracking down bottlenecks with SnoopCompile and using precompile directives from PrecompileTools.jl)","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"Or study new metric learning approach on application in animation description","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"apply machine learning on slides within presentation provide by PowToon","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"If it sounds interesting, get in touch with lecturer/lab assistant, who will connect you with Simon Mandlik.","category":"page"},{"location":"projects/#Project-requirements","page":"Projects","title":"Project requirements","text":"","category":"section"},{"location":"projects/","page":"Projects","title":"Projects","text":"The goal of the semestral project is to create a Julia pkg with reusable, properly tested and documented code. We have given you some options of topics, as well as the freedom to choose something that could be useful for your research or other subjects. In general we are looking for something where performance may be crucial such as data processing, optimization or equation solving.","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"In practice the project should follow roughly this tree structure","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":".\n├── scripts\n│\t├── run_example.jl\t\t\t# one or more examples showing the capabilities of the pkg\n│\t├── Project.toml \t\t\t# YOUR_PROJECT should be added here with develop command with rel path\n│\t└── Manifest.toml \t\t\t# should be committed as it allows to reconstruct the environment exactly\n├── src\n│\t├── YOUR_PROJECT.jl \t\t# ideally only some top level code such as imports and exports, rest of the code included from other files\n│\t├── src1.jl \t\t\t\t# source files structured in some logical chunks\n│\t└── src2.jl\n├── test\n│\t├── runtest.jl # contains either all the tests or just includes them from other files\n│\t├── Project.toml \t\t\t# lists some additional test dependencies\n│\t└── Manifest.toml \t\t# usually not committed to git as it is generated on the fly\n├── README.md \t\t\t\t\t# describes in short what the pkg does and how to install pkg (e.g. some external deps) and run the example\n├── Project.toml \t\t\t\t# lists all the pkg dependencies\n└── Manifest.toml \t\t\t\t# usually not committed to git as the requirements may be to restrictive","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"The first thing that we will look at is README.md, which should warn us if there are some special installation steps, that cannot be handled with Julia's Pkg system. For example if some 3rd party binary dependency with license is required. Secondly we will try to run tests in the test folder, which should run and not fail and should cover at least some functionality of the pkg. Thirdly and most importantly we will instantiate environment in scripts and test if the example runs correctly. Lastly we will focus on documentation in terms of code readability, docstrings and inline comments. ","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"Only after all this we may look at the extent of the project and it's difficulty, which may help us in deciding between grades. ","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"Nice to have things, which are not strictly required but obviously improves the score.","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"Ideally the project should be hosted on GitHub, which could have the continuous integration/testing set up.\nInclude some benchmark and profiling code in your examples, which can show us how well you have dealt with the question of performance.\nSome parallelization attempts either by multi-processing, multi-threadding, or CUDA. Do not forget to show the improvement.\nDocumentation with a webpage using Documenter.jl.","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"Here are some examples of how the project could look like:","category":"page"},{"location":"projects/","page":"Projects","title":"Projects","text":"ImageInspector","category":"page"},{"location":"lecture_01/outline/#Course-outline","page":"Outline","title":"Course outline","text":"","category":"section"},{"location":"lecture_01/outline/","page":"Outline","title":"Outline","text":"Introduction\nType system\nuser: tool for abstraction\ncompiler: tool for memory layout\nDesign patterns (mental setup)\nJulia is a type-based language\nmultiple-dispatch generalizes OOP and FP\nPackages\nway how to organize code\ncode reuse (alternative to libraries)\nexperiment reproducibility\nBenchmarking\nhow to measure code efficiency\nIntrospection\nunderstand how the compiler process the data\nMacros\nautomate writing of boring the boilerplate code\ngood macro create cleaner code\nAutomatic Differentiation\nTheory: difference between the forward and backward mode\nImplementation techniques\nIntermediate representation\nhow to use internal the representation of the code \nexample in automatic differentiation\nParallel computing\nthreads, processes\nGraphics card coding\ntypes for GPU\nspecifics of architectures\nOrdinary Differential Equations\nsimple solvers\nerror propagation\nData driven ODE\ncombine ODE with optimization\nautomatic differentiation (adjoints)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"using Plots","category":"page"},{"location":"lecture_08/lecture/#Automatic-Differentiation","page":"Lecture","title":"Automatic Differentiation","text":"","category":"section"},{"location":"lecture_08/lecture/#Motivation","page":"Lecture","title":"Motivation","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"It supports a lot of modern machine learning by allowing quick differentiation of complex mathematical functions. The 1st order optimization methods are ubiquitous in finding parameters of functions (not only in deep learning).\nAD is interesting to study from both mathermatical and implementation perspective, since different approaches comes with different trade-offs. Julia offers many implementations (some of them are not maintained anymore), as it showed to implement (simple) AD is relatively simple.\nWe (authors of this course) believe that it is good to understand (at least roughly), how the methods work in order to use them effectively in your work.\nJulia is unique in the effort separating definitions of AD rules from AD engines that use those rules to perform the AD and the backend which executes the rules This allows authors of generic libraries to add new rules that would be compatible with many frameworks. See juliadiff.org for a list.","category":"page"},{"location":"lecture_08/lecture/#Theory","page":"Lecture","title":"Theory","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"The differentiation is routine process, as most of the time we break complicated functions down into small pieces that we know, how to differentiate and from that to assemble the gradient of the complex function back. Thus, the essential piece is the differentiation of the composed function f mathbbR^n rightarrow mathbbR^m","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"f(x) = f_1(f_2(f_3(ldots f_n(x)))) = (f_1 circ f_2 circ ldots circ f_n)(x)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"which is computed by chainrule. Before we dive into the details, let's define the notation, which for the sake of clarity needs to be precise. The gradient of function f(x) with respect to x at point x_0 is denoted as leftfracpartial fpartial xright_x^0","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"For a composed function f(x) the gradient with respect to x at point x_0 is equal to","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"leftfracpartial fpartial xright_x^0 = leftfracf_1partial y_1right_y_1^0 times leftfracf_2partial y_2right_y_2^0 times ldots times leftfracf_npartial y_nright_y_n^0","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"where y_i denotes the input of function f_i and","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"beginalignat*2\ny_i^0 = left(f_i+1 circ ldots circ f_nright) (x^0) \ny_n^0 = x^0 \ny_0^0 = f(x^0) \nendalignat*","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"How leftfracf_ipartial y_iright_y_i^0 looks like? ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"If f_i mathbbR rightarrow mathbbR, then fracf_ipartial y_i in mathbbR is a real number mathbbR and we live in a high-school world, where it was sufficient to multiply real numbers.\nIf f_i mathbbR^m_i rightarrow mathbbR^n_i, then mathbfJ_i = leftfracf_ipartial y_iright_y_i^0 in mathbbR^n_im_i is a matrix with m_i rows and n_i columns. ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"The computation of gradient fracpartial fpartial x theoretically boils down to ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"computing Jacobians leftmathbfJ_iright_i=1^n \nmultiplication of Jacobians as it holds that leftfracpartial fpartial xright_y_0 = J_1 times J_2 times ldots times J_n. ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"The complexity of the computation (at least one part of it) is therefore therefore determined by the Matrix multiplication, which is generally expensive, as theoretically it has complexity at least O(n^23728596) but in practice a little bit more as the lower bound hides the devil in the O notation. The order in which the Jacobians are multiplied has therefore a profound effect on the complexity of the AD engine. While determining the optimal order of multiplication of sequence of matrices is costly, in practice, we recognize two important cases.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Jacobians are multiplied from right to left as J_1 times (J_2 times ( ldots times (J_n-1 times J_n) ldots)) which has the advantage when the input dimension of f mathbbR^n rightarrow mathbbR^m is smaller than the output dimension, n m. - referred to as the FORWARD MODE\nJacobians are multiplied from left to right as ( ldots ((J_1 times J_2) times J_3) times ldots ) times J_n which has the advantage when the input dimension of f mathbbR^n rightarrow mathbbR^m is larger than the output dimension, n m. - referred to as the BACKWARD MODE","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"The ubiquitous in machine learning to minimization of a scalar (loss) function of a large number of parameters. Also notice that for f of certain structures, it pays-off to do a mixed-mode AD, where some parts are done using forward diff and some parts using reverse diff. ","category":"page"},{"location":"lecture_08/lecture/#Example","page":"Lecture","title":"Example","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Let's workout an example","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"z = xy + sin(x)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"How it maps to the notation we have used above? Particularly, what are f_1 f_2 ldots f_n and the corresponding y_i_i=1^n, such that (f_1 circ f_2 circ ldots circ f_n)(xy) = xy + sin(x) ?","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"beginalignat*6\nf_1mathbbR^2 rightarrow mathbbR quadf_1(y_1) = y_11 + y_12 quad y_0 = (xy + sin(x)) \nf_2mathbbR^3 rightarrow mathbbR^2 quadf_2(y_2) = (y_21y_22 y_23) quad y_1 = (xy sin(x))\nf_3 mathbbR^2 rightarrow mathbbR^3 quadf_3(y_3) = (y_31 y_32 sin(y_31)) quad y_2 = (x y sin(x))\nendalignat*","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"The corresponding jacobians are ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"beginalignat*4\nf_1(y_1) = y_11 + y_12 quad mathbfJ_1 = beginbmatrix 1 1 endbmatrix \nf_2(y_2) = (y_21y_22 y_23) quad mathbfJ_2 = beginbmatrix y_2 2 0 y_21 0 0 1 endbmatrix\nf_3(y_3) = (y_31 y_32 sin(y_31)) quad mathbfJ_3 = beginbmatrix 1 0 cos(y_31) 0 1 0 endbmatrix \nendalignat*","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"and for the gradient it holds that","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"beginbmatrix fracpartial f(x y)partialx fracpartial f(xy)partialy endbmatrix = mathbfJ_3 times mathbfJ_2 times mathbfJ_1 = beginbmatrix 1 0 cos(x) 0 1 0 endbmatrix times beginbmatrix y 0 x 0 0 1 endbmatrix times beginbmatrix 1 1 endbmatrix = beginbmatrix y cos(x) x 0 endbmatrix times beginbmatrix 1 1 endbmatrix = beginbmatrix y + cos(x) x endbmatrix","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Note that from theoretical point of view this decomposition of a function is not unique, however as we will see later it usually given by the computational graph in a particular language/environment.","category":"page"},{"location":"lecture_08/lecture/#Calculation-of-the-Forward-mode","page":"Lecture","title":"Calculation of the Forward mode","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"In theory, we can calculate the gradient using forward mode as follows Initialize the Jacobian of y_n with respect to x to an identity matrix, because as we have stated above y^0_n = x, i.e. fracpartial y_npartial x = mathbbI. Iterate i from n down to 1 as","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"calculate the next intermediate output as y^0_i-1 = f_i(y^0_i) \ncalculate Jacobian J_i = leftfracf_ipartial y_iright_y^0_i\npush forward the gradient as leftfracpartial y_i-1partial xright_x = J_i times leftfracpartial y_npartial xright_x","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Notice that ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"on the very end, we are left with y = y^0_0 and with fracpartial y_0partial x, which is the gradient we wanted to calculate;\nif y is a scalar, then fracpartial y_0partial x is a matrix with single row\nthe Jacobian and the output of the function is calculated in one sweep.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"The above is an idealized computation. The real implementation is a bit different, as we will see later.","category":"page"},{"location":"lecture_08/lecture/#Implementation-of-the-forward-mode-using-Dual-numbers","page":"Lecture","title":"Implementation of the forward mode using Dual numbers","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Forward modes need to keep track of the output of the function and of the derivative at each computation step in the computation of the complicated function f. This can be elegantly realized with a dual number, which are conceptually similar to complex numbers, but instead of the imaginary number i dual numbers use epsilon in its second component:","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"x = v + dot v epsilon","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"where (vdot v) in mathbb R and by definition epsilon^2=0 (instead of i^2=-1 in complex numbers). What are the properties of these Dual numbers?","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"beginalign\n(v + dot v epsilon) + (u + dot u epsilon) = (v + u) + (dot v + dot u)epsilon \n(v + dot v epsilon)(u + dot u epsilon) = vu + (udot v + dot u v)epsilon + dot v dot u epsilon^2 = vu + (udot v + dot u v)epsilon \nfracv + dot v epsilonu + dot u epsilon = fracv + dot v epsilonu + dot u epsilon fracu - dot u epsilonu - dot u epsilon = fracvu - frac(dot u v - u dot v)epsilonu^2\nendalign","category":"page"},{"location":"lecture_08/lecture/#How-are-dual-numbers-related-to-differentiation?","page":"Lecture","title":"How are dual numbers related to differentiation?","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Let's evaluate the above equations at (v dot v) = (v 1) and (u dot u) = (u 0) we obtain ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"beginalign\n(v + dot v epsilon) + (u + dot u epsilon) = (v + u) + 1epsilon \n(v + dot v epsilon)(u + dot u epsilon) = vu + uepsilon\nfracv + dot v epsilonu + dot u epsilon = fracvu + frac1u epsilon\nendalign","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"and notice that terms (1 u frac1u) corresponds to gradient of functions (u+v uv fracvu) with respect to v. We can repeat it with changed values of epsilon as (v dot v) = (v 0) and (u dot u) = (u 1) and we obtain","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"beginalign\n(v + dot v epsilon) + (u + dot u epsilon) = (v + u) + 1epsilon \n(v + dot v epsilon)(u + dot u epsilon) = vu + vepsilon\nfracv + dot v epsilonu + dot u epsilon = fracvu - fracvu^2 epsilon\nendalign","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"meaning that at this moment we have obtained gradients with respect to u.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"All above functions (u+v uv fracuv) are of mathbbR^2 rightarrow mathbbR, therefore we had to repeat the calculations twice to get gradients with respect to both inputs. This is inline with the above theory, where we have said that if input dimension is larger then output dimension, the backward mode is better. But consider a case, where we have a function ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"f(v) = (v + 5 5*v 5 v) ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"which is mathbbR rightarrow mathbbR^3. In this case, we obtain the Jacobian 1 5 -frac5v^2 in a single forward pass (whereas the reverse would require three passes over the backward calculation, as will be seen later).","category":"page"},{"location":"lecture_08/lecture/#Does-dual-numbers-work-universally?","page":"Lecture","title":"Does dual numbers work universally?","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Let's first work out polynomial. Let's assume the polynomial","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"p(v) = sum_i=1^n p_iv^i","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"and compute its value at v + dot v epsilon (note that we know how to do addition and multiplication)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"beginsplit\np(v) = \n sum_i=0^n p_i(v + dotv epsilon )^i = \n sum_i=0^n leftp_i sum_j=0^nbinomijv^i-j(dot v epsilon)^iright = \n p_0 + sum_i=1^n leftp_i sum_j=0^1binomijv^i-j(dot v epsilon)^jright = \n = p_0 + sum_i=1^n p_i(v^i + i v^i-1 dot v epsilon ) \n = p(v) + left(sum_i=1^n ip_i v^i-1right) dot v epsilon\nendsplit","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"where in the multiplier of dotv epsilon: sum_i=1^n ip_i v^i - 1, we recognize the derivative of p(v) with respect to v. This proves that Dual numbers can be used to calculate the gradient of polynomials.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Let's now consider a general function fmathbbR rightarrow mathbbR. Its value at point v + dot v epsilon can be approximated using Taylor expansion at function at point v as","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"f(v+dot v epsilon) = sum_i=0^infty fracf^i(v)dot v^iepsilon^ni\n = f(v) + f(v)dot vepsilon","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"where all higher order terms can be dropped because epsilon^i=0 for i1. This shows that we can calculate the gradient of f at point v by calculating its value at f(v + epsilon) and taking the multiplier of epsilon.","category":"page"},{"location":"lecture_08/lecture/#Implementing-Dual-number-with-Julia","page":"Lecture","title":"Implementing Dual number with Julia","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"To demonstrate the simplicity of Dual numbers, consider following definition of Dual numbers, where we define a new number type and overload functions +, -, *, and /. In Julia, this reads:","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"struct Dual{T<:Number} <: Number\n x::T\n d::T\nend\n\nBase.:+(a::Dual, b::Dual) = Dual(a.x+b.x, a.d+b.d)\nBase.:-(a::Dual, b::Dual) = Dual(a.x-b.x, a.d-b.d)\nBase.:/(a::Dual, b::Dual) = Dual(a.x/b.x, (a.d*b.x - a.x*b.d)/b.x^2) # recall (a/b) = a/b + (a'b - ab')/b^2 ϵ\nBase.:*(a::Dual, b::Dual) = Dual(a.x*b.x, a.d*b.x + a.x*b.d)\n\n# Let's define some promotion rules\nDual(x::S, d::T) where {S<:Number, T<:Number} = Dual{promote_type(S, T)}(x, d)\nDual(x::Number) = Dual(x, zero(typeof(x)))\nDual{T}(x::Number) where {T} = Dual(T(x), zero(T))\nBase.promote_rule(::Type{Dual{T}}, ::Type{S}) where {T<:Number,S<:Number} = Dual{promote_type(T,S)}\nBase.promote_rule(::Type{Dual{T}}, ::Type{Dual{S}}) where {T<:Number,S<:Number} = Dual{promote_type(T,S)}\n\n# and define api for forward differentionation\nforward_diff(f::Function, x::Real) = _dual(f(Dual(x,1.0)))\n_dual(x::Dual) = x.d\n_dual(x::Vector) = _dual.(x)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"And let's test the Babylonian Square Root (an algorithm to compute sqrt x):","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"babysqrt(x, t=(1+x)/2, n=10) = n==0 ? t : babysqrt(x, (t+x/t)/2, n-1)\n\nforward_diff(babysqrt, 2) \nforward_diff(babysqrt, 2) ≈ 1/(2sqrt(2))\nforward_diff(x -> [1 + x, 5x, 5/x], 2) ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"We now compare the analytic solution to values computed by the forward_diff and byt he finite differencing","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"f(x) = sqrtx qquad f(x) = frac12sqrtx","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"using FiniteDifferences\nforward_dsqrt(x) = forward_diff(babysqrt,x)\nanalytc_dsqrt(x) = 1/(2babysqrt(x))\nforward_dsqrt(2.0)\nanalytc_dsqrt(2.0)\ncentral_fdm(5, 1)(babysqrt, 2.0)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"plot(0.0:0.01:2, babysqrt, label=\"f(x) = babysqrt(x)\", lw=3)\nplot!(0.1:0.01:2, analytc_dsqrt, label=\"Analytic f'\", ls=:dot, lw=3)\nplot!(0.1:0.01:2, forward_dsqrt, label=\"Dual Forward Mode f'\", lw=3, ls=:dash)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"","category":"page"},{"location":"lecture_08/lecture/#Takeaways","page":"Lecture","title":"Takeaways","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Forward mode f is obtained simply by pushing a Dual through babysqrt\nTo make the forward diff work in Julia, we only need to overload a few operators for forward mode AD to work on any function. Therefore the name of the approach is called operator overloading.\nFor vector valued function we can use Hyperduals\nForward diff can differentiation through the setindex! (called each time an element is assigned to a place in array, e.g. x = [1,2,3]; x[2] = 1)\nForwardDiff is implemented in ForwardDiff.jl, which might appear to be neglected, but the truth is that it is very stable and general implementation.\nForwardDiff does not have to be implemented through Dual numbers. It can be implemented similarly to ReverseDiff through multiplication of Jacobians, which is what is the community work on now (in Diffractor, Zygote with rules defined in ChainRules).","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"","category":"page"},{"location":"lecture_08/lecture/#Reverse-mode","page":"Lecture","title":"Reverse mode","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"In reverse mode, the computation of the gradient follow the opposite order. We initialize the computation by setting mathbfJ_0 = fracpartial ypartial y_0 which is again an identity matrix. Then we compute Jacobians and multiplications in the opposite order. The problem is that to calculate J_i we need to know the value of y_i^0, which cannot be calculated in the reverse pass. The backward pass therefore needs to be preceded by the forward pass, where y_i^0_i=1^n are calculated.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"The complete reverse mode algorithm therefore proceeds as ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Forward pass: iterate i from n down to 1 as\ncalculate the next intermediate output as y^0_i-1 = f_i(y^0_i) \nBackward pass: iterate i from 1 down to n as\ncalculate Jacobian J_i = leftfracf_ipartial y_iright_y_i^0 at point y_i^0\npull back the gradient as leftfracpartial f(x)partial y_iright_y^0_i = leftfracpartial y_0partial y_i-1right_y^0_i-1 times J_i","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"The need to store intermediate outs has a huge impact on memory requirements, which particularly on GPU is a big deal. Recall few lectures ago we have been discussing how excessive memory allocations can be damaging for performance, here we are given an algorithm where the excessive allocation is by design.","category":"page"},{"location":"lecture_08/lecture/#Tricks-to-decrease-memory-consumptions","page":"Lecture","title":"Tricks to decrease memory consumptions","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Define custom rules over large functional blocks. For example while we can auto-grad (in theory) matrix product, it is much more efficient to define make a matrix multiplication as one large function, for which we define Jacobians (note that by doing so, we can dispatch on Blas). e.g","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"beginalignat*2\n mathbfC = mathbfA * mathbfB \n fracpartialmathbfCpartial mathbfA = mathbfB \n fracpartialmathbfCpartial mathbfB = mathbfA^mathrmT \nendalignat*","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"When differentiating Invertible functions, calculate intermediate outputs from the output. This can lead to huge performance gain, as all data needed for computations are in caches. \nCheckpointing does not store intermediate ouputs after larger sequence of operations. When they are needed for forward pass, they are recalculated on demand.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Most reverse mode AD engines does not support mutating values of arrays (setindex! in julia). This is related to the memory consumption, where after every setindex! you need in theory save the full matrix. Enzyme differentiating directly LLVM code supports this, since in LLVM every variable is assigned just once. ForwardDiff methods does not suffer this problem, as the gradient is computed at the time of the values.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\nReverse mode AD was first published in 1976 by Seppo Linnainmaa[1], a finnish computer scientist. It was popularized in the end of 80s when applied to training multi-layer perceptrons, which gave rise to the famous backpropagation algorithm[2], which is a special case of reverse mode AD.[1]: Linnainmaa, S. (1976). Taylor expansion of the accumulated rounding error. BIT Numerical Mathematics, 16(2), 146-160.[2]: Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\nThe terminology in automatic differentiation is everything but fixed. The community around ChainRules.jl went a great length to use something reasonable. They use pullback for a function realizing vector-Jacobian product in the reverse-diff reminding that the gradient is pulled back to the origin of the computation. The use pushforward to denote the same operation in the ForwardDiff, as the gradient is push forward through the computation.","category":"page"},{"location":"lecture_08/lecture/#Implementation-details-of-reverse-AD","page":"Lecture","title":"Implementation details of reverse AD","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Reverse-mode AD needs to record operations over variables when computing the value of a differentiated function, such that it can walk back when computing the gradient. This record is called tape, but it is effectively a directed acyclic graph. The construction of the tape can be either explicit or implicit. The code computing the gradient can be produced by operator-overloading or code-rewriting techniques. This give rise of four different takes on AD, and Julia has libraries for alll four.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Yota.jl: explict tape, code-rewriting\nTracker.jl, AutoGrad.jl: implict tape, operator overloading\nReverseDiff.jl: explict tape, operator overloading\nZygote.jl: implict tape, code-rewriting","category":"page"},{"location":"lecture_08/lecture/#Graph-based-AD","page":"Lecture","title":"Graph-based AD","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"In Graph-based approach, we start with a complete knowledge of the computation graph (which is known in many cases like classical neural networks) and augment it with nodes representing the computation of the computation of the gradient (backward path). We need to be careful to add all edges representing the flow of information needed to calculate the gradient. Once the computation graph is augmented, we can find the subgraph needed to compute the desired node(s). ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Recall the example from the beginning of the lecture f(x y) = sin(x) + xy, let's observe, how the extension of the computational graph will look like. The computation graph of function f looks like","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"(Image: diff graph)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"where arrows rightarrow denote the flow of operations and we have denoted the output of function f as z and outputs of intermediate nodes as h_i standing for hidden.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"We start from the top and add a node calculating fracpartial zpartial h_3 which is an identity, needed to jump-start the differentiation. ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"(Image: diff graph)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"We connect it with the output of h_3, even though technically in this case it is not needed, as the z = h_3. We then add a node calculating fracpartial h_3partial h_2 for which we only need information about h_2 and mark it in the graph (again, this edge can be theoretically dropped due to being equal to one regardless the inputs). Following the chain rule, we need to combine fracpartial h_3partial h_2 with fracpartial zpartial h_3 to compute fracpartial zpartial h_2 which we note in the graph.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"(Image: diff graph)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"We continue with the same process with fracpartial h_3partial h_1, which we again combine with fracpartial zpartial h_1 to obtain fracpartial zpartial h_1. Continuing the reverse diff we obtain the final graph","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"(Image: diff graph) ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"containing the desired nodes fracpartial zpartial x and fracpartial zpartial y. This computational graph can be passed to the compiler to compute desired values.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"This approach to AD has been taken for example by Theano and by TensorFlow. In Tensorflow when you use functions like tf.mul( a, b ) or tf.add(a,b), you are not performing the computation in Python, but you are building the computational graph shown as above. You can then compute the values using tf.run with a desired inputs, but you are in fact computing the values in a different interpreter / compiler then in python.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Advantages:","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Knowing the computational graph in advance is great, as you can do expensive optimization steps to simplify the graph. \nThe computational graph have a simple semantics (limited support for loops, branches, no objects), and the compiler is therefore simpler than the compiler of full languages.\nSince the computation of gradient augments the graph, you can run the process again to obtain higher order gradients. \nTensorFlow allows you to specialize on sizes of Tensors, which means that it knows precisely how much memory you will need and where, which decreases the number of allocations. This is quite important in GPU.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Disadvantages:","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"You are restricted to fixed computation graph. It is generally difficult to implement if or while, and hence to change the computation according to values computed during the forward pass.\nDevelopment and debugging can be difficult, since you are not developing the computation graph in the host language.\nExploiting within computation graph parallelism might be difficult.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Comments:","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"DaggerFLux.jl use this approach to perform model-based paralelism, where parts of the computation graph (and especially parameters) can reside on different machines.\nUmlaut.jl allows to easily obtain the tape through tracing of the execution of a function, which can be then used to implement the AD as described above (see Yota's documentation for complete example).","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"using Umlaut\ng(x, y) = x * y\nf(x, y) = g(x, y)+sin(x)\ntape = trace(f, 1.0, 2.0)[2]","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Yota.jl use the tape to generate the gradient as ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"tape = Yota.gradtape(f, 1.0, 2.0; seed=1.0)\nUmlaut.to_expr(tape)","category":"page"},{"location":"lecture_08/lecture/#Tracking-based-AD","page":"Lecture","title":"Tracking-based AD","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Alternative to static-graph based methods are methods, which builds the graph during invocation of functions and then use this dynamically built graph to know, how to compute the gradient. The dynamically built graph is frequently called tape. This approach is used by popular libraries like PyTorch, AutoGrad, and Chainer in Python ecosystem, or by Tracker.jl (Flux.jl's former AD backend), ReverseDiff.jl, and AutoGrad.jl (Knet.jl's AD backend) in Julia. This type of AD systems is also called operator overloading, since in order to record the operations performed on the arguments we need to replace/wrap the original implementation.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"How do we build the tracing? Let's take a look what ReverseDiff.jl is doing. It defines TrackedArray (it also defines TrackedReal, but TrackedArray is more interesting) as","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"struct TrackedArray{T,N,V<:AbstractArray{T,N}} <: AbstractArray{T,N}\n value::V\n deriv::Union{Nothing,V}\n tape::Vector{Any}\n string_tape::String\nend","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"where in","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"value it stores the value of the array\nderiv will hold the gradient of the tracked array\ntape of will log operations performed with the tracked array, such that we can calculate the gradient as a sum of operations performed over the tape.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"What do we need to store on the tape? Let's denote as a the current TrackedArray. The gradient with respect to some output z is equal to fracpartial zpartial a = sum_g_i fracpartial zpartial g_i times fracpartial g_ipartial a where g_i is the output of any function (in the computational graph) where a was a direct input. The InstructionTape will therefore contain a reference to g_i (which has to be of TrackedArray and where we know fracpartial zpartial g_i will be stored in deriv field) and we also need to a method calculating fracpartial g_ipartial a, which can be stored as an anonymous function will accepting the grad as an argument.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"TrackedArray(a::AbstractArray, string_tape::String = \"\") = TrackedArray(a, similar(a) .= 0, [], string_tape)\nTrackedMatrix{T,V} = TrackedArray{T,2,V} where {T,V<:AbstractMatrix{T}}\nTrackedVector{T,V} = TrackedArray{T,1,V} where {T,V<:AbstractVector{T}}\nBase.show(io::IO, ::MIME\"text/plain\", a::TrackedArray) = show(io, a)\nBase.show(io::IO, a::TrackedArray) = print(io, \"TrackedArray($(size(a.value)))\")\nvalue(A::TrackedArray) = A.value\nvalue(A) = A\ntrack(A, string_tape = \"\") = TrackedArray(A, string_tape)\ntrack(a::Number, string_tape) = TrackedArray(reshape([a], 1, 1), string_tape)\n\nimport Base: +, *\nfunction *(A::TrackedMatrix, B::TrackedMatrix)\n a, b = value.((A, B))\n C = TrackedArray(a * b, \"($(A.string_tape) * $(B.string_tape))\")\n push!(A.tape, (C, ∂C -> ∂C * b'))\n push!(B.tape, (C, ∂C -> a' * ∂C))\n C\nend\n\nfunction *(A::TrackedMatrix, B::AbstractMatrix)\n a, b = value.((A, B))\n C = TrackedArray(a * b, \"($(A.string_tape) * B)\")\n push!(A.tape, (C, ∂C -> ∂C * b'))\n C\nend\n\nfunction *(A::Matrix, B::TrackedMatrix)\n a, b = value.((A, B))\n C = TrackedArray(a * b, \"A * $(B.string_tape)\")\n push!(A.tape, (C, ∂C -> ∂C * b'))\n C\nend\n\nfunction +(A::TrackedMatrix, B::TrackedMatrix)\n C = TrackedArray(value(A) + value(B), \"($(A.string_tape) + $(B.string_tape))\")\n push!(A.tape, (C, ∂C -> ∂C))\n push!(B.tape, (C, ∂C -> ∂C))\n C\nend\n\nfunction msin(A::TrackedMatrix)\n a = value(A)\n C = TrackedArray(sin.(a), \"sin($(A.string_tape))\")\n push!(A.tape, (C, ∂C -> cos.(a) .* ∂C))\n C\nend","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Let's observe that the operations are recorded on the tape as they should","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"a = rand()\nb = rand()\nA = track(a, \"A\")\nB = track(b, \"B\")\n# R = A * B + msin(A)\nC = A * B \nA.tape\nB.tape\nC.string_tape\nR = C + msin(A)\nA.tape\nB.tape\nR.string_tape","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Let's now implement a function that will recursively calculate the gradient of a term of interest. It goes over its childs, if they not have calculated the gradients, calculate it, otherwise it adds it to its own after if not, ask them to calculate the gradient and otherwise ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"function accum!(A::TrackedArray)\n isempty(A.tape) && return(A.deriv)\n A.deriv .= sum(g(accum!(r)) for (r, g) in A.tape)\n empty!(A.tape)\n A.deriv\nend","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"We can calculate the gradient by initializing the gradient of the result to vector of ones simulating the sum function","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"using FiniteDifferences\nR.deriv .= 1\naccum!(A)[1]\n∇a = grad(central_fdm(5,1), a -> a*b + sin(a), a)[1]\nA.deriv[1] ≈ ∇a\naccum!(B)[1]\n∇b = grad(central_fdm(5,1), b -> a*b + sin(a), b)[1]\nB.deriv[1] ≈ ∇b","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"The api function for computing the grad might look like","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"function trackedgrad(f, args...)\n args = track.(args)\n o = f(args...)\n fill!(o.deriv, 1)\n map(accum!, args)\nend","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"where we should assert that the output dimension is 1. In our implementation we dirtily expect the output of f to be summed to a scalar.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Let's compare the results to those computed by FiniteDifferences","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"A = rand(4,4)\nB = rand(4,4)\ntrackedgrad(A -> A * B + msin(A), A)[1]\ngrad(central_fdm(5,1), A -> sum(A * B + sin.(A)), A)[1]\ntrackedgrad(A -> A * B + msin(A), B)[1]\ngrad(central_fdm(5,1), A -> sum(A * B + sin.(A)), B)[1]","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"To make the above AD system really useful, we would need to ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Add support for TrackedReal, which is straightforward (we might skip the anonymous function, as the derivative of a scalar function is always a number).\nWe would need to add a lot of rules, how to work with basic values. This is why the the approach is called operator overloading since you need to overload a lot of functions (or methods or operators).","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"For example to add all combinations for +, we would need to add following rules.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"function +(A::TrackedMatrix, B::TrackedMatrix)\n C = TrackedArray(value(A) + value(B), \"($(A.string_tape) + $(B.string_tape))\")\n push!(A.tape, (C, ∂C -> ∂C ))\n push!(B.tape, (C, ∂C -> ∂C))\n C\nend\n\nfunction +(A::AbstractMatrix, B::TrackedMatrix)\n C = TrackedArray(A * value(B), \"(A + $(B.string_tape))\")\n push!(B.tape, (C, ∂C -> ∂C))\n C\nend","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Advantages:","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Debugging and development is nicer, as AD is implemented in the same language.\nThe computation graph, tape, is dynamic, which makes it simpler to take the gradient in the presence of if and while.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Disadvantages:","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"The computation graph is created and differentiated during every computation, which might be costly. In most deep learning applications, this overhead is negligible in comparison to time of needed to perform the operations itself (ReverseDiff.jl allows to compile the tape).\nThe compiler has limited options for optimization, since the tape is created during the execution.\nSince computation graph is dynamic, it cannot be optimized as the static graph, the same holds for the memory allocations. ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"A more complete example which allow to train feed-forward neural network on GPU can be found here.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\nThe difference between tracking and graph-based AD systems is conceptually similar to interpreted and compiled programming languages. Tracking AD systems interpret the time while computing the gradient, while graph-based AD systems compile the computation of the gradient.","category":"page"},{"location":"lecture_08/lecture/#ChainRules","page":"Lecture","title":"ChainRules","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"From our discussions about AD systems so far we see that while the basic, engine, part is relatively straightforward, the devil is in writing the rules prescribing the computation of gradients. These rules are needed for every system whether it is graph based, tracking, or Wengert list based. ForwardDiff also needs a rule system, but rules are a bit different (as they are pushing the gradient forward rather than pulling it back). It is obviously a waste of effort for each AD system to have its own set of rules. Therefore the community (initiated by Catherine Frames White backed by Invenia) have started to work on a unified system to express differentiation rules, such that they can be shared between systems. So far, they are supported by Zygote.jl, Nabla.jl, ReverseDiff.jl and Diffractor.jl, suggesting that the unification approach is working (but not by Enzyme.jl).","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"The definition of reverse diff rules follows the idea we have nailed above (we refer readers interested in forward diff rules to official documentation).","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"ChainRules defines the reverse rules for function foo in a function rrule with the following signature","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"function rrule(::typeof(foo), args...; kwargs...)\n ...\n return y, pullback\nend","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"where","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"the first argument ::typeof(foo) allows to dispatch on the function for which the rules is written\nthe output of function foo(args...) is returned as the first argument\npullback(Δy) takes the gradient of upstream functions with respect to the output of foo(args) and returns it multiplied by the jacobian of the output of foo(args) with respect to parameters of the function itself (recall the function can have parameters, as it can be a closure or a functor), and with respect to the arguments.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"function pullback(Δy)\n ...\n return ∂self, ∂args...\nend","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Notice that key-word arguments are not differentiated. This is a design decision with the explanation that parametrize the function, but most of the time, they are not differentiable.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"ChainRules.jl provides support for lazy (delayed) computation using Thunk. Its argument is a function, which is not evaluated until unthunk is called. There is also a support to signal that gradient is zero using ZeroTangent (which can save valuable memory) or to signal that the gradient does not exist using NoTangent. ","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"How can we use ChainRules to define rules for our AD system? Let's first observe the output","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"using ChainRulesCore, ChainRules\nr, g = rrule(*, rand(2,2), rand(2,2))\ng(r)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"With that, we can extend our AD system as follows","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"import Base: *, +, -\nfor f in [:*, :+, :-]\n @eval function $(f)(A::TrackedMatrix, B::AbstractMatrix)\n C, pullback = rrule($(f), value(A), B)\n C = track(C)\n push!(A.tape, (C, Δ -> pullback(Δ)[2]))\n C\n end\n\n @eval function $(f)(A::AbstractMatrix, B::TrackedMatrix)\n C, pullback = rrule($(f), A, value(B))\n C = track(C)\n push!(B.tape, (C, Δ -> pullback(Δ)[3]))\n C\n end\n\n @eval function $(f)(A::TrackedMatrix, B::TrackedMatrix)\n C, pullback = rrule($(f), value(A), value(B))\n C = track(C)\n push!(A.tape, (C, Δ -> pullback(Δ)[2]))\n push!(B.tape, (C, Δ -> pullback(Δ)[3]))\n C\n end\nend","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"and we need to modify our accum! code to unthunk if needed","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"function accum!(A::TrackedArray)\n isempty(A.tape) && return(A.deriv)\n A.deriv .= sum(unthunk(g(accum!(r))) for (r, g) in A.tape)\nend","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"A = rand(4,4)\nB = rand(4,4)\ngrad(A -> (A * B + msin(A))*B, A)[1]\ngradient(A -> sum(A * B + sin.(A)), A)[1]\ngrad(A -> A * B + msin(A), B)[1]\ngradient(A -> sum(A * B + sin.(A)), B)[1]","category":"page"},{"location":"lecture_08/lecture/#Source-to-source-AD-using-Wengert","page":"Lecture","title":"Source-to-source AD using Wengert","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Recall the compile stages of julia and look, how the lowered code for","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"f(x,y) = x*y + sin(x)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"looks like","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_lowered f(1.0, 1.0)\nCodeInfo(\n1 ─ %1 = x * y\n│ %2 = Main.sin(x)\n│ %3 = %1 + %2\n└── return %3\n)","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"This form is particularly nice for automatic differentiation, as we have on the left hand side always a single variable, which means the compiler has provided us with a form, on which we know, how to apply AD rules.","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"What if we somehow be able to talk to the compiler and get this form from him?","category":"page"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"simplest viable implementation","category":"page"},{"location":"lecture_08/lecture/#Sources-for-this-lecture","page":"Lecture","title":"Sources for this lecture","text":"","category":"section"},{"location":"lecture_08/lecture/","page":"Lecture","title":"Lecture","text":"Mike Innes' diff-zoo\nWrite Your Own StS in One Day\nBuild your own AD with Umlaut\nZygote.jl Paper and Zygote.jl Internals\nKeno's Talk\nChris' Lecture\nAutomatic-Differentiation-Based-on-Computation-Graph","category":"page"},{"location":"lecture_08/lab/#Lab-08-Reverse-Mode-Differentiation","page":"Lab","title":"Lab 08 - Reverse Mode Differentiation","text":"","category":"section"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"(Image: descend)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"In the lecture you have seen how to implement forward-mode automatic differentiation (AD). Assume you want to find the derivative fracdfdx of the function fmathbb R^2 rightarrow mathbb R","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"f(x,y) = x*y + sin(x)\nnothing # hide","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"If we have rules for *, +, and sin we could simply seed the function with Dual(x,one(x)) and read out the derivative fracdfdx from the Dual that is returned by f. If we are also interested in the derivative fracdfdy we will have to run f again, this time seeding the second argument with Dual(y,one(y)). Hence, we have to evaluate f twice if we want derivatives w.r.t to both its arguments which means that forward differentiation scales as O(N) where N is the number of inputs to f.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"dfdx = f(Dual(x,one(x)), Dual(y,zero(y)))\ndfdy = f(Dual(x,zero(x)), Dual(y,one(y)))","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Reverse-mode AD can compute gradients of functions with many inputs and one output in one go. This is great because very often we want to optimize loss functions which are exactly that: Functions with many input variables and one loss output.","category":"page"},{"location":"lecture_08/lab/#Reverse-Mode-AD","page":"Lab","title":"Reverse Mode AD","text":"","category":"section"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"With functions fmathbb R^Nrightarrowmathbb R^M and gmathbb R^Lrightarrow mathbb R^N with an input vector bm x we can define the composition of f and g as","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"bm z = (f circ g)(bm x) qquad textwhere qquad bm y=g(bm x) qquad bm z = f(bm y)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"The multivariate chainrule reads","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"leftfracpartial z_ipartial x_jright_bm x =\n sum_k=1^N leftfracpartial z_ipartial y_kright_bm y\n leftfracpartial y_kpartial x_iright_bm x","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"If you want to read about where this comes from you can check here or here. It is essentially one row of the Jacobian matrix J. Note that in order to compute the derivative we always have to know the input to the respective function, because we can only compute the derivative at a specific point (denoted by the _x _ notation). For our example","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"z = f(xy) = xy + sin(x)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"with the sub-functions g(xy)=xy and h(x)=sin(x) we get","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"leftfrac dfdxright_xy\n = leftfrac dfdgright_g(xy)cdot leftfrac dgdxright_xy\n + leftfrac dfdhright_h(x)cdot leftfrac dhdxright_x\n = 1 cdot y _y + 1cdotcos(x)_x","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"You can see that, in order to implement reverse-mode AD we have to trace and remember all inputs to our intermediate functions during the forward pass such that we can compute their gradients during the backward pass. The simplest way of doing this is by dynamically building a computation graph which tracks how each input variable affects its output variables. The graph below represents the computation of our function f.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"z = x*y + sin(x)\n\n# as a Wengert list # Partial derivatives\na = x*y # da/dx = y; da/dy = x\nb = sin(x) # db/dx = cos(x)\nz = a + b # dz/da = 1; dz/db = 1","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"(Image: graph)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"In the graph you can see that the variable x can directly affect b and a. Hence, x has two children a and b. During the forward pass we build the graph, keeping track of which input affects which output. Additionally we include the corresponding local derivatives (which we can already compute). To implement a dynamically built graph we can introduce a new number type TrackedReal which has three fields:","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"data contains the value of this node in the computation graph as obtained in the forward pass.\ngrad is initialized to nothing and will later hold the accumulated gradients (the sum in the multivariate chain rule)\nchildren is a Dict that keeps track which output variables are affected by the current node and also stores the corresponding local derivatives fracpartial fpartial g_k.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"mutable struct TrackedReal{T<:Real}\n data::T\n grad::Union{Nothing,T}\n children::Dict\n # this field is only need for printing the graph. you can safely remove it.\n name::String\nend\n\ntrack(x::Real,name=\"\") = TrackedReal(x,nothing,Dict(),name)\n\nfunction Base.show(io::IO, x::TrackedReal)\n t = isempty(x.name) ? \"(tracked)\" : \"(tracked $(x.name))\"\n print(io, \"$(x.data) $t\")\nend","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"The backward pass is nothing more than the application of the chainrule. To compute the derivative. Assuming we know how to compute the local derivatives fracpartial fpartial g_k for simple functions such as +, *, and sin, we can write a simple function that implements the gradient accumulation from above via the chainrule","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"leftfracpartial fpartial x_iright_bm x =\n sum_k=1^N leftfracpartial fpartial g_kright_bm g(bm x)\n leftfracpartial g_kpartial x_iright_bm x","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"We just have to loop over all children, collect the local derivatives, and recurse:","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"function accum!(x::TrackedReal)\n if isnothing(x.grad)\n x.grad = sum(w*accum!(v) for (v,w) in x.children)\n end\n x.grad\nend\nnothing # hide","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"where w corresponds to fracpartial fpartial g_k and accum!(v) corresponds to fracpartial g_kpartial x_i. At this point we have already implemented the core functionality of our first reverse-mode AD! The only thing left to do is implement the reverse rules for basic functions. Via recursion the chainrule is applied until we arrive at the final output z. This final output has to be seeded (just like with forward-mode) with fracpartial zpartial z=1.","category":"page"},{"location":"lecture_08/lab/#Writing-Reverse-Rules","page":"Lab","title":"Writing Reverse Rules","text":"","category":"section"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Lets start by overloading the three functions +, *, and sin that we need to build our computation graph. First, we have to track the forward computation and then we register the output z as a child of its inputs by using z as a key in the dictionary of children. The corresponding value holds the derivatives, in the case of multiplication case we simply have","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"z = a cdot b","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"for which the derivatives are","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"fracpartial zpartial a=b qquad\nfracpartial zpartial b=a","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Knowing the derivatives of * at a given point we can write our reverse rule","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"function Base.:*(a::TrackedReal, b::TrackedReal)\n z = track(a.data * b.data, \"*\")\n a.children[z] = b.data # dz/da=b\n b.children[z] = a.data # dz/db=a\n z\nend","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Creating two tracked numbers and adding them results in","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"x = track(2.0)\ny = track(3.0)\nz = x*y\nx.children\ny.children","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Implement the two remaining rules for + and sin by overloading the appropriate methods like we did for *. First you have to compute the tracked forward pass, and then register the local derivatives in the children of your input variables. Remember to return the tracked result of the forward pass in the end.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"function Base.:+(a::TrackedReal{T}, b::TrackedReal{T}) where T\n z = track(a.data + b.data, \"+\")\n a.children[z] = one(T)\n b.children[z] = one(T)\n z\nend\n\nfunction Base.sin(x::TrackedReal)\n z = track(sin(x.data), \"sin\")\n x.children[z] = cos(x.data)\n z\nend","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_08/lab/#Forward-and-Backward-Pass","page":"Lab","title":"Forward & Backward Pass","text":"","category":"section"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"To visualize that with reverse-mode AD we really do save computation we can visualize the computation graph at different stages. We start with the forward pass and keep observing x","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"using AbstractTrees\nAbstractTrees.children(v::TrackedReal) = v.children |> keys |> collect\nfunction AbstractTrees.printnode(io::IO,v::TrackedReal)\n print(io,\"$(v.name) data: $(round(v.data,digits=2)) grad: $(v.grad)\")\nend","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"x = track(2.0,\"x\");\ny = track(3.0,\"y\");\na = x*y;\nprint_tree(x)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"We can see that we x now has one child a which has the value 2.0*3.0==6.0. All the gradients are still nothing. Computing another value that depends on x will add another child.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"b = sin(x)\nprint_tree(x)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"In the final step we compute z which does not mutate the children of x because it does not depend directly on it. The result z is added as a child to both a and b.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"z = a + b\nprint_tree(x)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"For the backward pass we have to seed the initial gradient value of z and call accum! on the variable that we are interested in.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"z.grad = 1.0\ndx = accum!(x)\ndx ≈ y.data + cos(x.data)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"By accumulating the gradients for x, the gradients in the sub-tree connected to x will be evaluated. The parts of the tree that are only connected to y stay untouched.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"print_tree(x)\nprint_tree(y)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"If we now accumulate the gradients over y we re-use the gradients that are already computed. In larger computations this will save us a lot of effort!","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"info: Info\nThis also means that we have to re-build the graph for every new set of inputs!","category":"page"},{"location":"lecture_08/lab/#Optimizing-2D-Functions","page":"Lab","title":"Optimizing 2D Functions","text":"","category":"section"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Implement a function gradient(f, args::Real...) which takes a function f and its corresponding arguments (as Real numbers) and outputs the corresponding gradients","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"function gradient(f, args::Real...)\n ts = track.(args)\n y = f(ts...)\n y.grad = 1.0\n accum!.(ts)\nend\nnothing # hide","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"f(x,y) = x*y + sin(x)\ngradient(f, 2.0, 3.0)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"As an example we can find a local minimum of the function g (slightly modified to show you that we can now actually do automatic differentiation).","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"g(x,y) = y*y + sin(x)\n\nusing Plots\ncolor_scheme = cgrad(:RdYlBu_5, rev=true)\ncontour(-4:0.1:4, -2:0.1:2, g, fill=true, c=color_scheme, xlabel=\"x\", ylabel=\"y\")","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"We can find a local minimum of g by starting at an initial point (x_0y_0) and taking small steps in the opposite direction of the gradient","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"beginalign\nx_i+1 = x_i - lambda fracpartial fpartial x_i \ny_i+1 = y_i - lambda fracpartial fpartial y_i\nendalign","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"where lambda is the learning rate that has to be tuned manually.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Implement a function descend performs one step of Gradient Descent (GD) on a function f with an arbitrary number of inputs. For GD you also have to specify the learning rate lambda so the function signature should look like this","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"descend(f::Function, λ::Real, args::Real...)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"function descend(f::Function, λ::Real, args::Real...)\n Δargs = gradient(f, args...)\n args .- λ .* Δargs\nend\nnothing # hide","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Running one descend step should result in two new inputs with a smaller output for g","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"g(1.0, 1.0)\n(x,y) = descend(g, 0.2, 1.0, 1.0)\ng(x,y)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"You can minimize a g starting from an initial value. Below is a code snippet that performs a number of descend steps on two different initial points and creates an animation of each step of the GD algorithm.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"function minimize(f::Function, args::T...; niters=20, λ=0.01) where T<:Real\n paths = ntuple(_->Vector{T}(undef,niters), length(args))\n for i in 1:niters\n args = descend(f, λ, args...)\n @info f(args...)\n for j in 1:length(args)\n paths[j][i] = args[j]\n end\n end\n paths\nend\n\nxs1, ys1 = minimize(g, 1.5, -2.4, λ=0.2, niters=34)\nxs2, ys2 = minimize(g, 1.8, -2.4, λ=0.2, niters=16)\n\np1 = contour(-4:0.1:4, -2:0.1:2, g, fill=true, c=color_scheme, xlabel=\"x\", ylabel=\"y\")\nscatter!(p1, [xs1[1]], [ys1[1]], mc=:black, marker=:star, ms=7, label=\"Minimum\")\nscatter!(p1, [xs2[1]], [ys2[1]], mc=:black, marker=:star, ms=7, label=false)\nscatter!(p1, [-π/2], [0], mc=:red, marker=:star, ms=7, label=\"Initial Point\")\nscatter!(p1, xs1[1:1], ys1[1:1], mc=:black, label=\"GD Path\", xlims=(-4,4), ylims=(-2,2))\n\n@gif for i in 1:max(length(xs1), length(xs2))\n if i <= length(xs1)\n scatter!(p1, xs1[1:i], ys1[1:i], mc=:black, lw=3, xlims=(-4,4), ylims=(-2,2), label=false)\n end\n if i <= length(xs2)\n scatter!(p1, xs2[1:i], ys2[1:i], mc=:black, lw=3, label=false)\n end\n p1\nend","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"(Image: descend)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"At this point you can move to the homework of this lab. If you want to know how to generalize this simple reverse AD to work with functions that operate on Arrays, feel free to continue with the remaining volutary part of the lab.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"","category":"page"},{"location":"lecture_08/lab/#Naively-Vectorized-Reverse-AD","page":"Lab","title":"Naively Vectorized Reverse AD","text":"","category":"section"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"A naive solution to use our TrackedReal number type to differentiate functions that operate on vectors is to just use Array{<:TrackedReal}. Unfortunately, this means that we have to replace the fast BLAS matrix operations with our own matrix multiplication methods that know how to deal with TrackedReals. This results in large performance hits and your task during the rest of the lab is to implement a smarter solution to this problem.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"using LinearAlgebra\nBase.zero(::TrackedReal{T}) where T = TrackedReal(zero(T))\nLinearAlgebra.adjoint(x::TrackedReal) = x\ntrack(x::Array) = track.(x)\naccum!(xs::Array{<:TrackedReal}) = accum!.(xs)\n\nconst VecTracked = AbstractVector{<:TrackedReal}\nconst MatTracked = AbstractMatrix{<:TrackedReal}\n\nLinearAlgebra.dot(xs::VecTracked, ys::VecTracked) = mapreduce(*, +, xs, ys)\nBase.:*(X::MatTracked, y::VecTracked) = map(x->dot(x,y), eachrow(X))\nBase.:*(X::MatTracked, Y::MatTracked) = mapreduce(y->X*y, hcat, eachcol(Y))\nBase.sum(xs::AbstractArray{<:TrackedReal}) = reduce(+,xs)\n\nfunction reset!(x::TrackedReal)\n x.grad = nothing\n reset!.(keys(x.children))\n x.children = Dict()\nend\n\nX = rand(2,3)\nY = rand(3,2)\n\nfunction run()\n Xv = track(X)\n Yv = track(Y)\n z = sum(Xv * Yv)\n z.grad = 1.0\n accum!(Yv)\nend","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"julia> using BenchmarkTools\njulia> @benchmark run()\nBenchmarkTools.Trial: 10000 samples with 1 evaluation.\n Range (min … max): 44.838 μs … 8.404 ms ┊ GC (min … max): 0.00% … 98.78%\n Time (median): 48.680 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 53.048 μs ± 142.403 μs ┊ GC (mean ± σ): 4.61% ± 1.71%\n\n ▃▆█▃ \n ▂▁▁▂▂▃▆████▇▅▄▄▄▄▄▅▅▄▄▄▄▃▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂ ▃\n 44.8 μs Histogram: frequency by time 66.7 μs <\n\n Memory estimate: 26.95 KiB, allocs estimate: 392.","category":"page"},{"location":"lecture_08/lab/#Reverse-AD-with-TrackedArrays","page":"Lab","title":"Reverse AD with TrackedArrays","text":"","category":"section"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"To make use of the much faster BLAS methods we have to implement a custom array type which will offload the heavy matrix multiplications to the normal matrix methods. Start with a fresh REPL and possibly a new file that only contains the definition of our TrackedReal:","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"mutable struct TrackedReal{T<:Real}\n data::T\n grad::Union{Nothing,T}\n children::Dict\nend\n\ntrack(x::Real) = TrackedReal(x, nothing, Dict())\nnothing # hide","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Define a new TrackedArray type which subtypes and AbstractArray{T,N} and contains the three fields: data, grad, and children. Which type should grad have?","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Additionally define track(x::Array), and forward size, length, and eltype to x.data (maybe via metaprogrammming? ;).","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"mutable struct TrackedArray{T,N,A<:AbstractArray{T,N}} <: AbstractArray{T,N}\n data::A\n grad::Union{Nothing,A}\n children::Dict\nend\n\ntrack(x::Array) = TrackedArray(x, nothing, Dict())\ntrack(x::Union{TrackedArray,TrackedReal}) = x\n\nfor f in [:size, :length, :eltype]\n\teval(:(Base.$(f)(x::TrackedArray, args...) = $(f)(x.data, args...)))\nend\n\n# only needed for hashing in the children dict...\nBase.getindex(x::TrackedArray, args...) = getindex(x.data,args...)\n\n# pretty print TrackedArray\nBase.show(io::IO, x::TrackedArray) = print(io, \"Tracked $(x.data)\")\nBase.print_array(io::IO, x::TrackedArray) = Base.print_array(io, x.data)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Creating a TrackedArray should work like this:","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"track(rand(2,2))","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"function accum!(x::Union{TrackedReal,TrackedArray})\n if isnothing(x.grad)\n x.grad = sum(λ(accum!(Δ)) for (Δ,λ) in x.children)\n end\n x.grad\nend","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"To implement the first rule for * i.e. matrix multiplication we would first have to derive it. In the case of general matrix multiplication (which is a function (R^Ntimes M R^Mtimes L) rightarrow R^Ntimes L) we are not dealing with simple derivatives anymore, but with a so-called pullback which takes a wobble in the output space R^Ntimes L and returns a wiggle in the input space (either R^Ntimes M or R^Mtimes L).","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Luckily ChainRules.jl has a nice guide on how to derive array rules, so we will only state the solution for the reverse rule such that you can implement it. They read:","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"bar A = barOmega B^T qquad bar B = A^TbarOmega","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Where barOmega is the given output wobble, which in the simplest case can be the seeded value of the final node. The crucial problem to note here is that the two rules rely in barOmega being multiplied from different sides. This information would be lost if would just store B^T as the pullback for A. Hence we will store our pullbacks as closures:","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Ω̄ -> Ω̄ * B'\nΩ̄ -> A' * Ω̄","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Define the pullback for matrix multiplication i.e. Base.:*(A::TrackedArray,B::TrackedArray) by computing the primal and storing the partials as closures.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"function Base.:*(X::TrackedArray, Y::TrackedArray)\n Z = track(X.data * Y.data)\n X.children[Z] = Δ -> Δ * Y.data'\n Y.children[Z] = Δ -> X.data' * Δ\n Z\nend","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"X = rand(2,3) |> track\nY = rand(3,2) |> track\nZ = X*Y\nf = X.children[Z]\nΩ̄ = ones(size(Z)...)\nf(Ω̄)\nΩ̄*Y.data'","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Implement rules for sum, +, -, and abs2.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"function Base.sum(x::TrackedArray)\n z = track(sum(x.data))\n x.children[z] = Δ -> Δ*ones(eltype(x), size(x)...)\n z\nend\n\nfunction Base.:+(X::TrackedArray, Y::TrackedArray)\n Z = track(X.data + Y.data)\n X.children[Z] = Δ -> Δ\n Y.children[Z] = Δ -> Δ\n Z\nend\n\nfunction Base.:-(X::TrackedArray, Y::TrackedArray)\n Z = track(X.data - Y.data)\n X.children[Z] = Δ -> Δ\n Y.children[Z] = Δ -> -Δ\n Z\nend\n\nfunction Base.abs2(x::TrackedArray)\n y = track(abs2.(x.data))\n x.children[y] = Δ -> Δ .* 2x.data\n y\nend","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"X = rand(2,3)\nY = rand(3,2)\nfunction run()\n Xv = track(X)\n Yv = track(Y)\n z = sum(Xv * Yv)\n z.grad = 1.0\n accum!(Yv)\nend\nnothing # hide","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"julia> using BenchmarkTools\njulia> @benchmark run()\nBenchmarkTools.Trial: 10000 samples with 6 evaluations.\n Range (min … max): 5.797 μs … 1.618 ms ┊ GC (min … max): 0.00% … 98.97%\n Time (median): 6.530 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 7.163 μs ± 22.609 μs ┊ GC (mean ± σ): 4.42% ± 1.40%\n\n ▆█▇▇▇▆▅▄▃▃▂▂▂▁▁ ▁▁ ▂\n █████████████████████▇▇▇▆▆▅▅▅▅▆▅▄▅▅▄▁▃▁▁▄▁▃▁▁▁▃▃▄▁▁▁▄▁▃▁▅▄ █\n 5.8 μs Histogram: log(frequency) by time 15.8 μs <\n\n Memory estimate: 3.08 KiB, allocs estimate: 31.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"Even for this tiny example we are already 10 times faster than with the naively vectorized approach!","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"In order to implement a full neural network we need two more rules. One for the non-linearity and one for concatentation of individual training points to a batch.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"σ(x::Real) = 1/(1+exp(-x))\nσ(x::AbstractArray) = σ.(x)\nfunction σ(x::TrackedArray)\n z = track(σ(x.data))\n d = z.data\n x.children[z] = Δ -> Δ .* d .* (1 .- d)\n z\nend\n\nfunction Base.hcat(xs::TrackedArray...)\n y = track(hcat(data.(xs)...))\n stops = cumsum([size(x,2) for x in xs])\n starts = vcat([1], stops[1:end-1] .+ 1)\n for (start,stop,x) in zip(starts,stops,xs)\n x.children[y] = function (Δ)\n δ = if ndims(x) == 1\n Δ[:,start]\n else\n ds = map(_ -> :, size(x)) |> Base.tail |> Base.tail\n Δ[:, start:stop, ds...]\n end\n δ\n end\n end\n y\nend","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"You can see a full implementation of our tracing based AD here and a simple implementation of a Neural Network that can learn an approximation to the function g here. Running the latter script will produce an animation that shows how the network is learning.","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"(Image: anim)","category":"page"},{"location":"lecture_08/lab/","page":"Lab","title":"Lab","text":"This lab is heavily inspired by Rufflewind","category":"page"},{"location":"lecture_03/hw/#Homework-3","page":"Homework","title":"Homework 3","text":"","category":"section"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"In this homework we will implement a function find_food and practice the use of closures. The solution of lab 3 can be found here. You can use this file and add the code that you write for the homework to it.","category":"page"},{"location":"lecture_03/hw/#How-to-submit?","page":"Homework","title":"How to submit?","text":"","category":"section"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"Put all your code (including your or the provided solution of lab 2) in a script named hw.jl. Zip only this file (not its parent folder) and upload it to BRUTE.","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"projdir = dirname(Base.active_project())\ninclude(joinpath(projdir,\"src\",\"lecture_03\",\"Lab03Ecosystem.jl\"))\n\nfunction find_food(a::Animal, w::World)\n as = filter(x -> eats(a,x), w.agents |> values |> collect)\n isempty(as) ? nothing : rand(as)\nend\n\neats(::Animal{Sheep},g::Plant{Grass}) = g.size > 0\neats(::Animal{Wolf},::Animal{Sheep}) = true\neats(::Agent,::Agent) = false\n\nfunction every_nth(f::Function, n::Int)\n i = 1\n function callback(args...)\n # display(i) # comment this out to see out the counter increases\n if i == n\n f(args...)\n i = 1\n else\n i += 1\n end\n end\nend\n\nnothing # hide","category":"page"},{"location":"lecture_03/hw/#Agents-looking-for-food","page":"Homework","title":"Agents looking for food","text":"","category":"section"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"
\n
Homework:
\n
","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"Implement a method find_food(a::Animal, w::World) returns one randomly chosen agent from all w.agents that can be eaten by a or nothing if no food could be found. This means that if e.g. the animal is a Wolf you have to return one random Sheep, etc.","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"Hint: You can write a general find_food method for all animals and move the parts that are specific to the concrete animal types to a separate function. E.g. you could define a function eats(::Animal{Wolf}, ::Animal{Sheep}) = true, etc.","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"You can check your solution with the public test:","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"sheep = Sheep(1,pf=1.0)\nworld = World([Grass(2), sheep])\nfind_food(sheep, world) isa Plant{Grass}","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"
","category":"page"},{"location":"lecture_03/hw/#Callbacks-and-Closures","page":"Homework","title":"Callbacks & Closures","text":"","category":"section"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"
\n
Homework:
\n
","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"Implement a function every_nth(f::Function,n::Int) that takes an inner function f and uses a closure to construct an outer function g that only calls f every nth call to g. For example, if n=3 the inner function f be called at the 3rd, 6th, 9th ... call to g (not at the 1st, 2nd, 4th, 5th, 7th... call).","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"Hint: You can use splatting via ... to pass on an unknown number of arguments from the outer to the inner function.","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"
","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"You can use every_nth to log (or save) the agent count only every couple of steps of your simulation. Using every_nth will look like this:","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"w = World([Sheep(1), Grass(2), Wolf(3)])\n# `@info agent_count(w)` is executed only every 3rd call to logcb(w)\nlogcb = every_nth(w->(@info agent_count(w)), 3);\n\nlogcb(w); # x->(@info agent_count(w)) is not called\nlogcb(w); # x->(@info agent_count(w)) is not called\nlogcb(w); # x->(@info agent_count(w)) *is* called","category":"page"},{"location":"lecture_03/hw/","page":"Homework","title":"Homework","text":"","category":"page"},{"location":"lecture_05/lecture/#perf_lecture","page":"Lecture","title":"Benchmarking, profiling, and performance gotchas","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"This class is a short introduction to writing a performant code. As such, we cover","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"how to identify weak spots in the code\nhow to properly benchmark\ncommon performance anti-patterns\nJulia's \"performance gotchas\", by which we mean performance problems specific for Julia (typical caused by the lack of understanding of Julia or by a errors in conversion from script to functions)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Though recall the most important rule of thumb: Never optimize code from the very beginning. A much more productive workflow is ","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"write the code that is idiomatic and easy to understand\nmeticulously cover the code with unit test, such that you know that the optimized code works the same as the original\noptimize the code","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Premature optimization frequently backfires, because:","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"you might end-up optimizing wrong thing, i.e. you will not optimize performance bottleneck, but something very different\noptimized code can be difficult to read and reason about, which means it is more difficult to make it right.","category":"page"},{"location":"lecture_05/lecture/#Optimize-for-your-mode-of-operation","page":"Lecture","title":"Optimize for your mode of operation","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Let's for fun measure a difference in computation of a simple polynomial over elements of arrays between numpy, jax, default Julia, and Julia with LoopVectorization library. ","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"import numpy as np\nimport jax\nfrom jax import jit\nimport jax.numpy as jnp\njax.config.update(\"jax_enable_x64\", True)\n\n@jit\ndef f(x):\n return 3*x**3 + 2*x**2 + x + 1\n\ndef g(x):\n return 3*x**3 + 2*x**2 + x + 1\n\nx = np.random.rand(10)\nf(x)\nx = random.uniform(key, shape=(10,), dtype=jnp.float64)\ng(x)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function f(x)\n @. 3*x^3 + 2*x^2 + x + 1\nend\n\nusing LoopVectorization\nfunction f_turbo(x)\n @turbo @. 3*x^3 + 2*x^2 + x + 1\nend\n\nfunction f_tturbo(x)\n @tturbo @. 3*x^3 + 2*x^2 + x + 1\nend\n\nx = rand(10)\nf(x)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"A complete implementations can be found here: Julia and Python. Julia should be executed with multithreaded support, in the case of below image it used four threads on MacBook PRO with M1 processor with four performant and four energy efficient cores. Below figure shows the minimum execution time with respect to the","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"(Image: figure)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"It frequently happens that Julia newbies asks on forum that their code in Julia is slow in comparison to the same code in Python (numpy). Most of the time, they make trivial mistakes and it is very educative to go over their mistakes","category":"page"},{"location":"lecture_05/lecture/#Numpy-10x-faster-than-julia-what-am-i-doing-wrong?-(solved-julia-faster-now)-[1]","page":"Lecture","title":"Numpy 10x faster than julia what am i doing wrong? (solved julia faster now) [1]","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"[1]: Adapted from Julia's discourse thread","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function f(p) # line 1 \n t0,t1 = p # line 2\n m0 = [[cos(t0) - 1im*sin(t0) 0]; [0 cos(t0) + 1im*sin(t0)]] # line 3\n m1 = [[cos(t1) - 1im*sin(t1) 0]; [0 cos(t1) + 1im*sin(t1)]] # line 4\n r = m1*m0*[1. ; 0.] # line 5\n return abs(r[1])^2 # line 6\nend\n\nfunction g(p,n)\n return [f(p[:,i]) for i=1:n]\nend\n\ng(rand(2,3),3) # call to force jit compilation\n\nn = 10^6\np = 2*pi*rand(2,n)\n\n@elapsed g(p,n)\n","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Let's first use Profiler to identify, where the function spends most time.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"note: Note\nJulia's built-in profilerJulia's built-in profiler is part of the standard library in the Profile module implementing a fairly standard sampling based profiler. It a nutshell it asks at regular intervals, where the code execution is currently and marks it and collects this information in some statistics. This allows us to analyze, where these \"probes\" have occurred most of the time which implies those parts are those, where the execution of your function spends most of the time. As such, the profiler has two \"controls\", which is the delay between two consecutive probes and the maximum number of probes n (if the profile code takes a long time, you might need to increase it).using Profile\nProfile.init(; n = 989680, delay = 0.001))\n@profile g(p,n)\nProfile.clear()\n@profile g(p,n)Making sense of profiler's outputThe default Profile.print function shows the call-tree with count, how many times the probe occurred in each function sorted from the most to least. The output is a little bit difficult to read and orient in, therefore there are some visualization options.What are our options?ProfileView is the workhorse with a GTK based API and therefore recommended for those with working GTK\nProfileSVG is the ProfileView with the output exported in SVG format, which is viewed by most browser (it is also very convenient for sharing with others)\nPProf.jl is a front-end to Google's PProf profile viewer https://github.com/JuliaPerf/PProf.jl\nStatProfilerHTML https://github.com/tkluck/StatProfilerHTML.jlBy personal opinion I mostly use ProfileView (or ProfileSVG) as it indicates places of potential type instability, which as will be seen later is very useful feature. Profiling caveatsThe same function, but with keyword arguments, can be used to change these settings, however these settings are system dependent. For example on Windows, there is a known issue that does not allow to sample faster than at 0.003s and even on Linux based system this may not do much. There are some further caveat specific to Julia:When running profile from REPL, it is usually dominated by the interactive part which spawns the task and waits for it's completion.\nCode has to be run before profiling in order to filter out all the type inference and interpretation stuff. (Unless compilation is what we want to profile.)\nWhen the execution time is short, the sampling may be insufficient -> run multiple times.We will use ProfileSVG for its simplicity (especially installation). It shows the statistics in form of a flame graph which read as follows: , where . The hierarchy is expressed as functions on the bottom calls functions on the top. reads as follows:each function is represented by a horizontal bar\nfunction in the bottom calls functions above\nthe width of the bar corresponds to time spent in the function\nred colored bars indicate type instabilities\nfunctions in bottom bars calls functions on top of upper barsFunction name contains location in files and particular line number called. GTK version is even \"clickable\" and opens the file in default editor.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Let's use the profiler on the above function g to find potential weak spots","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"using Profile, ProfileSVG\nProfile.clear()\n@profile g(p, n)\nProfileSVG.save(\"profile.svg\")","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"The output can be seen here","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"We can see that the function is type stable and 2/3 of the time is spent in lines 3 and 4, which allocates arrays","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"[[cos(t0) - 1im*sin(t0) 0]; \n [0 cos(t0) + 1im*sin(t0)]]","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"and","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"[[cos(t1) - 1im*sin(t1) 0]; \n [0 cos(t1) + 1im*sin(t1)]]","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Since version 1.8, Julia offers a memory profiler, which helps to identify parts of the code allocating memory on heap. Unfortunately, ProfileSVG does not currently visualize its output, hence we are going to use PProf.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"using Profile, PProf\nProfile.Allocs.@profile g(p,n)\nPProf.Allocs.pprof(Profile.Allocs.fetch(), from_c=false)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"PProf by default shows outputs in call graph (how to read it can be found here), but also supports the flamegraph (fortunately). Investigating the output we found that most allocations are caused by concatenation of arrays on lines 3 and 4.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Scrutinizing the function f, we see that in every call, it has to allocate arrays m0 and m1 on the heap. The allocation on heap is expensive, because it might require interaction with the operating system and it potentially stress garbage collector. Can we avoid it? Repeated allocation can be frequently avoided by:","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"preallocating arrays (if the arrays are of the fixed dimensions)\nor allocating objects on stack, which does not involve interaction with OS (but can be used in limited cases.)","category":"page"},{"location":"lecture_05/lecture/#Adding-preallocation","page":"Lecture","title":"Adding preallocation","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function f!(m0, m1, p, u) \t\t\t\t\t\t\t\t\t\t# line 1 \n t0,t1 = p \t\t\t\t\t\t\t\t\t\t\t\t\t\t# line 2\n m0[1,1] = cos(t0) - 1im*sin(t0)\t\t\t\t\t\t\t\t\t# line 3\n m0[2,2] = cos(t0) + 1im*sin(t0)\t\t\t\t\t\t\t\t\t# line 4\n m1[1,1] = cos(t1) - 1im*sin(t1)\t\t\t\t\t\t\t\t\t# line 5\n m1[2,2] = cos(t1) + 1im*sin(t1)\t\t\t\t\t\t\t\t\t# line 6\n r = m1*m0*u \t\t\t\t\t\t\t\t\t\t\t\t\t# line 7\n return abs(r[1])^2 \t\t\t\t\t\t\t\t\t\t\t\t# line 8\nend\n\nfunction g2(p,n)\n u = [1. ; 0.]\n m0 = [[cos(p[1]) - 1im*sin(p[1]) 0]; [0 cos(p[1]) + 1im*sin(p[1])]]\t# line 3\n m1 = [[cos(p[2]) - 1im*sin(p[2]) 0]; [0 cos(p[2]) + 1im*sin(p[2])]]\n return [f!(m0, m1, p[:,i], u) for i=1:n]\nend","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"note: Note\nBenchmarkingThe simplest benchmarking can be as simple as writing repetitions = 100\nt₀ = time()\nfor in 1:100\n\tg(p, n)\nend\n(time() - t₀) / n where we add repetitions to calibrate for background processes that can step in the precise measurements (recall that your program is not allone). Writing the above for benchmarking is utterly boring. Moreover, you might want to automatically determine the number of repetitions (the shorter time the more repetitions you want), take care of compilation of the function outside measured loop, you might want to have more informative output, for example median, mean, and maximum time of execution, information about number of allocation, time spent in garbage collector, etc. This is in nutshell what BenchmarkTools.jl offers, which we consider an essential tool for anyone interesting in tuning its code.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"We will using macro @benchmark from BenchmarkTools.jl to observe the speedup we will get between g and g2.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"using BenchmarkTools\n\njulia> @benchmark g(p,n)\nBenchmarkTools.Trial: 5 samples with 1 evaluation.\n Range (min … max): 1.168 s … 1.199 s ┊ GC (min … max): 11.57% … 13.27%\n Time (median): 1.188 s ┊ GC (median): 11.91%\n Time (mean ± σ): 1.183 s ± 13.708 ms ┊ GC (mean ± σ): 12.10% ± 0.85%\n\n █ █ █ █ █\n █▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█ ▁\n 1.17 s Histogram: frequency by time 1.2 s <\n\n Memory estimate: 1.57 GiB, allocs estimate: 23000002.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @benchmark g2(p,n)\nBenchmarkTools.Trial: 11 samples with 1 evaluation.\n Range (min … max): 413.167 ms … 764.393 ms ┊ GC (min … max): 6.50% … 43.76%\n Time (median): 426.728 ms ┊ GC (median): 6.95%\n Time (mean ± σ): 460.688 ms ± 102.776 ms ┊ GC (mean ± σ): 12.85% ± 11.04%\n\n ▃█ █\n ██▇█▁▁▁▁▁▁▁▁▇▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▇ ▁\n 413 ms Histogram: frequency by time 764 ms <\n\n Memory estimate: 450.14 MiB, allocs estimate: 4000021.\n","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"We can see that we have approximately 3-fold improvement.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Let's profile again, not forgetting to use Profile.clear() to clear already stored probes.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Profile.clear()\n@profile g2(p,n)\nProfileSVG.save(\"/tmp/profile2.svg\")","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"What the profiler tells is now (clear here to see the output)? \t- we spend a lot of time in similar in matmul, which is again an allocation of results for storing output of multiplication on line 7 matrix r. \t- the trigonometric operations on line 3-6 are very costly \t- Slicing p always allocates a new array and performs a deep copy.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Let's get rid of memory allocations at the expense of the code clarity","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"using LinearAlgebra\n@inline function initm!(m, t)\n st, ct = sincos(t) \n @inbounds m[1,1] = Complex(ct, -st)\t\t\t\t\t\t\t\t\n @inbounds m[2,2] = Complex(ct, st) \t\t\t\t\t\t\t\t\nend\n\nfunction f1!(r1, r2, m0, m1, t0, t1, u) \t\t\t\t\t\n initm!(m0, t0)\n initm!(m1, t1)\n mul!(r1, m0, u)\n mul!(r2, m1, r1)\n return @inbounds abs(@inbounds r2[1])^2\nend\n\nfunction g3(p,n)\n u = [1. ; 0.]\n m0 = [cos(p[1]) - 1im*sin(p[1]) 0; 0 cos(p[1]) + 1im*sin(p[1])]\n m1 = [cos(p[2]) - 1im*sin(p[2]) 0; 0 cos(p[2]) + 1im*sin(p[2])]\n r1 = m0*u\n r2 = m1*r1\n return [f1!(r1, r2, m0, m1, p[1,i], p[2,i], u) for i=1:n]\nend","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @benchmark g3(p,n)\n Range (min … max): 193.922 ms … 200.234 ms ┊ GC (min … max): 0.00% … 1.67%\n Time (median): 195.335 ms ┊ GC (median): 0.00%\n Time (mean ± σ): 196.003 ms ± 1.840 ms ┊ GC (mean ± σ): 0.26% ± 0.61%\n\n █▁ ▁ ██▁█▁ ▁█ ▁ ▁ ▁ ▁ ▁ ▁ ▁▁ ▁ ▁ ▁\n ██▁▁█▁█████▁▁██▁█▁█▁█▁▁▁▁▁▁▁▁█▁▁▁█▁▁▁▁▁█▁▁▁██▁▁▁█▁▁█▁▁▁▁▁▁▁▁█ ▁\n 194 ms Histogram: frequency by time 200 ms <\n\n Memory estimate: 7.63 MiB, allocs estimate: 24.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Notice that now, we are about six times faster than the first solution, albeit passing the preallocated arrays is getting messy. Also notice that we spent a very little time in garbage collector. Running the profiler, ","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Profile.clear()\n@profile g3(p,n)\nProfileSVG.save(\"/tmp/profile3.svg\")","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"we see here that there is a very little what we can do now. May-be, remove bounds checks (more on this later) and make the code a bit nicer.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Let's look at solution from a Discourse","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"using StaticArrays, BenchmarkTools\n\nfunction f(t0,t1)\n cis0, cis1 = cis(t0), cis(t1)\n m0 = @SMatrix [ conj(cis0) 0 ; 0 cis0]\n m1 = @SMatrix [ conj(cis1) 0 ; 0 cis1]\n r = m1 * (m0 * @SVector [1. , 0.])\n return abs2(r[1])\nend\n\ng(p) = [f(p[1,i],p[2,i]) for i in axes(p,2)]","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @benchmark g(p)\n Range (min … max): 36.076 ms … 43.657 ms ┊ GC (min … max): 0.00% … 9.96%\n Time (median): 37.948 ms ┊ GC (median): 0.00%\n Time (mean ± σ): 38.441 ms ± 1.834 ms ┊ GC (mean ± σ): 1.55% ± 3.60%\n\n █▃▅ ▅▂ ▂\n ▅▇▇███████▅███▄████▅▅▅▅▄▇▅▇▄▇▄▁▄▇▄▄▅▁▄▁▄▁▄▅▅▁▁▅▁▁▅▄▅▄▁▁▁▁▁▅ ▄\n 36.1 ms Histogram: frequency by time 43.4 ms <\n\n Memory estimate: 7.63 MiB, allocs estimate: 2.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"We can see that it is six-times faster than ours while also being much nicer to read and having almost no allocations. Where is the catch? It uses StaticArrays which offers linear algebra primitices performant for vectors and matrices of small size. They are allocated on stack, therefore there is no pressure of GarbageCollector and the type is specialized on size of matrices (unlike regular matrices) works on arrays of an sizes. This allows the compiler to perform further optimizations like unrolling loops, etc.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"What we have learned so far?","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Profiler is extremely useful in identifying functions, where your code spends most time.\nMemory allocation (on heap to be specific) can be very bad for the performance. We can generally avoided by pre-allocation (if possible) or allocating on the stack (Julia offers increasingly larger number of primitives for hits. We have already seen StaticArrays, DataFrames now offers for example String3, String7, String15, String31).\nBenchmarking is useful for comparison of solutions","category":"page"},{"location":"lecture_05/lecture/#Replacing-deep-copies-with-shallow-copies-(use-view-if-possible)","page":"Lecture","title":"Replacing deep copies with shallow copies (use view if possible)","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Let's look at the following function computing mean of a columns","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function cmean(x::AbstractMatrix{T}) where {T}\n\to = zeros(T, size(x,1))\n\tfor i in axes(x, 2)\n\t\to .+= x[:,i]\t\t\t\t\t\t\t\t# line 4\n\tend\n\tn = size(x, 2)\n\tn > 0 ? o ./ n : o \nend\nx = randn(2, 10000)\n","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"@benchmark cmean(x)\nBenchmarkTools.Trial: 10000 samples with 1 evaluation.\n Range (min … max): 371.018 μs … 3.291 ms ┊ GC (min … max): 0.00% … 83.30%\n Time (median): 419.182 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 482.785 μs ± 331.939 μs ┊ GC (mean ± σ): 9.91% ± 12.02%\n\n ▃█▄▃▃▂▁ ▁\n ████████▇▆▅▃▁▁▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▃▇██ █\n 371 μs Histogram: log(frequency) by time 2.65 ms <\n\n Memory estimate: 937.59 KiB, allocs estimate: 10001.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"What we see that function is performing more than 10000 allocations. They come from x[:,i] which allocates a new memory and copies the content. In this case, this is completely unnecessary, as the content of the array x is never modified. We can avoid it by creating a view into an x, which you can imagine as a pointer to x which automatically adjust the bounds. Views can be constructed either using a function call view(x, axes...) or using a convenience macro @view which turns the usual notation x[...] to view(x, ...)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function view_cmean(x::AbstractMatrix{T}) where {T}\n\to = zeros(T, size(x,1))\n\tfor i in axes(x, 2)\n\t\to .+= @view x[:,i]\n\tend\n\tn = size(x,2)\n\tn > 0 ? o ./ n : o \nend","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"We obtain instantly a 10-fold speedup","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @benchmark view_cmean(x)\nBenchmarkTools.Trial: 10000 samples with 1 evaluation.\n Range (min … max): 36.802 μs … 166.260 μs ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 41.676 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 42.936 μs ± 9.921 μs ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n ▂ █▆█▆▂ ▁▁ ▁ ▁ ▂\n █▄█████████▇▅██▆█▆██▆▆▇▆▆▆▆▇▆▅▆▆▅▅▁▅▅▆▇▆▆▆▆▄▃▆▆▆▄▆▄▅▅▄▆▅▆▅▄▆ █\n 36.8 μs Histogram: log(frequency) by time 97.8 μs <\n\n Memory estimate: 96 bytes, allocs estimate: 1.","category":"page"},{"location":"lecture_05/lecture/#Traverse-arrays-in-the-right-order","page":"Lecture","title":"Traverse arrays in the right order","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Let's now compute rowmean using the function similar to cmean and since we have learnt from the above, we use the view to have non-allocating version","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function rmean(x::AbstractMatrix{T}) where {T}\n\to = zeros(T, size(x,2))\n\tfor i in axes(x, 1)\n\t\to .+= @view x[i,:]\n\tend\n\tn = size(x,1)\n\tn > 0 ? o ./ n : o \nend\nx = randn(10000, 2)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"x = randn(10000, 2)\n@benchmark rmean(x)\nBenchmarkTools.Trial: 10000 samples with 1 evaluation.\n Range (min … max): 44.165 μs … 194.395 μs ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 46.654 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 48.544 μs ± 10.940 μs ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n █▆█▇▄▁ ▁ ▂\n ██████▇▇▇▇▆▇▅██▇█▇█▇▆▅▄▄▅▅▄▄▄▄▂▄▅▆▅▅▅▆▅▅▅▆▄▆▄▄▅▅▄▅▄▄▅▅▅▅▄▄▃▅ █\n 44.2 μs Histogram: log(frequency) by time 108 μs <\n\n Memory estimate: 192 bytes, allocs estimate: 2.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"The above seems OK and the speed is comparable to our tuned cmean. But, can we actually do better? We have to realize that when we are accessing slices in the matrix x, they are not aligned in the memory. Recall that Julia is column major (like Fortran and unlike C and Python), which means that consecutive arrays of memory are along columns. i.e for a matrix with n rows and m columns they are aligned as ","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"1 | n + 1 | 2n + 1 | ⋯ | (m-1)n + 1\n2 | n + 2 | 2n + 2 | ⋯ | (m-1)n + 2\n3 | n + 3 | 2n + 3 | ⋯ | (m-1)n + 3\n⋮ | ⋮ | ⋮ | ⋯ | ⋮ \nn | 2n | 3n | ⋯ | mn","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"accessing non-consecutively is really bad for cache, as we have to load the memory into a cache line and use a single entry (in case of Float64 it is 8 bytes) out of it, discard it and load another one. If cache line has length 32 bytes, then we are wasting remaining 24 bytes. Therefore, we rewrite rmean to access the memory in consecutive blocks as follows, where we essentially sum the matrix column by columns.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function aligned_rmean(x::AbstractMatrix{T}) where {T}\n\to = zeros(T, size(x,2))\n\tfor i in axes(x, 2)\n\t\to[i] = sum(@view x[:, i])\n\tend\n\tn = size(x, 1)\n\tn > 0 ? o ./ n : o \nend\n\naligned_rmean(x) ≈ rmean(x)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @benchmark aligned_rmean(x)\nBenchmarkTools.Trial: 10000 samples with 10 evaluations.\n Range (min … max): 1.988 μs … 11.797 μs ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 2.041 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 2.167 μs ± 568.616 ns ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n █▇▄▂▂▁▁ ▁ ▂▁ ▂\n ██████████████▅▅▃▁▁▁▁▁▄▅▄▁▅▆▆▆▇▇▆▆▆▆▅▃▅▅▄▅▅▄▄▄▃▃▁▁▁▄▁▁▄▃▄▃▆ █\n 1.99 μs Histogram: log(frequency) by time 5.57 μs <\n\n Memory estimate: 192 bytes, allocs estimate: 2.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Running the benchmark shows that we have about 20x speedup and we are on par with Julia's built-in functions.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Remark tempting it might be, there is actually nothing we can do to speed-up the cmean function. This trouble is inherent to the processor design and you should be careful how you align things in the memory, such that it is performant in your project","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Detecting this type of inefficiencies is generally difficult, and requires processor assisted measurement. LIKWID.jl is a wrapper for a LIKWID library providing various processor level statistics, like throughput, cache misses","category":"page"},{"location":"lecture_05/lecture/#Type-stability","page":"Lecture","title":"Type stability","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Sometimes it happens that we create a non-stable code, which might be difficult to spot at first, for a non-trained eye. A prototypical example of such bug is as follows","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function poor_sum(x)\n s = 0\n for xᵢ in x\n s += xᵢ\n end\n s\nend","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"x = randn(10^8);\njulia> @benchmark poor_sum(x)\nBenchmarkTools.Trial: 23 samples with 1 evaluation.\n Range (min … max): 222.055 ms … 233.552 ms ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 225.259 ms ┊ GC (median): 0.00%\n Time (mean ± σ): 225.906 ms ± 3.016 ms ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n ▁ ▁ ▁▁█ ▁▁ ▁ ▁█ ▁ ▁ ▁ ▁ ▁ ▁▁▁▁ ▁ ▁\n █▁█▁███▁▁██▁▁█▁██▁█▁█▁█▁█▁█▁▁▁▁████▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁█ ▁\n 222 ms Histogram: frequency by time 234 ms <\n\n Memory estimate: 16 bytes, allocs estimate: 1.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Can we do better? Let's look what profiler says.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"using Profile, ProfileSVG\nProfile.clear()\n@profile poor_sum(x)\nProfileSVG.save(\"/tmp/profile4.svg\")","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"The profiler (output here) does not show any red, which means that according to the profilerthe code is type stable (and so does the @code_typed poor_sum(x) does not show anything bad.) Yet, we can see that the fourth line of the poor_sum function takes unusually long (there is a white area above, which means that the time spend in childs of that line (iteration and sum) does the sum to the time spent in the line, which is fishy). ","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"A close lookup on the code reveals that s is initialized as Int64, because typeof(0) is Int64. But then in the loop, we add to s a Float64 because x is Vector{Float64}, which means during the execution, the type s changes the type.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"So why nor compiler nor @code_typed(poor_sum(x)) warns us about the type instability? This is because of the optimization called small unions, where Julia can optimize \"small\" type instabilitites (recall the second lecture).","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"We can fix it for example by initializing x to be the zero of an element type of the array x (though this solution technically assumes x is an array, which means that poor_sum will not work for generators)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function stable_sum(x)\n s = zero(eltype(x))\n for xᵢ in x\n s += xᵢ\n end\n s\nend","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"But there is no difference, due to small union optimization (the above would kill any performance in older versions.)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @benchmark stable_sum(x)\nBenchmarkTools.Trial: 42 samples with 1 evaluation.\n Range (min … max): 119.491 ms … 123.062 ms ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 120.535 ms ┊ GC (median): 0.00%\n Time (mean ± σ): 120.687 ms ± 819.740 μs ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n █\n ▅▁▅▁▅▅██▅▁█▁█▁██▅▁█▅▅▁█▅▁█▁█▅▅▅█▁▁▁▁▁▁▁▅▁▁▁▁▁▅▁▅▁▁▁▁▁▁▅▁▁▁▁▁▅ ▁\n 119 ms Histogram: frequency by time 123 ms <\n\n Memory estimate: 16 bytes, allocs estimate: 1.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\nThe optimization of small unions has been added in Julia 1.0. If we compare the of the same function in Julia 0.6, the difference is strikingjulia> @time poor_sum(x)\n 1.863665 seconds (300.00 M allocations: 4.470 GiB, 4.29% gc time)\n9647.736705951513\njulia> @time stable_sum(x)\n 0.167794 seconds (5 allocations: 176 bytes)\n9647.736705951513The optimization of small unions is a big deal. It simplifies implementation of arrays with missing values, or allows to signal that result has not been produced by returning missing. In case of arrays with missing values, the type of element is Union{Missing,T} where T is the type of non-missing element.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"We can tell Julia that it is safe to vectorize the code. Julia tries to vectorize anyway, but @simd macro allows more aggressive operations, such as instruction reordering, which might change the output due imprecision of representation of real numbers in Floats.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function simd_sum(x)\n s = zero(eltype(x))\n @simd for xᵢ in x\n s += xᵢ\n end\n s\nend","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @benchmark simd_sum(x)\nBenchmarkTools.Trial: 90 samples with 1 evaluation.\n Range (min … max): 50.854 ms … 62.260 ms ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 54.656 ms ┊ GC (median): 0.00%\n Time (mean ± σ): 55.630 ms ± 3.437 ms ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n █ ▂ ▄ ▂ ▂ ▂ ▄\n ▄▆█▆▁█▄██▁▁█▆██▆▄█▁▆▄▁▆▆▄▁▁▆▁▁▁▁▄██▁█▁▁█▄▄▆▆▄▄▁▄▁▁▁▄█▁▆▁▆▁▆ ▁\n 50.9 ms Histogram: frequency by time 62.1 ms <\n\n Memory estimate: 16 bytes, allocs estimate: 1.","category":"page"},{"location":"lecture_05/lecture/#Untyped-global-variables-introduce-type-instability","page":"Lecture","title":"Untyped global variables introduce type instability","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function implicit_sum()\n\ts = zero(eltype(y))\n\t@simd for yᵢ in y\n\t\ts += yᵢ\n\tend\n\ts\nend","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> y = randn(10^8);\njulia> @benchmark implicit_sum()\nBenchmarkTools.Trial: 1 sample with 1 evaluation.\n Single result which took 10.837 s (11.34% GC) to evaluate,\n with a memory estimate of 8.94 GiB, over 499998980 allocations.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"What? The same function where I made the parameters to be implicit has just turned nine orders of magnitude slower? ","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Let's look what the profiler says","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Profile.clear()\ny = randn(10^4)\n@profile implicit_sum()\nProfileSVG.save(\"/tmp/profile5.svg\")","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"(output available here) which does not say anything except that there is a huge type-instability (red bar). In fact, the whole computation is dominated by Julia constantly determining the type of something.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"How can we determine, where is the type instability?","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"@code_typed implicit_sum() is \nCthulhu as @descend implicit_sum()\nJET available for Julia 1.7 onward","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\nJETJET is a code analyzer, which analyze the code without actually invoking it. The technique is called \"abstract interpretation\" and JET internally uses Julia's native type inference implementation, so it can analyze code as fast/correctly as Julia's code generation. JET internally traces the compiler's knowledge about types and detects, where the compiler cannot infer the type (outputs Any). Note that small unions are no longer considered type instability, since as we have seen above, the performance bottleneck is small. We can use JET as \tusing JET\n\t@report_opt implicit_sum()","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"All of these tools tells us that the Julia's compiler cannot determine the type of x. But why? I can just invoke typeof(x) and I know immediately the type of x. ","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"To understand the problem, you have to think about the compiler.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"You define function implicit_sum().\nIf you call implicit_sum and y does not exist, Julia will happily crash.\nIf you call implicit_sum and y exist, the function will give you the result (albeit slowly). At this moment, Julia has to specialize implicit_sum. It has two options how to behave with respect to y. \na. The compiler can assume that type of y is the current typeof(y) but that would mean that if a user redefines y and change the type, the specialization of the function implicit_sum will assume the wrong type of y and it can have unexpected results.\nb. The compiler take safe approach and determine the type of y inside the function implicit_sum and behave accordingly (recall that julia is dynamically typed). Yet, not knowing the type precisely is absolute disaster for performance. You can see this assumption for yourself by typing @code_typed implicit_sum().","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Notice the compiler dispatches on the name of the function and type of its arguments, hence, the compiler cannot create different versions of implicit_sum for different types of y, since it is not an argument, hence the dynamic resolution of types y inside implicit_sum function.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Julia takes the safe approach, which we can verify that although the implicit_sum was specialized (compiled) when y was Vector{Float64}, it works for other types","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"y = rand(Int, 1000)\nimplicit_sum() ≈ sum(y)\ny = map(x -> Complex(y...), zip(rand(1000), rand(1000)))\nimplicit_sum() ≈ sum(y)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"This means, using global variables inside functions without passing them as arguments ultimately leads to type-instability. What are the solutions?","category":"page"},{"location":"lecture_05/lecture/#Julia-1.7-and-below-Declaring-y-as-const","page":"Lecture","title":"Julia 1.7 and below => Declaring y as const","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"We can declare y as const, which tells the compiler that y will not change (and for the compiler mainly indicates that type of y will not change).","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Let's see that, but restart the julia before trying. After defining y as const, we see that the speed is the same as of simd_sum().","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @benchmark implicit_sum()\nBenchmarkTools.Trial: 99 samples with 1 evaluation.\n Range (min … max): 47.864 ms … 58.365 ms ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 50.042 ms ┊ GC (median): 0.00%\n Time (mean ± σ): 50.479 ms ± 1.598 ms ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n ▂ █▂▂▇ ▅ ▃\n ▃▁▃▁▁▁▁▇██████▅█▆██▇▅▆▁▁▃▅▃▃▁▃▃▁▃▃▁▁▃▁▃▁▁▁▁▁▁▁▁▁▁▁▁▁▃▁▁▁▁▁▃ ▁\n 47.9 ms Histogram: frequency by time 57.1 ms <\n\n Memory estimate: 0 bytes, allocs estimate: 0.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Also notice the difference in @code_typed implicit_sum()","category":"page"},{"location":"lecture_05/lecture/#Julia-1.8-and-above-Provide-type-to-y","page":"Lecture","title":"Julia 1.8 and above => Provide type to y","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Julia 1.8 added support for typed global variables which solves the above problem as can be seen from (do not forget to restart julia)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"y::Vector{Float64} = rand(10^8);\n``julia\n@benchmark implicit_sum()\n@code_typed implicit_sum()","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Unlike in const, we are free to change the bindings if it is possible to convert it to typeof(y)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"y = [1.0,2.0]\ntypeof(y)\ny = [1,2]\ntypeof(y)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"but y = [\"1\",\"2\"] will issue an error, since String has no default conversion rule to Float64 (you can overwrite this by defining Base.convert(::Type{Float64}, s::String) = parse(Float64, s) but it will likely lead to all kinds of shenanigans).","category":"page"},{"location":"lecture_05/lecture/#Barier-function","page":"Lecture","title":"Barier function","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Recall the reason, why the implicit_sum is so slow is that every time the function invokes getindex and +, it has to resolve types. The solution would be to limit the number of resolutions, which can done by passing all parameters to inner function as follows (do not forget to restart julia if you have defined y as const before).","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"using BenchmarkTools\nfunction barrier_sum()\n simd_sum(y)\nend\ny = randn(10^8);","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"@benchmark barrier_sum()\nBenchmarkTools.Trial: 93 samples with 1 evaluation.\n Range (min … max): 50.229 ms … 58.484 ms ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 53.882 ms ┊ GC (median): 0.00%\n Time (mean ± σ): 54.064 ms ± 2.892 ms ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n ▂▆█ ▆▄\n ▆█████▆▄█▆▄▆▁▄▄▄▄▁▁▄▁▄▄▆▁▄▄▄▁▁▄▁▁▄▁▁▆▆▁▁▄▄▁▄▆████▄▆▄█▆▄▄▄▄█ ▁\n 50.2 ms Histogram: frequency by time 58.4 ms <\n\n Memory estimate: 16 bytes, allocs estimate: 1.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"using JET\n@report_opt barrier_sum()","category":"page"},{"location":"lecture_05/lecture/#Checking-bounds-is-expensive","page":"Lecture","title":"Checking bounds is expensive","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"By default, julia checks bounds on every access to a location on an array, which can be difficult. Consider a following quicksort","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function qsort!(a,lo,hi)\n i, j = lo, hi\n while i < hi\n pivot = a[(lo+hi)>>>1]\n while i <= j\n while a[i] < pivot; i = i+1; end\n while a[j] > pivot; j = j-1; end\n if i <= j\n a[i], a[j] = a[j], a[i]\n i, j = i+1, j-1\n end\n end\n if lo < j; qsort!(a,lo,j); end\n lo, j = i, hi\n end\n return a\nend\n\nqsort!(a) = qsort!(a,1,length(a))","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"On lines 6 and 7 the qsort! accesses elements of array and upon every access julia checks bounds. We can signal to the compiler that it is safe not to check bounds using macro @inbounds as follows","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function inqsort!(a,lo,hi)\n i, j = lo, hi\n @inbounds while i < hi\n pivot = a[(lo+hi)>>>1]\n while i <= j\n while a[i] < pivot; i = i+1; end\n while a[j] > pivot; j = j-1; end\n if i <= j\n a[i], a[j] = a[j], a[i]\n i, j = i+1, j-1\n end\n end\n if lo < j; inqsort!(a,lo,j); end\n lo, j = i, hi\n end\n return a\nend\n\ninqsort!(a) = inqsort!(a,1,length(a))","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"We @benchmark to measure the impact","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> a = randn(1000);\njulia> @benchmark qsort!($(a))\nBenchmarkTools.Trial: 10000 samples with 4 evaluations.\n Range (min … max): 7.324 μs … 41.118 μs ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 7.415 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 7.666 μs ± 1.251 μs ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n ▇█ ▁ ▁ ▁ ▁ ▁\n ██▄█▆█▆▆█▃▇▃▁█▆▁█▅▃█▆▁▆▇▃▄█▆▄▆█▇▅██▄▃▃█▆▁▁▃▄▃▁▃▁▆▅▅▅▁▃▃▅▆▆ █\n 7.32 μs Histogram: log(frequency) by time 12.1 μs <\n\n Memory estimate: 0 bytes, allocs estimate: 0.\n\n\njulia> @benchmark inqsort!($(a))\nBenchmarkTools.Trial: 10000 samples with 7 evaluations.\n Range (min … max): 4.523 μs … 873.401 μs ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 4.901 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 5.779 μs ± 9.165 μs ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n ▄█▇▅▁▁▁▁ ▁▁▂▃▂▁▁ ▁ ▁\n █████████▆▆▆▆▆▇██████████▇▇▆▆▆▇█▇▆▅▅▆▇▅▅▆▅▅▅▇▄▅▆▅▃▅▅▆▅▄▄▃▅▅ █\n 4.52 μs Histogram: log(frequency) by time 14.8 μs <\n\n Memory estimate: 0 bytes, allocs estimate: 0.\n","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"and see that by not checking bounds, the code is 33% faster.","category":"page"},{"location":"lecture_05/lecture/#Boxing-in-closure","page":"Lecture","title":"Boxing in closure","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Recall closure is a function which contains some parameters contained ","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"An example of closure (adopted from JET.jl)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function abmult(r::Int)\n if r < 0\n r = -r\n end\n # the closure assigned to `f` make the variable `r` captured\n f = x -> x * r\n return f\nend;","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Another example of closure counting the error and printing it every steps","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function initcallback(; steps = 10)\n i = 0\n ts = time()\n y = 0.0\n cby = function evalcb(_y)\n i += 1.0\n y += _y\n if mod(i, steps) == 0\n l = y / steps\n y = 0.0\n println(i, \": loss: \", l,\" time per step: \",round((time() - ts)/steps, sigdigits = 2))\n ts = time()\n end\n end\n cby\nend\n\n\ncby = initcallback()\n\nfor i in 1:100\n cby(rand())\nend","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function simulation()\n cby = initcallback(;steps = 10000)\t#intentionally disable printing\n for i in 1:1000\n cby(sin(rand()))\n end\nend\n\n@benchmark simulation()","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"using Profile, ProfileSVG\nProfile.clear()\n@profile (for i in 1:100; simulation(); end)\nProfileSVG.save(\"/tmp/profile.svg\")","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"We see a red bars in lines 4 and 8 of evalcb, which indicates the type instability hindering the performance. Why they are there? The answer is tricky.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"In closures, as the name suggest, function closes over (or captures) some variables defined in the function outside the function that is returned. If these variables are of primitive types (think Int, Float64, etc.), the compiler assumes that they might be changed. Though when primitive types are used in calculations, the result is not written to the same memory location but to a new location and the name of the variable is made to point to this new variable location (this is called rebinding). We can demonstrate it on this example [2].","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"[2]: Invenia blog entry","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> x = [1];\n\njulia> objectid(x)\n0x79eedc509237c203\n\njulia> x .= [10]; # mutating contents\n\njulia> objectid(x)\n0x79eedc509237c203\n\njulia> y = 100;\n\njulia> objectid(y)\n0xdb216d4e5c739c77\n\njulia> y = y + 100; # rebinding the variable name\n\njulia> objectid(y)\n0xb642af5f06b41e88","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Since the inner function needs to point to the same location, julia uses Box container which can be seen as a translation, where the pointer inside the Box can change while the inner function contains the same pointer to the Box. This makes possible to change the captured variables and tracks changes in the point. Sometimes (it can happen many time) the compiler fails to determine that the captured variable is read only, and it wrap it (box it) in the Box wrapper, which makes it type unstable, as Box does not track types (it would be difficult as even the type can change in the inner function). This is what we can see in the first example of abmult. In the second example, the captured variable y and i changes and the compiler is right.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"What can we do?","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"The first difficulty is to even detect this case. We can spot it using @code_typed and of course JET.jl can do it and it will warn us. Above we have seen the effect of the profiler.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Using @code_typed","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_typed abmult(1)\nCodeInfo(\n1 ─ %1 = Core.Box::Type{Core.Box}\n│ %2 = %new(%1, r@_2)::Core.Box\n│ %3 = Core.isdefined(%2, :contents)::Bool\n└── goto #3 if not %3\n2 ─ goto #4\n3 ─ $(Expr(:throw_undef_if_not, :r, false))::Any\n4 ┄ %7 = Core.getfield(%2, :contents)::Any\n│ %8 = (%7 < 0)::Any\n└── goto #9 if not %8\n5 ─ %10 = Core.isdefined(%2, :contents)::Bool\n└── goto #7 if not %10\n6 ─ goto #8\n7 ─ $(Expr(:throw_undef_if_not, :r, false))::Any\n8 ┄ %14 = Core.getfield(%2, :contents)::Any\n│ %15 = -%14::Any\n└── Core.setfield!(%2, :contents, %15)::Any\n9 ┄ %17 = %new(Main.:(var\"#5#6\"), %2)::var\"#5#6\"\n└── return %17\n) => var\"#5#6\"","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Using Jet.jl (recall it requires the very latest Julia 1.7)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @report_opt abmult(1)\n═════ 3 possible errors found ═════\n┌ @ REPL[15]:2 r = Core.Box(:(_7::Int64))\n│ captured variable `r` detected\n└──────────────\n┌ @ REPL[15]:2 Main.<(%7, 0)\n│ runtime dispatch detected: Main.<(%7::Any, 0)\n└──────────────\n┌ @ REPL[15]:3 Main.-(%14)\n│ runtime dispatch detected: Main.-(%14::Any)\n└──────────────","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Sometimes, we do not have to do anything. For example the above example of evalcb function, we assume that all the other code in the simulation would take much more time so a little type instability is not important.\nAlternatively, we can explicitly use Ref instead of the Box, which are typed wrappers, but they are awkward to use. ","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function ref_abmult(r::Int)\n if r < 0\n r = -r\n end\n rr = Ref(r)\n f = x -> x * rr[]\n return f\nend;","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"We can see in @code_typed that the compiler is happy as it can resolve the types correctly","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_typed ref_abmult(1)\nCodeInfo(\n1 ─ %1 = Base.slt_int(r@_2, 0)::Bool\n└── goto #3 if not %1\n2 ─ %3 = Base.neg_int(r@_2)::Int64\n3 ┄ %4 = φ (#2 => %3, #1 => _2)::Int64\n│ %5 = %new(Base.RefValue{Int64}, %4)::Base.RefValue{Int64}\n│ %6 = %new(var\"#7#8\"{Base.RefValue{Int64}}, %5)::var\"#7#8\"{Base.RefValue{Int64}}\n└── return %6\n) => var\"#7#8\"{Base.RefValue{Int64}}","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Jet is also happy.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"\njulia> @report_opt ref_abmult(1)\nNo errors !\n","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"So when you use closures, you should be careful of the accidental boxing, since it can inhibit the speed of code. This is a big deal in Multithreadding and in automatic differentiation, both heavily uses closures. You can track the discussion here.","category":"page"},{"location":"lecture_05/lecture/#NamedTuples-are-more-efficient-that-Dicts","page":"Lecture","title":"NamedTuples are more efficient that Dicts","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"It happens a lot in scientific code, that some experiments have many parameters. It is therefore very convenient to store them in Dict, such that when adding a new parameter, we do not have to go over all defined functions and redefine them.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Imagine that we have a (nonsensical) simulation like ","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"settings = Dict(:stepsize => 0.01, :h => 0.001, :iters => 500, :info => \"info\")\nfunction find_min!(f, x, p)\n for i in 1:p[:iters]\n x̃ = x + p[:h]\n fx = f(x) # line 4\n x -= p[:stepsize] * (f(x̃) - fx)/p[:h] # line 5\n end\n x\nend","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Notice the parameter p is a Dict and that it can contain arbitrary parameters, which is useful. Hence, Dict is cool for passing parameters. Let's now run the function through the profiler","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"x₀ = rand()\nf(x) = x^2\nProfile.clear()\n@profile find_min!(f, x₀, settings)\nProfileSVG.save(\"/tmp/profile6.svg\")","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"from the profiler's output here we can see some type instabilities. Where they come from? The compiler does not have any information about types stored in settings, as the type of stored values are Any (caused by storing String and Int).","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> typeof(settings)\nDict{Symbol, Any}","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"The second problem is get operation on dictionaries is very time consuming operation (although technically it is O(1)), because it has to search the key in the list. Dicts are designed as a mutable container, which is not needed in our use-case, as the settings are static. For similar use-cases, Julia offers NamedTuple, with which we can construct settings as ","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"nt_settings = (;stepsize = 0.01, h=0.001, iters=500, :info => \"info\")","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"The NamedTuple is fully typed, but which we mean the names of fields are part of the type definition and fields are also part of type definition. You can think of it as a struct. Moreover, when accessing fields in NamedTuple, compiler knows precisely where they are located in the memory, which drastically reduces the access time. Let's see the effect in BenchmarkTools.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"julia> @benchmark find_min!(x -> x^2, x₀, settings)\nBenchmarkTools.Trial: 10000 samples with 1 evaluation.\n Range (min … max): 86.350 μs … 4.814 ms ┊ GC (min … max): 0.00% … 97.61%\n Time (median): 90.747 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 102.405 μs ± 127.653 μs ┊ GC (mean ± σ): 4.69% ± 3.75%\n\n ▅██▆▂ ▁▁ ▁ ▂\n ███████▇▇████▇███▇█▇████▇▇▆▆▇▆▇▇▇▆▆▆▆▇▆▇▇▅▇▆▆▆▆▄▅▅▄▅▆▆▅▄▅▃▅▃▅ █\n 86.4 μs Histogram: log(frequency) by time 209 μs <\n\n Memory estimate: 70.36 KiB, allocs estimate: 4002.\n\njulia> @benchmark find_min!(x -> x^2, x₀, nt_settings)\nBenchmarkTools.Trial: 10000 samples with 7 evaluations.\n Range (min … max): 4.179 μs … 21.306 μs ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 4.188 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 4.493 μs ± 1.135 μs ┊ GC (mean ± σ): 0.00% ± 0.00%\n\n █▃▁ ▁ ▁ ▁ ▁\n ████▇████▄██▄█▃██▄▄▇▇▇▇▅▆▆▅▄▄▅▄▅▅▅▄▁▅▄▁▄▄▆▆▇▄▅▆▄▄▃▄▆▅▆▁▄▄▄ █\n 4.18 μs Histogram: log(frequency) by time 10.8 μs <\n\n Memory estimate: 16 bytes, allocs estimate: 1.","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"Checking the output with JET, there is no type instability anymore","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"@report_opt find_min!(f, x₀, nt_settings)\nNo errors !","category":"page"},{"location":"lecture_05/lecture/#Don't-use-IO-unless-you-have-to","page":"Lecture","title":"Don't use IO unless you have to","text":"","category":"section"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"debug printing in performance critical code should be kept to minimum or using in memory/file based logger in stdlib Logging.jl","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function find_min!(f, x, p; verbose=true)\n\tfor i in 1:p[:iters]\n\t\tx̃ = x + p[:h]\n\t\tfx = f(x)\n\t\tx -= p[:stepsize] * (f(x̃) - fx)/p[:h]\n\t\tverbose && println(\"x = \", x, \" | f(x) = \", fx)\n\tend\n\tx\nend\n\n@btime find_min!($f, $x₀, $params_tuple; verbose=true)\n@btime find_min!($f, $x₀, $params_tuple; verbose=false)","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"interpolation of strings is even worse https://docs.julialang.org/en/v1/manual/performance-tips/#Avoid-string-interpolation-for-I/O","category":"page"},{"location":"lecture_05/lecture/","page":"Lecture","title":"Lecture","text":"function find_min!(f, x, p; verbose=true)\n\tfor i in 1:p[:iters]\n\t\tx̃ = x + p[:h]\n\t\tfx = f(x)\n\t\tx -= p[:stepsize] * (f(x̃) - fx)/p[:h]\n\t\tverbose && println(\"x = $x | f(x) = $fx\")\n\tend\n\tx\nend\n@btime find_min!($f, $x₀, $params_tuple; verbose=true)","category":"page"},{"location":"lecture_01/lab/#Lab-01:-Introduction-to-Julia","page":"Lab","title":"Lab 01: Introduction to Julia","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"This lab should get everyone up to speed in the basics of Julia's installation, syntax and basic coding. For more detailed introduction you can check out Lectures 1-3 of the bachelor course.","category":"page"},{"location":"lecture_01/lab/#Testing-Julia-installation-(custom-setup)","page":"Lab","title":"Testing Julia installation (custom setup)","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"In order to proceed further let's run a simple script to see, that the setup described in chapter Installation is working properly. After spawning a terminal/cmdline run this command:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"julia ./test_setup.jl","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The script does the following ","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"\"Tests\" if Julia is added to path and can be run with julia command from anywhere\nPrints Julia version info\nChecks Julia version.\nChecks git configuration (name + email)\nCreates an environment configuration files\nInstalls a basic pkg called BenchmarkTools, which we will use for benchmarking a simple function later in the labs.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"There are some quality of life improvements over long term support versions of Julia and thus throughout this course we will use the latest stable release of Julia 1.6.x.","category":"page"},{"location":"lecture_01/lab/#Polynomial-evaluation-example","page":"Lab","title":"Polynomial evaluation example","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Let's consider a common mathematical example for evaluation of nth-degree polynomial","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"f(x) = a_nx^n + a_n-1x^n-1 + dots + a_0x^0","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"where x in mathbbR and veca in mathbbR^n+1.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The simplest way of writing this in a generic fashion is realizing that essentially the function f is really implicitly containing argument veca, i.e. f equiv f(veca x), yielding the following Julia code","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"function polynomial(a, x)\n accumulator = 0\n for i in length(a):-1:1\n accumulator += x^(i-1) * a[i] # ! 1-based indexing for arrays\n end\n return accumulator\nend\nnothing #hide","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Evaluate the code of the function called polynomial in Julia REPL and evaluate the function itself with the following arguments.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"a = [-19, 7, -4, 6] # list coefficients a from a^0 to a^n\nx = 3 # point of evaluation\nnothing #hide","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The simplest way is to just copy&paste into an already running terminal manually. As opposed to the default Python REPL, Julia can deal with the blocks of code and different indentation much better without installation of an ipython-like REPL. There are ways to make this much easier in different text editors/IDEs:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"VSCode - when using Julia extension is installed and .jl file is opened, Ctrl/Cmd+Enter will spawn Julia REPL\nSublime Text - Ctrl/Cmd+Enter with Send Code pkg (works well with Linux terminal or tmux, support for Windows is poor)\nVim - there is a Julia language plugin, which can be combine with vimcmdline to gain similar functionality","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Either way, you should see the following:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"function polynomial(a, x)\n accumulator = 0\n for i in length(a):-1:1\n accumulator += x^(i-1) * a[i] # ! 1-based indexing for arrays\n end\n return accumulator\nend","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Similarly we enter the arguments of the function a and x:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"a = [-19, 7, -4, 6]\nx = 3","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Function call intuitively takes the name of the function with round brackets as arguments, i.e. works in the same way as majority of programming languages. The result is printed unless a ; is added at the end of the statement.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"polynomial(a, x) # function call","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Thanks to the high level nature of Julia language it is often the case that examples written in pseudocode are almost directly rewritable into the language itself without major changes and the code can be thus interpreted easily.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"(Image: polynomial_explained)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Due to the existence of the end keyword, indentation is not necessary as opposed to other languages such as Python, however it is strongly recommended to use it, see style guide.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Though there are libraries/IDEs that allow us to step through Julia code (Debugger.jl link and VSCode link), here we will explore the code interactively in REPL by evaluating pieces of code separately.","category":"page"},{"location":"lecture_01/lab/#Basic-types,-assignments-and-variables","page":"Lab","title":"Basic types, assignments and variables","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"When defining a variable through an assignment we get the representation of the right side, again this is different from the default behavior in Python, where the output of assignments a = [-19, 7, -4, 6] or x = 3, prints nothing. Internally Julia returns the result of the display function.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"a = [-19, 7, -4, 6]\ndisplay(a) # should return the same thing as the line above","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"As you can see, the string that is being displayed contains information about the contents of a variable along with it's type in this case this is a Vector/Array of Int types. If the output of display is insufficient the type of variable can be checked with the typeof function:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"typeof(a)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Additionally for collection/iterable types such as Vector there is also the eltype function, which returns the type of elements in the collection.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"eltype(a)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"In most cases variables store just a reference to a place in memory either stack/heap (exceptions are primitive types such as Int, Float) and therefore creating an array a, \"storing\" the reference in b with an assignment and changing elements of b, e.g. b[1] = 2, changes also the values in a.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Create variables x and accumulator, storing floating point 3.0 and integer value 0 respectively. Check the type of variables using typeof function.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"x = 3.0\naccumulator = 0\ntypeof(x), typeof(accumulator)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_01/lab/#For-cycles-and-ranges","page":"Lab","title":"For cycles and ranges","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Moving further into the polynomial function we encounter the definition of a for cycle, with the de facto standard syntax","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"for iteration_variable in iterator\n # do something\nend","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"As an example of iterator we have used an instance of a range type ","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"r = length(a):-1:1\ntypeof(r)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"As opposed to Python, ranges in Julia are inclusive, i.e. they contain number from start to end - in this case running from 4 to 1 with negative step -1, thus counting down. This can be checked with the collect and/or length functions.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"collect(r)\nlength(r)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Create variable c containing an array of even numbers from 2 to 42. Furthermore create variable d that is different from c only at the 7th position, which will contain 13.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"HINT: Use collect function for creation of c and copy for making a copy of c.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"c = collect(2:2:42)\nd = copy(c)\nd[7] = 13\nd","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_01/lab/#Functions-and-operators","page":"Lab","title":"Functions and operators","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Let us now move from the function body to the function definition itself. From the picture at the top of the page, we can infer the general syntax for function definition:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"function function_name(arguments)\n # do stuff with arguments and define output value `something`\n return something\nend","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The return keyword can be omitted, if the last line being evaluated contains the result.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"By creating the function polynomial we have defined a variable polynomial, that from now on always refers to a function and cannot be reassigned to a different type, like for example Int.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"polynomial = 42","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"This is caused by the fact that each function defines essentially a new type, the same like Int ~ Int64 or Vector{Int}.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"typeof(polynomial)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"You can check that it is a subtype of the Function abstract type, with the subtyping operator <:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"typeof(polynomial) <: Function","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"These concepts will be expanded further in the type system lecture, however for now note that this construction is quite useful for example if we wanted to create derivative rules for our function derivativeof(::typeof(polynomial), ...).","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Looking at mathematical operators +, *, we can see that in Julia they are also standalone functions. ","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"+\n*","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The main difference from our polynomial function is that there are multiple methods, for each of these functions. Each one of the methods coresponds to a specific combination of arguments, for which the function can be specialized to using multiple dispatch. You can see the list by calling a methods function:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"julia> methods(+)\n# 190 methods for generic function \"+\": \n[1] +(x::T, y::T) where T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8} in Base at\n int.jl:87 \n[2] +(c::Union{UInt16, UInt32, UInt64, UInt8}, x::BigInt) in Base.GMP at gmp.jl:528 \n[3] +(c::Union{Int16, Int32, Int64, Int8}, x::BigInt) in Base.GMP at gmp.jl:534\n...","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"One other notable difference is that these functions allow using both infix and postfix notation a + b and +(a,b), which is a specialty of elementary functions such as arithmetic operators or set operation such as ∩, ∪, ∈. ","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The functionality of methods is complemented with the reverse lookup methodswith, which for a given type returns a list of methods that can be called with it as an argument.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"julia> methodswith(Int)\n[1] +(x::T, y::T) where T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8} in Base at int.jl:87\n[2] +(c::Union{Int16, Int32, Int64, Int8}, x::BigInt) in Base.GMP at gmp.jl:534\n[3] +(c::Union{Int16, Int32, Int64, Int8}, x::BigFloat) in Base.MPFR at mpfr.jl:384\n[4] +(x::BigFloat, c::Union{Int16, Int32, Int64, Int8}) in Base.MPFR at mpfr.jl:379\n[5] +(x::BigInt, c::Union{Int16, Int32, Int64, Int8}) in Base.GMP at gmp.jl:533\n...","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Define function called addone with one argument, that adds 1 to the argument.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"function addone(x)\n x + 1\nend\naddone(1) == 2","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_01/lab/#Calling-for-help","page":"Lab","title":"Calling for help","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"In order to better understand some keywords we have encountered so far, we can ask for help in the Julia's REPL itself with the built-in help terminal. Accessing help terminal can be achieved by writing ? with a query keyword after. This searches documentation of all the available source code to find the corresponding keyword. The simplest way to create documentation, that can be accessed in this way, is using so called docstrings, which are multiline strings written above function or type definition. ","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"\"\"\"\n polynomial(a, x)\n\nReturns value of a polynomial with coefficients `a` at point `x`.\n\"\"\"\nfunction polynomial(a, x)\n # function body\nend","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"More on this in lecture 4 about pkg development.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Lookup docstring for the basic functions that we have introduced in the previous exercises: typeof, eltype, length, collect, copy, methods and methodswith. ","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"BONUS: Try it with others, for example with the subtyping operator <:.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Example docstring for typeof function.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":" typeof(x)\n\n Get the concrete type of x.\n\n Examples\n ≡≡≡≡≡≡≡≡≡≡\n\n julia> a = 1//2;\n \n julia> typeof(a)\n Rational{Int64}\n \n julia> M = [1 2; 3.5 4];\n \n julia> typeof(M)\n Matrix{Float64} (alias for Array{Float64, 2})","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_01/lab/#Testing-waters","page":"Lab","title":"Testing waters","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"As the arguments of the polynomial functions are untyped, i.e. they do not specify the allowed types like for example polynomial(a, x::Number) does, the following exercise explores which arguments the function accepts, while giving expected result.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Choose one of the variables af to ac representing polynomial coefficients and try to evaluate it with the polynomial function at point x=3 as before. Lookup the type of coefficient collection variable itself with typeof and the items in the collection with eltype. In this case we allow you to consult your solution with the expandable solution bellow to find out more information about a particular example.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"af = [-19.0, 7.0, -4.0, 6.0]\nat = (-19, 7, -4, 6)\nant = (a₀ = -19, a₁ = 7, a₂ = -4, a₃ = 6)\na2d = [-19 -4; 7 6]\nac = [2i^2 + 1 for i in -2:1]\nnothing #hide","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"typeof(af), eltype(af)\npolynomial(af, x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"As opposed to the basic definition of a type the array is filled with Float64 types and the resulting value gets promoted as well to the Float64.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"typeof(at), eltype(at)\npolynomial(at, x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"With round brackets over a fixed length vector we get the Tuple type, which is so called immutable \"array\" of a fixed size (its elements cannot be changed, unless initialized from scratch). Each element can be of a different type, but here we have only one and thus the Tuple is aliased into NTuple. There are some performance benefits for using immutable structure, which will be discussed later.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Defining key=value pairs inside round brackets creates a structure called NamedTuple, which has the same properties as Tuple and furthermore its elements can be conveniently accessed by dot syntax, e.g. ant.a₀.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"typeof(ant), eltype(ant)\npolynomial(ant, x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Defining a 2D array is a simple change of syntax, which initialized a matrix row by row separated by ; with spaces between individual elements. The function returns the same result because linear indexing works in 2d arrays in the column major order.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"typeof(a2d), eltype(a2d)\npolynomial(a2d, x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The last example shows so called array comprehension syntax, where we define and array of known length using and for loop iteration. Resulting array/vector has integer elements, however even mixed type is possible yielding Any, if there isn't any other common supertype to promote every entry into. (Use ? to look what promote and promote_type does.)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"typeof(ac), eltype(ac)\npolynomial(ac, x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"So far we have seen that polynomial function accepts a wide variety of arguments, however there are some understandable edge cases that it cannot handle.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Consider first the vector/array of characters ach","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"ach = ['1', '2', '3', '4']","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"which themselves have numeric values (you can check by converting them to Int Int('1') or convert(Int, '1')). In spite of that, our untyped function cannot process such input, as there isn't an operation/method that would allow multiplication of Char and Int type. Julia tries to promote the argument types to some common type, however checking the promote_type(Int, Char) returns Any (union of all types), which tells us that the conversion is not possible automatically.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"typeof(ach), eltype(ach)\npolynomial(ach, x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"In the stacktrace we can see the location of each function call. If we include the function polynomial from some file poly.jl using include(\"poly.jl\"), we will see that the location changes from REPL[X]:10 to the actual file name.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"By swapping square brackets for round in the array comprehension ac above, we have defined so called generator/iterator, which as opposed to original variable ac does not allocate an array, only the structure that produces it.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"ag = (2i^2 + 1 for i in -2:1)\ntypeof(ag), eltype(ag)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"You may notice that the element type in this case is Any, which means that a function using this generator as an argument cannot specialize based on the type and has to infer it every time an element is generated/returned. We will touch on how this affects performance in one of the later lectures.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"polynomial(ag, x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The problem that we face during evaluation is that generator type is missing the getindex operation, as they are made for situations where the size of the collection may be unknown and the only way of obtaining particular elements is through sequential iteration. Generators can be useful for example when creating batches of data for a machine learning training. We can \"fix\" the situation using collect function, mentioned earlier, however that again allocates an array.","category":"page"},{"location":"lecture_01/lab/#Extending/limiting-the-polynomial-example","page":"Lab","title":"Extending/limiting the polynomial example","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Following up on the polynomial example, let's us expand it a little further in order to facilitate the arguments, that have been throwing exceptions. The first direction, which we will move forward to, is providing the user with more detailed error message when an incorrect type of coefficients has been provided.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Design an if-else condition such that the array of Char example throws an error with custom string message, telling the user what went wrong and printing the incorrect input alongside it. Confirm that we have not broken the functionality of other examples from previous exercise.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Throw the ArgumentError(msg) with throw function and string message msg. More details in help mode ? or at the end of this document.\nStrings are defined like this s = \"Hello!\"\nUse string interpolation to create the error message. It allows injecting an expression into a string with the $ syntax b = 1; s = \"Hellow Number $(b)\"\nCompare eltype of the coefficients with Char type.\nThe syntax for if-else:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"if condition\n println(\"true\") # true branch code\nelse\n println(\"false\") # false branch code\nend","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Not equal condition can be written as a != b.\nThrowing an exception automatically returns from the function. Use return inside one of the branches to return the correct value.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The simplest way is to wrap the whole function inside an if-else condition and returning only when the input is \"correct\" (it will still fail in some cases).","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"function polynomial(a, x)\n if eltype(a) != Char\n accumulator = 0\n for i in length(a):-1:1\n accumulator += x^(i-1) * a[i] # ! 1-based indexing for arrays\n end\n return accumulator\n else\n throw(ArgumentError(\"Invalid coefficients $(a) of type Char!\"))\n end\nend\nnothing #hide","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Now this should show our predefined error message. ","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"polynomial(ach, x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Testing on other examples should pass without errors and give the same output as before.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"polynomial(a, x)\npolynomial(af, x)\npolynomial(at, x)\npolynomial(ant, x)\npolynomial(a2d, x)\npolynomial(ac, x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The second direction concerns the limitation to index-able structures, which the generator example is not. For this we will have to rewrite the whole loop in a more functional programming approach using map, anonymous function and other concepts.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Rewrite the following code inside our original polynomial function with map, enumerate and anonymous function.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"accumulator = 0\nfor i in length(a):-1:1\n accumulator += x^(i-1) * a[i] # ! 1-based indexing for arrays\nend","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"note: Anonymous functions reminder\nx -> x + 1 # unless the reference is stored it cannot be called\nplusone = x -> x + 1 # the reference can be stored inside a variable\nplusone(x) # calling with the same syntax","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Use enumerate to obtain iterator over a that returns a tuple of ia = (i, aᵢ). With Julia 1-based indexing i starts also from 1 and goes up to length(a).\nPass this into a map with either in-place or predefined anonymous function that does the operation of x^(i-1) * aᵢ.\nUse sum to collect the resulting array into accumulator variable or directly into the return command.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"BONUS: Can you figure out how to use the mapreduce function here? See entry in the help mode ?.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Ordered from the longest to the shortest, here are three examples with the same functionality (and there are definitely many more). Using the map(iterable) do itervar ... end syntax, that creates anonymous function from the block of code.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"function polynomial(a, x)\n powers = map(enumerate(a)) do (i, aᵢ)\n x^(i-1) * aᵢ\n end\n accumulator = sum(powers)\n return accumulator\nend\nnothing #hide","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Using the default syntax for map and storing the anonymous into a variable","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"function polynomial(a, x)\n polypow(i,aᵢ) = x^(i-1) * aᵢ\n powers = map(polypow, enumerate(a))\n return sum(powers)\nend\nnothing #hide","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"As the function polypow is used only once, there is no need to assign it to a local variable. Note the sightly awkward additional parenthesis in the argument of the lambda function.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"function polynomial(a, x)\n powers = map(((i,aᵢ),) -> x^(i-1) * aᵢ, enumerate(a))\n sum(powers)\nend\nnothing #hide","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Checking the behavior on all the inputs.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"polynomial(a, x)\npolynomial(af, x)\npolynomial(at, x)\npolynomial(ant, x)\npolynomial(a2d, x)\npolynomial(ach, x)\npolynomial(ac, x)\npolynomial(ag, x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"BONUS: You may have noticed that in the example above, the powers variable is allocating an additional, unnecessary vector. With the current, scalar x, this is not such a big deal. But in your homework you will generalize this function to matrix inputs of x, which means that powers becomes a vector of (potentially very large) matrices. This is a very natural use case for the mapreduce: function:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"polynomial(a, x) = mapreduce(+, enumerate(a), init=zero(x)) do (i, aᵢ)\n x^(i-1) * aᵢ\nend\n\npolynomial(a, x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Let's unpack what is happening here. If the function mapreduce(f, op, itr) is called with op=+ it returns the same result as sum(map(f, itr)). In contrast to sum(map(f, itr)) (which allocates a vector as a result of map and then sums) mapreduce applies f to an element in itr and immediately accumulates the result with the given op=+.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"polynomial(a, x) = sum(ia -> x^(ia[1]-1) * ia[2], enumerate(a))\nnothing #hide","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_01/lab/#How-to-use-code-from-other-people","page":"Lab","title":"How to use code from other people","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The script that we have run at the beginning of this lab has created two new files inside the current folder:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"./\n ├── Manifest.toml\n └── Project.toml","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Every folder with a toml file called Project.toml, can be used by Julia's pkg manager into setting so called environment, which contains a list of pkgs to be installed. Setting up or more often called activating an environment can be done either before starting Julia itself by running julia with the --project XXX flag or from within the Julia REPL, by switching to Pkg mode with ] key (similar to the help mode activated by pressing ?) and running command activate.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"So far we have used the general environment (depending on your setup), which by default does not come with any 3rd party packages and includes only the base and standard libraries - already quite powerful on its own. ","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"In order to find which environment is currently active, run the following:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"pkg> status","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The output of such command usually indicates the general environment located at .julia/ folder (${HOME}/.julia/ or ${APPDATA}/.julia/ in case of Unix/Windows based systems respectively)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"pkg> status\nStatus `~/.julia/environments/v1.6/Project.toml` (empty project)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Generally one should avoid working in the general environment, with the exception of some generic pkgs, such as PkgTemplates.jl, which is used for generating library templates/folder structure like the one above (link), more on this in the lecture on pkg development. ","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Activate the environment inside the current folder and check that the BenchmarkTools package has been installed. Use BenchmarkTools pkg's @btime to benchmark our polynomial function with the following arguments.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"aexp = ones(10) ./ factorial.(0:9)\nx = 1.1\nnothing #hide","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"HINTS:","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"In pkg mode use the command activate and status to check the presence. \nIn order to import the functionality from other package, lookup the keyword using in the repl help mode ?. \nThe functionality that we want to use is the @btime macro (it acts almost like a function but with a different syntax @macro arg1 arg2 arg3 ...). More on macros in lecture 7.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"BONUS: Compare the output of polynomial(aexp, x) with the value of exp(x), which it approximates.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"note: Broadcasting\nIn the assignment's code, we are using quite ubiquitous concept in Julia called broadcasting or simply the dot-syntax - represented here by ./, factorial.. This concept allows to map both simple arithmetic operations as well as custom functions across arrays, with the added benefit of increased performance, when the broadcasting system can merge operations into a more efficient code. More information can be found in the official documentation or section of our bachelor course.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"There are other options to import a function/macro from a different package, however for now let's keep it simple with the using Module syntax, that brings to the REPL, all the variables/function/macros exported by the BenchmarkTools pkg. If @btime is exported, which it is, it can be accessed without specification i.e. just by calling @btime without the need for BenchmarkTools.@btime. More on the architecture of pkg/module loading in the package developement lecture.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"julia> using BenchmarkTools\n\njulia> @btime polynomial(aexp, x)\n 97.119 ns (1 allocation: 16 bytes)\n3.004165230550543","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The output gives us the time of execution averaged over multiple runs (the number of samples is defined automatically based on run time) as well as the number of allocations and the output of the function, that is being benchmarked.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"BONUS: The difference between our approximation and the \"actual\" function value computed as a difference of the two. ","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"polynomial(aexp, x) - exp(x)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"The apostrophes in the previous sentence are on purpose, because implementation of exp also relies on a finite sum, though much more sophisticated than the basic Taylor expansion.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_01/lab/#Discussion-and-future-directions","page":"Lab","title":"Discussion & future directions","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Instead of if-else statements that would throw an error for different types, in Julia, we generally see the pattern of typing the function in a way, that for other than desirable types MethodError is emitted with the information about closest matching methods. This is part of the design process in Julia of a function and for the particular functionality of the polynomial example, we can look into the Julia itself, where it has been implemented in the evalpoly function","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"methods(evalpoly)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Another avenue, that we have only touched with the BenchmarkTools, is performance and will be further explored in the later lectures.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"With the next lecture focused on typing in Julia, it is worth noting that polynomials lend themselves quite nicely to a definition of a custom type, which can help both readability of the code as well further extensions.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"struct Polynom{C}\n coefficients::{C}\nend\n\nfunction (p:Polynom)(x)\n polynomial(p.coefficients, x)\nend","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"","category":"page"},{"location":"lecture_01/lab/#Useful-resources","page":"Lab","title":"Useful resources","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Getting Started tutorial from JuliaLang documentation - Docs\nConverting syntax between MATLAB ↔ Python ↔ Julia - Cheatsheet\nBachelor course for refreshing your knowledge - Course\nStylistic conventions - Style Guide\nReserved keywords - List\nOfficial cheatsheet with basic syntax - link","category":"page"},{"location":"lecture_01/lab/#lab_errors","page":"Lab","title":"Various errors and how to read them","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"This section summarizes most commonly encountered types of errors in Julia and how to resolve them or at least understand, what has gone wrong. It expands a little bit the official documentation, which contains the complete list with examples. Keep in mind again, that you can use help mode in the REPL to query error types as well.","category":"page"},{"location":"lecture_01/lab/#MethodError","page":"Lab","title":"MethodError","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"This type of error is most commonly thrown by Julia's multiple dispatch system with a message like no method matching X(args...), seen in two examples bellow.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"2 * 'a' # many candidates\ngetindex((i for i in 1:4), 3) # no candidates","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Both of these examples have a short stacktrace, showing that the execution failed on the top most level in REPL, however if this code is a part of some function in a separate file, the stacktrace will reflect it. What this error tells us is that the dispatch system could not find a method for a given function, that would be suitable for the type of arguments, that it has been given. In the first case Julia offers also a list of candidate methods, that match at least some of the arguments","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"When dealing with basic Julia functions and types, this behavior can be treated as something given and though one could locally add a method for example for multiplication of Char and Int, there is usually a good reason why Julia does not support such functionality by default. On the other hand when dealing with user defined code, this error may suggest the developer, that either the functions are too strictly typed or that another method definition is needed in order to satisfy the desired functionality.","category":"page"},{"location":"lecture_01/lab/#InexactError","page":"Lab","title":"InexactError","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"This type of error is most commonly thrown by the type conversion system (centered around convert function), informing the user that it cannot exactly convert a value of some type to match arguments of a function being called.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Int(1.2) # root cause\nappend!([1,2,3], 1.2) # same as above but shows the root cause deeper in the stack trace","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"In this case the function being Int and the value a floating point. The second example shows InexactError may be caused deeper inside an inconspicuous function call, where we want to extend an array by another value, which is unfortunately incompatible.","category":"page"},{"location":"lecture_01/lab/#ArgumentError","page":"Lab","title":"ArgumentError","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"As opposed to the previous two errors, ArgumentError can contain user specified error message and thus can serve multiple purposes. It is however recommended to throw this type of error, when the parameters to a function call do not match a valid signature, e.g. when factorial were given negative or non-integer argument (note that this is being handled in Julia by multiple dispatch and specific DomainError).","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"This example shows a concatenation of two 2d arrays of incompatible sizes 3x3 and 2x2.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"hcat(ones(3,3), zeros(2,2))","category":"page"},{"location":"lecture_01/lab/#KeyError","page":"Lab","title":"KeyError","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"This error is specific to hash table based objects such as the Dict type and tells the user that and indexing operation into such structure tried to access or delete a non-existent element.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"d = Dict(:a => [1,2,3], :b => [1,23])\nd[:c]","category":"page"},{"location":"lecture_01/lab/#TypeError","page":"Lab","title":"TypeError","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Type assertion failure, or calling an intrinsic function (inside LLVM, where code is strictly typed) with incorrect argument type. In practice this error comes up most often when comparing value of a type against the Bool type as seen in the example bellow.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"if 1 end # calls internally typeassert(1, Bool)\ntypeassert(1, Bool)","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"In order to compare inside conditional statements such as if-elseif-else or the ternary operator x ? a : b the condition has to be always of Bool type, thus the example above can be fixed by the comparison operator: if 1 == 1 end (in reality either the left or the right side of the expression contains an expression or a variable to compare against).","category":"page"},{"location":"lecture_01/lab/#UndefVarError","page":"Lab","title":"UndefVarError","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"While this error is quite self-explanatory, the exact causes are often quite puzzling for the user. The reason behind the confusion is to do with code scoping, which comes into play for example when trying to access a local variable from outside of a given function or just updating a global variable from within a simple loop. ","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"In the first example we show the former case, where variable is declared from within a function and accessed from outside afterwards.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"function plusone(x)\n uno = 1\n return x + uno\nend\nuno # defined only within plusone","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Unless there is variable I_am_not_defined in the global scope, the following should throw an error.","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"I_am_not_defined","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"Often these kind of errors arise as a result of bad code practices, such as long running sessions of Julia having long forgotten global variables, that do not exist upon new execution (this one in particular has been addressed by the authors of the reactive Julia notebooks Pluto.jl).","category":"page"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"For more details on code scoping we recommend particular places in the bachelor course lectures here and there.","category":"page"},{"location":"lecture_01/lab/#ErrorException-and-error-function","page":"Lab","title":"ErrorException & error function","text":"","category":"section"},{"location":"lecture_01/lab/","page":"Lab","title":"Lab","text":"ErrorException is the most generic error, which can be thrown/raised just by calling the error function with a chosen string message. As a result developers may be inclined to misuse this for any kind of unexpected behavior a user can run into, often providing out-of-context/uninformative messages.","category":"page"},{"location":"lecture_01/demo/#Extensibility-of-the-language","page":"Examples","title":"Extensibility of the language","text":"","category":"section"},{"location":"lecture_01/demo/#DifferentialEquations","page":"Examples","title":"DifferentialEquations","text":"","category":"section"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"A package for solving differential equations, similar to odesolve in Matlab.","category":"page"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"Example:","category":"page"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"using DifferentialEquations\nfunction lotka_volterra(du,u,p,t)\n x, y = u\n α, β, δ, γ = p\n du[1] = dx = α*x - β*x*y\n du[2] = dy = -δ*y + γ*x*y\nend\nu0 = [1.0,1.0]\ntspan = (0.0,10.0)\np = [1.5,1.0,3.0,1.0]\nprob = ODEProblem(lotka_volterra,u0,tspan,p)\n\nsol = solve(prob)\nusing Plots\nplot(sol)","category":"page"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"(Image: )","category":"page"},{"location":"lecture_01/demo/#Measurements","page":"Examples","title":"Measurements","text":"","category":"section"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"A package defining \"numbers with precision\" and complete algebra on these numbers:","category":"page"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"using Measurements\n\na = 4.5 ± 0.1\nb = 3.8 ± 0.4\n\n2a + b\nsin(a)/cos(a) - tan(a)","category":"page"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"It also defines recipes for Plots.jl how to plot such numbers.","category":"page"},{"location":"lecture_01/demo/#Starting-ODE-from-an-interval","page":"Examples","title":"Starting ODE from an interval","text":"","category":"section"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"using Measurements\nu0 = [1.0±0.1,1.0±0.01]\n\nprob = ODEProblem(lotka_volterra,u0,tspan,p)\nsol = solve(prob)\nplot(sol,denseplot=false)","category":"page"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"(Image: )","category":"page"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"all algebraic operations are defined, \npasses all grid refinement techniques\nplot uses the correct plotting for intervals","category":"page"},{"location":"lecture_01/demo/#Integration-with-other-toolkits","page":"Examples","title":"Integration with other toolkits","text":"","category":"section"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"Flux: toolkit for modelling Neural Networks. Neural network is a function.","category":"page"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"integration with Measurements,\nIntegration with ODE (think of NN as part of the ODE)","category":"page"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"Turing: Probabilistic modelling toolkit","category":"page"},{"location":"lecture_01/demo/","page":"Examples","title":"Examples","text":"integration with FLux (NN)\ninteration with ODE\nusing arbitrary bijective transformations, Bijectors.jl","category":"page"},{"location":"lecture_02/lab/#lab02","page":"Lab","title":"Lab 2: Predator-Prey Agents","text":"","category":"section"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"In the next labs you will implement your own predator-prey model. The model will contain wolves, sheep, and - to feed your sheep - some grass. The final simulation will be turn-based and the agents will be able to eat each other, reproduce, and die in every iteration. At every iteration of the simulation each agent will step forward in time via the agent_step! function. The steps for the agent_step! methods of animals and plants are written below in pseudocode.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"# for animals:\nagent_step!(animal, world)\n decrement energy by 1\n find & eat food (with probability pf)\n die if no more energy\n reproduce (with probability pr)\n\n# for plants:\nagent_step!(plant, world)\n grow if not at maximum size","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"The world in which the agents live will be the simplest possible world with zero dimensions (i.e. a Dict of ID=>Agent). Running and plotting your final result could look something like the plot below.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"(Image: img)","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"We will start implementing the basic functionality for each Agent like eat!ing, reproduce!ing, and a very simplistic World for your agents to live in. In the next lab you will refine both the type hierarchy of your Agents, as well as the design of the World in order to leverage the power of Julia's type system and compiler.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"We start with a very basic type hierarchy:","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"abstract type Agent end\nabstract type Animal <: Agent end\nabstract type Plant <: Agent end","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"We will implement the World for our Agents later, but it will essentially be implemented by a Dict which maps unique IDs to an Agent. Hence, every agent will need an ID.","category":"page"},{"location":"lecture_02/lab/#The-Grass-Agent","page":"Lab","title":"The Grass Agent","text":"","category":"section"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Let's start by implementing some Grass which will later be able to grow during each iteration of our simulation.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Define a mutable struct called Grass which is a subtype of Plant has the fields id (the unique identifier of this Agent - every agent needs one!), size (the current size of the Grass), and max_size. All fields should be integers.\nDefine a constructor for Grass which, given only an ID and a maximum size m, will create an instance of Grass that has a randomly initialized size in the range 1m. It should also be possible to create Grass, just with an ID and a default max_size of 10.\nImplement Base.show(io::IO, g::Grass) to get custom printing of your Grass such that the Grass is displayed with its size in percent of its max_size.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Hint: You can implement a custom show method for a new type MyType like this:","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"struct MyType\n x::Bool\nend\nBase.show(io::IO, a::MyType) = print(io, \"MyType $(a.x)\")","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Since Julia 1.8 we can also declare some fields of mutable structs as const, which can be used both to prevent us from mutating immutable fields (such as the ID) but can also be used by the compiler in certain cases.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"mutable struct Grass <: Plant\n const id::Int\n size::Int\n const max_size::Int\nend\n\nGrass(id,m=10) = Grass(id, rand(1:m), m)\n\nfunction Base.show(io::IO, g::Grass)\n x = g.size/g.max_size * 100\n # hint: to type the leaf in the julia REPL you can do:\n # \\:herb:\n print(io,\"🌿 #$(g.id) $(round(Int,x))% grown\")\nend","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Creating a few Grass agents can then look like this:","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Grass(1,5)\ng = Grass(2)\ng.id = 5","category":"page"},{"location":"lecture_02/lab/#Sheep-and-Wolf-Agents","page":"Lab","title":"Sheep and Wolf Agents","text":"","category":"section"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Animals are slightly different from plants. They will have an energy E, which will be increase (or decrease) if the agent eats (or reproduces) by a certain amount Delta E. Later we will also need a probability to find food p_f and a probability to reproduce p_r.c","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Define two mutable structs Sheep and Wolf that are subtypes of Animal and have the fields id, energy, Δenergy, reprprob, and foodprob.\nDefine constructors with the following default values:\nFor 🐑: E=4, Delta E=02, p_r=08, and p_f=06.\nFor 🐺: E=10, Delta E=8, p_r=01, and p_f=02.\nOverload Base.show to get pretty printing for your two new animals.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Solution for Sheep","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"mutable struct Sheep <: Animal\n const id::Int\n const energy::Float64\n Δenergy::Float64\n const reprprob::Float64\n const foodprob::Float64\nend\n\nSheep(id, e=4.0, Δe=0.2, pr=0.8, pf=0.6) = Sheep(id,e,Δe,pr,pf)\n\nfunction Base.show(io::IO, s::Sheep)\n e = s.energy\n d = s.Δenergy\n pr = s.reprprob\n pf = s.foodprob\n print(io,\"🐑 #$(s.id) E=$e ΔE=$d pr=$pr pf=$pf\")\nend","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Solution for Wolf:","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"mutable struct Wolf <: Animal\n const id::Int\n energy::Float64\n const Δenergy::Float64\n const reprprob::Float64\n const foodprob::Float64\nend\n\nWolf(id, e=10.0, Δe=8.0, pr=0.1, pf=0.2) = Wolf(id,e,Δe,pr,pf)\n\nfunction Base.show(io::IO, w::Wolf)\n e = w.energy\n d = w.Δenergy\n pr = w.reprprob\n pf = w.foodprob\n print(io,\"🐺 #$(w.id) E=$e ΔE=$d pr=$pr pf=$pf\")\nend","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Sheep(4)\nWolf(5)","category":"page"},{"location":"lecture_02/lab/#The-World","page":"Lab","title":"The World","text":"","category":"section"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Before our agents can eat or reproduce we need to build them a World. The simplest (and as you will later see, somewhat suboptimal) world is essentially a Dict from IDs to agents. Later we will also need the maximum ID, lets define a world with two fields:","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"mutable struct World{A<:Agent}\n agents::Dict{Int,A}\n max_id::Int\nend","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise:
\n
","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Implement a constructor for the World which accepts a vector of Agents.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"function World(agents::Vector{<:Agent})\n max_id = maximum(a.id for a in agents)\n World(Dict(a.id=>a for a in agents), max_id)\nend\n\n# optional: overload Base.show\nfunction Base.show(io::IO, w::World)\n println(io, typeof(w))\n for (_,a) in w.agents\n println(io,\" $a\")\n end\nend","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_02/lab/#Sheep-eats-Grass","page":"Lab","title":"Sheep eats Grass","text":"","category":"section"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"We can implement the behaviour of our various agents with respect to each other by leveraging Julia's multiple dispatch.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Implement a function eat!(::Sheep, ::Grass, ::World) which increases the sheep's energy by Delta E multiplied by the size of the grass.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"After the sheep's energy is updated the grass is eaten and its size counter has to be set to zero.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Note that you do not yet need the world in this function. It is needed later for the case of wolves eating sheep.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"function eat!(sheep::Sheep, grass::Grass, w::World)\n sheep.energy += grass.size * sheep.Δenergy\n grass.size = 0\nend\nnothing # hide","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Below you can see how a fully grown grass is eaten by a sheep. The sheep's energy changes size of the grass is set to zero.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"grass = Grass(1)\nsheep = Sheep(2)\nworld = World([grass, sheep])\neat!(sheep,grass,world);\nworld","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Note that the order of the arguments has a meaning here. Calling eat!(grass,sheep,world) results in a MethodError which is great, because Grass cannot eat Sheep.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"eat!(grass,sheep,world);","category":"page"},{"location":"lecture_02/lab/#Wolf-eats-Sheep","page":"Lab","title":"Wolf eats Sheep","text":"","category":"section"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"The eat! method for wolves increases the wolf's energy by sheep.energy * wolf.Δenergy and kills the sheep (i.e. removes the sheep from the world). There are other situationsin which agents die , so it makes sense to implement another function kill_agent!(::Animal,::World).","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Hint: You can use delete! to remove agents from the dictionary in your world.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"function eat!(wolf::Wolf, sheep::Sheep, w::World)\n wolf.energy += sheep.energy * wolf.Δenergy\n kill_agent!(sheep,w)\nend\n\nkill_agent!(a::Agent, w::World) = delete!(w.agents, a.id)\nnothing # hide","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"With a correct eat! method you should get results like this:","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"grass = Grass(1);\nsheep = Sheep(2);\nwolf = Wolf(3);\nworld = World([grass, sheep, wolf])\neat!(wolf,sheep,world);\nworld","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"The sheep is removed from the world and the wolf's energy increased by Delta E.","category":"page"},{"location":"lecture_02/lab/#Reproduction","page":"Lab","title":"Reproduction","text":"","category":"section"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Currently our animals can only eat. In our simulation we also want them to reproduce. We will do this by adding a reproduce! method to Animal.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
Exercise
\n
","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"Write a function reproduce! that takes an Animal and a World. Reproducing will cost an animal half of its energy and then add an almost identical copy of the given animal to the world. The only thing that is different from parent to child is the ID. You can simply increase the max_id of the world by one and use that as the new ID for the child.","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"
\n
\nSolution:

","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"function reproduce!(a::Animal, w::World)\n a.energy = a.energy/2\n new_id = w.max_id + 1\n â = deepcopy(a)\n â.id = new_id\n w.agents[â.id] = â\n w.max_id = new_id\nend","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"You can avoid mutating the id field (which could be considered bad practice) by reconstructing the child from scratch:","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"function reproduce!(a::A, w::World) where A<:Animal\n a.energy = a.energy/2\n a_vals = [getproperty(a,n) for n in fieldnames(A) if n!=:id]\n new_id = w.max_id + 1\n â = A(new_id, a_vals...)\n w.agents[â.id] = â\n w.max_id = new_id\nend\nnothing # hide","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"

","category":"page"},{"location":"lecture_02/lab/","page":"Lab","title":"Lab","text":"s1, s2 = Sheep(1), Sheep(2)\nw = World([s1, s2])\nreproduce!(s1, w);\nw","category":"page"},{"location":"lecture_07/lecture/#macro_lecture","page":"Lecture","title":"Macros","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"What is macro? In its essence, macro is a function, which ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"takes as an input an expression (parsed input)\nmodify the expressions in argument\ninsert the modified expression at the same place as the one that is parsed.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Macros are necessary because they execute after the code is parsed (2nd step in conversion of source code to binary as described in last lect, after Meta.parse) therefore, macros allow the programmer to generate and include fragments of customized code before the full program is compiled run. Since they are executed during parsing, they do not have access to the values of their arguments, but only to their syntax.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"To illustrate the difference, consider the following example:","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"A very convenient and highly recommended ways to write macros is to write functions modifying the Expression and then call that function in the macro. Let's demonstrate on an example, where every occurrence of sin is replaced by cos. We defined the function recursively traversing the AST and performing the substitution","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"replace_sin(x::Symbol) = x == :sin ? :cos : x\nreplace_sin(e::Expr) = Expr(e.head, map(replace_sin, e.args)...)\nreplace_sin(u) = u","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"and then we define the macro","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro replace_sin(ex)\n\treplace_sin(esc(ex))\nend\n\n@replace_sin(cosp1(x) = 1 + sin(x))\ncosp1(1) == 1 + cos(1)","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"notice the following","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"the definition of the macro is similar to the definition of the function with the exception that instead of the keyword function we use keyword macro\nwhen calling the macro, we signal to the compiler our intention by prepending the name of the macro with @. \nthe macro receives the expression(s) as the argument instead of the evaluated argument and also returns an expression that is placed on the position where the macro has been called\nwhen you are using macro, you should be as a user aware that the code you are entering can be arbitrarily modified and you can receive something completely different. This meanst that @ should also serve as a warning that you are leaving Julia's syntax. In practice, it make sense to make things akin to how they are done in Julia or to write Domain Specific Language with syntax familiar in that domain.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Inspecting the lowered code","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Meta.@lower @replace_sin( 1 + sin(x))","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"We observe that there is no trace of macro in lowered code (compare to Meta.@lower 1 + cos(x), which demonstrates that the macro has been expanded after the code has been parsed but before it has been lowered. In this sense macros are indispensible, as you cannot replace them simply by the combination of Meta.parse end eval. You might object that in the above example it is possible, which is true, but only because the effect of the macro is in the global scope.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"ex = Meta.parse(\"cosp1(x) = 1 + sin(x)\")\nex = replace_sin(ex)\neval(ex)","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"The following example cannot be achieved by the same trick, as the output of the macro modifies just the body of the function","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"function cosp2(x)\n\t@replace_sin 2 + sin(x)\nend\ncosp2(1) ≈ (2 + cos(1))","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"This is not possible","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"function parse_eval_cosp2(x)\n\tex = Meta.parse(\"2 + sin(x)\")\n\tex = replace_sin(ex)\n\teval(ex)\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"as can be seen from","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"julia> @code_lowered cosp2(1)\nCodeInfo(\n1 ─ %1 = Main.cos(x)\n│ %2 = 2 + %1\n└── return %2\n)\n\njulia> @code_lowered parse_eval_cosp2(1)\nCodeInfo(\n1 ─ %1 = Base.getproperty(Main.Meta, :parse)\n│ ex = (%1)(\"2 + sin(x)\")\n│ ex = Main.replace_sin(ex)\n│ %4 = Main.eval(ex)\n└── return %4\n)","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\nScope of evaleval function is always evaluated in the global scope of the Module in which the macro is called (note that there is that by default you operate in the Main module). Moreover, eval takes effect after the function has been has been executed. This can be demonstrated as add1(x) = x + 1\nfunction redefine_add(x)\n eval(:(add1(x) = x - 1))\n add1(x)\nend\njulia> redefine_add(1)\n2\n\njulia> redefine_add(1)\n0\n","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Macros are quite tricky to debug. Macro @macroexpand allows to observe the expansion of macros. Observe the effect as","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"@macroexpand @replace_sin(cosp1(x) = 1 + sin(x))","category":"page"},{"location":"lecture_07/lecture/#What-goes-under-the-hood-of-macro-expansion?","page":"Lecture","title":"What goes under the hood of macro expansion?","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Let's consider that the compiler is compiling","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"function cosp2(x)\n\t@replace_sin 2 + sin(x)\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"First, Julia parses the code into the AST as","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"ex = Meta.parse(\"\"\"\n function cosp2(x)\n\t @replace_sin 2 + sin(x)\nend\n\"\"\") |> Base.remove_linenums!\ndump(ex)","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"We observe that there is a macrocall in the AST, which means that Julia will expand the macro and put it in place","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"ex.args[2].args[1].head \t# the location of the macrocall\nex.args[2].args[1].args[1] # which macro to call\nex.args[2].args[1].args[2] # line number\nex.args[2].args[1].args[3]\t# on which expression","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"We can manullay run replace_sin and insert it back on the relevant sub-part of the sub-tree","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"ex.args[2].args[1] = replace_sin(ex.args[2].args[1].args[3])\nex |> dump","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"now, ex contains the expanded macro and we can see that it correctly defines the function","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"eval(ex)","category":"page"},{"location":"lecture_07/lecture/#Calling-macros","page":"Lecture","title":"Calling macros","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Macros can be called without parentheses","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro showarg(ex)\n\tprintln(\"single argument version\")\n\t@show ex\n\tex\nend\n@showarg(1 + 1)\n@showarg 1 + 1","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Macros use the very same multiple dispatch as functions, which allows to specialize macro calls","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro showarg(x1, x2::Symbol)\n\tprintln(\"two argument version, second is Symbol\")\n\t@show x1\n\t@show x2\n\tx1\nend\nmacro showarg(x1, x2::Expr)\n\tprintln(\"two argument version, second is Expr\")\n\t@show x1\n\t@show x2\n\tx1\nend\n@showarg(1 + 1, x)\n@showarg(1 + 1, 1 + 3)\n@showarg 1 + 1, 1 + 3\n@showarg 1 + 1 1 + 3","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"(the @showarg(1 + 1, :x) raises an error, since :(:x) is of Type QuoteNode). ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Observe that macro dispatch is based on the types of AST that are handed to the macro, not the types that the AST evaluates to at runtime.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"List of all defined versions of macro","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"methods(var\"@showarg\")","category":"page"},{"location":"lecture_07/lecture/#lec7_quotation","page":"Lecture","title":"Notes on quotation","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"In the previous lecture we have seen that we can quote a block of code, which tells the compiler to treat the input as a data and parse it. We have talked about three ways of quoting code.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":":(quoted code)\nMeta.parse(input_string)\nquote ... end","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"The truth is that Julia does not do full quotation, but a quasiquotation as it allows you to interpolate expressions inside the quoted code using $ symbol similar to the string. This is handy, as sometimes, when we want to insert into the quoted code an result of some computation / preprocessing. Observe the following difference in returned code","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"a = 5\n:(x = a)\n:(x = $(a))\nlet y = :x\n :(1 + y), :(1 + $y)\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"In contrast to the behavior of :() (or quote ... end, true quotation would not perform interpolation where unary $ occurs. Instead, we would capture the syntax that describes interpolation and produce something like the following:","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"(\n :(1 + x), # Quasiquotation\n Expr(:call, :+, 1, Expr(:$, :x)), # True quotation\n)","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"for (v, f) in [(:sin, :foo_sin)]\n\tquote\n\t\t$(f)(x) = $(v)(x)\n\tend |> Base.remove_linenums! |> dump\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"When we need true quoting, i.e. we need something to stay quoted, we can use QuoteNode as","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro true_quote(e)\n QuoteNode(e)\nend\n\nlet y = :x\n (\n @true_quote(1 + $y),\n :(1 + $y),\n )\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"At first glance, QuoteNode wrapper seems to be useless. But QuoteNode has clear value when it's used inside a macro to indicate that something should stay quoted even after the macro finishes its work. Also notice that the expression received by macro are quoted, not quasiquoted, since in the latter case $y would be replaced. ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"We can demonstrate it by defining a new macro no_quote which will just return the expression as is ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro no_quote(ex)\n ex\nend\n\nlet y = :x\n @no_quote(1 + $y)\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"The error code snippet errors telling us that the expression \"$\" is outside of a quote block. This is because the macro @no_quote has returned a block with $ occuring outside of quote or string definition.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\nSome macros like @eval (recall last example)for f in [:setindex!, :getindex, :size, :length]\n @eval $(f)(A::MyMatrix, args...) = $(f)(A.x, args...)\nendor @benchmark support interpolation of values. This interpolation needs to be handled by the logic of the macro and is not automatically handled by Julia language.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Macros do not know about runtime values, they only know about syntax trees. When a macro receives an expression with a x in it, it can't interpolate the value of x into the syntax tree because it reads the syntax tree before x ever has a value! ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Instead, when a macro is given an expression with $ in it, it assumes you're going to give your own meaning to x. In the case of BenchmarkTools.jl they return code that has to wait until runtime to receive the value of x and then splice that value into an expression which is evaluated and benchmarked. Nowhere in the actual body of the macro do they have access to the value of x though.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\nWhy $ for interpolation?The $ string for interpolation was used as it identifies the interpolation inside the string and inside the command. For examplea = 5\ns = \"a = $(a)\"\ntypoef(s)\nprintln(s)\nfilename = \"/tmp/test_of_interpolation\"\nrun(`touch $(filename)`)","category":"page"},{"location":"lecture_07/lecture/#lec7_hygiene","page":"Lecture","title":"Macro hygiene","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Macro hygiene is a term coined in 1986 addressing the following problem: if you're automatically generating code, it's possible that you will introduce variable names in your generated code that will clash with existing variable names in the scope in which a macro is called. These clashes might cause your generated code to read from or write to variables that you should not be interacting with. A macro is hygienic when it does not interact with existing variables, which means that when macro is evaluated, it should not have any effect on the surrounding code. ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"By default, all macros in Julia are hygienic which means that variables introduced in the macro have automatically generated names, where Julia ensures they will not collide with user's variable. These variables are created by gensym function / macro. ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"info: Info\ngensymgensym([tag]) Generates a symbol which will not conflict with other variable names.julia> gensym(\"hello\")\nSymbol(\"##hello#257\")","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Let's demonstrate it on our own version of an macro @elapsed which will return the time that was needed to evaluate the block of code.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro tooclean_elapsed(ex)\n\tquote\n\t\ttstart = time()\n\t\t$(ex)\n\t\ttime() - tstart\n\tend\nend\n\nfib(n) = n <= 1 ? n : fib(n-1) + fib(n - 2)\nlet \n\ttstart = \"should not change the value and type\"\n\tt = @tooclean_elapsed r = fib(10)\n\tprintln(\"the evaluation of fib took \", t, \"s and result is \", r)\n\t@show tstart\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"We see that variable r has not been assigned during the evaluation of macro. We have also used let block in orders not to define any variables in the global scope. The problem with the above is that it cannot be nested. Why is that? Let's observe how the macro was expanded","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"julia> Base.remove_linenums!(@macroexpand @tooclean_elapsed r = fib(10))\nquote\n var\"#12#tstart\" = Main.time()\n var\"#13#r\" = Main.fib(10)\n Main.time() - var\"#12#tstart\"\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"We see that tstart in the macro definition was replaced by var\"#12#tstart\", which is a name generated by Julia's gensym to prevent conflict. The same happens to r, which was replaced by var\"#13#r\". This names are the result of Julia's hygiene-enforcing pass, which is intended to prevent us from overwriting existing variables during macro expansion. This pass usually makes our macros safer, but it is also a source of confusion because it introduces a gap between the expressions we generate and the expressions that end up in the resulting source code. Notice that in the case of tstart, we actually wanted to replace tstart with a unique name, such that if we by a bad luck define tstart in our code, it would not be affected, as we can see in this example.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"let \n\ttstart = \"should not change the value and type \"\n\tt = @tooclean_elapsed r = fib(10)\n\tprintln(tstart, \" \", typeof(tstart))\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"But in the second case, we would actually very much like the variable r to retain its name, such that we can accesss the results (and also, ex can access and change other local variables). Julia offer a way to escape from the hygienic mode, which means that the variables will be used and passed as-is. Notice the effect if we escape just the expression ex ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro justright_elapsed(ex)\n\tquote\n\t\ttstart = time()\n\t\t$(esc(ex))\n\t\ttime() - tstart\n\tend\nend\n\nlet \n\ttstart = \"should not change the value and type \"\n\tt = @justright_elapsed r = fib(10)\n\tprintln(\"the evaluation of fib took \", t, \"s and result is \", r)\n\tprintln(tstart, \" \", typeof(tstart))\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"which now works as intended. We can inspect the output again using @macroexpand","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"julia> Base.remove_linenums!(@macroexpand @justright_elapsed r = fib(10))\nquote\n var\"#19#tstart\" = Main.time()\n r = fib(10)\n Main.time() - var\"#19#tstart\"\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"and compare it to Base.remove_linenums!(@macroexpand @justright_elapsed r = fib(10)). We see that the expression ex has its symbols intact. To use the escaping / hygience correctly, you need to have a good understanding how the macro evaluation works and what is needed. Let's now try the third version of the macro, where we escape everything as","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro toodirty_elapsed(ex)\n\tex = quote\n\t\ttstart = time()\n\t\t$(ex)\n\t\ttime() - tstart\n\tend\n\tesc(ex)\nend\n\nlet \n\ttstart = \"should not change the value and type \"\n\tt = @toodirty_elapsed r = fib(10)\n\tprintln(\"the evaluation of fib took \", t, \"s and result is \", r)\n\tprintln(tstart, \" \", typeof(tstart))\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Using @macroexpand we observe that @toodirty_elapsed does not have any trace of hygiene.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"julia> Base.remove_linenums!(@macroexpand @toodirty_elapsed r = fib(10))\nquote\n tstart = time()\n r = fib(10)\n time() - tstart\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"From the above we can also see that hygiene-pass occurs after the macro has been applied but before the code is lowered. esc is inserted to AST as a special node Expr(:escape,...), which can be seen from the follows.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"julia> esc(:x)\n:($(Expr(:escape, :x)))","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"The definition in essentials.jl:480 is pretty simple as esc(@nospecialize(e)) = Expr(:escape, e), but it does not tell anything about the actual implementation, which is hidden probably in the macro-expanding logic.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"With that in mind, we can now understand our original example with @replace_sin. Recall that we have defined it as ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro replace_sin(ex)\n\treplace_sin(esc(ex))\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"where the escaping replace_sin(esc(ex)) in communicates to compiler that ex should be used as without hygienating the ex. Indeed, if we lower it","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"function cosp2(x)\n\t@replace_sin 2 + sin(x)\nend\n\njulia> @code_lowered(cosp2(1.0))\nCodeInfo(\n1 ─ %1 = Main.cos(x)\n│ %2 = 2 + %1\n└── return %2\n)","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"we see it works as intended. Whereas if we use hygienic version","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro hygienic_replace_sin(ex)\n\treplace_sin(ex)\nend\n\nfunction hcosp2(x)\n\t@hygienic_replace_sin 2 + sin(x)\nend\n\njulia> @code_lowered(hcosp2(1.0))\nCodeInfo(\n1 ─ %1 = Main.cos(Main.x)\n│ %2 = 2 + %1\n└── return %2\n)","category":"page"},{"location":"lecture_07/lecture/#Why-hygienating-the-function-calls?","page":"Lecture","title":"Why hygienating the function calls?","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"function foo(x)\n\tcos(x) = exp(x)\n\t@replace_sin 1 + sin(x)\nend\n\nfoo(1.0) ≈ 1 + exp(1.0)\n\nfunction foo2(x)\n\tcos(x) = exp(x)\n\t@hygienic_replace_sin 1 + sin(x)\nend\n\nx = 1.0\nfoo2(1.0) ≈ 1 + cos(1.0)","category":"page"},{"location":"lecture_07/lecture/#Can-I-do-the-hygiene-by-myself?","page":"Lecture","title":"Can I do the hygiene by myself?","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Yes, it is by some considered to be much simpler (and safer) then to understand, how macro hygiene works.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro manual_elapsed(ex)\n x = gensym()\n esc(quote\n \t\t$(x) = time()\n \t$(ex)\n \ttime() - $(x)\n end\n )\nend\n\nlet \n\tt = @manual_elapsed r = fib(10)\n\tprintln(\"the evaluation of fib took \", t, \"s and result is \", r)\nend\n","category":"page"},{"location":"lecture_07/lecture/#How-macros-compose?","page":"Lecture","title":"How macros compose?","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro m1(ex)\n\tprintln(\"m1: \")\n\tdump(ex)\n\tex\nend\n\nmacro m2(ex)\n\tprintln(\"m2: \")\n\tdump(ex)\n\tesc(ex)\nend\n\n@m1 @m2 1 + sin(1)","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"which means that macros are expanded in the order from the outer most to inner most, which is exactly the other way around than functions.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"@macroexpand @m1 @m2 1 + sin(1)","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"also notice that the escaping is only partial (running @macroexpand @m2 @m1 1 + sin(1) would not change the results).","category":"page"},{"location":"lecture_07/lecture/#Write-@exfiltrate-macro","page":"Lecture","title":"Write @exfiltrate macro","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Since Julia's debugger is a complicated story, people have been looking for tools, which would simplify the debugging. One of them is a macro @exfiltrate, which copies all variables in a given scope to a safe place, from where they can be collected later on. This helps you in evaluating the function. F","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Whyle a full implementation is provided in package Infiltrator.jl, we can implement such functionality by outselves.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"We collect names and values of variables in a given scope using the macro Base.@locals\nWe store variables in some global variable in some module, such that we have one place from which we can retrieve them and we are certain that this storage would not interact with any existing code.\nIf the @exfiltrate should be easy, ideally called without parameters, it has to be implemented as a macro to supply the relevant variables to be stored.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"module Exfiltrator\n\nconst environment = Dict{Symbol, Any}()\n\nfunction copy_variables!(d::Dict)\n\tforeach(k -> delete!(environment, k), keys(environment))\n\tfor (k, v) in d\n\t\tenvironment[k] = v\n\tend\nend\n\nmacro exfiltrate()\n\tv = gensym(:vars)\n\tquote\n\t\t$(v) = $(esc((Expr(:locals))))\n\t\tcopy_variables!($(v))\n\tend\nend\n\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Test it to ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"using Main.Exfiltrator: @exfiltrate\nlet \n\tx,y,z = 1,\"hello\", (a = \"1\", b = \"b\")\n\t@exfiltrate\nend\n\nExfiltrator.environment\n\nfunction inside_function()\n\ta,b,c = 1,2,3\n\t@exfiltrate\nend\n\ninside_function()\n\nExfiltrator.environment\n\nfunction a()\n\ta = 1\n\t@exfiltrate\nend\n\nfunction b()\n\tb = 1\n\ta()\nend\nfunction c()\n\tc = 1\n\tb()\nend\n\nc()\nExfiltrator.environment","category":"page"},{"location":"lecture_07/lecture/#Domain-Specific-Languages-(DSL)","page":"Lecture","title":"Domain Specific Languages (DSL)","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Macros are convenient for writing domain specific languages, which are languages designed for specific domain. This allows them to simplify notation and / or make the notation familiar for people working in the field. For example in Turing.jl, the model of coinflips can be specified as ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"@model function coinflip(y)\n\n # Our prior belief about the probability of heads in a coin.\n p ~ Beta(1, 1)\n\n # The number of observations.\n N = length(y)\n for n in 1:N\n # Heads or tails of a coin are drawn from a Bernoulli distribution.\n y[n] ~ Bernoulli(p)\n end\nend;","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"which resembles, but not copy Julia's syntax due to the use of ~. A similar DSLs can be seen in ModelingToolkit.jl for differential equations, in Soss.jl again for expressing probability problems, in Metatheory.jl / SymbolicUtils.jl for defining rules on elements of algebras, or JuMP.jl for specific mathematical programs.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"One of the reasons for popularity of DSLs is that macro system is very helpful in their implementation, but it also contraints the DSL, as it has to be parseable by Julia's parser. This is a tremendous helps, because one does not have to care about how to parse numbers, strings, parenthesess, functions, etc. (recall the last lecture about replacing occurences of i variable).","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Let's jump into the first example adapted from John Myles White's howto. We would like to write a macro, which allows us to define graph in Graphs.jl just by defining edges.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"@graph begin \n\t1 -> 2\n\t2 -> 3\n\t3 -> 1\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"The above should expand to","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"using Graphs\ng = DiGraph(3)\nadd_edge!(g, 1,2)\nadd_edge!(g, 2,3)\nadd_edge!(g, 3,1)\ng","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Let's start with easy and observe, how ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"ex = Meta.parse(\"\"\"\nbegin \n\t1 -> 2\n\t2 -> 3\n\t3 -> 1\nend\n\"\"\")\nex = Base.remove_linenums!(ex)","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"is parsed to ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"quote\n 1->begin\n 2\n end\n 2->begin\n 3\n end\n 3->begin\n 1\n end\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"We see that ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"the sequence of statements is parsed to block (we know that from last lecture).\n-> is parsed to ->, i.e. ex.args[1].head == :-> with parameters being the first vertex ex.args[1].args[1] == 1 and the second vertex is quoted to ex.args[1].args[2].head == :block. ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"The main job will be done in the function parse_edge, which will parse one edge. It will check that the node defines edge (otherwise, it will return nothing, which will be filtered out)","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"function parse_edge(ex)\n\t#checking the syntax\n\t!hasproperty(ex, :head) && return(nothing)\n\t!hasproperty(ex, :args) && return(nothing)\n\tex.head != :-> && return(nothing)\n\tlength(ex.args) != 2 && return(nothing)\n\t!hasproperty(ex.args[2], :head) && return(nothing)\n\tex.args[2].head != :block && length(ex.args[2].args) == 1 && return(nothing)\n\n\t#ready to go\n\tsrc = ex.args[1]\n\t@assert src isa Integer\n\tdst = ex.args[2].args[1]\n\t@assert dst isa Integer\n\t:(add_edge!(g, $(src), $(dst)))\nend\n\nfunction parse_graph(ex)\n\t@assert ex.head == :block\n\tex = Base.remove_linenums!(ex)\n\tedges = filter(!isnothing, parse_edge.(ex.args))\n\tn = maximum(e -> maximum(e.args[3:4]), edges)\n\tquote\n g = Graphs.DiGraph($(n))\n $(edges...)\n g\n end\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Once we have the first version, let's make everything hygienic","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"function parse_edge(g, ex::Expr)\n\t#checking the syntax\n\tex.head != :-> && return(nothing)\n\tlength(ex.args) != 2 && return(nothing)\n\t!hasproperty(ex.args[2], :head) && return(nothing)\n\tex.args[2].head != :block && length(ex.args[2].args) == 1 && return(nothing)\n\n\t#ready to go\n\tsrc = ex.args[1]\n\t@assert src isa Integer\n\tdst = ex.args[2].args[1]\n\t@assert dst isa Integer\n\t:(add_edge!($(g), $(src), $(dst)))\nend\nparse_edge(g, ex) = nothing\n\nfunction parse_graph(ex)\n\t@assert ex.head == :block\n\tg = gensym(:graph)\n\tex = Base.remove_linenums!(ex)\n\tedges = filter(!isnothing, parse_edge.(g, ex.args))\n\tn = maximum(e -> maximum(e.args[3:4]), edges)\n\tquote\n $(g) = Graphs.DiGraph($(n))\n $(edges...)\n $(g)\n end\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"and we are ready to go","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro graph(ex)\n\tparse_graph(ex)\nend\n\n@graph begin\n\t1 -> 2\n\t2 -> 3\n\t3 -> 1\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"and we can check the output with @macroexpand.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"julia> @macroexpand @graph begin\n 1 -> 2\n 2 -> 3\n 3 -> 1\n end\nquote\n #= REPL[173]:8 =#\n var\"#27###graph#273\" = (Main.Graphs).DiGraph(3)\n #= REPL[173]:9 =#\n Main.add_edge!(var\"#27###graph#273\", 1, 2)\n Main.add_edge!(var\"#27###graph#273\", 2, 3)\n Main.add_edge!(var\"#27###graph#273\", 3, 1)\n #= REPL[173]:10 =#\n var\"#27###graph#273\"\nend","category":"page"},{"location":"lecture_07/lecture/#non-standard-string-literals","page":"Lecture","title":"non-standard string literals","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Julia allows to customize parsing of strings. For example we can define regexp matcher as r\"^\\s*(?:#|$)\", i.e. using the usual string notation prepended by the string r.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"You can define these \"parsers\" by yourself using the macro definition with suffix _str","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro debug_str(p)\n\t@show p\n p\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"by invoking it","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"debug\"hello\"","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"we see that the string macro receives string as an argument. ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Why are they useful? Sometimes, we want to use syntax which is not compatible with Julia's parser. For example IntervalArithmetics.jl allows to define an interval open only from one side, for example [a, b), which is something that Julia's parser would not like much. String macro solves this problem by letting you to write the parser by your own.","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"struct Interval{T}\n\tleft::T\n\tright::T\n\tleft_open::Bool\n\tright_open::Bool\nend\n\nfunction Interval(s::String)\n\ts[1] == '(' || s[1] == '[' || error(\"left nterval can be only [,(\")\n\ts[end] == ')' || s[end] == ']' || error(\"left nterval can be only ],)\")\n \tleft_open = s[1] == '(' ? true : false\n \tright_open = s[end] == ')' ? true : false\n \tss = parse.(Float64, split(s[2:end-1],\",\"))\n \tlength(ss) != 2 && error(\"interval should have two numbers separated by ','\")\n \tInterval(ss..., left_open, right_open)\nend\n\nfunction Base.show(io::IO, r::Interval)\n\tlb = r.left_open ? \"(\" : \"[\"\n\trb = r.right_open ? \")\" : \"]\"\n\tprint(io, lb,r.left,\",\",r.right,rb)\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"We can check it does the job by trying Interval(\"[1,2)\"). Finally, we define a string macro as ","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"macro int_str(s)\n\tInterval(s)\nend","category":"page"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"which allows us to define interval as int\"[1,2)\".","category":"page"},{"location":"lecture_07/lecture/#Sources","page":"Lecture","title":"Sources","text":"","category":"section"},{"location":"lecture_07/lecture/","page":"Lecture","title":"Lecture","text":"Great discussion on evaluation of macros.","category":"page"},{"location":"","page":"Home","title":"Home","text":"\"Scientific\n\"Scientific","category":"page"},{"location":"","page":"Home","title":"Home","text":"","category":"page"},{"location":"","page":"Home","title":"Home","text":"using Plots\nENV[\"GKSwstype\"] = \"100\"\ngr()","category":"page"},{"location":"","page":"Home","title":"Home","text":"Scientific Programming requires the highest performance but we also want to write very high level code to enable rapid prototyping and avoid error prone, low level implementations.","category":"page"},{"location":"","page":"Home","title":"Home","text":"The Julia programming language is designed with exactly those requirements of scientific computing in mind. In this course we will show you how to make use of the tools and advantages that jit-compiled Julia provides over dynamic, high-level languages like Python or lower level languages like C++.","category":"page"},{"location":"","page":"Home","title":"Home","text":"
\n \n
\n Learn the power of abstraction.\n Example: The essence of forward mode automatic differentiation.\n
\n
","category":"page"},{"location":"","page":"Home","title":"Home","text":"Before joining the course, consider reading the following two blog posts to figure out if Julia is a language in which you want to invest your time.","category":"page"},{"location":"","page":"Home","title":"Home","text":"What is great about Julia.\nWhat is bad about Julia.","category":"page"},{"location":"#What-will-you-learn?","page":"Home","title":"What will you learn?","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"First and foremost you will learn how to think julia - meaning how write fast, extensible, reusable, and easy-to-read code using things like optional typing, multiple dispatch, and functional programming concepts. The later part of the course will teach you how to use more advanced concepts like language introspection, metaprogramming, and symbolic computing. Amonst others you will implement your own automatic differetiation (the backbone of modern machine learning) package based on these advanced techniques that can transform intermediate representations of Julia code.","category":"page"},{"location":"#Organization","page":"Home","title":"Organization","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"This course webpage contains all information about the course that you need, including lecture notes, lab instructions, and homeworks. The official format of the course is 2+2 (2h lectures/2h labs per week) for 4 credits.","category":"page"},{"location":"","page":"Home","title":"Home","text":"The official course code is: B0M36SPJ and the timetable for the winter semester 2022 can be found here.","category":"page"},{"location":"","page":"Home","title":"Home","text":"The course will be graded based on points from your homework (max. 20 points) and points from a final project (max. 30 points).","category":"page"},{"location":"","page":"Home","title":"Home","text":"Below is a table that shows which lectures have homeworks (and their points).","category":"page"},{"location":"","page":"Home","title":"Home","text":"Homework 1 2 3 4 5 6 7 8 9 10 11 12 13\nPoints 2 2 2 2 2 2 2 2 - 2 - 2 -","category":"page"},{"location":"","page":"Home","title":"Home","text":"Hint: The first few homeworks are easier. Use them to fill up your points.","category":"page"},{"location":"#final_project","page":"Home","title":"Final project","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"The final project will be individually agreed on for each student. Ideally you can use this project to solve a problem you have e.g. in your thesis, but don't worry - if you cannot come up with an own project idea, we will suggest one to you. More info and project suggestion can be found here.","category":"page"},{"location":"#Grading","page":"Home","title":"Grading","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"Your points from the homeworks and the final project are summed and graded by the standard grading scale below.","category":"page"},{"location":"","page":"Home","title":"Home","text":"Grade A B C D E F\nPoints 45-50 40-44 35-39 30-34 25-29 0-25","category":"page"},{"location":"#emails","page":"Home","title":"Teachers","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"– E-mail Room Role\nTomáš Pevný pevnak@protonmail.ch KN:E-406 Lecturer\nVašek Šmídl smidlva1@fjfi.cvut.cz KN:E-333 Lecturer\nMatěj Zorek zorekmat@fel.cvut.cz KN:E-333 Lab Instructor\nNiklas Heim heimnikl@fel.cvut.cz KN:E-333 Lab Instructor","category":"page"},{"location":"#Prerequisites","page":"Home","title":"Prerequisites","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"There are no hard requirements to take the course, but if you are not at all familiar with Julia we recommend you to take Julia for Optimization and Learning before enrolling in this course. The Functional Programming course also contains some helpful concepts for this course. And knowledge about computer hardware, namely basics of how CPU works, how it interacts with memory through caches, and basics of multi-threadding certainly helps.","category":"page"},{"location":"#References","page":"Home","title":"References","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"Official documentation\nModern Julia Workflows\nWorkflow tips, and what is new in v1.9\nThink Julia: How to Think Like a Computer Scientist\nFrom Zero to Julia!\nWikiBooks\nJustin Krumbiel's excellent introduction to the package manager.\njuliadatascience.io contains an excellent introduction to plotting with Makie.\nThe art of multiple dispatch\nMIT Course: Julia Computation\nTim Holy's Advanced Scientific Computing","category":"page"},{"location":"lecture_06/hw/#Homework-6:-Find-variables","page":"Homework","title":"Homework 6: Find variables","text":"","category":"section"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"Following the lab exercises, you may think that metaprogramming is a fun little exercise. Let's challenge this notion in this homework, where YOU are being trusted with catching all the edge cases in an AST.","category":"page"},{"location":"lecture_06/hw/#How-to-submit?","page":"Homework","title":"How to submit?","text":"","category":"section"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"Put the code of the compulsory task inside hw.jl. Zip only this file (not its parent folder) and upload it to BRUTE. Your file should not use any 3rd party dependency.","category":"page"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"
\n
Homework (2 points)
\n
","category":"page"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"Your task is to find all single letter variables in an expression, i.e. for example when given expression","category":"page"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"x + 2*y*z - c*x","category":"page"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"return an array of unique alphabetically sorted symbols representing variables in an expression.","category":"page"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"[:c, :x, :y, :z]","category":"page"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"Implement this in a function called find_variables. Note that there may be some edge cases that you may have to handle in a special way, such as ","category":"page"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"variable assignments r = x*x should return the variable on the left as well (r in this case)\nignoring symbols representing single letter function calls such as f(x)","category":"page"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"
\n","category":"page"},{"location":"lecture_06/hw/#Voluntary-exercise","page":"Homework","title":"Voluntary exercise","text":"","category":"section"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"
\n
Voluntary exercise
\n
","category":"page"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"Create a function that replaces each of +, -, * and / with the respective checked operation, which checks for overflow. E.g. + should be replaced by Base.checked_add.","category":"page"},{"location":"lecture_06/hw/","page":"Homework","title":"Homework","text":"
\n","category":"page"},{"location":"lecture_05/hw/#Homework-5:-Root-finding-of-polynomials","page":"Homework","title":"Homework 5: Root finding of polynomials","text":"","category":"section"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"This homework should test your ability to use the knowledge of benchmarking, profiling and others to improve an existing implementation of root finding methods for polynomials. The provided code is of questionable quality. In spite of the artificial nature, it should simulate a situation in which you may find yourself quite often, as it represents some intermediate step of going from a simple script to something, that starts to resemble a package.","category":"page"},{"location":"lecture_05/hw/#How-to-submit?","page":"Homework","title":"How to submit?","text":"","category":"section"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"Put the modified root_finding.jl code inside hw.jl. Zip only this file (not its parent folder) and upload it to BRUTE. Your file should not use any dependency other than those already present in the root_finding.jl.","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"
\n
Homework (2 points)
\n
","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"Use profiler on the find_root function to find a piece of unnecessary code, that takes more time than the computation itself. The finding of roots with the polynomial ","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"p(x) = (x - 3)(x - 2)(x - 1)x(x + 1)(x + 2)(x + 3) = x^7 - 14x^5 + 49x^3 - 36x","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"should not take more than 50μs when running with the following parameters","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"atol = 1e-12\nmaxiter = 100\nstepsize = 0.95\n\nx₀ = find_root(p, Bisection(), -5.0, 5.0, maxiter, stepsize, atol)\nx₀ = find_root(p, Newton(), -5.0, 5.0, maxiter, stepsize, atol)\nx₀ = find_root(p, Secant(), -5.0, 5.0, maxiter, stepsize, atol)","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"Remove obvious type instabilities in both find_root and step! functions. Each variable with \"inferred\" type ::Any in @code_warntype will be penalized.","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"HINTS:","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"running the function repeatedly 1000x helps in the profiler sampling\nfocus on parts of the code that may have been used just for debugging purposes","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"
\n","category":"page"},{"location":"lecture_05/hw/#Voluntary-exercise","page":"Homework","title":"Voluntary exercise","text":"","category":"section"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"
\n
Voluntary exercise
\n
","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"Use Plots.jl to plot the polynomial p on the interval -5 5 and visualize the progress/convergence of each method, with a dotted vertical line and a dot on the x-axis for each subsequent root approximation x̃.","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"HINTS:","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"plotting scalar function f - plot(r, f), where r is a range of x values at which we evaluate f\nupdating an existing plot - either plot!(plt, ...) or plot!(...), in the former case the plot lives in variable plt whereas in the latter we modify some implicit global variable\nplotting dots - for example with scatter/scatter!\nplot([(1.0,2.0), (1.0,3.0)], ls=:dot) will create a dotted line from position (x=1.0,y=2.0) to (x=1.0,y=3.0)","category":"page"},{"location":"lecture_05/hw/","page":"Homework","title":"Homework","text":"
","category":"page"}] }