Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding the matmul benchmark from plb2 #526

Merged
merged 2 commits into from
Feb 14, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions MANIFEST
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ examples/Benchmark/Bench.pm
examples/Benchmark/Bench.xs
examples/Benchmark/Makefile.PL
examples/Benchmark/README.md
examples/Benchmark/tasks/matrix_multiplication.pl
examples/Benchmark/tasks/README.md
examples/Benchmark/time.pl
examples/earth-interp.pl
examples/earth.txt
Expand Down
61 changes: 61 additions & 0 deletions examples/Benchmark/tasks/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# A Guide to Benchmarking with PDL

## Tasks

This is a collection of benchmarking tasks collected from comparison sites,
such as [plb2](https://github.com/attractivechaos/plb2),
[FPBench](https://github.com/FPBench/FPBench) and
the [Benchmarks Game](https://benchmarksgame-team.pages.debian.net/benchmarksgame/index.html).
Yes, we want PDL to be the fastest dog in the race, but more importantly it lets
our developers know when their latest commit has degraded performance.

### Matrix multiplication

From [plb2](https://github.com/attractivechaos/plb2/tree/master/src/perl), it creates
two square matrices and multiplies them together (the inner product, `x`).

This PDL script is more than 65 times faster than their Perl script.
I've tried a few variations to find a faster version, but most gains are within
the timing variation.

Initial measurement for PDL 2.095
```
Benchmark 1: ./matrix_multiplication.pl
Time (mean ± σ): 3.569 s ± 0.063 s [User: 3.656 s, System: 0.088 s]
Range (min … max): 3.521 s … 3.710 s 10 runs
```

## Benchmarking

You can run a comparison with the Perl script using [hyperfine](https://github.com/sharkdp/hyperfine)

```
hyperfine --warmup 1 'YOUR_ENV_VAR=1 path/to/matmul.pl 300' 'path/to/matrix_multiplication.pl 300'
```

I recommend low values at first because, using the default (N=1500),
hyperfine takes over 10 minutes to measure the Perl script because it's default
is to run each program 10 times (changed with the **[mMr]** options).
Ideally, you want to run this on a quiet system with few other processes running.

Other benchmarks in pure Perl can be found at [plb2](https://github.com/attractivechaos/plb2/tree/master/src/perl),

### Strategies

If you want to compare two different branches against each other,
consider using something like `--setup 'git checkout HEAD^'` or perhaps
just running hyperfine with the single benchmark while searching for
the offending commit with `git bisect`. Let us know how _you_ do benchmarking.

## Profiling

Use [Devel::NYTProf](https://metacpan.org/pod/Devel::NYTProf) to find out where
your script is spending its time and how many times a line is run. Running the
profiler really slows down your code, so I will run both commands on the same line
and come back later.
```
perl -d:NYTProf matrix_multiplication.pl 500; nytprofhtml --no-flame
```

* [profiling](https://github.com/PDLPorters/pdl/issues/451)
* [perl v java](https://charlesreid1.github.io/perl-vs-java-n-queens-problem.html)
26 changes: 26 additions & 0 deletions examples/Benchmark/tasks/matrix_multiplication.pl
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
#!/usr/bin/perl
#
# Matrix multiplication of 2 square matrices
# Implements matmul from https://github.com/attractivechaos/plb2
# Boyd Duffee, 2025

use warnings;
use strict;
use PDL;

my $n = $ARGV[0] || 1500;
$n = int($n/2) * 2;

my $a = generate_matrix($n);
my $b = generate_matrix($n);

my $x = $a x $b;
print $x->at($n/2, $n/2), "\n";

sub generate_matrix {
my $n = shift;
my $c = 1 / $n / $n;
my $i = xvals($n, $n);
my $j = yvals($n, $n);
return $c * ($i - $j) * ($i + $j);
}
Loading