Adding the matmul benchmark from plb2

[duffee]

Getting a start on #518. Adding matmul from the Programming Language Benchmark, I've created a tasks directory under examples/Benchmark for the scripts and included a guide to hold notes for hopeful benchmarkers.

If there's a better way of creating the matrices, let me know.

You can run a comparison with the Perl script using hyperfine

hyperfine --warmup 3 'path/to/matmul.pl 300' 'path/to/matrix_multiplication.pl 300'

I recommend low values at first because at N=1500, hyperfine takes over 10 minutes to measure the Perl script.

[mohawk2]

Please could you add the note you included in the comment here (about hyperfine) to the guide you made? And rename that guide to README.md, since that's effectively what it is?

[mohawk2]

You'll note this is failing one of the release tests, which you can repro locally with prove -l xt. The solution is to run rm MANIFEST; make manifest

[duffee]

Adding notes to the new README.md:

I need a suggestion on how other devs can test two different versions of PDL on the same benchmark on the same command line. Different environment variables are a possibility, as are different paths. How do you work with 2 PDLs in the same git repo? Hyperfine takes the commands to run in single quotes, like so

hyperfine 'DEV_ENV=1 PERL5_LIB=over/here ./matrix_multiplication.pl' '...'

Thoughts?

Edit: hyperfine has --setup and --prepare options that allow you to run commands before timing runs so PDL could be re-compiled between comparisons.

[mohawk2]

How do you work with 2 PDLs in the same git repo?

It's not really a "thing". Why would it be?

By the way, your most recent commit is a merge instead of a rebase. Please rebase and force-push, like the other times. Please make this part of your normal practice.

[mohawk2]

You'll note this is failing one of the release tests, which you can repro locally with prove -l xt. The solution is to run rm MANIFEST; make manifest

This is still not fixed.

[duffee]

For 2 PDLs, I'm thinking of the case where you want to know if your code change has affected the benchmark.

I've added this to my git config (edited)

[pull]
      rebase = true

MANIFEST is fixed locally and will go up in the next commit. If you've no bright ideas, I'll push the README I've got so far.

[mohawk2]

For 2 PDLs, I'm thinking of the case where you want to know if your code change has affected the benchmark.

In the normal way, I'd run the benchmark, make the change, run it again. But one can achieve similar with git checkout HEAD^ or any other ref. This is another use-case for git bisect, of course, to find where things changed. If Hyperfine is really taking 10 mins to speed-check a simple matmult, is it really worth using?

MANIFEST is fixed locally and will go up in the next commit. If you've no bright ideas, I'll push the README I've got so far.

My approach would be (and is) to push changes as I have them. Small, beneficial changes should be the "unit" of a commit.

[duffee]

Requested changes made to MANIFEST and README.md

PR submitted to attractivechaos/plb2

[coveralls]

coverage: 67.081%. remained the same
when pulling c9d5f5c on duffee:benchmarking-matmul
into a47768e on PDLPorters:master.

[mohawk2]

This PR is now in much better shape, thank you. Your PR on plb2 looks good, too.

Are you planning to add more to this, or shall I merge it?

[duffee]

You can merge this. I'll come back to the other benchmarks after I've got my modules passing cpants again.