HerbBenchmarks.jl

A collection of useful program synthesis benchmarks. Each folder contains a different benchmark and a README.

A benchmark has:

• A readme, including description of data fields and dimensionality
• (multiple) sets of input output examples
• file that references where we found that
• evaluation function/interpretation function
• program grammar

Optional:

• a data grammar or data generation scripts, ideally with a notion of increasing complexity
• download script to download that specific dataset, if too big for Github
• test-/train-split getter functions if specified in the original data
• landmarks for planning problems

How to use:

HerbBenchmarks is still not yet complete and is lacking crucial benchmarking functionality. However, if you want to test on a single problem and grammar, you can do the following

Select your favourite benchmark, we use the string transformation benchmark from the SyGuS challenge:

using HerbSpecification, HerbGrammar

using HerbBenchmarks.PBE_SLIA_Track_2019

# The id has to be matching
grammar = PBE_SLIA_Track_2019.grammar_11604909
problem = PBE_SLIA_Track_2019.problem_11604909

# Print out the grammar and problem in readable format
println("grammar:", grammar)
println("problem:", problem.examples)

For some benchmarks there is only a single grammar for all problems.