Repository with the results from OpenTau

Link to the main repository: https://github.com/GammaTauAI/opentau

Link to the dataset used for evaluation: https://huggingface.co/datasets/nuprl/ts-eval

744-dataset Results table

Configuration	Percent Typecheck	Avg Heuristic	Avg type errors
Santacoder TS FT - Baseline - 2048	245/744 (32.9%)	200.7	4.7
Santacoder FIT FT - Baseline - 2048	297/744 (39.9%)	200.9	5.2
Santacoder FIT FT - Baseline - 1024	248/744 (33.3%)	200.7	5.1
Santacoder FIT FT - Baseline - 512	178/744 (23.9%)	201.2	6.3
Santacoder FIT FT - Tree, no usages - 2048	274/744 (36.8%)	168.4	3.7
Santacoder FIT FT - Tree, usages - 2048	353/744 (47.4%)	154.6	3.3

Configuration to file mapping

Configuration	File name in the repository
Santacoder TS FT - Baseline - 2048	santacoder_baseline_finetuned_ts_ts-eval.json
Santacoder FIT FT - Baseline - 2048	santacoder_baseline_finetuned_fim_2048_ts-eval.json
Santacoder FIT FT - Baseline - 1024	santacoder_baseline_finetuned_fim_1024_ts-eval.json
Santacoder FIT FT - Baseline - 512	santacoder_baseline_finetuned_fim_512_ts-eval.json
Santacoder FIT FT - Tree, no usages	santacoder_tree_nousages_ts-eval.json
Santacoder FIT FT - Tree, usages	santacoder_tree_ts-eval.json

Notes about the results

• Metrics in bold are the best results for each column.
• All rows for the 744 runs had 0 average syntax errors, thus we did not include the column in the table.
• FIT FT means that the model was fine-tuned for Fill-In-the-Type, for more information about the technique, please refer to the repository: https://github.com/GammaTauAI/santacoder-finetuning
• TS FT means that the model was fine-tuned for TypeScript code generation, NOT for Fill-In-the-Type (FIT) or Fill-In-the-Middle (FIM).
• The numbers at the end (e.g. 2048, 1024, 512) are the maximum number of characters for the context window when running inference with the model.

50-dataset Results table (Parser Disabled)

Both rows used the Baseline and the maximum context window size was 2048 characters.

Configuration	Percent Typecheck	Avg Heuristic	Avg type errors	Avg syntax errors
Santacoder TS FT	1/50 (2.0%)	0	121.2	42.1
Santacoder FIT FT	25/50 (50.0%)	230.0	4.6	0.2

Configuration to file mapping

Configuration	File name in the repository
Santacoder TS FT	santacoder_baseline_noparser_normal_ts-eval-subset.json
Santacoder FIT FT	santacoder_baseline_noparser_fim_finetuned_ts-eval-subset.json