BioHackathon 3, Kassel, Dec 2024

[slobentanzer]

This discussion is to fix in writing ideas and directions to pursue in the BioHackathon, in order to facilitate grouping of participants and follow-up and recording of implementations.

For each idea / subtopic / team / objective, we can start a discussion thread below. From this first step, we can derive more concrete issues and, ultimately, pull requests implementing solutions for these issues.

Contributions can target the BioChatter repository directly (for instance, in the case of additional API calling modules), or, if the implementation is not a logical direct addition to biochatter, the repository created for the hackathon. Issues can also be created in both repos, accordingly.

Once these issues and PRs are created, please link them in the corresponding discussion thread below.

[slobentanzer]

Subproject idea: end-to-end proof of concept of a pipeline that aids in applying an scverse package. Start with scanpy.

[slobentanzer]

Meta-issue to collect all API calling module issues: biocypher/biochatter#223

[slobentanzer]

PR for plotting module: biocypher/biochatter#226

[noahbruderer]

Issue for anndata: biocypher/biochatter/#224

[mengerj]

Issue for scanpy tools: biocypher/biochatter#222

[anisdismail]

new issues for concatenation operations in anndata biocypher/biochatter#235

[slobentanzer]

Insight from Tuesday: it is not possible to use the function calling (tool calling) functionality through langchain without going to the actual lowest level of granularity. The method (https://python.langchain.com/docs/how_to/tool_calling/) assumes that the parameters of the Pydantic class are exactly reflecting the parameters (arguments) of the function the LLM should use. As such, we are now exclusive exploring two options:

• manually or semi-automatically defining the Pydantic classes based on the documentation of scverse functions. Advantages: independence of package imports, high degree of manual intervention (e.g., adjusting descriptions to make the LLM work better). Disadvantages: maintenance burden.
• automatically parsing the python functions or modules that should be parameterised by using the Pydantic functionality, directly yielding Pydantic classes for all methods in the module. Advantages: highly dynamic and scaleable to any software (that is well-documented). Disadvantages: no intervention of content or extent of the information given to the LLM.

[slobentanzer]

Option: implement an abstract base class for the specific Pydantic BaseModel implementation of the different APIs, enforcing standard parameters:

• REST API: base_url, question_uuid
• Python API: method_name, question_uuid

[slobentanzer]

Post-hackathon objective: identify LLMs with tool calling capabilities and extend the conversation class to allow using different LLM vendors as useful for the community.

[anisdismail]

Further unit test development: for every class implementing BaseQueryBuilder

It is needed to test both create_runnable and parameterise_query. Currently, only unit tests of parameterise_query for AnnDataIOQueryBuilder and ScanpyPlQueryBuilder are implemented. Further implementation is needed for create_runnable unit tests, but it might be more complicated to devise the mocking inside.

[noahbruderer]

Key Updates:

Pydantic Class Updates and Serialization Fixes
1. Updated Pydantic Classes:
• Added defaults for optional fields (None or placeholders) to fix serialization issues.
• Resolved FieldInfo leakage in serialized outputs.
2. Import Change:
• Important: Now importing Field directly from pydantic instead of langchain_core.pydantic_v1.
3. Method Update:
• format_as_python_call now explicitly takes BaseAPIModel as the argument type.

Next steps:

General:
0. Change the BaseAPIModel method_name into title field.
• To be consistent with the automatic pydanitc class generation
For the concatenation: biocypher/biochatter#235
1. Integrate format_as_python_call in the general workflow:
• Ensure the method outputs the correct Python code as a string in all relevant use cases.
2. Extend Updates for New Concatenation Methods:
• Write the pydantic classes
• Write the QueryBuilder
• Follow the same structure as the updates to the Pydantic classes (add defaults, ensure proper serialization, and adapt methods).
3. Testing and Validation:
• Validate outputs with an LLM -> local scripting
• Benchmark the new methods to confirm functionality and performance.

[anisdismail]

@noahbruderer and I are working today on issue biocypher/biochatter#235, working on branch anndata-concatentation-integration
Also wrapping up the issue biocypher/scverse-x-biochatter#4 to extend the notebook for anndata operations

[anisdismail]

Added issue for adding benchmarking unit tests for AnnData IO Operations here biocypher/biochatter#250

[mengerj]

Template for creating pydantic classes for functions to be used as tools for bind_tools method:
https://github.com/biocypher/scverse-x-biochatter/blob/main/functions/dict_to_pydanctic_template.py

[anisdismail]

Day 4 Breakdown

@noahbruderer and @anisdismail

Insight from Thursday:

• Added support for concatenation + mapping using pydantic classes
• Added test use cases in benchmark for reading, concatenation and mapping
  Updates are reflected in PR Anndata concatentation + mapping integration biochatter#257
• It was really challenging to resolve all conflicts especially with tools maker integration in anndata_agent.py, so we have to revert to pydantic classes definition instead to run the benchmark successfully.
• test_create_runnable was integrated to AnnData query builder and ScanpyTl query builder thanks to @vd-dragan21. Need to test runnable for all the remaining query builders.

Future Steps:

• Injecting the runtime variables into the LLM: This might allow the LLM to have a better overview of the active variables in the runtime
• Formatting series of methods: when faced with multiple methods like adata.loc[:,['var1',var2]].copy(), we have to modify the formatter to allow the create of multiple methods while respecting their order.
• Improving system prompt: important to study the impact of the system prompt and its variations on the LLM performance