Added gRPC endpoint and python client

[johnaohara]

This PR introduces a gRPC service and an example command line client. The gRPC service uses the same hpo_service instance as the REST api, so running experiments are available simultaneously through both api's

This allows for greater integration possibilities with other tooling as the service protobuf is consumable via many different languages which allows for easy creation of HPO clients

Protobuf also handles message encoding/decoding and date type validation at the protocol level

[johnaohara]

I am on PTO now until 19th April, so will respond to comments when I return

[dinogun]

@johnaohara Hi John, sorry just back from PTO, I see that there is a conflict, can you please resolve that. Also we are adding a demo / example script here, would suggest that we use that repo for demo scripts. (That repo will be renamed as kruize-demos as it will be housing demos for both autotune as well as hpoaas)

[dinogun]

@chandrams can you review from a test perspective, thanks

[johnaohara]

@dinogun is the test suite working now? There are no explicit tests for this PR, but if the test-suite works, I can add them

[chandrams]

@johnaohara the testsuite is working now.

To add a new test case into hpo test suite:

• Add a test case method in tests/scripts/hpo/hpo_api_tests.sh & add the function name into this array
  hpo_api_tests=("hpo_post_experiment" "hpo_get_trial_json" "hpo_post_exp_result" "hpo_sanity_test")

• Add the test description into hpo_api_test_description array in tests/scripts/hpo/constants/hpo_api_constants.sh

Then the test can be run using this command:
./test_hpo.sh -c native --resultsdir=/home//hpo_results --testsuite=hpo_api_tests --testcase=<grpc_test>

[chandrams]

Installed these modules - grpcio, google, google-api-core

Deployed the HPO service and accessed it using the gRPC client as below

(base) [csubrama@csubrama hpo]$ ./deploy_hpo.sh -c native 

Using version: 0.0.1

###   Installing HPO as a native App


### Installing dependencies..........


### Starting the service...

2022-04-22 11:15:14 - INFO - rest_service - Access server at http://localhost:8085
Starting gRPC server at http://:::50051


Experiment does not exist
2022-04-22 11:15:59 - ERROR - grpc._server - Exception calling application: 'NoneType' object has no attribute 'hpo_algo_impl'
Traceback (most recent call last):
  File "/home/csubrama/anaconda3/lib/python3.8/site-packages/grpc/_server.py", line 443, in _call_behavior
    response_or_iterator = behavior(argument, context)
  File "/home/csubrama/github/john/hpo/src/grpc_service.py", line 80, in GetTrialConfig
    data = hpo_service.instance.get_trial_json_object(request.experiment_name)
  File "/home/csubrama/github/john/hpo/src/hpo_service.py", line 68, in get_trial_json_object
    if experiment.hpo_algo_impl in ("optuna_tpe", "optuna_tpe_multivariate", "optuna_skopt"):
AttributeError: 'NoneType' object has no attribute 'hpo_algo_impl'

(base) [csubrama@csubrama src]$ python grpc_client.py new --file=/home/csubrama/github/john/hpo/tests/resources/searchspace_jsons/newExperiment.json
 Adding new experiment: petclinic-sample-2-75884c5549-npvgd
Trial Number: 0
(base) [csubrama@csubrama src]$ python grpc_client.py config
 Experiment name: petclinic-sample-2-75884c5549-npvgd
 Trial number: 0
Traceback (most recent call last):
  File "grpc_client.py", line 135, in <module>
    main()
  File "/home/csubrama/anaconda3/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/csubrama/anaconda3/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/csubrama/anaconda3/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/csubrama/anaconda3/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/csubrama/anaconda3/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "grpc_client.py", line 87, in config
    trial_config: hpo_pb2.TrialConfig = run(fun)
  File "grpc_client.py", line 126, in run
    return func(stub)
  File "grpc_client.py", line 86, in <lambda>
    fun = lambda stub : stub.GetTrialConfig(expr)
  File "/home/csubrama/anaconda3/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/csubrama/anaconda3/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "Exception calling application: 'NoneType' object has no attribute 'hpo_algo_impl'"
	debug_error_string = "{"created":"@1650606359.811651371","description":"Error received from peer ipv4:127.0.0.1:50051","file":"src/core/lib/surface/call.cc","file_line":903,"grpc_message":"Exception calling application: 'NoneType' object has no attribute 'hpo_algo_impl'","grpc_status":2}"

@khansaad - John mentioned this was due to using experiment_id to lookup rather than experiment_name. I think addressing #28 which is to replace experiment_id with experiment_name in HPO REST APIs will address this.

[khansaad]

Ok.
I will take that up now.

[johnaohara]

@dinogun / @chandrams are there anything outstanding issues on this PR?

[dinogun]

@johnaohara as mentioned earlier, wanted to use experiment_name as the key for all object retrieval before merging this. That has now been merged (PR #31). We should now be able to merge this

[johnaohara]

@dinogun ok, I will rebase and fix any conflicts

[johnaohara]

@johnaohara as mentioned earlier, wanted to use experiment_name as the key for all object retrieval before merging this. That has now been merged (PR #31). We should now be able to merge this
@dinogun is the plan to drop experiment_id completely?

[dinogun]

@johnaohara as mentioned earlier, wanted to use experiment_name as the key for all object retrieval before merging this. That has now been merged (PR #31). We should now be able to merge this
@dinogun is the plan to drop experiment_id completely?

From the input API yes, as id is something that ideally should be generated by HPOaaS. The generated id in this case is a combination of trial_id and trial_num.

[chandrams]

Tested it manually changes look fine.

[chandrams]

LGTM

[dinogun]

LGTM

[johnaohara]

Thanks a lot

[dinogun]

Thank you for helping HPO get off the ground!