completed job folder doesn't save in the server

[avramit5674]

Python version (python3 -V)

3.10

NVFlare version (python3 -m pip list | grep "nvflare")

2.4.1

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

2.3

Operating system

Ubuntu 24.04

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

I've successfully completed execute xgboost regression job of a FL system consisting 2 docker clients running on windows and 1 server running on ubuntu and got the following results:

2024-07-01 15:51:09,689 - ScatterAndGather - INFO - [identity=example_project, run=dcde19e3-245a-4962-a04f-21902ae17a4c, wf=scatter_and_gather]: End aggregation.
2024-07-01 15:51:09,691 - ScatterAndGather - INFO - [identity=example_project, run=dcde19e3-245a-4962-a04f-21902ae17a4c, wf=scatter_and_gather]: Start persist model on server.
2024-07-01 15:51:09,692 - XGBModelPersistor - INFO - Saving received model to /home/amit/Desktop/NVFlare/workspace/example_project/prod_01/server1/dcde19e3-245a-4962-a04f-21902ae17a4c/app_server/final_xgboost_model.json
2024-07-01 15:51:09,710 - ScatterAndGather - INFO - [identity=example_project, run=dcde19e3-245a-4962-a04f-21902ae17a4c, wf=scatter_and_gather]: End persist model on server.
2024-07-01 15:51:09,711 - ScatterAndGather - INFO - [identity=example_project, run=dcde19e3-245a-4962-a04f-21902ae17a4c, wf=scatter_and_gather]: Round 3 finished.
2024-07-01 15:51:09,854 - ScatterAndGather - INFO - [identity=example_project, run=dcde19e3-245a-4962-a04f-21902ae17a4c, wf=scatter_and_gather]: Finished ScatterAndGather Training.
2024-07-01 15:51:09,855 - ServerRunner - INFO - [identity=example_project, run=dcde19e3-245a-4962-a04f-21902ae17a4c, wf=scatter_and_gather]: Workflow: scatter_and_gather finalizing ...
2024-07-01 15:51:10,019 - ServerRunner - INFO - [identity=example_project, run=dcde19e3-245a-4962-a04f-21902ae17a4c, wf=scatter_and_gather]: ABOUT_TO_END_RUN fired
2024-07-01 15:51:10,029 - ServerRunner - INFO - [identity=example_project, run=dcde19e3-245a-4962-a04f-21902ae17a4c, wf=scatter_and_gather]: Firing CHECK_END_RUN_READINESS ...
2024-07-01 15:51:10,033 - ServerRunner - INFO - [identity=example_project, run=dcde19e3-245a-4962-a04f-21902ae17a4c, wf=scatter_and_gather]: END_RUN fired
2024-07-01 15:51:10,034 - ReliableMessage - INFO - ReliableMessage is shutdown
2024-07-01 15:51:10,035 - ServerRunner - INFO - [identity=example_project, run=dcde19e3-245a-4962-a04f-21902ae17a4c, wf=scatter_and_gather]: Server runner finished.

But the file "/home/amit/Desktop/NVFlare/workspace/example_project/prod_01/server1/dcde19e3-245a-4962-a04f-21902ae17a4c/app_server/final_xgboost_model.json" doesn't exists and even the job folder which located at "/home/amit/Desktop/NVFlare/workspace/example_project/prod_01/server1/dcde19e3-245a-4962-a04f-21902ae17a4c" doesn't exist either. In addition, the clients also doesn't save the final_xgboost_model.json in the their job folder which created in both dockers (job folder created but without the final_xgboost_model.json). I'm not sure if the lack of final global model json file from the clients is excepted behavior and I'd appreciate clarification.
So the real issue is the server doesn't save the job folder which contains the complete global model

[chesterxgchen]

@avramit5674 you said "I've successfully completed execute xgboost regression job of a FL system consisting 2 docker clients running on windows and 1 server running on ubuntu", in discussion #2666 you said you still have trouble running this. can you close the 2666 and explain how did you resolve the issue ?

[chesterxgchen]

We will take a look at this issue

[nvkevlu]

@avramit5674 I am trying to reproduce the issue to investigate, can you please provide some more details about the job and the environment you are using? Is the job one of the provided examples or is this a custom job? For the docker image, are you using an provided image from Docker Hub or did you build your own docker image?