Support Sharding with PP_Degree > 1

[HappyAmazonian]

Description

Support Sharding with PP_Degree > 1

• make tp_degree and pp_degree member variables to NeoShardingService class
• remove pipeline_parallel_degree validation for sharding
• change save_configs so it save info to model config obj gradually. It will only output the file to disk when it gets the last pp_rank.
• add pp_rank_to_shard param to shard_lmi_dist_model
  • when pp_degree = 1, the behavior should be the same
• in run_sharding, we create a loop of pp_degree, so it calls shard_lmi_dist_model for each pp_rank.

• If this change is a backward incompatible change, why must this change be made?
  this will break if the lmi_dist is not updated

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

Checklist:

• Please add the link of Integration Tests Executor run with related tests.
• Have you manually built the docker image and verify the change?
• Have you run related tests? Check how to set up the test environment here; One example would be pytest tests.py -k "TestCorrectnessLmiDist" -m "lmi_dist"
• Have you added tests that prove your fix is effective or that this feature works?
• Has code been commented, particularly in hard-to-understand areas?
• Have you made corresponding changes to the documentation?

Feature/Issue validation/testing

Tested with

llama3 405b pp=2 tp=8, sharded on one instance, and tested serving with two p4de instances.

llama3 8b pp=2 tp=4, sharded on one instance, and tested serving with two g6.12xlarge instances.

llama3 8b pp=1 tp=4, sharded on one instance, and tested serving with one g6.12xlarge instances.