Block proposer error when weak head detected

[twoeths]

Describe the bug

On one of the server we enabled chain.proposerBoostReorg flag, weak head detected

however there was a block proposer error at the same time:

Expected behavior

when weak head detected, we still need to be able to propose block

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

v1.21.0

[twoeths]

got another case:

2025-01-13 20:06:44.081	
Jan-13 13:06:44.081[chain]         verbose: Weak head detected. May build on this block instead: proposerHeadSlot=3405932, proposerHeadRoot=0xf06d0c432fca89eab063767b62b75faca6d0956acb11e5dd1b1bfdc29f5b6f31, headSlot=3405933, headRoot=0x108e157c63aab008212a619804105e9fcafe291e901bcaec5ec0bad3ba6d4499

error in PrepareNextSlot

2025-01-13 20:06:44.086	
Jan-13 13:06:44.086[chain]           error: Failed to run prepareForNextSlot nextEpoch=106436, isEpochTransition=false, prepareSlot=3405934 - Received invalid payloadId=null
2025-01-13 20:06:44.079	
Jan-13 13:06:44.078[chain]         verbose: Running prepareForNextSlot nextEpoch=106436, prepareSlot=3405934, headSlot=3405933, headRoot=0x108e157c63aab008212a619804105e9fcafe291e901bcaec5ec0bad3ba6d4499, isEpochTransition=false

same error when producing block

 error: Error proposing block slot=3405934, validator=0x97aa…b461 - produceBlockV3 failed with status 500: Builder and engine both failed to produce the block - Failed to produce block

2025-01-13 20:06:48.975	
Jan-13 13:06:48.974[api]              warn: Engine failed to produce the block slot=3405934, fork=deneb, builderSelection=default, isBuilderEnabled=true, isEngineEnabled=true, strictFeeRecipientCheck=false, builderBoostFactor=90, durationMs=5 - Received invalid payloadId=null
2025-01-13 20:06:48.201	
Jan-13 13:06:48.201[chain]         verbose: Producing beacon block body fork=deneb, blockType=Full, slot=3405934
2025-01-13 20:06:48.199	
Jan-13 13:06:48.199[chain]         verbose: Producing beacon block body fork=deneb, blockType=Blinded, slot=3405934
2025-01-13 20:06:48.197	
Jan-13 13:06:48.197[api]           verbose: Block production race (builder vs execution) starting slot=3405934, fork=deneb, builderSelection=default, isBuilderEnabled=true, isEngineEnabled=true, strictFeeRecipientCheck=false, builderBoostFactor=90, cutoffMs=1803, timeoutMs=12000
2025-01-13 20:06:48.197	
Jan-13 13:06:48.197[api]             debug: Produced common block body slot=3405934, fork=deneb, builderSelection=default, isBuilderEnabled=true, isEngineEnabled=true, strictFeeRecipientCheck=false, builderBoostFactor=90
2025-01-13 20:06:48.086	
Jan-13 13:06:48.086[api]           verbose: Assembling block with produceEngineOrBuilderBlock slot=3405934, fork=deneb, builderSelection=default, isBuilderEnabled=true, isEngineEnabled=true, strictFeeRecipientCheck=false, builderBoostFactor=90
2025-01-13 20:06:48.001	
Jan-13 13:06:48.001[chain]         verbose: Clock slot slot=3405934

[twoeths]

the problem is we call engine_forkchoiceUpdated with the weak head 3405933 already
then when preparing payload for slot 3405934, we tried to build on slot 3405932 so geth logged this and returned no payload

2025-01-13 20:06:48.202 | INFO [01-13\|13:06:48.202] Ignoring beacon update to old head       number=3,137,042 hash=b89f15..77aa7b age=24s    have=3,137,043 |  
-- | -- | --
  |   | 2025-01-13 20:06:44.085 | INFO [01-13\|13:06:44.085] Ignoring beacon update to old head       number=3,137,042 hash=b89f15..77aa7b age=20s    have=3,137,043

here's geth code: https://github.com/ethereum/go-ethereum/blob/e26dd774a9c0f77f814673d64870d84da79cd549/eth/catalyst/api.go#L392

[twoeths]

according to this spec https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#engine_forkchoiceupdatedv1

Client software MAY skip an update of the forkchoice state and MUST NOT begin a payload build process if forkchoiceState.headBlockHash references a VALID ancestor of the head of canonical chain, i.e. the ancestor passed [payload validation](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#payload-validation) process and deemed VALID. In the case of such an event, client software MUST return {payloadStatus: {status: VALID, latestValidHash: forkchoiceState.headBlockHash, validationError: null}, payloadId: null}.

We should not be rush notifying EL right after importing a block, lighthouse even do it 500ms before the next slot