Epoch processing performance optimizations

[rodrigo-o]

After some work on reducing slot processing time, which greatly enhance block processing we still have a big bottleneck on every epoch processing. After some measures I was able to pinpoint the specific part of the code were 95% of the time was taken in the epoch processing execution and it was in a particular zip operation under rewards and penalties calculation as show by the following logs:

This could be found here, but the line was modified to an Enum.zip instead of Stream, to be able to measure the bottleneck.

Something more caught my eyes when following the logs, there were 3 things that could be potential candidates for performance optimizations:

1. I'm not completely sure but as far as the log shows, epoch processing occurs more than once within 3/4 blocks of difference instead of happening once every 32 blocks, there might be something involved related to the triggering of the epoch processing
2. committee prefetch happens more than once nearly at the dame time due to the cache not storing the previous process calculating it in the meantime
3. slot processing recalculates block and state root due to them not being available which apparently just happens near the epoch processing as shown in these logs

This could easily become an umbrella Issue with sub task for tackling the aforementioned discoveries one at a time.

[rodrigo-o]

After spending some time looking at the bottleneck, I saw it happened in the Enum.zip due to it receiving a stream of deltas, so most of the time was spent on get_flag_index_deltas, particularly in the pre compute base rewards for all validators.