Investigating performance

[conradoplg]

Things we know to be true:

• Sync seems slower than usual. We need accurate benchmarks but we have multiple reports of sync going slowly and getting stuck. The symptoms seem largely similar to what was described in this old issue.
• Sync-to-mandantory-checkpoint got slower; it used to take less than 3h but now takes ~5h (we know this because the CI job that ran it had a 3h timeout). Note that pre-checkpoint sync is not influenced by signature and proof verification performance, and that the mandatory checkpoint (Canopy activation) is also way before the spam attack.

Ideas to start up the investigation:

• Run a full mainnet zebrad sync with checkpoint_sync = true and a full zcashd sync (also with checkpoints enabled? There is a -ibdskiptxverification flag but I don't know if it's really it and whether it is enabled by default or not) and get their logs. Compare the sync progression. Ideally also enable metrics and make prometheus scrape them so we can analyze the metrics with grafana.
• Optional/less priority: the same but with checkpoints disabled. This is not as important since it's not how people deploy the nodes, but might give interesting data.
• The same but limited to the mandatory checkpoint (or simply extract the data up to the mandatory checkpoint from the full sync data, if possible), so that we can analyze the sync-to-mandatory-checkpoint performance.
• Profile zebrad and enable flamegraphs. That might be unwiedly for the full sync, so we might want to pinpoint some specific parts of the sync that we want to profile. Suggestions: genesis up to some reasonable height (1000?); and some part of the spam + 1000-ish blocks. The latter one will require a partially-synced state. Note that it should be possible to use ldb checkpoint command to save and restore state between profiling runs.
• Do some or all of the above, but for the testnet.

Links / data points

• Last successful full sync from CI. It took 1d 21h. (Weirdly, a subsequent run got cancelled for reasons we're investigating but ran for 2d 5h and reached only height 2515206)
• (There are a bunch of old issues related to performance that I want to dig into and list here)

[conradoplg]

Comparison made when preparing the 2.4.0 release.

Logs	Desc	Time	Rate	Commit
Logs	last successful sync from CI	45.8 hours	~63k blocks/hour	cbd1bb7
Logs	manual sync before 2.4.0 release	69.5 hours	~42k blocks/hour	70909eb

Some observations:

• The second run stalled near the end due to bug: high memory usage #9699. However I think the rest of the syncs shouldn't have been affected by it.
• Everything is slower in the second run, including the before-spam period. That seems to indicate a general issue in the overall sync.

[upbqdn]

Seems like a duplicate of #7718. It might be worth merging the discussions and closing one of them.