CBG-4636: have check for skipped sequences on recent sequence handling

[gregns1]

CBG-4636

We need to check if its possible that a recent_sequence got pushed to skipped on recent sequence handing, when pending gets to max or is flushed due to age we move the next expected sequence at the cache to the oldest pending sequence. Hence this loop was not running on some sequences that were in recent sequences on a doc but were also present in skipped sequences. This meant skipped would grow exponentially with no prospect of ever removing the sequences until we abandoned them. This lead to high sequence stable being stuck until abandoned sequences job was run.

I chose not to do an actual check in skipped given this is done inside processEntry and it would be less expensive to just check if seq in recent sequence is less than current sequence and it is greater than or equal to the oldest skipped sequence.

Before changes high_seq_stable:

After changes high_seq_stable:

Pre-review checklist

• Removed debug logging (fmt.Print, log.Print, ...)
• Logging sensitive data? Make sure it's tagged (e.g. base.UD(docID), base.MD(dbName))
• Updated relevant information in the API specifications (such as endpoint descriptions, schemas, ...) in docs/api

Integration Tests

• GSI=true,xattrs=true https://jenkins.sgwdev.com/job/SyncGateway-Integration/3114/

[bbrks]

Have a question about how this change interacts when we don't have any skipped sequences but do have recent sequences present on a doc