Skip to content

Use go 1.17 ability to re-use LZW writer#1

Open
relistan wants to merge 3 commits intomasterfrom
relistan/reuse-lzw
Open

Use go 1.17 ability to re-use LZW writer#1
relistan wants to merge 3 commits intomasterfrom
relistan/reuse-lzw

Conversation

@relistan
Copy link

@relistan relistan commented Apr 9, 2022

A busy Sidecar is spending like 6% of its CPU time allocating LZW writers! These had a dumb interface in the stdlib and it was improved in 1.17 with the addition of a Reset() function. This update re-uses both the bytes.Buffer and the lzw.Writer which should save a lot of allocs! This is only called from a loop over a channel so there should not be synchronization issues.

Saves a ton of 64k allocs

if ack.SeqNo != ping.SeqNo {
return false, fmt.Errorf("Sequence number from ack (%d) doesn't match ping (%d)", ack.SeqNo, ping.SeqNo, LogConn(conn))
return false, fmt.Errorf("Sequence number from ack (%d) doesn't match ping (%d) %s", ack.SeqNo, ping.SeqNo, LogConn(conn))
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updating Go found all these issues in the existing code

@relistan relistan requested a review from mihaitodor April 9, 2022 18:19
Copy link

@mihaitodor mihaitodor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Nice find. Looks like this should be fixed upstream too: https://github.com/hashicorp/memberlist/blob/696ff46201c1b64d31b60d57971d07d8cf15df7a/util.go#L240

Not sure if relevant, but I noticed this error is unchecked with golangci-lint:

_, publicIfs, err := sockaddr.IfByRFC("6890", ifAddrs)

Otherwise :shipit:

@relistan
Copy link
Author

relistan commented Apr 9, 2022

Awesome! Nice find. Looks like this should be fixed upstream too: https://github.com/hashicorp/memberlist/blob/696ff46201c1b64d31b60d57971d07d8cf15df7a/util.go#L240

@mihaitodor thanks, yeah I had looked at the upstream first and saw they hadn't done anything to improve this. Because the upstream SWIM implementation does nowhere near as much gossiping, I suspect that this is not as much of an issue.

@relistan
Copy link
Author

relistan commented Apr 9, 2022

Sadly, looks like this does need a pooling setup. Appears to be used in a goroutine.

@mihaitodor
Copy link

Ugh... You know, I was thinking to ask about that when I reviewed it. I recall implementing something like that somewhere, but I can't recall where.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants