Add the client IP address to each message if requested in the configuration#3
Add the client IP address to each message if requested in the configuration#3stanhu wants to merge 1 commit into
Conversation
…ration file. Requires an update to the Lumberjack server to pass along the address.
|
Given this protocol uses TLS for authentication, it seems like it would provide more accuracy to use the client's SSL certificate subject instead of the client ip (because the real client address would be obscured by folks using proxies, etc). Thoughts? |
|
I think there is still value in getting the IP address, even if proxies get in the way. Right now I have to workaround this by having each logstash-forwarder node perform an external IP lookup. Someone else also ran into this issue on Stack Overflow. Are SSL client certificates being used here? If they are not, how would the server get the client's SSL certificate subject? |
|
@stanhu you can also set any arbitrary field you wish in the logstash-forwarder config. This would easily allow you to, on every node running lsf, set a special field that includes an IP, a name, or anything. |
|
@jordansissel, thanks, I did see that. As far as I can tell, the key/values must be statically configured in the logstash-forwarder configuration. Since many of the machines I use have ephemeral IP addresses, having 100 different configuration files that need to kept up-to-date doesn't work so well. |
Apologies if this sounds like a rude question, but why not use configuration management to solve this for you? Generate your config files based on the system and everything will be groovy :) |
|
We do use Ansible and it could populate the forwarder with the right info, but that's another moving piece that we have to get right. I'd rather have logstash just take care of it in one place. |
Well, someone has to get it right, somewhere. The purpose of the 'fields' setting in logstash-forwarder was for exactly these scenarios where you need to ship additional data along with events (perhaps cluster name, datacenter name, region name, application name, or other identification information). |
|
Yes, it's useful for doing that, and we do use the extra fields for those kinds of things. The one challenge is that across machine reboots, ephemeral IP addresses may change, and so the configuration has to be be updated each time to reflect that and require a restart of the forwarder process. In any case, an application may want to know that exactly which IP address from which the SSL server received the data. The data is readily available by the socket, and it should be made available. As you mentioned, there may be proxies that may be changing the actual IP address seen by the server. This may be another use case for this PR. |
|
It feels like we're both standing here saying "I don't want to do any work" to each other :\ |
|
I'm open to adding IP source, but I'm not sure about the implementation here. I would prefer exposing everything we can know about the connection (certificate, etc) rather than focusing specifically on just one attribute like IP. |
|
:) I like that idea. Another alternative I considered was to have the callback use the SSLSocket instance, but that potentially gives the input filter the ability to close/open a new connection. We could whittle down the callback to a map of client data. Looking at the SSLSocket interface, I see a number of things that could be sent:
Anything I missed? |
+::Dir.glob('vendor/*')
Remove erroneous files reference
Fixes logstash-plugins#3
|
Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run; then say 'jenkins, test it'. |
|
is this supported yet? Why is this not merged |
3 participants
Requires an update to the Lumberjack server (see https://github.com/stanhu/logstash-forwarder/tree/add-client-address) to pass along the address. I'm not quite sure how to tie the server version to the input filter version.
I considered putting this functionality in the Lumberjack server itself, but it seems like the input filter should decide this here, just as the TCP input filter does.