We have deployed Sensu out to a small environment for testing purposes before replacing our Nagios server.
The sensu server has some re-mediation handlers on it that restart services / reboot servers for X failed checks - which works really well for us.
Unfortunately we have noticed that a sensu client can pretend to be any client it wishes as their is no validation on the client hostname / IP address in the check return payload. The SSL certificates appear to only create a secure connection between the client -> rabbit and not all the way to the Sensu sever.
As such we have had to shutdown all the sensu clients+servers as this for us, is a massive security issue.
Is their anything inbuilt in to Sensu today that can allow the sensu server to authenticate the client/trust the client calling in is indeed who it says it is - as to avoid a client sending critical for all our hosts and causing the entire infrastructure to reboot
There currently isn’t anything in core to verify the result source. I imagine you could use local check definitions to add a custom attribute “signature” w/ a value that is registered where the remediation handler(s) can query to validate the event’s authenticity. But if the signature is know, that to can be falsified. If someone is able to inject results onto the RabbitMQ exchange, you probably have larger concerns, or a serious people problem. I will look into utilizing the AMQP message headers, at least to provide some light reassurance.
···
On Thu, Dec 5, 2013 at 1:23 AM, Nathan Flynn eper@eper.net wrote:
Hello,
We have deployed Sensu out to a small environment for testing purposes before replacing our Nagios server.
The sensu server has some re-mediation handlers on it that restart services / reboot servers for X failed checks - which works really well for us.
Unfortunately we have noticed that a sensu client can pretend to be any client it wishes as their is no validation on the client hostname / IP address in the check return payload. The SSL certificates appear to only create a secure connection between the client → rabbit and not all the way to the Sensu sever.
As such we have had to shutdown all the sensu clients+servers as this for us, is a massive security issue.
Is their anything inbuilt in to Sensu today that can allow the sensu server to authenticate the client/trust the client calling in is indeed who it says it is - as to avoid a client sending critical for all our hosts and causing the entire infrastructure to reboot
Yeah, that’s tricky. Each message would have to be signed individually to prevent injection. Pre-reboot, the remediator could do it’s own verification, i.e. ssh’ing into the box and running the check a few times in succession, but that’s definitely not as clean.
If you’re using the sensu-check-based-remediator, you could define the remediation action for service X as “check-service-x.sh || restart_service_x.sh” to make sure that it fails before restarting it.
···
On Thursday, December 5, 2013 1:43:41 AM UTC-8, portertech wrote:
There currently isn’t anything in core to verify the result source. I imagine you could use local check definitions to add a custom attribute “signature” w/ a value that is registered where the remediation handler(s) can query to validate the event’s authenticity. But if the signature is know, that to can be falsified. If someone is able to inject results onto the RabbitMQ exchange, you probably have larger concerns, or a serious people problem. I will look into utilizing the AMQP message headers, at least to provide some light reassurance.
On Thu, Dec 5, 2013 at 1:23 AM, Nathan Flynn ep...@eper.net wrote:
Hello,
We have deployed Sensu out to a small environment for testing purposes before replacing our Nagios server.
The sensu server has some re-mediation handlers on it that restart services / reboot servers for X failed checks - which works really well for us.
Unfortunately we have noticed that a sensu client can pretend to be any client it wishes as their is no validation on the client hostname / IP address in the check return payload. The SSL certificates appear to only create a secure connection between the client → rabbit and not all the way to the Sensu sever.
As such we have had to shutdown all the sensu clients+servers as this for us, is a massive security issue.
Is their anything inbuilt in to Sensu today that can allow the sensu server to authenticate the client/trust the client calling in is indeed who it says it is - as to avoid a client sending critical for all our hosts and causing the entire infrastructure to reboot
How do others run Sensu in their infrastructure even if not running re-mediation tasks? Is it acceptable where you run it? We would fail ISO27001 at current if we deployed it
···
On Thursday, 5 December 2013 16:19:16 UTC, Nick Stielau wrote:
Yeah, that’s tricky. Each message would have to be signed individually to prevent injection. Pre-reboot, the remediator could do it’s own verification, i.e. ssh’ing into the box and running the check a few times in succession, but that’s definitely not as clean.
If you’re using the sensu-check-based-remediator, you could define the remediation action for service X as “check-service-x.sh || restart_service_x.sh” to make sure that it fails before restarting it.
On Thursday, December 5, 2013 1:43:41 AM UTC-8, portertech wrote:
There currently isn’t anything in core to verify the result source. I imagine you could use local check definitions to add a custom attribute “signature” w/ a value that is registered where the remediation handler(s) can query to validate the event’s authenticity. But if the signature is know, that to can be falsified. If someone is able to inject results onto the RabbitMQ exchange, you probably have larger concerns, or a serious people problem. I will look into utilizing the AMQP message headers, at least to provide some light reassurance.
On Thu, Dec 5, 2013 at 1:23 AM, Nathan Flynn ep...@eper.net wrote:
Hello,
We have deployed Sensu out to a small environment for testing purposes before replacing our Nagios server.
The sensu server has some re-mediation handlers on it that restart services / reboot servers for X failed checks - which works really well for us.
Unfortunately we have noticed that a sensu client can pretend to be any client it wishes as their is no validation on the client hostname / IP address in the check return payload. The SSL certificates appear to only create a secure connection between the client → rabbit and not all the way to the Sensu sever.
As such we have had to shutdown all the sensu clients+servers as this for us, is a massive security issue.
Is their anything inbuilt in to Sensu today that can allow the sensu server to authenticate the client/trust the client calling in is indeed who it says it is - as to avoid a client sending critical for all our hosts and causing the entire infrastructure to reboot
We’re in the same situation. As it is now any client can push anything to sensu. We also need to verify that clients are who they say they are. Ideally we’d like that clients have to be registered first with a key and that all messages are somehow signed with that key with HMAC or something. As it is now it can be used to “test” it but we can’t deploy in production := Monitoring data is really critical since a piece of misinformation can make an operator bring a service down without knowing.
Cheers,
A*
···
On Thursday, 5 December 2013 10:23:36 UTC+1, Nathan Flynn wrote:
Hello,
We have deployed Sensu out to a small environment for testing purposes before replacing our Nagios server.
The sensu server has some re-mediation handlers on it that restart services / reboot servers for X failed checks - which works really well for us.
Unfortunately we have noticed that a sensu client can pretend to be any client it wishes as their is no validation on the client hostname / IP address in the check return payload. The SSL certificates appear to only create a secure connection between the client → rabbit and not all the way to the Sensu sever.
As such we have had to shutdown all the sensu clients+servers as this for us, is a massive security issue.
Is their anything inbuilt in to Sensu today that can allow the sensu server to authenticate the client/trust the client calling in is indeed who it says it is - as to avoid a client sending critical for all our hosts and causing the entire infrastructure to reboot
We’re in the same situation. As it is now any client can push anything to sensu. We also need to verify that clients are who they say they are. Ideally we’d like that clients have to be registered first with a key and that all messages are somehow signed with that key with HMAC or something. As it is now it can be used to “test” it but we can’t deploy in production := Monitoring data is really critical since a piece of misinformation can make an operator bring a service down without knowing.
Cheers,
A*
On Thursday, 5 December 2013 10:23:36 UTC+1, Nathan Flynn wrote:
Hello,
We have deployed Sensu out to a small environment for testing purposes before replacing our Nagios server.
The sensu server has some re-mediation handlers on it that restart services / reboot servers for X failed checks - which works really well for us.
Unfortunately we have noticed that a sensu client can pretend to be any client it wishes as their is no validation on the client hostname / IP address in the check return payload. The SSL certificates appear to only create a secure connection between the client → rabbit and not all the way to the Sensu sever.
As such we have had to shutdown all the sensu clients+servers as this for us, is a massive security issue.
Is their anything inbuilt in to Sensu today that can allow the sensu server to authenticate the client/trust the client calling in is indeed who it says it is - as to avoid a client sending critical for all our hosts and causing the entire infrastructure to reboot
We’re in the same situation. As it is now any client can push anything to sensu. We also need to verify that clients are who they say they are. Ideally we’d like that clients have to be registered first with a key and that all messages are somehow signed with that key with HMAC or something. As it is now it can be used to “test” it but we can’t deploy in production := Monitoring data is really critical since a piece of misinformation can make an operator bring a service down without knowing.
Cheers,
A*
On Thursday, 5 December 2013 10:23:36 UTC+1, Nathan Flynn wrote:
Hello,
We have deployed Sensu out to a small environment for testing purposes before replacing our Nagios server.
The sensu server has some re-mediation handlers on it that restart services / reboot servers for X failed checks - which works really well for us.
Unfortunately we have noticed that a sensu client can pretend to be any client it wishes as their is no validation on the client hostname / IP address in the check return payload. The SSL certificates appear to only create a secure connection between the client → rabbit and not all the way to the Sensu sever.
As such we have had to shutdown all the sensu clients+servers as this for us, is a massive security issue.
Is their anything inbuilt in to Sensu today that can allow the sensu server to authenticate the client/trust the client calling in is indeed who it says it is - as to avoid a client sending critical for all our hosts and causing the entire infrastructure to reboot
In our infrastructure we already have an internal CA and use it extensively so every host has its own node key+cert signed by that CA. And anyone who uses chef server or puppet with a puppet master should also have client certs since both of those setups issue keys and certs to clients.
Configuring each sensu-client to use the per-node certs to connect to rabbitmq (and ensuring rabbitmq is configured with the ca.crt and set to validate clients) should be similar to achieving your goal of authenticating messages from clients. You are at least ensured that whomever is connecting to rabbitmq has been issued a cert by your CA and thus all messages are from trusted clients.
I would be more concerned about the messages coming from sensu-servers than clients since they can initiate actions on clients. An additional layer of security could be a special sensu-server cert that is used to sign messages from the sensu-server instances with clients verifying the sensu-server’s signature. Each client would have a copy of that sensu-server cert.
We’re in the same situation. As it is now any client can push anything to sensu. We also need to verify that clients are who they say they are. Ideally we’d like that clients have to be registered first with a key and that all messages are somehow signed with that key with HMAC or something. As it is now it can be used to “test” it but we can’t deploy in production := Monitoring data is really critical since a piece of misinformation can make an operator bring a service down without knowing.
Cheers,
A*
On Thursday, 5 December 2013 10:23:36 UTC+1, Nathan Flynn wrote:
Hello,
We have deployed Sensu out to a small environment for testing purposes before replacing our Nagios server.
The sensu server has some re-mediation handlers on it that restart services / reboot servers for X failed checks - which works really well for us.
Unfortunately we have noticed that a sensu client can pretend to be any client it wishes as their is no validation on the client hostname / IP address in the check return payload. The SSL certificates appear to only create a secure connection between the client → rabbit and not all the way to the Sensu sever.
As such we have had to shutdown all the sensu clients+servers as this for us, is a massive security issue.
Is their anything inbuilt in to Sensu today that can allow the sensu server to authenticate the client/trust the client calling in is indeed who it says it is - as to avoid a client sending critical for all our hosts and causing the entire infrastructure to reboot
Authentication of sensu-server is achieved by the client cert it uses to connect to RabbitMQ. (Assuming you have SSL verification enabled as you well should.) I am almost certain that this satisfies authentication and authorization requirements of ISO 27001, but I have not read the entire standard in quite some time. I am working on getting a copy to confirm what I just said.
Effectively, what you have is an agreement to trust everything in RabbitMQ – which is perfectly reasonable as there is no unauthenticated access to RabbitMQ and the Sensu vhost can be restricted with not only PSK, but you could even do unique username and password combinations per host, but that is both cumbersome and unnecessary.
This approach requires that you are very careful about your SSL keys and the security of the PSK so that someone cannot easily impersonate Sensu. Even with a hashing algorithm, once an attacker has the PSK and SSL private key of a sensu-server, the end is nigh.
Anyhow. Hth.
···
On Monday, January 20, 2014, Joe Miller joeym@joeym.net wrote:
I would be more concerned about the messages coming from sensu-servers than clients since they can initiate actions on clients. An additional layer of security could be a special sensu-server cert that is used to sign messages from the sensu-server instances with clients verifying the sensu-server’s signature. Each client would have a copy of that sensu-server cert.
I run from the concept of what can occur if a host is compromised on the network with root level access.
In this scenario he can now send anything in to the rabbitmq queue pretending to be any source.
I’d be happy with a queue per hostname and sensu-server uses the queue name as the source-fqdn in messages not what a client says.
As said, most people already have client certs signed by a CA from puppet, so these could be reused to sign the message prior to being put in the queue with sensu-server decrypting them and using the cert-fqdn again rather than what the client told it.
I also think the idea of having a client side unique key with a lookup config on the sensu-server to validate is a simple for effective alternative.
On Thursday, 5 December 2013 09:23:36 UTC, Nathan Flynn wrote:
Hello,
We have deployed Sensu out to a small environment for testing purposes before replacing our Nagios server.
The sensu server has some re-mediation handlers on it that restart services / reboot servers for X failed checks - which works really well for us.
Unfortunately we have noticed that a sensu client can pretend to be any client it wishes as their is no validation on the client hostname / IP address in the check return payload. The SSL certificates appear to only create a secure connection between the client → rabbit and not all the way to the Sensu sever.
As such we have had to shutdown all the sensu clients+servers as this for us, is a massive security issue.
Is their anything inbuilt in to Sensu today that can allow the sensu server to authenticate the client/trust the client calling in is indeed who it says it is - as to avoid a client sending critical for all our hosts and causing the entire infrastructure to reboot
As Nathan says we need to assume that a host gets compromised. There are 3 things we need:
Server must check that client data comes from the client it’s supposed to. Probably signed with client key although this means the server must have all the clients certs to check signatures.
Clients must check that requests come from the server. Signed with server key, thus all clients need to have the server cert
Prevent poisoning of the channel. In this case rabbitmq. How to prevent that a valid client starts flooding the channel? If a host gets compromised and you have a PSK you can say bye-bye to whatever you have since it can impersonate the server and make clients do whatever he wants.
Cheers,
A*
···
On 21 January 2014 09:20, Nathan Flynn eper@eper.net wrote:
I run from the concept of what can occur if a host is compromised on the network with root level access.
In this scenario he can now send anything in to the rabbitmq queue pretending to be any source.
I’d be happy with a queue per hostname and sensu-server uses the queue name as the source-fqdn in messages not what a client says.
As said, most people already have client certs signed by a CA from puppet, so these could be reused to sign the message prior to being put in the queue with sensu-server decrypting them and using the cert-fqdn again rather than what the client told it.
I also think the idea of having a client side unique key with a lookup config on the sensu-server to validate is a simple for effective alternative.
On Thursday, 5 December 2013 09:23:36 UTC, Nathan Flynn wrote:
Hello,
We have deployed Sensu out to a small environment for testing purposes before replacing our Nagios server.
The sensu server has some re-mediation handlers on it that restart services / reboot servers for X failed checks - which works really well for us.
Unfortunately we have noticed that a sensu client can pretend to be any client it wishes as their is no validation on the client hostname / IP address in the check return payload. The SSL certificates appear to only create a secure connection between the client → rabbit and not all the way to the Sensu sever.
As such we have had to shutdown all the sensu clients+servers as this for us, is a massive security issue.
Is their anything inbuilt in to Sensu today that can allow the sensu server to authenticate the client/trust the client calling in is indeed who it says it is - as to avoid a client sending critical for all our hosts and causing the entire infrastructure to reboot
**This message contains information which may be confidential and privileged. **Unless you are the intended addressee (or authorized to receive for the intended addressee), **you may not use, copy or disclose to anyone the message or any information contained in the message. **If you have received the message in error, please advise the sender by reply legal@cloudjutsu.com and delete the message.
Any compromised system cannot be trusted regardless of any arbitrarily complex attempt to authenticate messages. The best effort here is to properly secure your system and harden the communication channel for Sensu.
Non-root compromise is significantly more likely than root compromise, and likewise an internal threat is more likely than an external threat. Furthermore, the security of these message signatures is akin to double encrypting messages. You have signed your message once with SSL. Signing it again is simply redundant.
···
On 21 January 2014 09:20, Nathan Flynn eper@eper.net wrote:
I run from the concept of what can occur if a host is compromised on the network with root level access.
In this scenario he can now send anything in to the rabbitmq queue pretending to be any source.
I’d be happy with a queue per hostname and sensu-server uses the queue name as the source-fqdn in messages not what a client says.
As said, most people already have client certs signed by a CA from puppet, so these could be reused to sign the message prior to being put in the queue with sensu-server decrypting them and using the cert-fqdn again rather than what the client told it.
I also think the idea of having a client side unique key with a lookup config on the sensu-server to validate is a simple for effective alternative.
On Thursday, 5 December 2013 09:23:36 UTC, Nathan Flynn wrote:
Hello,
We have deployed Sensu out to a small environment for testing purposes before replacing our Nagios server.
The sensu server has some re-mediation handlers on it that restart services / reboot servers for X failed checks - which works really well for us.
Unfortunately we have noticed that a sensu client can pretend to be any client it wishes as their is no validation on the client hostname / IP address in the check return payload. The SSL certificates appear to only create a secure connection between the client → rabbit and not all the way to the Sensu sever.
As such we have had to shutdown all the sensu clients+servers as this for us, is a massive security issue.
Is their anything inbuilt in to Sensu today that can allow the sensu server to authenticate the client/trust the client calling in is indeed who it says it is - as to avoid a client sending critical for all our hosts and causing the entire infrastructure to reboot
**This message contains information which may be confidential and privileged. **Unless you are the intended addressee (or authorized to receive for the intended addressee), **you may not use, copy or disclose to anyone the message or any information contained in the message. **If you have received the message in error, please advise the sender by reply legal@cloudjutsu.com and delete the message.
Obviously a compromised system cannot be trusted. But the point here is to mitigate as much as possible the threat. Keep in mind that you won’t be notified when the system gets compromised. It can manipulate any system it has access to and you may now know it.
Currently nobody is signing anything relevant. Clients and server are just using a PSK to connect to rabbitmq. The pre shared key (PSK) right now is in the form of a certificate/key pair and a login/pass. But it still is a PSK. Once a third party gets it, you’re screwed because they can be anybody in your sensu system. Since there’s rabbitmq between the sensu server and the clients, you don’t know who sent what. sensu is trusting that the data is good.
If each component has it’s own key pair and those are used for signing messages, everyone can check that the data is coming from who we expect and that it hasn’t been modified.
Yet another point of concern is if the compromised component can spy on what the rest of the components are sending. That is also a real threat. I’m not that familiar with rabbitmq to know how to configure component isolation to prevent it.
Any compromised system cannot be trusted regardless of any arbitrarily complex attempt to authenticate messages. The best effort here is to properly secure your system and harden the communication channel for Sensu.
Non-root compromise is significantly more likely than root compromise, and likewise an internal threat is more likely than an external threat. Furthermore, the security of these message signatures is akin to double encrypting messages. You have signed your message once with SSL. Signing it again is simply redundant.
As Nathan says we need to assume that a host gets compromised. There are 3 things we need:
Server must check that client data comes from the client it’s supposed to. Probably signed with client key although this means the server must have all the clients certs to check signatures.
Clients must check that requests come from the server. Signed with server key, thus all clients need to have the server cert
Prevent poisoning of the channel. In this case rabbitmq. How to prevent that a valid client starts flooding the channel? If a host gets compromised and you have a PSK you can say bye-bye to whatever you have since it can impersonate the server and make clients do whatever he wants.
**This message contains information which may be confidential and privileged. **Unless you are the intended addressee (or authorized to receive for the intended addressee), **you may not use, copy or disclose to anyone the message or any information contained in the message. **If you have received the message in error, please advise the sender by reply legal@cloudjutsu.com and delete the message.
On 21 January 2014 09:20, Nathan Flynn eper@eper.net wrote:
I run from the concept of what can occur if a host is compromised on the network with root level access.
In this scenario he can now send anything in to the rabbitmq queue pretending to be any source.
I’d be happy with a queue per hostname and sensu-server uses the queue name as the source-fqdn in messages not what a client says.
As said, most people already have client certs signed by a CA from puppet, so these could be reused to sign the message prior to being put in the queue with sensu-server decrypting them and using the cert-fqdn again rather than what the client told it.
I also think the idea of having a client side unique key with a lookup config on the sensu-server to validate is a simple for effective alternative.
On Thursday, 5 December 2013 09:23:36 UTC, Nathan Flynn wrote:
Hello,
We have deployed Sensu out to a small environment for testing purposes before replacing our Nagios server.
The sensu server has some re-mediation handlers on it that restart services / reboot servers for X failed checks - which works really well for us.
Unfortunately we have noticed that a sensu client can pretend to be any client it wishes as their is no validation on the client hostname / IP address in the check return payload. The SSL certificates appear to only create a secure connection between the client → rabbit and not all the way to the Sensu sever.
As such we have had to shutdown all the sensu clients+servers as this for us, is a massive security issue.
Is their anything inbuilt in to Sensu today that can allow the sensu server to authenticate the client/trust the client calling in is indeed who it says it is - as to avoid a client sending critical for all our hosts and causing the entire infrastructure to reboot
**This message contains information which may be confidential and privileged. **Unless you are the intended addressee (or authorized to receive for the intended addressee), **you may not use, copy or disclose to anyone the message or any information contained in the message. **If you have received the message in error, please advise the sender by reply legal@cloudjutsu.com and delete the message.
