SSH: Design and implement counter-DoS measures

Discuss this help topic in SecureBlackbox Forum

The main problem about DoS and 'try-and-guess' attacks is that they put a huge burden on the server's computational and networking resources. This is especially applicable to secure servers, as every instance of such server, besides allocating networking resources, also needs to maintain a set of cryptographic parameters, making per-session computational and resource costs much higher.

The exact measures to choose for counteracting DoS attacks depend on the specifics and circumstances of every particular service and the environment it is supposed to work in (e.g., they will obviously differ for a service that runs behind an IPS and a service publicly visible from the Internet and not protected by anything). Still, there are certain general rules that are applicable nearly everywhere, and will increase the DoS tolerance of any secure service significantly.

The general approach here is to kick the attacker out as early down the protocol as possible. The main points to consider are:

A newly allocated secure environment consumes several hundred KBs of memory more than a fresh accepted TCP connection, so rejecting supposed attackers just after the TCP accept stage will save you a lot of memory.
SSH/SSL key exchange is a heavy routine involving asymmetric cryptographic operations, so getting rid of potential attackers before the protocol reaches this stage decreases the CPU burden drastically.
SSH authentication phase may be used by the attacker to 'hold' the session (they may emulate the 'alive' state by issuing continuous login attempts). This may lead to exhaustion of server's network resources.

Taking the above points into account, employing the following techniques may be considered:

Keep a log of all incoming connections and introduce the black list of IP addresses. If many consecutive fake connections are coming from the same IP address and/or subnet, add the address/subnet to that list and simply deny all subsequent connections originating from that range on the network layer (i.e. straight after the Socket.Accept() call, without promoting the connection up to the SSH layer). The length of time to keep the IP in the black list can be chosen basing on the specifics of your service and the 'molestation level' of the particular IP address and can range from 5 minutes to infinite presence.
Think about the geographic scope of IP addresses where 'legal' connections to your service can come from. Is there a chance that your customers will connect to your server from e.g. China, Philippines or Russia? If not, it makes sense to reject connections originating from the corresponding geographic IP ranges even before looking for them in the black list.
Do not allow more than a reasonable amount of parallel unauthenticated connections (i.e. those still in the negotiation phase) from the same IP address. Limiting them to 2 would be fair enough.
Introduce a specific timeout for the negotiation phase. If the negotiation does not complete for 1 minute, there is a chance that a hacker is trying to consume the network resources of your server.
Do not allow more than 3 unsuccessful login attempts per session and terminate the session immediately after the last unsuccessful attempt.
Do not allow too many sessions authenticated with the same user credentials. A large number of sessions (in a majority of cases, more than 1) indicates that the credentials have probably been stolen and are now used to attack your server.
Temporarily block credentials which are supposed to be stolen.
Control the number of active SSH sessions made to your server. Introduce emergency threshold values after which no further sessions are accepted and/or a notification is sent to the server administrator.

The above techniques are the universal ones. Other methods of controlling the load and deciding between legal and illegal connections that consider the specifics of particular environments can also be used.

How To articles about SSH server

Discuss this help topic in SecureBlackbox Forum