Last week when I tried to login to a remote test server, SSH timeout exception occured quite frequently. I didn’t pay much attention untils an internal monitoring system detected there were some potential malicious activities in that server.

I noticed that when I logined into the server, SSH told me that there were about 2k failed SSH login attempts before this success one.

Someone is hacking the server.

I went to /var/log and checked the login history:

1
2
grep -i fail /var/log/secure | less
grep -i fail /var/log/secure | wc -l

Holy, there were about 40k failed attempts within last two days. In order to get more details, I need to figure out where did these requests come from and what did they do.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# This script generates the list of users
# that was used by these malicious login attempts.
# Results are printed in a reverse order by frequency.
grep -i "Failed password for invalid user" /var/log/secure |
  cut -d " " -f 11 |
  uniq |
  while read name
  do
    grep "$name" /var/log/secure | wc -l | tr -d "\n"
    echo " $name"
  done | sort -n -r | uniq

# This script generates the list of IPs
# that was used by these malicious login attempts.
# Results are printed in a reverse order by frequency.
grep -i "Failed password" /var/log/secure |
  cut -d " " -f 13 |
  uniq |
  while read ip
  do
    grep "$ip" /var/log/secure | wc -l | tr -d "\n"
    echo " $ip"
  done | sort -n -r | uniq

Here is a list of top users,

Amount Username Amount IPs
75381 1 2913 59.56.110.209
53914 user 2501 59.14.66.191
12097 log 383 82.235.192.233
2313 unknown 198 61.188.189.7
1156 service 163 123.31.34.215
570 admin 122 123.16.141.221
289 linux 115 198.167.140.168
235 test 91 202.96.30.210
232 app 90 91.224.160.184
205 oracle 77 222.215.118.68
179 ubnt 72 195.225.58.195
166 pos 64 179.43.141.216
153 postgres 57 179.43.141.225
148 temp 50 123.31.34.44
146 monitor 39 183.131.83.224
146 cisco 37 185.128.41.117
143 ziyuan 20 163.172.16.102
143 superman 14 171.251.76.109
143 sever 10 45.63.61.171
143 root123dmk.com 10 202.196.0.243
143 linuxer 4 118.193.214.29

Like these strange usernames, I can not get any useful information from IP distribution, most of them point to proxy server. The only thing that I can do is to block these frequent IPs to reduce resource cost by using fail2ban.