As stated in Ghost as a sub directory using NGINX - #4 by Hannah, Ghost only configures the webserver to proxy requests to Ghost for the domain, not the server. What this means for you is you can’t access Ghost by direct IP access - you need to access it by the domain.
Since you deleted the nginx is installed page, nginx (your webserver) has nothing to serve, so it responds with a 404 because no resource exists to respond with.