I know there’s a limit for ghost(pro).
Is there also a limit to the amount of possible staff members on a self-hosted instance of ghost?
There are no limits to self-hosted Ghost instances 
More specifically, there are different limits on self-hosted instances.
On Ghost(Pro) pricing is tiered by:
Amount of staff users, amount of members, advanced configurations - while all resource usage is essentially unlimited.
When self hosting pricing is tiered by:
Amount of RAM, amount of CPU, amount of bandwidth, CDN regions, amount of storage, number of emails sent, number of backups - while product usage is essentially unlimited.
Different setups are suited to different types of users, depending on requirements. Typically self-hosting is very well-suited to advanced/technical users who prefer managing their own infrastructure - while Ghost(Pro) is designed for people who prefer not to spend time managing those things.