what is the correct config to use a custom (mailcow) mailserver? I can only find docs for SaaS mail providers, but I have my own mailserver so why should I use them?
You need a “service” key there, but i cannot find documentation about possible values.
I think this might answer your second question:
While Mailgun is preferable, in my experimentation in fixing an email problem, I tried just about every mail config setting possible. I’ve found that removing the service config option and just including the SMTP server host and port, you can use it for transactional email. It won’t allow you to send email newsletters. That still requires mailgun, but setting an SMTP server will allow you to send staff emails and handle subscriber emails.