So I did a fresh reinstall and a new database, changed the config SMTP settings and tried to register a new user.
1) The new user does not get created at all now and gets the same error mentioned before. Tried testing all the settings again, changed ports, security modes, alternate domain for mail, nothing works. Using the next C# code to send mail from my own website when registering (same smtp settings yet it works):
var mail = new MailMessage();
mail.To.Add("whatever@mail.com");
mail.From = new MailAddress("some@mail.com");
//subject body etc.
mail.DeliveryNotificationOptions = DeliveryNotificationOptions.OnSuccess;
var client = new SmtpClient();
client.EnableSsl = false;
client.Host = "wixteria host name";
client.Port = 26
client.Credentials = new NetworkCredential(user,pass); //from config same as host and other hardcoded values
client.DeliveryMethod = SmtpDeliveryMethod.Network;
client.Send(mail);

2) Solved the issue with image upload and it not displaying, seems the ShowBlogGravatars set to true was overriding everything. Setting it to false fixes the image issue.

3) passwordFormat="Hashed", enablePasswordRetrieval="false". Admins still able to see user passwords in plain text...

I'll try to see what's in the logs tomorrow, I can't open/copy the logs from current day since those are in use by the application.

watch out for the From field - some mail servers require that the e-mail there is the same as the one for the credentials you used to log in to the mail server

Hello,

Can you check the most recent logs and see if the same error is thrown. Also, can you check and see if this setting in the web.config is also set to false:
Regards,
Mario

Hello,

I did try to toggle that option to true/false too, doesn't seem to impact anything, but I just checked the logs right after fresh install and I get an System.ApplicationException: An unknown error occured. The account could not be created, please try again later. If the problem persists, please contact us. [InvalidAnswer].
Attached the full log.
Seems like it's not even getting to the mail sending now.

Hello,

MonoX doesn't use the following settings out of the box. Please set the requiresQuestionAndAnswer from true to false in web.config. Ex:
Regards,
Mario

MZILIC, thanks, requiresQuestionAndAnswer="false" registers the user now (strange how those settings worked before the reinstall). Still getting error with the emails, tried these settings:
<add key="EnableSmtpSSL" value="false" /> , port="26" enableSsl="false"
<add key="EnableSmtpSSL" value="true" /> , port="26" enableSsl="true"
<add key="EnableSmtpSSL" value="false" /> , port="465" enableSsl="false"
<add key="EnableSmtpSSL" value="true" /> , port="465" enableSsl="true"

On port 26 (which works with the code I posted before) I get instant response with the error.
On port 465 (provided by wisteria for secure smtp) I get to wait about 30 seconds for the same error.

Not sure anymore if I'm missing something in the settings or it's just the SMTP server not working for some reason...

The code sample you posted, did you run that from the same server where MonoX is installed?

Regards,
Mario

Yes, it's from the same server, just different sub-domain.
Port 465 (secure) gets a timeout error in the logs, explains the waiting time for the response at least.

Hello,

Is the mail server on a different domain? Could you post the logs again please? Last error logs indicated that your SMTP server was refusing connection to MonoX. Also can you share more information regarding the server where MonoX is hosted and mailserver.

Regards,
Mario

Attaching the log where I tried to use multiple settings, some time out, some say a secure connection is required.
mail server: http://mail.mydomain.com/
website http://social.mydomain.com/

It's all on a shared environment from arvixe.com so definitely same server.