So I did a fresh reinstall and a new database, changed the config SMTP settings and tried to register a new user.
1) The new user does not get created at all now and gets the same error mentioned before. Tried testing all the settings again, changed ports, security modes, alternate domain for mail, nothing works. Using the next C# code to send mail from my own website when registering (same smtp settings yet it works):
var mail = new MailMessage();
mail.From = new MailAddress("email@example.com");
//subject body etc.
mail.DeliveryNotificationOptions = DeliveryNotificationOptions.OnSuccess;
var client = new SmtpClient();
client.EnableSsl = false;
client.Host = "wixteria host name";
client.Port = 26
client.Credentials = new NetworkCredential(user,pass); //from config same as host and other hardcoded values
client.DeliveryMethod = SmtpDeliveryMethod.Network;
2) Solved the issue with image upload and it not displaying, seems the ShowBlogGravatars set to true was overriding everything. Setting it to false fixes the image issue.
3) passwordFormat="Hashed", enablePasswordRetrieval="false". Admins still able to see user passwords in plain text...
I'll try to see what's in the logs tomorrow, I can't open/copy the logs from current day since those are in use by the application.