Overview
MacOS Venutra was released on 10/24/2022 and if you are like me you likely upgraded at least one of your machines to see what impact it would have on your day to day life. Well, buckle up because this is going to be quite the adventure but don’t worry! You have a friend in your corner who has already resolved the issue for you and also shared these details with WP Engine!
So, What happened?
With the new update to MacOS Ventura the underlying OpenSSL package got an upgrade. Normally this wouldn’t introduce that big of a deal and life goes on normally. This time however, it actually was a breaking change. What I mean to say is, the update makes your SSH commands to WP Engine fail because it no longer supports those RSA based SSH keys by default. I found this out the hard way this morning when my work machine upgraded to the latest version and I had to access an install to troubleshoot an issue for a customer and I couldn’t connect.
I kept seeing an error saying, “Permission denied (publickey).” Great, so now my file permissions need to be fixed after the update? Nope, nothing regarding my SSH keys changed so what is wrong?
I checked my file permissions and that the private key was using the 400 mask and still refused. So I reached out to WP Engine to see if they could help identify any known issues on their end. At this point I had forgotten about the OS upgrade being a factor and was just working through the process. If it’s not me then maybe its the server.
While we are digging into things we gave the old, make a new key and add it to your account trick, a go. That didn’t work either and both of us are beginning to scratch our heads about what could possibly be preventing the key from authenticating. The config file aligns with the documentation, the ssh key looks correct, the key was added to the agent, the file permissions are correct, it can’t be the client machine so it must be the server side.
While the tech was looking into possible server issues I began to dig deeper on my end to see if I could fine any further details around what might be causing this. Everything was fine the day before… the upgrade! What could have changed between the last version of MacOS and this new one?!
I hit the rumor mill, Google, and began to sift through the results to see what might be going on. Then I stumbled upon an article at macrumors.com that said that SSH was “broken” in Ventura! And towards the bottom of this article was the first glimpse at what the issue was and how to fix it. “Ventura ships […] with OpenSSH_9.0p1…” So, I checked my other machine that was still running Catalina and found that it was running OpenSSH 8.9p1. So what changed in these versions?
Looking at the release notes for OpenSSH we see the issue called out in version 9.0 where they talk about ssh using the hybrid Streamlined NTRU Prime + x25519 key exchange method by default.
Okay nerd, but what does that mean to me?
Well, to put it simply, the old key methods that most folks still use today is RSA 4096 and that is what even WP Engine has recommended but we have slowly been transitioning to a newer more secure method called ed25519. So the new default method is trying to send my RSA style key as if it was ed25519 and it wasn’t.
How do we fix it?
As that last comment on the macrumors page mentioned, there is an additional argument that we can add to our ssh config file that will allow us to use the older method but I think its time we all practice a bit of security awareness and go ahead and make that newer key anyway just to be sure we cycle our keys every once in awhile with a fresh one. You never know when this could save your bacon and you really should be rotating your SSH keys out just as you would a password.
This act alone WILL NOT fix this issue for you though! You still need to make some adjustments to your configuration file and I will drop that little addition below this paragraph. You need to update your config file to pass the accepted key method to WP Engine because they are still expecting an RSA connection by default even though your system is trying to send an ed25519 key by default. You can see the disconnection here!
Host *.ssh.wpengine.net
IdentityFile ~/.ssh/key_name
IdentitiesOnly yes
UseKeychain yes
PubkeyAcceptedKeyTypes ssh-rsa,ssh-ed25519
The crucial piece here is that last line for PubkeyAcceptedKeyTypes. In my example I’m including both the old RSA key type as well as the new one for those folks like me who needed to just get this fixed now and address any old keys when I have a spare moment. Once that setting is in place you should be able to establish a working connection to WP Engine once again!
I did that… It still doesn’t work!
If this didn’t resolve the issue for you, you may want to check and make sure that the key is properly secured with the correct permission mask. I would run the command below just to make sure.
sudo chmod 400 ~/.ssh/key_nameYou can also add the key to your keychain again just to be sure it is available by running the following command and if the permissions of your key are not read only to your user it will give you an error letting you know that the permissions are not secure enough.
ssh-add --apple-use-keychain ~/.ssh/key_nameAfter that, try the ssh connection again and if it still fails try running it with “-v” at the end to enable the verbose mode and debug output. Sifting through that output can help you figure out what additional issues the key is having. You may find that you created a new key but forgot to add the new public key to your WP Engine user profile and your key is being rejected by WP Engine and isn’t authorized. Once, the key is added to your profile you should be all set!
Conclusion
Every update and upgrade out there has something in it that doesn’t work with everything that preceded it and these kinds of things don’t just happen with operating systems. They occur within our WordPress sites as well. A plugin might have an update and we want to use the latest most secure version whenever we can but that update might have a dependency in it that breaks other themes and plugins. That is pretty much what we saw here in this MacOS update where the SSH system that depended on OpenSSL wasn’t updated but OpenSSL was and it had new default settings that we were not aware of and those had unintended consequences.
This is why it is always a great practice to test out these updates and upgrades on another device or platform before you make them your day to day. WP Mechanic offers maintenance services for WordPress websites with that methodology in mind so that an update can always be rolled back to a previous good state and bring back functionality that may have been impacted by these types of situations. Coming soon we will also be offering managed hosting solutions and security packages to help check your site for security issues and protect them from becoming issues in the first place.
Stay tuned for updates regarding our upgraded service options and hopefully you found this post helpful with your own SSH woes after upgrading to MacOS Ventura!
