Always blaming the network team first is tried and true common knowledge in the cloud industry.
As an alumni CCIE, I would like to avoid this tendency, but reality raises its ugly head more often than not.
I have encountered a frustrating problem with a customer using a vCenter converter to migrate(P2V) old physical servers to VMC(VMware Cloud) On AWS.
The customer received the error “A file I/O error occurred while accessing.”
It took blood and tears to realize we found an asymmetric routing edge case caused by the vTGW(AKA transit connect).
Disclaimer vCenter converted is an end-of-life and end-of-support product. Unfortunately, today is the only VMware product to perform P2V conversion, and as a recent HCX update, HCX provides best-effort support for vSphere 5.5, thus making v2v legacy migration hard.
Alongside manual export, such as OVF export.
vCenter converter is an excellent way for small-scale P2V and legacy V2V migrations.
If the customer can handle this manual effort and where downtime is not a show stopper.
How things were working before vCenter converter error caused by vTGW asymmetric routing problem has started
This customer has successfully migrated in a PoC several physical workstations. All went successfully smoothly.
We have followed the excellent vCenter converter blog post of Nico Vibert.
Explaining how to use the vCenter converted in VMware Cloud on AWS and its common pitfalls. As well as this excellent blog post from VMwareArena.
The customer has connected through a simple RPVPN(route-based site-to-site VPN) to the VMware Cloud on AWS from his on-prem environment.
It’s important to note that the internal management subnet of the SDDC, specifically the ESXi management subnet and the vCenter, needs to be accessible and resolvable by the vCenter converted client.
The management subnets are advertised through the VPN, and the firewall rules are open to port 902 towards the ESXi hosts in the management gateway.
Note that you cannot access the ESXi hosts over the internet over NAT only through RBVPN, PBVPN, DX, vTGW, and Sidecar VPC.
vCenter converter requirements.
How things stopped working and vCenter converter error started caused by vTGW asymmetric routing
The next day all of a sudden, after connecting a vTGW to an AWS native VPC, to access an isolated CRM application hosted on native AWS.
The customer’s vCenter converter started to get an error of “A file I/O error occurred while accessing.”
Getting to the bottom of the vCenter converter error
We have tried several troubleshooting steps to eliminate routing and firewall rule issues on the NSX side. The vCenter was accessible and pingable from on-prem the whole time, but the ESXi servers were not.
To troubleshoot the networking path, we tried to ping from the sidecar VPC through an AWS SSL VPN and successfully did so.
After some internal consulting, I realized we had stumbled on a previously undocumented constrain. Because of this case, VMware updated the documentation. Please see the vTGW documentation below.
Creating and Managing SDDC Deployment Groups with VMware Transit Connect™
Suppose you connect a vTGW or DX to an existing SDDC with other connectivity in your topology, such as a VPN.
The ESXi hosts traffic will always be forced to be egress the SDDC through the vTGW/DX interface. It is regardless of how traffic has entered, either if it is through RBVPN or PVBPN.
Creating asymmetric routing or even routing into a black hole, in our case, into an AWS native VPC that doesn’t know how to route back the traffic to the on-prem environment.
Fixing the vTGW asymmetric routing problem
Once we had temporarily disabled the vTGW SDDC group attachment, the vCenter converter returned to work as expected.
To address this issue from an architecture perspective, we suggest either from on-prem terminating the site-to-site VPN on the AWS VPC or creating a transit security VPC supported topology since our latest M16 release.
Or remove the vTGW altogether and continue working with a site-to-site VPN from the VMC(VMware cloud) on AWS NSX-T to the Native AWS TGW if its bandwidth disadvantages are not a show stopper.
To read more about peering VPC, please read Gilles Chekroun blog post here
Now you know how asymmetric routing may occur due to poor design in VMC(VMware cloud) on AWS, causing vCenter converter to present an error of “A file I/O error occurred while accessing.” and how to address it.
If you found this helpful, please leave let me know by leaving a comment.