Update - I found what what was causing the graying out of the target.

Even with the PackageHub Repo removed from the system I was trying to SP migrate there were 2 left over packages:

PackageHub-release SUSE:Backports:SLE-12-SP4 12.4-1.29
PackageHub-release-POOL SUSE:Backports:SLE-12-SP4 12.4-1.29

Not sure why they were still there nor do I know what exactly they are as they don't appear to have any actually packages. But once I removed them I was able to SP migrate the system without PackageHub downloaded.

Not sure if this means there is another issue or not.