I have a test controller in a domain and a test agent in a workgroup and I can’t register the agent with the controller.
Create shadow accounts on the agent machine and on the controller machine that are in the Administrators Group. Shadow accounts are accounts with the exact same name and password.
You can test your shadow account by trying to map to a share on the controller machine from the agent machine. Try to access \\controlerMachine\c$ when you are challenged for credentials use the shadow account. Note you may have to disable your firewall to perform this test.
Install and configure the controller on the controller machine. In the Configure Controller dialog use your TFS account and select the project collection. Using Microsoft Test Manager (MTM) under the Lab Center tab verify that your controller is registered on the controller tab.
Second log in to the test agent machine as the shadow account and install and configure the test agent with the test controller. When asked for a user name use the shadow account again. Using MTM refresh the contoller and you should see the Test Agent listed. This machine is now ready to be used in Lab to create a physical environment.