Problem:

I need to apply the same validation rule to every request of my web test to test for forbidden text.

Solution:

Create a web test plug-in to insert the validation rule before each request.

Code:

using System;
using System.ComponentModel;
using Microsoft.VisualStudio.TestTools.WebTesting;
using Microsoft.VisualStudio.TestTools.WebTesting.Rules;

namespace DLBRacingTest.Custom
{   
   [DisplayNameAttribute("Forbidden Text")]   
   [Description("Adds a validation rule on every request looking for forbidden text.")]   
   public class ForbiddenText : WebTestPlugin   
   {       
      /// <summary>       
      /// Optional , delimited list of forbidden text. If left blank 'Page Error' is the only       
      /// forbidden text.       
      /// </summary>       
      public string Values { get; set; }         
      public override void PreRequest(object sender, PreRequestEventArgs e)        
      {      
         base.PreRequest(sender, e);

         if (string.IsNullOrEmpty(Values))                
            Values = "Page Error";             

         foreach (string value in Values.Split(",".ToCharArray()))            
         {                
            ValidationRuleFindText pageErrorRule = new ValidationRuleFindText();                
            pageErrorRule.FindText = value;                
            pageErrorRule.IgnoreCase = true;                
            pageErrorRule.UseRegularExpression = false;                
            pageErrorRule.PassIfTextFound = false;                
            e.Request.ValidateResponse += new EventHandler<ValidationEventArgs>(pageErrorRule.Validate);            
         }        
      }   
   }
} 

Explanation:

While running test on my web site I noticed a test passed although one of the pages had a Page Error on it. The reason the test passed was because there were no required extraction rules, no validation rules and the paged it failed on was the intended page. However, this test obviously should have failed. The existence of the word “Error” on any of my sites pages is a failed test. Something has gone wrong and I need to be alerted.  By this point I have hundreds of web test each with many request. The thought of having to add a validation rule to every request of every test was very discouraging.

I just knew there had to be an easy way to add this validation rule to every request without doing it explicitly.  A web test plug-in actually has callback that is called before every request of your test called PreRequest.  So I simply add the validation rule when this callback is called.  Now I only have to add a single web test plug-in and every request of that test now has a validation rule that will fail if any forbidden text is found.

I added an optional parameter to the web test plug-in called Values. If you leave it blank a single validation rule is added that searches for the text “Page Error” and fails if it is found.  If you have other forbidden words you would like to search for simply set Values to a comma delimited list of values (i.e. “Page Error,Invalid”).

To use this web test plug-in you can either create a separate class library and reference it in your test project, or simply add this class to your test project. Once you do and build the code you can right click on the root of your web test and select Add Web Test Plug-in… from the context menu.  Select “Forbidden Text” from the list and click OK.

Problem:

I can’t see my custom build steps in the list of build steps during a team build.

Solution:

Use BuildStep element in tfsbuild.proj file.

Code:

<Target Name="MyCustomTarget">
   <BuildStep TeamFoundationServerUrl="$(TeamFoundationServerUrl)" BuildUri="$(BuildUri)" Name="TestBuildStep" Message="Running BVTs">
      <Output TaskParameter="Id" PropertyName="TestBuildStepId" />
   </BuildStep>

. . .

   <BuildStep TeamFoundationServerUrl="$(TeamFoundationServerUrl)" BuildUri="$(BuildUri)" Id="$(TestBuildStepId)" Status="Succeeded" />
   <OnError ExecuteTargets="OnTestTargetFail" />
</Target>

<Target Name="OnTestTargetFail">
   <BuildStep TeamFoundationServerUrl="$(TeamFoundationServerUrl)" BuildUri="$(BuildUri)" Id="$(TestBuildStepId)" Status="Failed" />
</Target>

Explanation:

Now when I run my builds I can see my custom steps with a Play icon while running and a check or X when finished.  As you can see below you can see the Running BVTs build step.  The “Copying files to IIS.” was also added this way.

Additional information about the BuildStep element can be found here http://msdn.microsoft.com/en-us/library/bb399129.aspx 

Problem:

I do not want to publish the test results with my team build.

Solution:

Override CoreTestConfiguration target and adjust the parameters for the call the TestToolsTask.

Code:

<Target Name="CoreTestConfiguration"
        DependsOnTargets="$(CoreTestConfigurationDependsOn)"
                Outputs="@(TestOutputs)">

<Warning Condition=" '$(V8TestToolsTask)'=='true' and '$(TestNames)'!='' "
         Text="Warning: The TestNames property cannot be used in in combination with the V8 TestToolsTask and will be ignored." />

<Warning Condition=" '$(V8TestToolsTask)'=='true' and '@(LocalTestContainer)' != '' "
         Text="Warning: The TestContainer item group cannot be used in combination with the V8 TestToolsTask and will be ignored." />

<PropertyGroup>
  <ContinueOnTestError Condition=" '$(StopOnTestFailure)' != 'true' ">true</ContinueOnTestError>
  <ContinueOnTestError Condition=" '$(StopOnTestFailure)' == 'true' ">false</ContinueOnTestError>
</PropertyGroup>

<!-- MetaDataFile tests for non-desktop builds. -->
<TestToolsTask
   Condition=" '$(IsDesktopBuild)'!='true' and '$(V8TestToolsTask)'!='true' and '%(LocalMetaDataFile.Identity)' != '' "
  SearchPathRoot="$(OutDir)"
  PathToResultsFilesRoot="$(TestResultsRoot)"
  MetaDataFile="%(LocalMetaDataFile.Identity)"
  RunConfigFile="$(RunConfigFile)"
  TestLists="%(LocalMetaDataFile.TestList)"
  TestNames="$(TestNames)"
    ContinueOnError="$(ContinueOnTestError)" />

<!-- 8.0 MetaDataFile tests for non-desktop builds. -->
<TestToolsTask
   Condition=" '$(IsDesktopBuild)'!='true' and '$(V8TestToolsTask)'=='true' and '%(LocalMetaDataFile.Identity)' != '' "
   SearchPathRoot="$(OutDir)"
   PathToResultsFilesRoot="$(TestResultsRoot)"
   MetaDataFile="%(LocalMetaDataFile.Identity)"
   RunConfigFile="$(RunConfigFile)"
   TestLists="%(LocalMetaDataFile.TestList)"
   ContinueOnError="$(ContinueOnTestError)" />

<!-- TestContainer tests for non-desktop builds. -->
<TestToolsTask
   Condition=" '$(IsDesktopBuild)'!='true' and '$(V8TestToolsTask)'!='true' and '@(LocalTestContainer)' != '' "
   SearchPathRoot="$(OutDir)"
   PathToResultsFilesRoot="$(TestResultsRoot)"
   RunConfigFile="$(RunConfigFile)"
   TestContainers="@(LocalTestContainer)"
   TestNames="$(TestNames)"
   ContinueOnError="$(ContinueOnTestError)" />

 <!-- MetaDataFile tests for desktop builds. -->
<TestToolsTask
   Condition=" '$(IsDesktopBuild)'=='true' and '$(V8TestToolsTask)'!='true' and '%(MetaDataFile.Identity)' != '' "
   SearchPathRoot="$(OutDir)"
   PathToResultsFilesRoot="$(TestResultsRoot)"
   MetaDataFile="%(MetaDataFile.Identity)"
   RunConfigFile="$(RunConfigFile)"
   TestLists="%(MetaDataFile.TestList)"
   TestNames="$(TestNames)"
   ContinueOnError="$(ContinueOnTestError)" />

<!-- 8.0 MetaDataFile tests for desktop builds. -->
<TestToolsTask
   Condition=" '$(IsDesktopBuild)'=='true' and '$(V8TestToolsTask)'=='true' and '%(MetaDataFile.Identity)' != '' "
   SearchPathRoot="$(OutDir)"
   PathToResultsFilesRoot="$(TestResultsRoot)"
   MetaDataFile="%(MetaDataFile.Identity)"
   RunConfigFile="$(RunConfigFile)"
   TestLists="%(MetaDataFile.TestList)"
   ContinueOnError="$(ContinueOnTestError)" />

  <!-- TestContainer tests for desktop builds. -->
<TestToolsTask
   Condition=" '$(IsDesktopBuild)'=='true' and '$(V8TestToolsTask)'!='true' and '@(LocalTestContainer)' != '' "
   SearchPathRoot="$(OutDir)"
   PathToResultsFilesRoot="$(TestResultsRoot)"
   RunConfigFile="$(RunConfigFile)"
   TestContainers="@(LocalTestContainer)"
   TestNames="$(TestNames)"
   ContinueOnError="$(ContinueOnTestError)" />

<!-- Populate TestOutputs with MetaData Files and Test Containers. -->
<ItemGroup>
   <!-- TestContainer tests for non-desktop and desktop builds. -->
   <TestOutputs Condition=" '@(LocalTestContainer)' != '' "
                Include="%(LocalTestContainer.Identity)">
      <Platform>$(Platform)</Platform>
      <Configuration>$(Configuration)</Configuration>
   </TestOutputs>

   <!-- MetaDataFile tests for non-desktop builds. -->
   <TestOutputs Condition=" '$(IsDesktopBuild)'!='true' and '%(LocalMetaDataFile.Identity)' != '' "
                Include="%(LocalMetaDataFile.Identity)">
       <Platform>$(Platform)</Platform>
       <Configuration>$(Configuration)</Configuration>
   </TestOutputs>

   <!-- MetaDataFile tests for desktop builds. -->
   <TestOutputs Condition=" '$(IsDesktopBuild)'=='true' and '%(MetaDataFile.Identity)' != '' "
                Include="%(MetaDataFile.Identity)">
       <Platform>$(Platform)</Platform>
       <Configuration>$(Configuration)</Configuration>
   </TestOutputs>
</ItemGroup>
</Target>

Explanation:

To override CoreTestConfiguration simply locate the Microsoft.TeamFoundation.Build.targets file and copy it from there into your TFSBuild.proj file.

To prevent build from publishing the test results simply remove the values for the following parameters from each instance of the TestToolsTask:

BuildFlavor="$(Configuration)"
Platform="$(Platform)"
PublishServer="$(TeamFoundationServerUrl)"
PublishBuild="$(BuildUri)"
TeamProject="$(TeamProject)"

If these parameters are left off the TestToolsTask the results will not be published.  The test are all executed and the build will fail if any of the test fail.  Note that you will not see the number of passed and failed test in the build report. It appears that the results must be published for that to happen. The results of the test can be reviewed in two ways.  First you can simply open the buildlog.txt file where the list of test and their results are listed.  If you need more detailed information you can locate the trx file in the TestResults folder on the build machine.

Problem:

Team build cannot create Scrum for Team System bug on build failure.

warning : TF42097: A work item could not be created due to a field error. The following fields have incorrect values: Field: 'Reason' Value: 'Build Failure'

Solution:

Add ‘Build Failure’ as a reason to the Bug Work Item type.

Code:

<TRANSITION from="" to="Not Done">
    <REASONS>
        <DEFAULTREASON value="New" />
        <REASON value="Build Failure" />
    </REASONS>
</TRANSITION>

Explanation:

The problem is that when the build script attempts to create a bug for a build failure it is trying to set the reason for the bug to ‘Build Failure’.  ‘Build Failure’ is not a valid reason in the Scrum for Team System bug work item.  To add ‘Build Failure’ as a valid reason start the Visual Studio 2008 Command Prompt and use witExport to export the bug work item type.  The witExport command you need to use is below:

witexport /f bug.xml /t http://tfsServer:8080 /p teamProject /n bug

Replace tfsServer with your TFS server name and replace teamProject with the name of the team project you want to export the work item from.  This command will result in a file bug.xml to be saved on your hard drive with the bug work item type definition.  Open the file with your favorite xml editor.

Change

<TRANSITION from="" to="Not Done">
    <REASONS>
        <DEFAULTREASON value="New" />
    </REASONS>
</TRANSITION>

To

<TRANSITION from="" to="Not Done">
    <REASONS>
        <DEFAULTREASON value="New" />
        <REASON value="Build Failure" />
    </REASONS>
</TRANSITION>

Save the file and use witImport to import the modified bug work item type into your team foundation server. The witImport command you need to use is below:

witimport /f bug.xml /t http://tfsServer:8080 /p teamProject

Replace tfsServer with your TFS server name and replace teamProject with the name of the team project you want to import the work item into.  This command will result in the new work item type definition being stored for that team project.

Now when your build fails the work item will be created.

Another alternative is to simply update the value passed as the reason of the bug from ‘Build Failure’ to ‘New’.  To do that you simply change the following line in your tfsbuild.proj file.

Change

<WorkItemFieldValues>System.Reason=Build Failure;System.Description=Start the build using Team Build</WorkItemFieldValues>

To

<WorkItemFieldValues>System.Reason=New;System.Description=Start the build using Team Build</WorkItemFieldValues>

I guess one final option is to simply skip creating the work item all together.  Add this to the first Property Group:

<SkipWorkItemCreation>true</SkipWorkItemCreation>

You can also change the template used to create your tfsbuild.proj file by changing C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\PrivateAssemblies\1033\TFSBuild.proj so you don’t have to change every tfsbuild.proj file after build creation.