18 min read

(For more resources related to this topic, see here.)

VSTest.Console utility

In Visual Studio 2012, the VSTest.Console command line utility is used for running the automated unit test and coded UI test. VSTest.Console is an optimized replacement for MSTest in Visual Studio 2012.

There are multiple options for the command line utility that can used in any order with multiple combinations. Running the command VSTest.Console /? at the command prompt shows the summary of available options and the usage message. These options are shown in the following screenshot:

Running tests using VSTest.Console

Running the test from the command prompt requires the expected parameters to be passed based on the options used along with the command. Some of the options available with VSTest.Console command are explained in the next few sections:

The /Tests option

This command is used to select particular tests from the list of tests in the test file. Specify the test names as parameters to the command, and separate the tests using commas when multiple tests are to be run. The next screenshot shows a couple of test methods that run from the test file:

The output shows the Test Run result for each of the tests along with the messages, if any. The summary of the tests is also shown at the end of the results sections with the time taken for the test execution.

The /ListTests option

This command is used to list all available tests within the test file. The following screenshot lists the tests from one of the Test Project file:

The next one is another command line utility, MSTest, which is used to run any automated tests.

MSTest utility

To access the MSTest tool, add the Visual Studio install directory to the path or open the Visual Studio Group from the Start menu, and then open the Tools section to access the Visual Studio command prompt. Use the command MSTest from the command prompt.

The MSTest command expects the name of the test as parameter to run the test. Just type MSTest /help or MSTest /? at the Visual Studio command prompt to get help and find out more about options.

The following table lists the different parameters that can be used with MSTest and the description of each parameter and its usage:

Option

Description

/help

This option displays the usage message for all parameters type /? or /h.

/nologo

This option disables the display of startup banner and the copyright message.

/testcontainer:[file name]

This option loads a file that contains tests; multiple test files can be specified to load multiple tests from the files, for example:

/tescontainer:mytestproject.dll/testcontainer:loadtest1.loadtest

/maxpriority:[priority]

/minpriority:[priority]

This option execute the tests with priority less than or equal to the value: /minpriority:0 /maxpriority:2.

/category

This filter is used to select tests and run, based on the category of each test. We can use logical operators (& and !) to construct the filters, or we can use the logical operators (| and &!) to filter the tests.

 

/category:Priority1 – any tests with category as priority1.

 

/category: “Priority1&MyTests”– any tests with multiple categories as priority1 and Mytests.

 

/category: “Priority1|Mytests” – Multiple tests with category as either Priority1 or MyTests.

 

/category:”Priority1&!MyTests” – Priority1 tests that do not have category MyTests

/testmetadata:[file name]

This option loads a metadata file. For example, /testmetadata:testproject1.vsmdi.

/testsettings:[file name]

This option uses the specified test settings file. For example, /testsettings:mysettings.testsettings.

/resultsfile:[file name]

This option saves the Test Run results to the specified file. for example, /resultsfile:c:tempmyresults.trx.

/testlist:[test list path]

The test list to run as specified in the metadata file; you can specify this option multiple times to run more than one test list. For example, /testlist:checkintests/clientteam.

/test:[file name]

This is the name of a test to be run; you can specify this option multiple times to run more than one test.

/unique

This option runs a test only if one unique match is found for any given /test.

/noisolation

This option runs a test within the MSTest.exe process. This choice improves Test Run speed, but increases risk to the MsTest process.

/noresults

This option does not save the Test Results in a TRX file; the choice improves Test Run speed, but does not save the Test Run results

/detail:[property id]

This parameter is used for getting value of additional property along with the test outcome. For example, the following command with the property is to get the error message from the Test Result: /detail:errormessage

Running a test from the command line

MSTest is only for automated tests. Even if the command is applied to a manual test, the tool will remove the non-automated test from the Test Run.

The /testcontainer option

The /testcontainer option requires the filename as parameter which contains information about tests that must be run. The /testcontainer file is an assembly that contains all the tests under the project, and each of the projects under a solution has its own container for the tests within the projects.

For example, the next screenshot shows the list of tests within the container unittestproject1.dll. MSTest executes all the tests within the container and shows the result as well. The summary of the Test Result is as shown in the next screenshot:

First, the MSTest will load all the tests within the project, then start executing them one by one. The result of each Test Run is shown but the detailed Test Run information is stored in the test trace file. The trace file can be loaded in Visual Studio to get the details of the Test Result.

The /testmetadata option

The /testmetadata option is used for running tests in multiple Test Projects under a solution. This is based on the metadata file, which is an XML file that has the list of all the tests created under the solution.

The /testcontainer option is specific to a Test Project, whereas /testmetadata is for multiple test containers with the flexibility of choosing tests from each container.

The /test option

There are instances where running all the tests within a test container is not required. To specify only the required tests, use the /test option with the /testmetadata option or the /testcontainer option. For example, the following command runs only the CodedUITest1 test from the list of all tests:

The /test option can be used along with /testmetadata or /testcontainer, but not both. There are different usages for the /test option:

  • Any number of tests can be specified using the /test option multiple times against the /testmetadata or /testcontainer option.
  • The name used against the /test option is the search keyword of the fully qualified test names. For example, if there are test names with fully qualified names such as: 
    UnitTestProject1.UnitTest1.CalculateTotalPriceTest 
UnitTestProject1.UnitTest1.CalculateTotalPricewithTaxTest 
UnitTestProject1.UnitTest1.GetTotalPriceTest

And if the command contains the option /test:UnitTestProject1, then all of the preceding three tests will run as the name contains the UnitTestProject1 string in it. Even though we specify only the name to the /test option, the result will display the fully qualified name of the tests run in the results window.

The /unique option

The /unique option will make sure that only one test which matches the given name, is run. In the preceding examples, there are different tests with the string UnitTestProject1 in its fully qualified name. Running the following command executes all the preceding tests:

mstest /testcontainer:c:SatheeshSharedAppsEmployeeMaintenance
UnitTestProject1bindebugunittestproject1.dll /test:Unittestproject1

But if the /unique option is specified along with the preceding command, the MSTest utility will return the message saying that more than one test was found with the same name. It means that the test will be successful only if the test name is unique.

The following command will execute successfully as there is only one test with the name GetTotalItemPriceTest.

The /noisolation option

The /noisolation option runs the tests within the MStest.exe process. This choice improves the Test Run speed, but increases risk to the MSTest.exe process.

Usually, the tests are run in a separate process that is allocated with separate memory from the system. By launching the MSTest.exe process with the /noisolation option, we avoid having a separate process created for the test.

The /testsettings option

The /testsettings option is used to specify the Test Run to use a specific test settings file. If the settings file is not specified, MSTest uses the default settings file. The following example forces the test to use the TestSettings1 settings file:

The /resultsfile option

In all the command executions, the MSTest utility stores the Test Results to a trace file. By default, the trace file name is assigned by MSTest using the login user ID, the machine name, and the current date and time. This can be customized to store the Test Results in a custom trace file using the /resultsfile option. For example, the next screenshot shows the custom trace file named as customtestresults.trx:

The preceding screenshot shows the Test Results stored at the c:Satheesh location in the results file, customtestresult.trx.

The /noresults option

The /noresults option informs the MSTest application not to store the Test Results to the TRX file. This option increases the performance of the test execution.

The /nologo option

The /nologo option is to inform the MSTest tool not to display the copyright information that is usually shown at the beginning of the Test Run.

The /detail option

The /detail option is used for collecting the property values from each Test Run result. Each Test Result provides information about the test such as error messages, start time, end time, test name, description, test type, and many more. The /detail option is useful to get the property values after the Test Run. For example, the following screenshot shows the start and end time of the Test Run, and also the type of the Test Run:

The /detail option can be specified multiple times to get multiple property values after the Test Run.

Publishing Test Results

Publishing Test Results is valid only if Team Explorer is installed, and if Visual Studio is connected to the Team Foundation Server (TFS). This is to publish the test data and results to the TFS Team Project. Please refer to Microsoft Developer Network (MSDN) for more information on installing and configuring TFS and Team Explorer.

Test Results can be published using the command line utility and the various options along with the utility. The /publish option with MSTest will first run the test, and then set the flavor and platform for the test before publishing the data to the TFS. Some of these options are mandatory for publishing the Test Run details.

The following are the different publishing options for the command line MSTest tool:

The /publish option

The /publish option should be followed by the uniform resource identifier (URI) of the TFS, if the TFS is not registered in the client. If it is registered, just use the name of the server to which the Test Result has to be published, as shown in the following command:

/publish:[server name]

Refer to the following examples:

  • If the TFS Server is not registered in the client, then: 
    /publish:http://MyTFSServer()
  • If the TFS Server is registered with the client, then: 
    /publish:MyTFSServer

The /publishbuild option

The /publishbuild option is used for publishing the builds. The parameter value is the unique name that identifies the build from the list of scheduled builds.

The /flavor option

Publishing the Test Rresults to TFS requires /flavor as mandatory. Flavor is a string value that is used in combination with the platform name, and should match with the completed build that can be identified by the /publishbuild option. The MSTest command will run the test, and then set the flavor and platform properties, before publishing the Test Run results to the TFS:

/flavour:[flavour string value]

For example:

  • /flavor:Release
  • /flavor:Debug

The /platform option

This is a mandatory string value used in combination with the /flavor option which should match the build option.

/platform:[string value]

For example:

  • /platform:Mixed Platforms
  • /platform:NET
  • /platform:Win32

The /publishresultsfile option

MSTest stores all the Test Results in the default trace files with the extension .trx. Using the /publishresultsfile option, the Test Results file can be published to TFS using the output/trace option. The name of the file is the input to this option. If the value is not specified, MSTest will publish the current Test Run trace file to TFS.

/publishresultsfile:[file name string]

For example, to publish the current Test Run trace file, use the /publishresultsfile option.

To publish the Test Result, one can use a combination of different options we saw in previous sections, along with the option /publishresultsfile.

The Test Results from the results file are published to the build output of the solution. The steps involved in publishing are to create the test, create a build definition, build the solution, execute the test, and then publish the result to the build output.

Step 1 – create/use existing Test Project

The following screenshot contains the solution EmployeeMaintenance. The solution contains a Test Project WebAndLoadTestProject1 with a web test WebTest2. The following screenshot shows the Test Project named WebAndLoadTestProject1:

Step 2 – running the test

On running the web test, by default the Test Result is stored in the trace file <file name>.trx.

Step 3 – creating a build

The /build service in Team Foundation Server has to be configured with a controller and agents. Each build controller manages a set of build agents. Unfortunately, the steps and the details behind creating the build types will not be covered in this article as it would be too long to discuss it. The following screenshot shows the /build service configured with controller and agents:

To create the build definition using the Team Explorer, navigate to the Build definitions in Builds folder, under Team Project. Select new build definition, and then configure the options by choosing the projects in TFS and the local folder. In one of the steps, you can see the following screenshot for selecting the project and setting the configuration information for the build process:

There are different configuration sections such as Required, Basic, and, Advanced, from where the project can be selected to include as part of this build definition setting such as build file formats, Agents Settings, work item creation on build failure, and other configurations.

Step 4 – building the project

Now that the project is created, configurations and properties are set, and we are ready to run the test, we will build and publish the Test Results. Select the New build definition and start the build queue process. The build service takes care of building the solution by applying the build definition, and on completion the result section shows the build summary.

Step 5 – publishing the result

So far, the test is run and the result is saved in the trace file, and also we have built the project using the build definition. The Test Run results should be published to the build. There are multiple options used for publishing the Test Results using the MSTest command line tool. The following command in the next screenshot publishes the Test Result to the specified build:

The command line options used in the preceding screenshot shows the Test Result trace file, TFS Team Project, and build against which the Test Result should be published. The command line also has the platform and the flavor values matching the build configurations.

After publishing the Test Results, if you open the build file, the test information along with the build summary is shown in the build summary. The information also contains a link to the trace file.

TCM command line utility

TCM is the command line utility used for importing automated tests to the Test Plan, running the test from the Test Plan, and then viewing a of tests and IDs corresponding to them. This utility is very useful if the IDE is not available. The /help or /? command is used to get the syntax and parameters for the tool. Following are the syntax and parameters for the tcm.exe tool:

Importing tests to a Test Plan

There wasn’t any test case for the unit test, and running the test case was also from Visual Studio IDE. This section explains how to import the tests to a Test Plan and create the test cases automatically while importing through the command line.

The Test Plans are created using the Test Manager to group the Test Suites and test cases. The following screenshot shows a few Test Plans created for the Team Project TeamProject1:

The EmployeeMaintenance solution contains the unit Test Project UnitTestProject1 with a few methods out of which there are methods such as CalculateTotalPriceTest() and CalculateTotalPricewithTaxTest() with their category defined as TotalPrice. So far there are no test cases defined in any of the Test Plans in the Test Manger for these tests. Refer to the following screenshot:

For any tests created using Visual Studio, the TCM utility can be used to import it to the Test Plan in Test Manager as test cases. The following command imports all tests with the category defined as TotalPrice from the UnitTestProject1 assembly into the Team Project TeamProject1. The category is defined to the tests to group it from all other available tests within the assembly. Refer to the following screenshot:

The command execution result shows the summary of the import, along with the names of the tests matching the command parameters.

Connect to the TeamProject1 using Test Manager and open any of the Test Plans within the project. On the Contents tab under the Plan option in Testing Center, click on Add from the toolbar in Test Suite section on the right. This will open up a new window to search for any available test cases to add to the Test Suite. By default, the Test Plan is the Test Suite, if no other Test Suite is created for the plan. In the new window, just click on the Run option to perform the default search with default parameters. You may notice that the search result shows two test cases in the name of the test methods which were imported from the Test Project. The test cases are named after the test method itself. Select either or both of the test cases and add them to the Test Suite.

After adding the test case to the Test Suite and Test Plan, open the test case using the Open toolbar option. There won’t be any step except the name of the test case and few other details. Include the details of the test steps to the test case, if required.

Running tests in a Test Plan

The tests cases associated with the tests can be run using the TCM command line utility without using the IDE. Whenever a test is run using the TCM, it requires additional information such as the environment and roles within the environment.

Running the test case using TCM requires Test Points or the Test Suite, and the configuration information. TCM requires the IDs of the Test Plan, Test Suite, and configuration. The TCM command line can be used to retrieve all these details.

To list all configurations from the Team Project, the TCM command is like the following result:

The following is the command and output for listing all the Test Plans within the Team Project:

To list all the Test Suites within the Plan, use the following TCM command with the options as shown in the next screenshot along with the Plan ID, collection, and the Team Project name. Use the Plan ID from the previous command output:

Use the Config ID, Plan ID, and the Suite ID collected by using the TCM utility from the collection and the Team Project to run the test. This will create a run as shown in the following screenshot:

The Test Run is created and the result can be viewed in Test Manager for analysis. Open the Test Manager and select the option Test under Testing Center. Select Analyze Test Runs from the menu bar. The Analyze Test Runs window shows the Test Runs for the Test Plan. The following screenshot shows a detailed view of the Test Run. The test is still in progress but you can see the test cases and the other details provided at the command:

The Test Agent needs to be set up to run as a process instead of a service to run the automated tests to interact with desktop.

Summary

This article explained the use of multiple command line utilities such as VSTest. Console, MSTest, and TCM for running the tests. These tools are very handy when there is no IDE. Lots of features are covered using the command line utility when compared to the earlier versions of Visual Studio. The VSTest.Console utility comes with multiple options to run automated tests such as unit tests and Coded UI tests. The MSTest utility provides options for backward compatibility along with multiple options to run automated tests and publish the results to the Team Foundation Server. The TCM utility is used for importing tests and creating test cases automatically to Test Plans. This utility is very useful, and saves lot of manual activities with Test Manager. Overall these utilities provide lot of features at the command line, and remove the IDE dependency.

Resources for Article:

Further resources on this subject:

Packt

LEAVE A REPLY

Please enter your comment!
Please enter your name here