Using SnagIt with Microsoft Test Manager and the Visual Studio TFS Feedback Client



I’m absolutely a fan of SnagIt.  It has a low cost and works super well for what it does!  If you want to use SnagIt with Microsoft Test Manager and the new Feedback Client for Visual Studio “11” and Team Foundation Server “11” then you are in luck!  By default, they both use MSPaint for editing screenshots but if you have SnagIt, you definitely want to use something more powerful to help you provide even better screenshots.

Configuring the TFS Feedback Client to Use SnagIt

This one is easy.  You’ll want to open up the client (even if you aren’t currently giving feedback) and then click the little gear icon at the top.

Configuring Screenshot Annotation Options for the TFS Feedback Client

Next, you’ll want to provide the path to the SnagIt Editor and then provide the command-line arguments which is just the filename placeholder for the screenshot.  If you don’t enable the checkbox for automatically opening the screenshot each time, don’t worry because you can always double-click any screenshot in the feedback client to open the SnagIt Editor.

Setting the Screenshot Annotation Tool for TFS Feedback Client

Configuring Microsoft Test Manager to Use SnagIt

Whether you want to walk through the manual execution of a test case or use the new Exploratory Testing features of Microsoft Test Manager, you can use SnagIt to help you annotate screenshots here as well.  It’s a very similar and easy process as well.  In this case, the gear icon is available at the bottom-right corner of the test runner window.

Configuring Screenshot Annotation Options for Microsoft Test Manager

You’ll then see a familiar dialog window to set the screenshot annotation options to SnagIt here as well.

Setting the Screenshot Annotation Tool for Microsoft Test Manager

Why use SnagIt with Screenshots?

The value propositions to me for being able to use SnagIt are the following.

  • Annotations – I can add arrows, highlights, text bubbles, etc. really easily.  Those can make a big difference for someone looking at screenshots.
  • Blurring Personally Identifying or Sensitive Information – This is a key one for me at times.  It is really easy to select an area of the screenshot and apply a blur effect

Ed Blankenship

Posted in TFS | VSTS | VSTS Testing


Difference Between Microsoft Test Manager and Visual Studio Test Professional



Microsoft Visual Studio Test Professional 2010 BoxWhen talking with our consulting customers and leading our 4-Hour Testing Workshops across North America, there is some common confusion about what exactly the differences are between Microsoft Test Manager and Visual Studio Test Professional.  It’s a pretty simple difference that I’ll see if I can help clarify and share some additional news for the vNext release of the Visual Studio family of products at the same time.  Once you get the difference, it will make perfect sense!


Visual Studio Test Professional

Visual Studio Test Professional is one of the different SKUs that you can purchase from the Visual Studio family of products.  What this means is that this is the “box” that you buy.  You can’t buy Microsoft Test Manager.  You buy an edition of Visual Studio that includes Microsoft Test Manager.  Visual Studio Test Professional happens to be the edition that many people will get when they are looking only to get Microsoft Test Manager (in addition to a TFS CAL and MSDN Subscription).


Microsoft Test Manager

Microsoft Test Manager IconMicrosoft Test Manager is the application you use to interact with Team Foundation Server to:

  • Create and manage test plans
  • Create, author, and maintain manual and automated test cases
  • Manage association of automation for test case artifacts
  • Run exploratory tests
  • Analyze previous manual, automated, or exploratory test runs
  • File rich actionable bugs while executing manual test runs or from automated test runs
  • Create and provision individual or shared lab environments with multiple machines for development or testing with Lab Management
  • Connect to physical or virtual lab environments for development or testing with Lab Management
  • Facilitate structured User Acceptance Testing (UAT) from business users
  • And lots more…

Microsoft Test Manager actually is included with two of the different SKUs or Visual Studio editions in the Visual Studio 2010 release.  You can acquire either Visual Studio 2010 Ultimate or Visual Studio 2010 Test Professional to receive Microsoft Test Manager.

Visual Studio “11” Release Changes – So Far

(Written for Beta) Microsoft has announced that Microsoft Test Manager will now be included in the Visual Studio Premium edition in addition to Visual Studio Ultimate and Visual Studio Test Professional.  That’s a great change and brings a lot of value to the Visual Studio Premium SKU!  You can find out some additional comparisons (that have been publicly announced so far) here on the Microsoft site:  http://www.microsoft.com/visualstudio/11/en-us/products/compare

Visual Studio "11" Beta Edition Comparison - Microsoft Test Manager

Need Help?

Let me know if you have any additional questions!  We help customers all the time figure out what types of licenses they should get, advocate to get the best pricing for their company, and sort through all of the licensing details to come up with the best scenario.  Contact me and I’ll be happy to get you started down the right path.  BTW – if there is also one thing to remember, don’t pay retail prices for any Visual Studio licenses before you come talk to us!

Ed Blankenship



Third-Party Support for Visual Studio Coded UI Testing



Bookmark this page! I’ll be updating this blog post article as I am informed of support for Coded UI testing in each of the products. Let me know if I’m missing any that should be in the list!

If you have been using Visual Studio 2010 and Microsoft Test Manager for automated UI testing, you may have experienced some issues with building the Coded UI tests if your application is using third-party components. It has been quite frustrating having been to several customer sites over the past year and running into this issue. My suggestion for anyone that has run into these issues is to open a support case with the component vendor and request that they add the necessary accessibility support to their component libraries. This will allow each of them to track the interest for this type of support in their products and then also provide a way to inform you whenever it is supported.

FYI – This is also the necessary implementation to get the “Fast-Forward” functionality from Microsoft Test Manager to work to record the actions of manual test cases.

There doesn’t seem to be a definitive list of the third-party controls that do or don’t support Coded UI testing at the moment. I have attempted to gather all of the information to provide a one-stop shop for this support. I’ll include links to updates, add-ons, etc. as I receive them.

Product Infragistics DevExpress Telerik ComponentOne Xceed
ASP.NET Controls Green Green Green Green N/A
Windows Forms Controls Green5 Green4 Green3 Red Red2
Windows Presentation Foundation (WPF)Controls Red Red Red Red Green1
Silverlight Controls Red Red Red Red Red2

Notes

  1. Added support Xceed WPF Controls: http://twitter.com/#!/Kosmatos/status/32527169318027264
  2. Added updates for other products for Xceed: http://twitter.com/#!/Kosmatos/status/32527901526065153
  3. Added update for Telerik Windows Forms controls. Now available in each release of Windows Forms after 2011 Q1. http://www.telerik.com/products/winforms/whats-new-full-story.aspx
  4. Added update for DevExpress Windows Forms controls. Introducing support in the DevExpress 12.1 release. Check out more information DevExpress here: http://devexpress.com/products/net/controls/winforms/coded-ui/. Additionally, Brian Harry has some details in his announcement as well: http://blogs.msdn.com/b/bharry/archive/2012/06/22/devexpress-announces-full-coded-ui-support.aspx
  5. Jason Beres has announced that support for Coded UI testing will be included with the Infragistics Windows Forms 12.2 release scheduled for early October 2012.  This first release will support the most popular subset of the Infragistics Windows Forms controls with more to follow in the future.  More information:  http://www.infragistics.com/community/blogs/jason_beres/archive/2012/08/31/announcing-coded-ui-support-for-windows-forms.aspx

Blog Post Updates

  • 2/1/2011: Updates for Xceed.
  • 2/1/2011: Added some links for implementing the appropriate accessibility hooks. Thanks to Jeff Levinson!
  • 2/16/2011: Added additional information about the Accessibility Checker. Thanks to Vinod Malhotra!
  • 3/24/2011: Update for Telerik Windows Forms controls.
  • 7/11/2012: Upgrade for DevExpress Windows Forms controls support.
  • 7/11/2012: Added additional details about the Coded UI Support Blog Post by Shubhra.
  • 10/1/2012:  Upgrade for Infragistics Windows Forms controls support.

Accessibility Support for Custom Controls

If you are building custom controls that are for internal use in your organization, you’ll want to make sure you implement the appropriate accessibility hooks to get full support from the Visual Studio Coded UI testing platform. Here are some links for you to get started:

Additionally, there is a tool called the UI Accessiblity Checker (also known as AccChecker) that is available to help you check your controls to make sure that the necessary accessibility implementation is correct. The tool can check for both MSAA and UIA implementations. It even has a plug-in architecture so that third parties can add additional verifications to the tool. All the source code and the binaries are available on the CodePlex site.

Additional Information

Shubhra Maji from the Visual Studio Test & Lab Management product team at Microsoft put together a great “basics” blog post about the different levels of Coded UI support for custom controls: http://blogs.msdn.com/b/visualstudioalm/archive/2011/10/28/coded-ui-test-extension-for-3rd-party-controls-the-basics-explained.aspx. The goal for all third-party custom controls should be to get to Level 4 which is the “Intent-Aware” stage.

clip_image002

Ed Blankenship

Posted in TFS | VSTS | VSTS Testing


Locating Controls Using UI Map Editor for Coded UI Tests



One of the great new features of Visual Studio 2010  and Microsoft Test Manager is the ability to record action recordings and then covert them into automated UI tests called “Coded UI Tests.”  The recorded steps turn into a UI Map that is an XML-based file format that is used by Visual Studio to generate source code.  Editing the UIMap was pretty tedious before so Microsoft released a new UI Map Editor in the latest feature packVisual Studio 2010 Feature Pack 2.  Feature Pack 2 is available to active MSDN Subscribers and contains the following additional features;

Cumulative feature pack that extends testing, code visualization and modeling capabilities in Visual Studio 2010.

Testing features:

  • Use Microsoft Test Manager to capture and playback action recordings for Silverlight 4 applications.
  • Create coded UI tests for Silverlight 4 applications with Visual Studio 2010 Premium or Visual Studio 2010 Ultimate.
  • Edit coded UI tests using a graphical editor with Visual Studio 2010 Premium or Visual Studio 2010 Ultimate.
  • Use action recordings to fast forward through manual tests that need to support Mozilla Firefox 3.5 and 3.6.
  • Run coded UI tests for web applications using Mozilla Firefox 3.5 and 3.6 with Microsoft Visual Studio 2010 Premium or Visual Studio 2010 Ultimate.

Code visualization and modeling features (requires Visual Studio 2010 Ultimate):

  • Use the Generate Code command to generate skeleton code from elements on UML class diagrams. You can use the default transformations, or you can write custom transformations to translate UML types into code.
  • Create UML class diagrams from existing code.
  • Explore the organization and relationships in C, C++, and ASP.NET projects by generating dependency graphs.
  • Import elements from UML sequence diagrams, class diagrams, and use case diagrams as XMI 2.1 files that are exported from other modeling tools.
  • Create links and view links from work items to model elements.
  • Create layer diagrams from C or C++ code and validate dependencies.
  • Write code to modify layer diagrams and to validate code against layer diagrams.

Once you get the Feature Pack installed, you will be able to open the UI Map and in a new editor as shown below.  It’s great because it allows you to see each of the methods that have been created and allows you to rename, remove, and edit the actions that have been recorded.  It even shows you all of the windows, controls, etc. that are included in the UI Map that describe the UI components in your application that are used by the Coded UI Tests.  You can also edit the properties for those so that you can maintain the automated tests to continue to work after UI changes are made in the application you are testing.

FeaturePack2-UIMapEditor

Some really cool features that I found today was the ability to locate a control on your application and also to verify that the UI Map is still correct by locating all of the controls.  First, you can find a specific UI control or element by choosing the “Locate the UI Control” toolbar button after you select the control you are interested.

FeaturePack2-LocateControl

You’ll want to have the particular application and window running before you do this (so it can find it) but it will put a little blue rectangle around the control as shown below.

FeaturePack2-LocateControlOnForm

Nice!  If you want to make sure that all of the UI control or elements on a window are still valid in the UI Map then you can select the window in the UI control’s tree and then choose the “Locate All” toolbar command.  You’ll see it go through each of the controls in the window and verify that it can still find it and mark it with a green check mark if it can be found.  How about that?

 

FeaturePack2-LocateControlsDialog

FeaturePack2-LocateControlsResults

 

Have fun!

Ed Blankenship

Posted in TFS | VSTS | VSTS Testing


Book Review for Wrox Professional Application Lifecycle Management with Visual Studio 2010



During the first week of April, a little package was sitting on my front porch with the first book to be released on the Visual Studio 2010 release that deals with the new Application Lifecycle Management (ALM) features.  For those of you who don’t know, this essentially means the former “Team System” line of products as we were exposed to it in the 2005 and 2008 releases.  Although the entire Visual Studio suite of products is considered something that helps you with ALM, the book primarily focused on Visual Studio 2010 Ultimate, Visual Studio 2010 Premium, Visual Studio 2010 Test Professional, Visual Studio 2010 Lab Management, and Team Foundation Server 2010.  During the Introduction, I even appreciated how the authors discussed about “where Team System went.”  It’s the best explanation of the branding change that I’ve seen to date.

I was extremely excited to start immediately reading the book.  Even though I have been closely involved with the 2010 release as a Microsoft MVP, when I started to read this book my goal was to be exposed deeper in the feature set being introduced in the 2010 release.

At the time of writing this blog post, the book was selling for $34.64 at Amazon.  The suggested retail price is $54.99.  It is currently #7 in the Software Development books category!

Strengths

If you are new to the ALM features in Visual Studio, I felt this book really offered you the ability to get the high-level overview of all of those features.  It’s essentially similar to a “survey” course that you would have taken in college.   It’s 696 pages that ends up going through all of the Visual Studio client and server features at just the right level of detail. There were even some areas that I felt that I learned more about and hadn’t been exposed to heavily in the past.

The architecture features were something that I had hoped to learn the most from.  They have just never been something that I dived into great detail during the 2010 release cycle.   All of the new UML diagrams that are available including the new architecture features like Use Case, Activity, Sequence, Component, Class, Dependency, and Layer Diagrams.  There was a also a great introduction to the Architecture Explorer.

The testing features have really been what has made up a majority of the Visual Studio 2010 release and the book definitely reflects that.  Going through the testing features, I really felt like I understood the end to end story.  It felt very rounded out!  These chapters are where I picked up a majority of the nuggets of information.  I can’t tell you how many times I said “wow, I didn’t know you could do that.”  I also feel like this is a great place to pick up some introductory knowledge about how Visual Studio Team Lab Management fits into the ALM story.  I also kept thinking how great this book would be for the testers on your team that are new to the Microsoft testing platform and Team Foundation Server.

There are so many changes to TFS, I can’t even begin to start describing them.  Thankfully, the book did a great job.  Especially with the revamp of Team Build to use Windows Workflow Foundation.  You can even download the Team Build chapter from the book for free here:  Team Foundation Build.  Other than automated builds, you’ll get a good pass by all of the rest of the new TFS 2010 features and architecture/topology changes.

There was a whole chapter dedicated to debugging with IntelliTrace!  That’s awesome.  I’m very much a fan of IntelliTrace and think that will truly change the way you develop.

Criticisms

I have been hoping to have a book available out there that really only discusses TFS.  The book definitely has a few chapters available on TFS and spends a good amount of time but that discussion is not the nitty gritty that I think some readers out there are really looking for.  With that said, I don’t think this book was positioned for the “TFS Administrator” exclusively.  Again, I really think this is a survey-level review of the entire ALM stack of features for Visual Studio.  That doesn’t allow you to go into the depths of any particular product.  There currently isn’t a book available for TFS 2010 with the level of detail that I am sure some readers out there are hoping for.  We’ll see what happens in the months to come…

My next criticism isn’t so much for the content of the book as what is media choices are available.  I own a Kindle DX and I imagine a few other techies in the world have some type of eBook reader as well.  I was hoping to have a CD that contained a DRM-free PDF that I could copy over to my Kindle DX whenever I’m traveling and need a quick resource for reference.  Wrox certainly does allow you to get a PDF of books but you have to order them separately even if you had purchased the hard copy.

Finally, the only other thing that I noticed was in that chapter about IntelliTrace (see above) there wasn’t a mention of Symbol & Source Server.  I couldn’t believe it.  There is definitely a discussion later in the book about Team Build’s integration with Symbol & Source server but I was hoping to have seen some more detail in the IntelliTrace chapter about the importance of having them setup for your organization.  You’ll want to put two and two together.

 

Now that I’m finished scrounging from the bottom of the barrel to find some criticisms… :)

My Recommendation

Hands down, get this book.  I think it’s well worth it.   I know each of the authors and it really looks like they put a tremendous amount of effort into writing the book.  The topics are really presented well and at the right level of detail for someone really wanting a crash course in all of the Visual Studio ALM features.  I can’t even tell you how many new nuggets of information that I ran across of things that I didn’t even realize were in the product.

It certainly gets my stamp of approval! :)  Kudos to the authors.

 

Very respectfully,

Ed Blankenship

Microsoft MVP of the Year, Visual Studio ALM and Team Foundation Server



Can I Use Microsoft Test Manager for User Acceptance Tests?



Microsoft Visual Studio Test Professional 2010 Box

That answer is a resoundingly yes, in my opinion, and I believe you would find some real value in having your UAT testers using Microsoft Test Manager to perform those UAT tests.  However, I think the real question that should be asked is do you have to purchase a license for Microsoft Visual Studio 2010 Test Professional to use Microsoft Test Manager to perform those User Acceptance Tests (UAT?)

 

Background

What are you talking about, Ed? 

There is a licensing exclusion that exists that really helps out when business users connect to development/test environments to perform user acceptance testing.  This exclusion really kicks in and helps when those development/test servers have used Operating Systems, SQL  Server, etc. licenses that conform to the MSDN EULA.  The exclusion basically says those non-technical business users don’t require an MSDN subscription to connect to those development/test servers if they are only performing user acceptance tests.  Normally, each person who connects to a development/test environment that has MSDN software installed on it (like the OS) requires that each of them has an active MSDN subscription.

Here’s the full description directly from the Visual Studio 2010 Licensing Whitepaper:

User Acceptance Testing

At the end of a software development project, end users (or team members acting as proxies for end users) typically review an application and determine whether it meets the necessary criteria for release—a process sometimes called user acceptance testing or UAT. MSDN software may be accessed by end users who do not have MSDN subscriptions for purposes of acceptance testing, provided that the use of the software otherwise complies with all MSDN licensing terms.

Under MSDN subscription licenses, user acceptance testing must not use live production data. If a copy of any live production data is used, then that copy of the data must be discarded after the testing is complete and cannot be incorporated back into the live production data.

So back to the real question…

 

Do you have to purchase a license to use Microsoft Test Manager to perform UAT?

Well that’s the question that a client brought up.  Did they have to purchase a license of at least Visual Studio 2010 Test Professional for those business users if they wanted to use Microsoft Test Manager to perform the UAT tests?

Microsoft’s answer is:  Yes

The fact that you are using Microsoft Test Manager, in their opinion, is that you are doing more technical & formalized testing than what they would consider to be in the UAT licensing exclusion for MSDN software.

 

Ed Blankenship



When Will Microsoft Test Manager and Testing Tools Support Silverlight?



Microsoft Visual Studio Test Professional 2010 Boxsilverlight_logoA lot of people have been asking about whether the new Coded UI automated testing & Microsoft Test Manager test runner features would support Silverlight and I haven’t really had a good answer for them other than “not at 2010 RTM.”  There’s been a ton of reasons why that wasn’t the case but thankfully we received a little more information about timeline for that type of support.  We were also able to make this information available publicly so check out the raw info coming in below.  Notice my particular emphasis added around the release mechanism that was mentioned.

Whenever we talk about platform coverage for UI automation one of the frequent requests is support for Silverlight app testing. We have been hard at work trying to cater to this need and I wanted to update you all on where we are with this effort and provide a roadmap.

We are working on adding support for Silverlight controls for “Fast forward for manual testing” and “Coded UI Tests”. The focus is on line-of-business applications built with Silverlight 4 for both in-browser and on the desktop. The initial investigation is in progress and we are working with the Silverlight team to close on the overall design. We are planning to release a CTP version of a plug-in by Q2CY2010. This will be delivered out of band to active MSDN subscribers (Visual Studio Test Professional or Visual Studio Ultimate) customers only. […]

- Ram Cherala, Visual Studio Team Test Product Team

Make sure you have active MSDN subscriptions if you want to get out of band value from the product teams.  Anyone have any questions?

Ed Blankenship



Can I Collect an IntelliTrace Log in Production?



I’ve been hearing this question quite a bit…  “Can I collect an IntelliTrace log in Production?”  This would be a really good idea especially now that there is a standalone command-line utility, IntelliTrace.exe, that you can run to collect IntelliTrace log files.  Unfortunately, it looks like the Visual Studio 2010 Licensing White Paper answers that question for us on page 28:

The IntelliTrace DDA and/or IntelliTrace.exe cannot be used:

  • On a device or server in a production environment.
  • For purposes of system or application monitoring.
  • In non-interactive scenarios other than as part of an automated test or debugging-data collection session.

Bummer! :(  Honestly, I imagine that has to do with something around how IntelliTrace works and Microsoft doesn’t feel comfortable the impact it may have on running Production environments.  Just my conjecture though…

You’ll notice that you can use IntelliTrace in other instances though; most notably on development & test environments!

The IntelliTrace diagnostic data adapter (DDA) and/or IntelliTrace.exe can be used for test and debugging purposes:

  • As part of an interactive test or debugging session.
  • As part of an automated test or debugging-data collection session that is authored by a licensed user and triggered by the same or another licensed user.

You can even share IntelliTrace files between two companies as long as both companies are properly licensed!

IntelliTrace files may be shared among two or more companies as long as all users capturing and debugging IntelliTrace files are licensed with either Visual Studio 2010 Ultimate or Visual Studio Test Professional 2010, depending on the activities they are performing. For example, a company can share IntelliTrace files with an external development consultant. Similarly. a company can use an external company for testing purposes and debug IntelliTrace files provided by that vendor.

Here were the common scenarios mentioned in the licensing white paper.  See if you happen to fit into one of them:

Example 1: Finding a defect in a test environment Company A is building a Web application. All the developers are licensed for Visual Studio 2010 Ultimate with MSDN, and the testers are licensed with Visual Studio Test Professional 2010 with MSDN. During a test run a defect is discovered in the test environment that is difficult to reproduce in a development environment. The test machines have previously been configured with the Visual Studio Test Agent 2010, which includes the IntelliTrace DDA. The tester uses the Microsoft Test Manager to execute the test case with the IntelliTrace diagnostic data adapter (DDA) enabled. When the defect is encountered, the tester files a new bug, with the IntelliTrace files from each of the test machines is automatically attached to the bug. When a developer opens the bug using Visual Studio Ultimate, he or she can open the IntelliTrace files and step through the execution.

Example 2: Working with an external consultant In Example 1, Company A uses an external consultant to help with development. If the external consultant is licensed for Visual Studio Ultimate, he or she can open and debug the IntelliTrace files provided by Company A.

Example 3: Working with an external test vendor In Example 1, Company A uses Company B as an outsourced test vendor. The two companies can work together using IntelliTrace as long as all developers at Company A and all testers at company B are licensed appropriately.

I’m not sure what the minimal technical footprint is to get IntelliTrace.exe to collect an iTrace file just yet but my answer right now will be to have one of these installed:

  • Visual Studio 2010 Ultimate
  • Visual Studio 2010 Test Professional
  • Visual Studio 2010 Test Agents (additional software) <--- probably the smallest impact to a system

If I find out some more information about this scenario, then I’ll be putting together a future blog post!

 

Take care,

Ed Blankenship



Test Scribe: Test Plan Documentation for TFS



Surprisingly, I’ve heard from several people that they still want a hard-copy document form of artifacts that are getting stored in TFS like Test Plan documents or Requirements documents.  I can understand some situations like if you need to follow certain regulatory requirements as so forth but don’t really see the need beyond that why you would ever want a hard-copy :)  Help me understand more if you happen to be in that boat!

Anyhow, if you need a hard-copy test plan document then you are in luck!  Test Scribe has just been released which will take your test plan information, artifacts, and progress from TFS and generate a nice Word document.  Quite handy!  If only we can get the Requirements document power tool now then we’ll satisfy that other group of people!

I'd like to announce the beta availability of Team Test's first Power Tool release for Visual Studio 2010: Test Scribe.  This tool allows users of Visual Studio 2010 Ultimate Beta 2 to generate a Word 2007-compatible Test Plan Document from their plan, suites, test cases, and other artifacts.  Using the tool is a fairly straightforward process, including:

   1. Launch the Test Scribe tool.
   2. Enter your server/collection URL (e.g. http://myserver:8080/tfs/DefaultCollection)
   3. Select a Project.
   4. Select a Test Plan (previously created in MTM).
   5. Click the Generate button.

The resulting document will contain (among other things) a list suites with test cases and steps detail and pie charts detailing the overall progress of your Test Plan.  You can see a screenshot below showing several sections of a generated document.  Feedback is welcome and appreciated, and you can find the tool download at http://visualstudiogallery.msdn.microsoft.com/en-us/d18873c7-909d-4788-a56e-0c496a1d8bb9.

image

Many thanks and appreciation to everyone who helped get this tool out the door.

More information available here:  http://blogs.msdn.com/vstsqualitytools/archive/2010/01/11/test-scribe-test-plan-documentation-for-mtlm-plans.aspx

 

Ed Blankenship

Posted in TFS | Tools | VSTS | VSTS Testing


Tip: Data Collectors Behave Differently Based on Test Pass or Fail – IntelliTrace and Test Impact



I learned a very important tip today about Microsoft Test & Lab Manager.  The data collectors that you can specify in your test runs (which by the way are extremely helpful) don’t necessarily attach data to the test run or bug all of the time.  It depends on the actual success of the test case during the test run.  Two example were mentioned today:Visual Studio Data Collectors in Microsoft Test and Lab Manager

  • IntelliTrace – The IntellIiTrace data collector will only output a trace log if the test case fails.
  • Test Impact Analysis – The Test Impact data collector will only publish test impact data back to Team Foundation Server if the test case succeeds.

So this leads us to the fact that we won’t ever see both of them together at the same time.  If you see an IntelliTrace trace log file for a test case then you won’t be getting any test impact data published to the server (and vice versa.)  Be sure to have both of them enabled though if you would like to get that type of data when the test case ends.  This also helps me in figuring out why I haven’t been getting impacted tests in my build reports and the handy view of “recommend tests” based on choosing a build in MTLM.  I need to actually pass some tests to get that data published back up to the server!

 

How about that?  Very handy to know.  The Visual Studio Quality Tools team has a blog that has been very active lately.  If you would like to learn more about Microsoft Test & Lab Manager head on over to the site.

Ed Blankenship

Posted in TFS | VSTS | VSTS Testing


Test Case Migration Tool for TFS 2010 and Excel



Looks like there is a new migration tool being made available that helps you with importing test cases into TFS 2010 which now has Test Case Management as one of it’s new features.  It’s now available on CodePlex and works with the Beta 2 release.  Here’s some of the features available:

  • Imports Test case information (along with Test Steps) present in Excel into TCM/TFS server – Note: This is ONE way migration only
  • Provides a wizard based UI to run the tool one file at a time
  • Provides a command line support to run the tool in a batch mode
  • Allows you to save your selections/configurations into a mappings file to be reused later
  • Default mapping file that works across both Agile based and CMMI based projects are shipped along with the tool

http://tcmimport.codeplex.com/

I believe some systems allow you to export your test cases into Excel (HP Quality Center?) so this might be a good migration option.  You can also use the TFS 2010 SDK to build a custom migration utility that will import from another system without having to go through Excel at first.  Take a look at it and see if it might be helpful for you!

 

Ed Blankenship



Test Categories and Running a Subset of Tests in Team Foundation Server 2010



Disclaimer:  I’m writing this at a time when only Beta 1 is available for Visual Studio Team System 2010 so the information may have changed by the time it has been released.  I have included links to the relevant MSDN articles which should remain valid after release time so just double-check.

This small little additional feature is actually one that I have been looking forward to for a long time.  In Visual Studio Team System and Team Foundation Server 2010, you will now be able to limit your test runs to specific test categories with a new command-line option on MSTest.exe and therefore in Team Build 2010 which calls MSTest.exe automatically for you.

Back in the day… You would need to create test lists (.VSMDI files) in VSTS 2005 and VSTS 2008 to basically “categorize” your automated unit tests by putting them into different lists.  One handy thing about them is that the lists could be hierarchical which helps out at build time.  When you wanted to run a specific subset of tests either locally using MSTest.exe or in Team Build, you would just specify the .VSMDI file to use and then the test list you wanted to run.  Not too bad, but it’s a pain to keep up with those test lists.  Serious pain.  However, the thing that I hated absolutely most about them is that you could only edit the .VSDMI files if you purchased Visual Studio Team Suite or the Tester Edition.  So that means that if you have just the VSTS Developer Edition then you are pretty much out of luck.  For most places that I have seen, it’s usually the developers maintaining those test list files not the testers.

For this reason I actually prefer and will be recommending the Test Container and Category approach going forward in 2010.  Test Containers are essentially files that contain tests in them.  For example, unit tests (and other compiled tests) are stored in .dll files and ordered tests are in .orderedtest files.  I like this approach.  In automated builds I just want to specify which files contain the tests that I want to run and then if I want to limit the test run to just a subset I can just list which categories to run. 

A great example of this is what I call the “BVT” category.  These are the tests that you have identified to be your “smoke” tests that make sure a build is okay.  If these tests fail then you’ve probably got a bad build.  (BVT = Build Verification Tests) So I would limit the test runs on any CI or even the new Gated Check-In builds to just those BVT tests.  Quick & dirty verification is really all you need for those builds leaving a more extensive automated test pass to happen during the nightly or weekly build.  If you’re not familiar with the new Gated Check-In feature in TFS 2010, check out Brian’s blog post or Patrick’s blog post for more information.  It’s a killer feature.

This does rely on one thing though… each “developer” of an automated test needs to make sure they add the correct attribute(s) to their test methods.  You don’t have to keep up with maintaining the .VSDMI files any longer but you do have to make sure you mark each method appropriately.

You can even use test categories with the new types of automated tests available in 2010 like Coded UI Tests.  It doesn’t just have to be unit tests.

How to Specify a Category in an Automated Test

This part is pretty easy.  You just add as many TestCategory attributes to the test method as you need.  Here’s an example in C# using multiple test categories for a test method called DebitTest:

[TestCategory("Nightly"), TestCategory("Weekly"), TestCategory("Monthly"), TestMethod()]
public void DebitTest()
{
}

Alternately, you can select a test in the Test View tool window and then set the category by using the Properties tool window in Visual Studio and it will add the appropriate attributes to the methods for you.

image

How to Specify which Categories to Run in an Automated Build with Team Build 2010

Okay… this part is easy too. :)  Build definitions now have build properties that can be exposed to the end user in the Build Definition Details dialog or in the Queue Build dialog.  This is handy because you could by default not set a filter to run under normal circumstances (triggered or default manual builds) or you can change it when manually queuing a build if you want that build to run differently.  Either way it’s the same for setting the categories.  If you’re using the default build process workflow that is available out of the box, then just scroll down through the property list until you reach the Testing section which includes a build property called Test Category.  Leave it blank if you want to run all tests or specify the categories you’d like to limit it too:

image image

According to the MSDN documentation for the Test Category switch, you can combine multiple categories in different combinations instead of just specifying one category.  Very handy – here’s some examples:

  • /category:group1 runs tests in the test category "group1".

  • /category:"group1&group2" runs tests that are in both test categories "group1" and "group2." Tests that are only in one of the specified test categories will not be run.

  • /category:"group1|group2" runs tests that are in test category "group1" or "group2". Tests that are in both test categories will also be run.

  • /category:"group1&!group2" runs tests from the test category "group1" that are not in the test category "group2." A test that is in both test category "group1" and "group2" will not be run.

  • What I’m not sure about is whether you can specify test categories when using the old Upgrade Build Workflow template .xaml file… I’ll check on that and then update the blog post.

    It’s worth noting that if you are going to use the test category method to limit test runs, you must use test containers.

    Limiting Test Runs Based on Test Priorities

    If you noticed in the screenshot above from Team Build, you can also limit your test run to tests that are in a specific priority range.  How do you specify the range for your test methods?  You can use the Properties window when selecting a test in the Test View tool window or you can add the Priority attribute manually to the test method.  After that you just specify the range of priorities to use in the test run.

    [TestCategory("Nightly"), TestCategory("Weekly"), TestCategory("Monthly")]
    [TestMethod()]
    [Priority(1)]
    public void DebitTest()
    {
    }

     

    Additional Note:  It appears that the product team is actually encouraging people to move away from the old .VSDMI approach in favor of categories.  Check their note out:

    NoteNote

    Test categories are recommended for use over the test lists functionality from earlier versions of Microsoft Visual Studio Team System Test, unless you have to create a check-in policy which requires a test list. For more information about check-in policies, see How to: Add Check-In Policies.

     

    Take care and happy testing,

    Ed Blankenship