Files
spring-net/build-support/tools/NCover/NCoverFAQ.html
eeichinger 47762e3f0f upgraded to nunit 2.5.1 for running unit tests
integrated ncover for code coverage analysis
switched COM integration tests to explicit
2009-07-14 22:37:59 +00:00

430 lines
28 KiB
HTML

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
<!-- $Id: NCoverFAQ.html 247 2007-04-26 15:17:46Z peterwald $ -->
<title></title>
<meta content="Microsoft Visual Studio .NET 7.1" name="GENERATOR">
<meta content="http://schemas.microsoft.com/intellisense/ie5" name="vs_targetSchema">
<style> body { font-size: 10pt; font-family: Verdana; }
p.title { font-size: 20pt; font-weight: bold; }
.subtitle { color: maroon; }
p.question { font-weight: bold; }
pre { font-size: 10pt; font-family: Courier; }
pre.usage { background-color: #F0F0F0; }
.quote { background-color: #F0F0F0; margin-left: 36pt;}
.method { color: maroon; font-size: 10pt; font-weight: bold; }
.hdrcell { background-color: #DDEEFF; font-size: 10pt; }
.datacell { background-color: #FFFFEE; text-align: right; font-size: 10pt; }
.hldatacell { background-color: #FFCCCC; text-align: right; font-size: 10pt; }
.box { border: 1px solid; padding: 10px; }
</style>
</head>
<body>
<P class="title">NCover FAQ</P>
<P>If you have questions that this document does not address, contact <A href="mailto:peter@waldschmidt.com">
Peter Waldschmidt</A> or try the <A href="http://ncover.org/">NCover Forums</A>.</P>
<P class="question">1. What is code coverage analysis?</P>
<P class="answer">A code coverage analyzer monitors your code at runtime and
records information about which lines of code were executed. NCover shows each
sequence point in your application along with the number of times that point
was executed. Sequence points are generated by the compiler and stored in the
debug information (.pdb) files. A sequence point basically corresponds to a
single program statement (often a line of code) in your high-level language.</P>
<P class="question">2. Why would I want to do code coverage analysis?</P>
<P class="answer">Unit test suites are often used as a quality tool during the
development process to keep the codebase stable as it changes and expands.
Tools such as <A href="http://nunit.org/">NUnit</A> are often used to run and
report on the test suites. However, when implementing unit testing in your
build process, you have no way of knowing how much of your code the unit tests
are actually testing. This is where code coverage comes in. You can run NUnit
within NCover and use the code coverage report to determine which code was not
tested by that particular test suite.</P>
<P class="question">3. What versions of the CLR does NCover support?</P>
<P class="answer">
NCover 1.5.x requires the .NET framework version 2.0.50727 to be installed; however,
the application being profiled can be written against any shipping version of the
framework. NCover
has been tested profiling coverage of .NET 2.0, .NET 1.1 and .NET 1.0 applications.</P>
<P class="question">4. Which version of NCover should I install?</p>
<P class="answer">
If you have the .NET 2.0 framework installed on your machine then you should use
the latest NCover version available. NCover as of version 1.5 can profile .NET 2.0, 1.1 and 1.0 applications.</p>
<p class="answer">
For development teams who do not have the .NET framework 2.0 installed but do have
the .NET framework version 1.1.4322, you can
try NCover 1.3.3. Note however that this version is no longer supported as
it has a number of known issues and limitations.</p>
<P class="question">5. What is the command line syntax for NCover?</P>
<P class="answer">Here is the usage info from the NCover command line (for NCover versions from 1.5.6
only):</P>
<pre class="usage">NCover.Console [&lt;command line&gt; [&lt;command args&gt;]]
[//svc &lt;service name&gt;]
[//iis]
[//a &lt;assembly list&gt;]
[//w &lt;working directory&gt;]
[//ea &lt;exclusion list&gt;]
[//reg]
[//x &lt;xml output file&gt;]
[//s [&lt;settings file&gt;]] [//r [&lt;settings file&gt;]]
[//v] [//q]
[//l &lt;log file&gt;]
//svc For profiling windows services
//iis For profiling web applications
//a List of assemblies to profile separated by semi-colons i.e. "MyAssembly1;MyAssembly2". Do not include paths or suffixes.
//w Working directory for profiled application
//ea List of attributes marking classes or methods to exclude from coverage
//reg Register profiler temporarily for user. (helps with xcopy deployment)
//x Specify coverage output file. (default: coverage.xml).
//pm Specify name of process to profile (i.e. myapp.exe)
//s Save settings to a file (defaults: NCover.Settings)
//r Use settings file, overriding other settings (default: NCover.Settings)
//l Specify profiler log file (default: coverage.log).
//q No logging (quiet)
//v Enable verbose logging (show instrumented code)
</pre>
<UL>
<LI>&lt;command line&gt; - This argument specifies the command-line of the .NET application
you want to analyze.
Any command line arguments not starting with // will be passed
through to that application. NCover will profile the running application until it has exited. See below for examples.<li>//svc - This option is an alternative to the &lt;command line&gt;
for profiling windows services, which cannot be run directly as executables. NCover
will start the service (stopping it first if already running) and profile coverage
until the windows service is stopped.</li>
<li>//iis - This option is an alternative to the &lt;command line&gt; for profiling
web applications. NCover will start the IISAdmin and W3C
services (stopping first if currently running) and profile coverage until the IISAdmin
service is stopped.<br />
</li>
<li>//a - This command-line argument specifies the assemblies that you want to analyze.
NCover can only analyze assemblies that have .pdb files included with them. If
you do not specify the //a argument, NCover will attempt to analyze every loaded
assembly that has debug information available. Note that the assembly name arguments are
the module name within the assembly, not the physical file name. e.g. "MyAssembly"
rather than "MyAssembly.dll".<li>//w - If the application being profiled requires the
working directory to be set to something other than the current directory you are
executing the command line from then you can override it with this argument.</li>
<li>//ea - You can choose to exclude classes and methods
from coverage statistics by defining .NET attribute(s) and applying it to the affected
code. When using this argument you must specify the full type namespace of these
attribute(s) separated by semi-colons. See below for an example.<br />
</li>
<li>//reg - NCover requires a COM registration of the CoverLib.dll assembly containing
the profiler, which is performed automatically by the default .msi installation.
If you require an xcopy style deployment of NCover like many other .NET tools, then
you can use this argument which will temporarily register the profiler while performing
coverage. This feature was added in NCover 1.5.6.</li>
<li>//x - The output of NCover is an xml file (example below). Use this argument to
specify an alternate filename to "coverage.xml" in the current directory.<br />
</li>
<li>//pm - This setting tells NCover to ignore processes that don't have the specified process module name.
This is the name of the executable (i.e. myapp.exe). This setting is useful in cases, where your NCover
command spawns a series of child processes. Using this setting will help NCover determine which process to profile.
</li>
<li>//s - You may find it more convenient to use a settings file rather than specifying
a long list of command line arguments for running NCover. If you get the NCover
command line working as you would like it and then use the //s argument it will
save the required arguments as an xml file that can then be used by the //r argument
below.</li>
<li>//r - For use when you have used //s to construct an NCover settings file containing
your command line arguments. e.g. "ncover.console.exe //r NCover.Settings"<br />
</li>
<li>//l - The coverage log file can provide an insight if the desired coverage output
is not obtained. Useful information you may find to assist you includes which assemblies
were loaded by NCover, their file paths and which of those it found the .pdb build
symbols for. Use this argument to specify an alternative log file name or location
to coverage.log in the current directory.</li>
<li>//q - Suppresses writing the coverage.log file.</li>
<li>//v - This command-line argument makes the profiler emit all the original IL and
modified IL instructions to the coverage log. This is useful for debugging
purposes. Beware that this can make your coverage log file very large!
</li>
</UL>
<P class="question">6. Does NCover required a special compilation step for my code?</P>
<P class="answer">No. Some code coverage tools change your source code and force
you to recompile it into a special build.&nbsp; NCover is designed to&nbsp;work
on shipping code.&nbsp; NCover uses the .NET Framework profiling API to monitor
your code. It does require build symbols, but can be run on release code
without any modifications.</P>
<P class="question">7. How does NCover work?</P>
<P class="answer">NCover uses the .NET Framework profiler API to monitor an
application's execution. When a method is loaded by the CLR, NCover retrieves
the IL and replaces it with instrumented IL code.&nbsp; NCover does not change
your original IL code, it simply inserts new code to update&nbsp;a visit
counter at each sequence point.&nbsp; Upon
request, (usually after the .NET process has shut down) the profiler outputs statistics
to the coverage file.
</P>
<P class="question">
8. What is the output of NCover?</P>
<P class="answer">NCover generally writes out three files after analysis
completes.
<ul>
<li>
Coverage.log - This file is a log of the events and messages from the profiler
during the analysis process. Most of the time, error messages are recorded in
this log. If you enable verbose logging, the coverage log will contain
disassembly of the original and instrumented IL code.&nbsp; Verbose logging is not recommended for
normal use.<li>
Coverage.xml - This file is the analysis output of NCover. You can see an
example of the output below.
<LI>
Coverage.xsl - This file is a simple XML transformation that makes the XML
output easily readable.
</LI>
</ul>
<span class="subtitle">Example XML output</span>
<div class="box"><pre>&lt;method class="NCoverTest.ClassLoaded" name="HasDeadCode"&gt;
&lt;seqpnt document="C:\Dev\Utilities\ncover\NCoverTest\NCoverTest.cs"
column="13" line="48" endcolumn="58" endline="48" visitcount="1" /&gt;
&lt;seqpnt document="C:\Dev\Utilities\ncover\NCoverTest\NCoverTest.cs"
column="13" line="49" endcolumn="22" endline="49" visitcount="1" /&gt;
&lt;seqpnt document="C:\Dev\Utilities\ncover\NCoverTest\NCoverTest.cs"
column="17" line="50" endcolumn="24" endline="50" visitcount="1" /&gt;
&lt;seqpnt document="C:\Dev\Utilities\ncover\NCoverTest\NCoverTest.cs"
column="13" line="51" endcolumn="48" endline="51" visitcount="0" /&gt;
&lt;seqpnt document="C:\Dev\Utilities\ncover\NCoverTest\NCoverTest.cs"
column="9" line="52" endcolumn="10" endline="52" visitcount="0" /&gt;
&lt;/method&gt;</pre>
</div>
<p></p>
<span class="subtitle">Example transformed output</span>
<div class="box">
<DIV class="method">NCoverTest.ClassLoaded.HasDeadCode</DIV>
<TABLE id="Table1" borderColor="black" cellSpacing="0" cellPadding="3" border="1">
<TBODY>
<TR>
<TD class="hdrcell">Visit Count</TD>
<TD class="hdrcell">Line</TD>
<TD class="hdrcell">Column</TD>
<TD class="hdrcell">End Line</TD>
<TD class="hdrcell">End Column</TD>
<TD class="hdrcell">Document</TD>
</TR>
<TR>
<TD class="datacell">1</TD>
<TD class="datacell">48</TD>
<TD class="datacell">13</TD>
<TD class="datacell">48</TD>
<TD class="datacell">58</TD>
<TD class="datacell">C:\Dev\Utilities\ncover\NCoverTest\NCoverTest.cs</TD>
</TR>
<TR>
<TD class="datacell">1</TD>
<TD class="datacell">49</TD>
<TD class="datacell">13</TD>
<TD class="datacell">49</TD>
<TD class="datacell">22</TD>
<TD class="datacell">C:\Dev\Utilities\ncover\NCoverTest\NCoverTest.cs</TD>
</TR>
<TR>
<TD class="datacell">1</TD>
<TD class="datacell">50</TD>
<TD class="datacell">17</TD>
<TD class="datacell">50</TD>
<TD class="datacell">24</TD>
<TD class="datacell">C:\Dev\Utilities\ncover\NCoverTest\NCoverTest.cs</TD>
</TR>
<TR>
<TD class="hldatacell">0</TD>
<TD class="datacell">51</TD>
<TD class="datacell">13</TD>
<TD class="datacell">51</TD>
<TD class="datacell">48</TD>
<TD class="datacell">C:\Dev\Utilities\ncover\NCoverTest\NCoverTest.cs</TD>
</TR>
<TR>
<TD class="hldatacell">0</TD>
<TD class="datacell">52</TD>
<TD class="datacell">9</TD>
<TD class="datacell">52</TD>
<TD class="datacell">10</TD>
<TD class="datacell">C:\Dev\Utilities\ncover\NCoverTest\NCoverTest.cs</TD>
</TR>
</TBODY>
</TABLE>
</div>
<P>Suggested usages of the coverage.xml output are to display it in the <a href="http://ncoverexplorer.org/">
NCoverExplorer</a> gui with the source
code highlighted, to generate html reports, or to include it in your continuous build server reports such as CruiseControl.Net.
For more information on these options see below in the FAQ.</P>
<P></P>
<P class="question">
9. How do I use coverage exclusions?</P>
<p>
First you should define an attribute to markup your excluded code with. You will
likely want to put this in a common assembly to make it reusable, or indeed within
a "CommonAssemblyInfo.cs" that you include in all your application assemblies.</p>
<P></P>
<pre class="usage">namespace MyNamespace {
class CoverageExcludeAttribute : System.Attribute { }
}</pre>
<p>
Apply the attribute to the C# classes and/or methods you wish to mark as excluded
from code coverage statistics:</p>
<P></P>
<pre class="usage"> [CoverageExclude]
private void SomeMethodToExclude() {} </pre>
<p>
Finally, ensure you pass the full qualified attribute information in the NCover command line:</p>
<P></P>
<pre class="usage"> NCover.Console MyApplication.exe //ea MyNamespace.CoverageExcludeAttribute </pre>
<p>
Note that if you are using the <a href="http://testdriven.net/">TestDriven.Net</a>
VS.Net add-in to "Test with Coverage" it will automatically
pass through "//ea CoverageExcludeAttribute"
which you should define without a namespace like above. For further information refer to this
<a href="http://weblogs.asp.net/nunitaddin/archive/2006/10/04/CoverageExclude.aspx">
blog entry</a>.</p>
<P class="question">
10. Examples</P>
<p>
Coverage while running a simple executable until it exits:</p>
<P></P>
<pre class="usage"> NCover.Console MyApplication.exe</pre>
<p>
Coverage while running all the unit tests in an assembly using NUnit, profiling
all loaded assemblies with .pdb build symbols:</p>
<P></P>
<pre class="usage"> NCover.Console nunit-console.exe MyApplication.Tests.dll</pre>
<p>
Coverage of only a subset of loaded assemblies while running unit tests:</p>
<P></P>
<pre class="usage"> NCover.Console nunit-console.exe MyApplication.Tests.dll //a MyApplication.Core;MyApplication.Utilities</pre>
<p>
Coverage of a windows service. Stop the service to generate the coverage output:</p>
<P></P>
<pre class="usage"> NCover.Console //svc MyServiceName</pre>
<p>
Coverage of an ASP.Net application. Stop the IIS service to generate the coverage
output:</p>
<P></P>
<pre class="usage"> NCover.Console //iis</pre>
<P class="question">
11. Where can I get help or support?</P>
<P class="answer">
Your best approach is to browse the <a href="http://ncover.org/site/forums/default.aspx">
NCover forums</a> as well as the <a href="http://ncover.org/SITE/blogs/default.aspx">
blog</a> by the author Peter Waldschmidt. If you cannot find a similar issue
mentioned feel free to post your query and perhaps someone can help.</P>
<P class="question">
12. How do I "xcopy deploy" NCover like my other build tools?</P>
<P class="answer">
Many developers prefer to have their build tools such as NUnit, NAnt etc stored
in source control in a Tools folder along with the source code. This ensures that
a new developer can obtain and build the application without having to install additional
tools on their own machines.</P>
<p>
NCover can also be deployed in this fashion. However the one gotcha with NCover
versus other tools is that the profiler within CoverLib.dll must be COM registered
on the local machine before you execute it. Prior to NCover 1.5.6 this was usually
achieved as part of your build script, which would call regsvr32 with the path to
the CoverLib.dll in your Tools folder. Alternatively the &lt;ncover&gt; NAnt and
MSBuild tasks described below will do this for you. As of NCover 1.5.6 you can also
use the //reg option in the command line arguments which will temporarily register
the profiler. Note that the //reg option will not work for IIS or Windows Service
profiling unless you are running NCover under the same Windows login account as
the IIS worker process, or your Windows Service.</p>
<P class="question">
13. How do I see my source code highlighted with the coverage results?</P>
<P class="answer">
<a href="http://ncoverexplorer.org/">NCoverExplorer</a> is a gui and console-based
.NET application developed by <a href="http://www.kiwidude.com/blog/">Grant Drake</a>. NCoverExplorer
parses the coverage.xml files output from NCover and displays the results integrated
with your source code. It also includes a number of additional features to merge,
filter, sort and generate html reports. The console version is
designed to be used as part of an automated build process. The support forums for
NCoverExplorer are located with the NCover ones at <a href="http://ncover.org">http://ncover.org/</a>.<strong>&nbsp;</strong></P>
<P class="question">
14. How do I run NCover from within the Visual Studio.Net IDE?</P>
<P class="answer">
The <a href="http://testdriven.net/">TestDriven.Net</a> add-in by <a href="http://weblogs.asp.net/nunitaddin/">
Jamie Cansdale</a> offers a right-click capability within the IDE to execute
your unit tests with code coverage. The results of the NCover code coverage are
displayed with the bundled NCoverExplorer gui for analysis and reporting.</P>
<P class="question">
15. How do I run NCover from a NAnt or MSBuild task?</P>
<P class="answer">
You can use an &lt;exec&gt; task with <a href="http://nant.sourceforge.net/">NAnt</a>
or an &lt;Exec&gt; task with MSBuild. Alternatively you may want to use the custom
&lt;ncover&gt; task for NAnt or &lt;NCover&gt; task for MSBuild developed by Grant
Drake for a more developer friendly syntax. The source code, compiled assemblies
and documentation are located in the NCoverExplorer.Extras.zip available from <a
href="http://ncoverexplorer.org/">http://ncoverexplorer.org/</a>.</P>
<P class="question">
16. How do I include NCover output in my CruiseControl.Net build reports?</P>
<P class="answer">
<a href="http://ccnet.thoughtworks.com/">CruiseControl.Net</a> is a continuous integration
build server which offers web-based reporting of the outputs of a build such as
unit test results and code coverage reporting. The default CruiseControl.Net installation
includes a basic stylesheet which works in combination with the standard coverage.xml
formatted output. So all you need to do is include the execution of NCover as part
of your build, then add a CruiseControl.Net merge file publisher task to integrate
the coverage.xml results into the build output.</P>
<p class="answer">
An improvement on the above to display more attractive and powerful reports as well
as minimize the build log size is to use NCoverExplorer. The NCoverExplorer.Console.exe
is designed to produce a more concise xml report summary that is combined with an
alternate xsl stylesheet for CruiseControl.Net. You can find more information and
screenshots in this <a href="http://www.kiwidude.com/blog/2006/04/ncoverexplorer-v133.html">
blog entry</a> - all the necessary tasks, examples and documentation are located
within NCoverExplorer.Extras.zip available from <a href="http://ncoverexplorer.org/">
http://ncoverexplorer.org/</a>.&nbsp;</p>
<P class="question">
17. How do I merge multiple NCover coverage.xml results?</P>
<P class="answer">
You can can use NCoverExplorer to merge the results of multiple coverage runs. For
more information refer to this <a href="http://www.kiwidude.com/blog/2006/10/ncoverexplorer-merging-ncover-reports.html">
blog entry</a>.</P>
<P class="question">
18. Troubleshooting: Why is my coverage.xml file empty?</P>
<ul>
<li>If using the command-line, did you COM register CoverLib.dll (or use the //reg option
from NCover 1.5.6)?</li>
<li>Did you generate build symbol files (.pdbs) for the profiled application?</li>
<li>If using the //a option, did you correctly list just the assembly names without
paths or .dll suffixes?</li>
</ul>
<P class="question">
19. Troubleshooting: I have coverage.xml output but my XYZ assembly is not included in it?</P>
<ul>
<li>NCover will only profile loaded assemblies - did your code execution path while
under coverage force that assembly to be loaded (e.g. by loading a type or calling
a method in that assembly)?&nbsp;</li>
<li>Did you generate build symbol files (.pdb files) for the missing assembly? </li>
<li>If using the //a option, did you correctly list the assembly names including the
one that is missing?</li>
<li>Can you see information about the assembly being loaded within the coverage.log?
Is the correct assembly being loaded (check the path) - if you have a version in
the GAC it may possibly prevent the .pdb file from being loaded.</li><li>If using the NCoverExplorer gui, have you got a coverage exclusion defined which
is hiding it from the display?</li>
</ul>
<P class="question">
20. Troubleshooting: After running NCover my coverage.log says "Failed to load symbols for module XYZ"?</P>
<ul>
<li>This message means that no .pdb build symbol file was found for that assembly so
it cannot be profiled for code coverage. If that assembly is part of the .NET framework
for instance like System.Data.dll, then this is an expected message and should not
cause concern.&nbsp;</li><li>If however the assembly belongs to your application, did you generate the
build symbol files (.pdb files) for it? </li>
</ul>
<P class="question">
21. Troubleshooting: I get a "Profiled process terminated. Profiler connection not
established" message?</P>
<ul>
<li>If using the command-line, did you COM register CoverLib.dll (or use the //reg option
from NCover 1.5.6)?</li><li>Are you running Windows XP 64-bit? You may want to take a look at
<a href="http://ncover.org/SITE/forums/thread/43.aspx">this thread</a></li></ul>
<P class="question">
22. Troubleshooting: My coverage exclusions are not working?</P>
<ul>
<li>Have you put the full namespace type name to the exclusion including the Attribute suffix in the //ea argument? See the "How
do I use coverage exclusions?" question above.</li></ul>
&nbsp;
</body>
</html>