Windows App Certification Kit: run appcert.exe from the command line

In 2016 I tested a UWP app before sending it to the Store, and on Windows Server 2016 the Windows App Certification Kit (WACK) window didn't even open:

The UI workflow of Windows App Certification Kit is not supported on this OS version.

The command-line version, appcert.exe, works there. It runs the same tests, and it's what you need on a build server anyway.

:: Command Prompt opened as administrator
cd /d "C:\Program Files (x86)\Windows Kits\10\App Certification Kit"

:: Microsoft's docs run reset before every test
appcert.exe reset

:: Test a package file that isn't installed
appcert.exe test -appxpackagepath "C:\build\MyApp_1.0.0.0_x64.msix" ^
    -reportoutputpath "C:\build\report.xml"

The kit comes with the Windows SDK, and C:\Program Files (x86)\Windows Kits\10\App Certification Kit\ is its default folder. appcert.exe /? lists all the parameters.

Test an installed app by its package full name

That's how I did it in 2016: install the package, then pass its full name.

1. Install the package. Visual Studio puts the Add-AppDevPackage.ps1 script in the package's _Test folder under AppPackages: right-click it and choose Run with PowerShell.

2. Find the PackageFullName:

Get-AppxPackage -Name "*Greets*"

3. Run the test with that name:

appcert.exe reset
appcert.exe test ^
    -packagefullname "1257MixVlad.Greets_2.2.0.0_x64__v53m0nncvwpbt" ^
    -reportoutputpath "C:\build\report.xml"

I kept it in a bat file next to the package:

@echo off
set KIT=C:\Program Files (x86)\Windows Kits\10\App Certification Kit
set PKG=1257MixVlad.Greets_2.2.0.0_x64__v53m0nncvwpbt
set REPORT=%~dp0report.xml

:: Delete the old report: the kit may refuse to overwrite it,
:: and you won't read stale results
del "%REPORT%" 2>nul

"%KIT%\appcert.exe" reset
"%KIT%\appcert.exe" test -packagefullname "%PKG%" -reportoutputpath "%REPORT%"
pause

%~dp0 is the folder of the bat file and already ends with a backslash. The report path must end in .xml, otherwise the kit rejects it.

For a desktop app with an installer, Microsoft's docs give this command:

appcert.exe test -apptype desktop -appusage peruser ^
    -setuppath "C:\build\setup.exe" ^
    -reportoutputpath "C:\build\report.xml"

Read the report

The root element REPORT has an OVERALL_RESULT attribute: PASS, WARNING or FAIL. Each test is a TEST element inside REQUIREMENTS/REQUIREMENT with its result in RESULT. In PowerShell:

[xml]$report = Get-Content -Raw "C:\build\report.xml"
$report.REPORT.OVERALL_RESULT

# Tests that didn't pass
$tests = $report.SelectNodes('/REPORT/REQUIREMENTS/REQUIREMENT/TEST')
foreach ($test in $tests) {
    $result = $test.SelectSingleNode('RESULT')
    if ($result -and $result.InnerText.Trim() -ne 'PASS') {
        '{0}: {1}' -f $test.GetAttribute('NAME'), $result.InnerText.Trim()
    }
}

The kit also saves an HTML copy of the report with the same name in %LOCALAPPDATA%\Microsoft\AppCertKit\.

On a build server

• Run as administrator.
• The docs require an active user session: a service may start the kit only inside a logged-in user's session, not in Session 0. If the kit fails on a build agent installed as a Windows service, run the agent as a regular process under a logged-in user. The same page says newer kits can run in automated testing without an interactive session, so try your version first.
• Turn on Developer Mode in Windows settings: it's in the kit's prerequisites.
• Performance test thresholds assume a low-power computer, so a fast build server can pass tests a weak laptop fails. Microsoft recommends testing on a low-power computer.