Test Reporting¶
Introduction¶
Arjuna uses pytest as its underlying test engine. The core of its reporting is driven by the facilities provided by pytest and its available plugins and options.
Arjuna makes use of various pytest hooks to add/tweak information that is included in the reports.
HTML and JUnit-Style XML Reporting¶
Arjuna currently supports two formats for reporting:
HTML: Arjuna wraps pytest-html reporting plugin for its HTML reports and adds its own JS/CSS extensions as well as more information controlled by Arjuna Options.
XML: Junit-style XMl report is generated for integration with CI applications like Jenkins.
Report Directory Structure¶
For each run, Arjuna generates reports in a directory that is controlled by the RUN ID set for the run.
- With defaults:
Default Run ID is mrun.
Arjuna appends current timestamp to the run id to generate the final run id.
This directory is created in <Project Root>/report directory.
- Following sub-directories are created in it:
html: HTML report(s)
xml: XML Report(s)
log: Arjuna log.
screenshot: Screenshots taken in this run.
You can provide a custom run id e.g. “abc-beta” using CLI’s -r / –run-d switches.
During development of scripts, you might want to update same reports. For this you can use CLI’s –update switch. If provided, Arjuna does not append timestamp to run id for directory name, thereby overwriting previous report files.
Report File Names¶
For run-project and run-selected Commands¶
The name of HTML report is report.html and for XML it is report.xml.
For run-session, run-stage and run-group Commands¶
The name of the report file is generated as:
report-<session name>-<stage-name>-<group-name>.{html/xml}
Choosing Report Format(s)¶
By default both report formats (XML and HTML) are generated.
You can control this by ArjunaOption.REPORT_FORMATS option in reference configuration.
You can also easily control this by CLI’s -o / –output-format switch.
Logging in the context of Reporting¶
The compelete log generated by Arjuna can be found in <report_dir>/log directory as arjuna.log.
The logging that is done during the duration of a test is captured and included in the HTML report as well in the HTML entry for that particular test.
Reporting Protocols¶
Additional contents in HTML reports are controlled using Arjuna’s reporting protocols.
Screenshooter Protocol¶
If the request object associated with a test resource or a test has a screen_shooter in its space, Arjuna takes a screenshot in case of failure and error situation and includes it in HTML report.
A screenshooter is any object that inherits and implements ScreenShooter base class.
Arjuna’s GuiApp and GuiPage classes implement this and hence are screenshooters.
A suggested pattern to enable this feature is to do the following in a test resource function:
request.space.screen_shooter = <app object or other screenshooter>
Auto-inclusion of screenshots with take_screenshot call¶
When you take a screenshot with Arjuna’s GuiApp or GuiPage using take_screenshot call, the screenshot is auto-included in the report.
Screenshots for Passed Tests¶
To keep the report size in control, by default screenshots are not included for passed tests.
You can ovverride this behavior by setting ArjunaOption.REPORT_SCREENSHOTS_ALWAYS to True.
Network Recorder Protocol¶
The HTML report can include network packets in the extras section.
A suggested pattern to enable this feature is to do the following in a test resource function:
request.space.network_recorder = app_object.network_recorder
Auto-inclusion Network Packets for HTTP Automation¶
When you do HTTP Automation using Arjuna’s HTTP lib, the network packets are auto-included in the report.
The redirections are also captured and reported.
Capturing Network Packets for Web Gui Automation¶
With Arjuna, you can capture network packets during Web Gui Automation as well.
Arjuna makes use of BrowserMobProxy for this.
Download and unarchive the BrowserMobProxy in <Your Test Project Root>/tools/bmproxy directory. Make sure it is setup properly with appropriate Java version.
By default, to keep report size in control, network recording is swtiched off. You can switch it on by setting ArjunaOption.BROWSER_NETWORK_RECORDER_ENABLED to True.
Reporting Network Packets for Passed Tests¶
To keep the report size in control, by default network packets are not included for passed tests.
You can ovverride this behavior by setting ArjunaOption.REPORT_NETWORK_ALWAYS to True.
Filtering of Network Packets¶
Bt default, Arjuna reports only packets which have HTML/JSON/XML as response content type.
To capture and report all packets, set ArjunaOption.REPORT_NETWORK_FILTER to False.
Auto-Inclusion of GuiApp/GuiPage packets¶
The network recording protocol is integrated with GuiApp/GuiPage objects in Arjuna.
To enable this integration, set ArjunaOption.BROWSER_NETWORK_RECORDER_AUTOMATIC to True.
Now, each time you launch a GuiApp, the recording will automatically be triggered with a bucket created with the title “Home”.
Each time you create a page, a bucket with the page label will be created.
Setting Title for Network Packet bucket¶
In a typical end user flow, there will be many packets sent by the browser.
You can organize these into buckets with titles for better organized reporting to aid in troubleshooting.
Doing this is straight-forward:
<app or page>.network_recorder.record("New bucket title")
This single instruction will register all packets in the previous buckets and create a new recording bucket with the tile provided.
Where is the reporting entry for Resource Tear Down?¶
Unless there is an error in tear down code, pytest does not include its execution details in the HTML report.
Accordingly, Arjuna is also not able to show this information.
However, you can find any logging done during resource teardown in arjuna.log file for a run.