The Upgrade Test Plan allows you to find differences in report, document (Webi), and view (Tableau) data, and run time between two different BI connections and users. One common scenario where Upgrade Test Plan can be used is in the case of a BI or system upgrade to compare reports in the newer version with the same reports in the older version. Another use case is report comparison across multiple environments.

User can drag reports from a BI Connection of BI Connection Explorer or add pages with prompted data using the intuitive user interface. The Test Plan also verifies that the reports are found in both the connections/users. Unlike the Report Test Plan, there is no benchmarking of the reports in the Upgrade Test Plan. When an Upgrade Test Plan is executed, the comparison takes place between two versions of the same report. If there is any difference found between the two versions, the difference will be shown in red color, in the pop-up.

After the Upgrade Test Plan is executed, the result data and the history of all the previous runs are saved.

For information on BI tools supported by the Upgrade Test Plan, please see here.

High-Level Steps for Adding an Upgrade Test Plan

Below are the high-level steps for adding an Upgrade Test Plan.

  1. On the side menu, click Test Plans.  
  2. Click Add New.
  3. In the New Test Plan page, select Upgrade Test Plan .
  4. In the Properties page, do the following,
    1. In the Basic Information screen, enter all the required details and click Next
    2. In the Options screen, complete the desired fields and click Save.
  5. In the Home page, do the following:
    1. Click Add test cases to add reports/views/documents to the test plan.
    2. Click Run to execute the test plan. 
  6. Click the > arrow menu button (next to Home) and select Run History to view the test plan results and manage test run history.

Setting up and Working with Upgrade Test Plan

This section walks you through the pages and the detailed explanation of options in an Upgrade Test Plan.

Set up Properties

The Properties are divided into Basic Information and Options.

Basic Information

In Basic Information, complete the following:

Test Plan Name

Enter a name for the test plan.


Create a label for easy identification and grouping of test plan. To create a label, click the Click to add label link, enter a name, and press Enter.  


Select a pre-upgrade environment for comparison with post-grade environment.


Select a post-upgrade environment for comparison with post-grade environment.

Bi Validator Groups

Assign groups to this test plan.

Select Default Format

Select a format from the drop-down list. Depending on the BI connection, the available formats can be PDF, Data, and Excel.


In Options, complete any of the following:

  • Report Timeout. The time limit for the report run before the request is killed if no response is received.
  • Parallel Reports. Number of pages within the Test Plan will execute in parallel when the Test Plan is benchmarking (baselining) the reports.
  • Select Run Type. Compare Both, Run Pre-Upgrade, Run Post-Upgrade to select run type. 
  • PDF Compare Type. There are two comparison options - 'Appearance' and 'Text'. The 'Appearance' option compares the Benchmark and Result PDF as an image (pixel to pixel) and marks the test run as 'Fail' if there are no differences. The 'Text' option compares the text in the report pages (including data) while ignoring differences due to 'Date Run' and marks the test case 'Fail', only if there is a 'Text' difference. 
  • Case-Sensitive. This option is used in the Text comparison type. Select this option to find differences for changes in case.
  • Run Time Variation Allowed. This is the percentage of variation in the dashboard page run time that is allowed from the benchmarked (or base-lined) value. For example, if the dashboard page took 10 seconds during the benchmark but took 21 seconds during the 'Run' step, the status of the Dashboard Page is marked as a 'Warning' provided there are no other failures. A message is also recorded in the 'Message' column stating the same. The default variation is 100%. 
  • Row Count. Only applicable to the 'Data' format. This option sets number of records to be downloaded for benchmarking and comparison during the test run.

Add Test Cases

In the Home page,  add reports / views / documents of the pre-upgrade connection to the test plan. Select the format type of the reports for comparision. There are three format options for comparing the reports - PDF, Data and Excel.

How to Add Test Cases?

  1. Ensure that you are on the Home page. If not, click the Home button.
  2. Do any one of the following:
  • Drag reports (or Documents) from the Connection Explorer (available on the left-hand side) of the BI Connection.
  • Click Add test cases. Add the report from within the browser window that opens, click Capture, and then click Add Test Page(s) to Test Plan.
  • Click the + icon to add pages from within a browser window that opens. This method allows users to select prompts for the dashboards and capture the dashboard page URLs before adding them to the Test Plan. This feature is only available for OBIEE connection.
  • Copy dashboard prompted links from the browser window of your BI application into a notepad and import them.

When a dashboard is added, underneath its reports are added. User can drag reports from the BI Connection Explorer or add pages with prompted data using the intuitive user interface. The Test Plan also verifies that the reports are found in both the connections/users.

Additional Tasks Post Test Case Addition 

After the test cases are added, you can perform the following additional tasks:

Add Description

Click the Edit link under the Description column if there are any details need to be added. After updating the description, click Save.

User selection

Select the users from the drop-down list under the user (Pre) and user (Post) columns to run the report for those particular users.
In case of OBIEE connection, a Create link will be displayed when report is added. When a user clicks the Create link, the View / Update link will be shown instead of the Create link if the View / Column selectors are available for that particular report. User can click the View link to select different option(s) in the selectors. The Update link will be used for updating the selected Selectors.

Add / Edit Options

Click the Edit link under Options column to open the pop-up containing parameters for any particular report. Users can add, edit, and delete parameters.


Click the  Delete icon to delete the selected reports.

Set Variance

Variance allows to mark the testcase as pass or fail if data in a particular column in the testcase satisfies that allowed variance percentile even though data is different on that column. This is used to select the variance against different columns. The Set Variance link under the Variance column is enabled when the format type is data. It is used to select the variance against different columns. For more information on Variance, see the how-to-procedure here.


You can set filters for Tableau reports. For more information on filters, see the how-to-procedure here

More Options

Clicking  More displays the following options:


Allows you to refresh the page (if any benchmark schedule is running in the background).


Allows you to import reports into the test plan using a text file containing report urls. For more information on Import, see the how-to-procedure here.


Allows you to export the selected reports. For information on where the export options are available in a test plan and how to use them, please see the how-to-procedure here

Configuration Notifications

Allows you to set up email addresses of the recipients for whom you want to send notifications about the test plan. To configure notifications, click  More available besides the Home button at the right-hand side and select Notify.

Schedule Test Plan

Allows you to schedule test plan run process at specified time/date selected from the schedule window. To schedule test plan, click  More available besides the Home button at the right-hand side and select Schedule.

Run Upgrade Test Plan

After adding the test cases, you click the Run button on the Home page to execute test cases. When you click Run, you will be navigated to the Run page, where you will need to click either Run or Run Test Cases. By any chance if you are not in the Home page, click  More available besides the Home button at the right-hand side and select Run. At any point, you can click the Stop button to stop the ongoing execution of test cases.

Information     Users in Multi-User environment can also run the test plan using the Command Line Interface.

When an Upgrade Test Plan is executed, the comparison takes place between two versions of the same report. If there is any difference found between the two versions, the difference will be shown in a window, where differences are shown in red color.

If there are any differences, a View link along with the red bubble appears in the Differences column. If there are no differences, then it shows "No differences" text along with the green bubble. For PDF, clicking the View link opens a window containing the current and benchmarked PDFs, in which differences are shown side by side. For Data, clicking the View link opens a window displaying records in pre-connection and post connection, matched and unmatched records, and queries for pre-connection, post connection, and differences. For Excel, the differences for benchmark and result are shown side-by-side and are marked in red color.

View Run History

In the Run History page, latest test plan run details are shown. Previous run details are also shown when the particular time stamp is selected from the left-hand side.  Also, counts for all status, passed, fail, warning, error and running status will be shown. When selected any status count, reports will be displayed based on the selected status category.

To view Run History, click  More available besides the Home button at the right-hand side and select Run History

The Run History page contains the following:


Click the  Refresh icon to refresh test plan results.


Select the time stamp and click the  Delete icon to remove the test run.

Delete All

Click the  Delete All icon to remove the complete test run history.


© Datagaps. All rights reserved.
Send feedback on this topic to Datagaps Support