Share:

A Functional Test Plan allows users to perform functional testing of Tableau Views. In each View selected, you test whether changes to visualization options conform to user or business requirements.  Using BI Validator's Functional Test Plan, you can test Parameters, Filters, Marks, Columns, Views, Worksheets, Snapshot, PDF, and Data in the Views. In addition, you can validate data by creating data rules on columns in Worksheets.

When you run a Functional Test Plan, you assert each visualization option and validate to find out its existence and all its associated values with the benchmarked result. You can build the PDF view to compare Text and Appearance changes or you can take a snapshot of the View and compare the visualization layout with expected results. In case of assertions for View, results provide validations for New, Missing, and Matching values.  In case of data assertions, the results (success or failure) that return upon execution of data rules determine differences in the data.

Information Functional Test Plan supports Tableau connection only.


High-Level Steps for Adding a Functional Test Plan

Below are the high-level steps to add a Functional Test Plan.

  1. On the side menu, click Test Plans.
  2. Click Add New. 
  3. In the New Test Plan page, select Functional Test Plan. The Functional test plan opens with the Provide Test Plan Details page.
    Information Alternate navigation methods to add a Functional Test Plan are described here.
  4. Enter basic details, select the connection name, workbook, and View options to test.  
  5. Click Create Test Plan. The Views contained in the Workbook are added to the Home page.
  6. Select the Views to test and click Create Tests. By default, all the views are selected automatically. Once the Create Tests button is clicked, a test case is created for each view and the views are benchmarked.
  7. Optional. At the top, click Benchmark to benchmark all of the views at once. Otherwise, click the Benchmark buton in the view row to benchmark a specific view.
  8. Click on the test name (or view name) to open its details.  
    1. As a result, the Get task is added and opens the Properties tab.
    2. In the Properties tab, select the required View Options and click the Benchmark button. 
    3. Click the Filters tab. Here, data for Parameters and Filters is extracted from the View after the test case (View) is benchmarked. Please note that Marks are extracted by using the Capture button only, which is discussed in Add Views below.
    4. Click the View Assertions tab. Assertions validate visualization options with the benchmarked data and return differences for missing, new, and matching data. In this tab, you will run assertions using the Assert button available for each View Option. 
    5. Click the Data Assertions tab. Here, you add rules based on the Worksheet columns to verify whether the data in the Worksheet is correct. For example, you might want to verify whether the filters applied in a View are working correctly or not.
    6. Drag and drop the Switch task if you want to apply global filters automatically when you switch to an another View.
    7. In the Properties tab, select a new View and the options to be validated. This View should be different from the View in the Get task.
    8. Click the View Assertions tab and assert the View Options.
    9. For each test case, work with Get and Switch tasks. 
  9. Click Run at the top to navigate to the Run page and click Run Test Plan. BI Validator executes all the test cases across the Get and Switch tasks.
  10. After the test plan is run, click on the test case name and view assertion results.
  11. Click the > arrow menu button (next to Home) and select Run History to view the test plan results and manage test run history.

Setting up and Working with Functional Test Plan

The pages you need to set up in the Functional Test Plan wizard are described in the sequence below.

Set up Properties

As soon as the Functional Test Plan is added, you will complete the following Properties and click Create Test Plan.

Test Plan Name

Enter a name for the Functional Test Plan. By default, an auto-generated name is assigned to the test plan. For each test plan, an ID is assigned and created when the test plan is saved.

BI Connection

Select a Tableau connection from the drop-down list.

Bi Validator Groups

Select groups that can access the test plan. Only users within the selected group can view and work on this test plan.

Workbook

Select the workbook in the Tableau connection.

How to select a Workbook?

  1. Click the magnifying glass icon.
  2. In the dialog that opens, from within the Project tab, select the desired project. When you select a project, you will be navigated to the Workbook tab. Select the workbook from the list. Alternatively, if you remember the workbook name, you can directly go to the Workbook tab and search and select the workbook.
  3. Once the workbook is selected, click Add.

Options

Select all the applicable visualization options to test in a Tableau Workbook. The following options are available:

  • Views - Validate the existence of Views
  • Worksheets - Validate the existence of Worksheets
  • Columns - Validate the list of Columns
  • Filters - Validate the existence of Filters and their values
  • Parameters - Validate the existence of Parameters and their values
  • PDF - Validate the differences between benchmarked PDF and current PDF View
  • Snapshot - Validate the differences between the captured portion of the benchmarked View and current View.
  • Data - Validate Worksheet data rules.

Label

Create a label for easy identification and grouping of test plan. To create a label, click the Click to add label link, enter a name, and press Enter.

Row Count

Enter how many number of rows to benchmark in a View. The test plan will validate the Visualization options based on these rows. By default, the count is 0, which means complete data is benchmarked.

Setting up Views

As soon as the Properties are saved,  all the Views present in the Workbook are added to the Home page. Click Create Tests for all the selected Views. BI Validator will create a test case for each View and all the Views benchmarked automatically. Along with the test cases created, BI Validator also adds a test case for Workbook called "Workbook_Assertions". This test case allows you to find differences in Views and Parameters at Workbook level.

For each test case created, you can delete or benchmark it. To benchmark all the test cases at once, you can prefer to use the Benchmark button available at the top of the test case grid. Once you benchmark the test cases, click Save to save the changes.

The following options are available:

Add Views

Allows you add more Views from the Workbook to the test plan.

Delete

Click the  Delete icon to delete the selected test case (View) from the test plan.

Benchmark

Allows you to benchmark test cases.

Save

Allows you to save the test plan.

Each test case contains at least one Get task. Optionally, you can also add a Switch task. For one test case, you can add more than one Get and Switch task. 

The details on Get and Switch task are as follows:

Working with Get Task

In the Get task, you run a View for functional and regression testing of visualization options. More than one Get task can be added to test the same View with different combination of Filters and Parameters.

In the Home page where test cases are created, click on the test name to open the Get task with the Properties tab displayed. The purpose and procedural tasks specific to each tab is described below.  

Properties Tab

In this tab, the Views added are shown. You will review the View options for testing, set the browser dimensions, benchmark data, and load Parameters, Filters and Marks for Worksheets.

View

The View that will be used for functional testing.

View Path

The path of the View in Tableau.

Browser Width / Browser Height 

Specify the width and height of the browser in pixels. This is the window size of the Selenium browser.

View Options

These are visualization options that you have selected in the Properties page. You can further select or clear any of the View Options (including Marks). Only the ones that are selected here will be asserted.

Benchmark

Allows you to benchmark the View Options for the selected View.

Filters Tab

In the Filters tab, you can observe Parameters, Filters, and Marks (provided these are selected in the Properties tab) on their own tabs available on the left side of the screen. These appear from the View when you click Benchmark in the Properties tab. If there is more than one worksheet, you can select it from the Worksheet drop-down list and click the Refresh icon to view Filters and Marks of that worksheet. 

In Parameters/Filters/Marks tabs, you can

  • Create a new filter/parameter/mark
  • Add a new value to the existing parameter/filter
  • Edit the filter/parameter
  • Select a parameter/filter value for deletion
  • Remove the parameter/filter value
The table below describes the tasks and their procedures for Filter, Parameter, and Marks:
View OptionTaskProcedure
Filter Add a Filter

In the Filters tab, in the pane containing Filters, click the + icon. Complete Filter Name, Filter Type and Data Type, and click Save.


Edit the Filter itself

In the Filters tab, in the pane containing Filters, click More options associated with the filter you wish to edit and select Edit. Edit the options and click Save.

Remove the Filter

In the Filters tab, in the pane containing Filters, click More options associated with the filter and select Remove. 


Create a Filter value

In the Filters tab, in the pane containing Filters, select the checkbox for the Filter. In the Filter Values pane, click the + icon and enter the Value.


Edit a Filter Value In the Filters tab, in the pane containing Filters, select the checkbox for the Filter. In the Filter Values pane, select the checkbox for Value, and edit the value. 
Remove the Filter value In the Filters tab, in the pane containing Filters, select the checkbox for the Filter. In the Filter Values pane, select the checkbox for Value, and click the Remove icon.  
Parameter Add a Parameter

In the Parameters tab, complete Name, String and Value, and click the + plus icon.


Edit a Parameter value ‚ÄčIn the Parameters tab, select the checkbox for the Parameter, and edit the value.
Delete a Parameter In the Parameters tab, select the checkbox for the Parameter, and click the delete icon. 
Mark Add a Mark In the Marks tab, complete Field Name, Field Value and Worksheet, and click the + plus icon.

When you make any changes in the Parameters, Filters and Marks tabs, click Apply. If you want to record Filters and Marks from the View, click Capture. Go to the required view option, make selections, and click Done. In the last, click Apply in the Filters/Marks tab to save the changes to test case.

Information Try to avoid inserting special characters and symbols in Filters, Parameters, and Marks. If included, BI Validator may not be able to run the test plan.

View Assertions Tab

In the View Assertions tab, you will observe that the View Options are arranged vertically on the left side of the screen. You navigate to each of the option and click Assert to find the differences. There is a Benchmark button in each of the visualization option tab so that you can specifically benchmark only the visualization option. At any time, you can click the Assert button to compare the visualization option with the previously benchmarked data. Otherwise, click the Benchmark button available in the visualization option tab and then compare it with the fresh benchmarked data.

Below are the different type of View Options and their Assertion details.

View OptionAssertion Details
Worksheets

All the Worksheets in the View that you have added are listed on the left-hand side. You can directly click the Assert button to validate the existence of Worksheets with the previously benchmarked data. Otherwise click Benchmark to fetch the fresh results.

If there are any differences, Assertions show New / Missing Views. If there are no differences, Assertions show Matched Views only.


Worksheet Columns

The columns are grouped by Worksheet and appear on the left-hand side. You can directly click the Assert button to validate the existence of Columns with the previously benchmarked data. Otherwise,  you may click Benchmark to fetch fresh results.

If there are any differences, Assertions show New / Missing Columns. Otherwise, Assertions show Matched Columns only.

Parameters The Parameters applied at Workbook level are listed on the left-hand side. You can directly click the Assert button to validate the existence of Parameters and their values with the previously benchmarked data.

If there are any differences, Assertions show New / Missing Parameters or their values. Otherwise, Assertions show Matched Parameters only.
Filters

The Filters applied in the View and at Workbook level are listed on the left-hand side. You can directly click the Assert button to validate the existence of Filters and their values with the previously benchmarked data.

Go to each Filter to know the differences. If there are any differences, Assertions show New or Missing Filters, or their values.

PDF

To find out the PDF differences between previous benchmarked and current PDF Views, you will add a rule by using a condition. For example, you can assert a PDF to validate the View for Text and Appearance differences.

Add a Rule

  1. Click ADD.
  2. Enter a name for the Rule.
  3. Select the condition and Assert Value. Some of the conditions available are STARTSWITH, ENDSWITH, EQUALS, NOTEQUALS, and so on. If you have selected the condition as PDF EQUALS, select the Comparison Type as Appearance or Text.
  4. Click Save.

At some point in future when you want to know the differences, you click the Assert button to validate the previous benchmarked PDF View with the fresh benchmarked PDF View.

After asserting PDF, click More options and select View Results.

Manage Rules

  • Edit - To edit a rule, click More options and select Edit. Change the existing rule, assert the PDF with a different rule condition, and click Save.
  • Remove - To remove a rule, click More options and select Remove
Snapshot

For Snapshot assertions, you add a rule to capture the portion of dashboard view (image). When you assert the Snapshot option, the results provide a holistic view of the changes to the new dashboard view as compared with the previous benchmarked dashboard view.

Add a Rule

  1. Click ADD .
  2. Enter a name for the Rule.
  3. Using the cursor, take a snapshot of the View.
  4. Click Save.

At some point in future when you want to know the differences, you can click the Assert button to validate the Snapshot with the fresh benchmarked View.

After asserting the Snapshot, click the More Options and select  to open the slider. Dragging the slider towards the Result will show the changes between the benchmarked and current state of the View.


Manage Rules

  • Edit - To edit a rule, click More options and select Edit. Change the existing rule, assert the image with a different selection, and click Save.
  • Remove - To remove a snapshot rule, click  More options and select Remove

Data Assertions Tab

In Data Assertions, you can validate changes by creating rules. For example, if you apply a filter on Region to show the data for South, you ensure this by creating a rule using the Region column available in the worksheet.

In the Data Assertions tab, all the worksheets are automatically selected. If you want to clear or select any new worksheets, click Add Worksheets to open the Add Worksheets dialog, select the ones that are needed and click Save Worksheets

When you save worksheets, the worksheets appear vertically in the left side of the Data Assertions tab.

If you have added any new worksheets, benchmark each worksheet individually or all worksheets at once. To benchmark a worksheet, click Benchmark in the worksheet row. To benchmark all worksheets at once, click Benchmark at the top-right corner outside the worksheet grid.

Data Tab

Selecting the worksheet displays the Data tab in which sample data is displayed. The data will appear after the worksheet is benchmarked. 

InformationThe data is displayed based on the number of  rows defined in the Row Count option available in the Properties.

Rules Tab

In the Rules tab, add Data Rule or SQL Rule. To add a new Data rule using the available columns, first select the column and click Add New Rule, and complete the rule. To add a SQL rule, click the plus icon. In the Rule Builder, select the columns, enter conditions if required, enter a name for the rule, and save the rule. When rules are added, they appear in the Rules tab. From this tab, you may want to delete rules before you run them.

Assert Tab

In the Assert tab, you click Assert to execute all the data rules.

Working with Switch Task

The Switch task provides the ability to apply global filters automatically when you switch to an another View. To add the Switch task, drag and drop the Switch task to the right side of the Get task. By doing so, the Switch task displays Properties and View Assertions tabs. However, please note that this task is optional in the Functional Test Plan.

The descriptions and purpose of these tabs are described below:

Properties

As soon as the Switch task is added, the Properties tab is displayed. In this tab, you select a new View from the drop-down and benchmark the View.

View

This is the View selected in the Switch task.

View Path

The path of the View in Tableau.

View Options

These are visualization options that you have selected in the Properties. You can further select or clear any of the View Options. Only the ones that are selected here will be asserted.

Benchmark

Allows you to benchmark the data for all the selected View Options.

View Assertions

In the View Assertions tab, you will observe that the View Options are arranged vertically on the left side of the screen. You navigate to each of the option and click Assert to find the changes between the previous and fresh benchmarked visualization results.

Configure Notifications

Allows you to add email addresses of recipients to whom you want to send notifications about the test plan results. To configure notifications, click More available besides the Home button at the right-hand side and select Notify.

Schedule Test Plan

This option allows you to schedule test plan run process at specified time/date selected from the schedule window. To schedule test plan, click More available besides the Home button at the right-hand side and select Schedule.

Run Functional Test Plan

After working with Get and Switch tasks for each test case, you are now ready to run the test plan. To run the test plan, click Run at the top-right corner or click the > arrow menu button available besides Home and select Run. In the Run page, click Run Test Plan. BI Validator now executes all the selected test cases. By default, the Functional Test Plan runs all the Views across the tasks. If you want to run specific test cases (Views), go to the Home page,  in the Test Cases pane, select only the ones you want to run, and click Save. At the top of the Run page, you will see the Get and Switch tasks. When you click Run, the Views by taking the Get and Switch tasks into account, will be executed in sequence, one after another.

After the test plan is executed, the Run Summary displays the run details of each test case. To view assertion results, click on the test case name or select the test case from the drop-down available at the top. Opening the run details for a test case displays assertions, where View Options appear vertically on the left side of the screen. Navigate to each View Option on the left and notice if there are any differences.

For Views in the Get task, you can also see Data Assertions. Clicking the Data Assertions tab displays assertions for data rules in Worksheets.

To view the Data Assertion results for a Worksheet, click the Worksheet on the left-hand side. The result "Success" indicates that there are no differences and "Failure" indicates that there are differences.

Information Users in Multi-User environment can also run the test plan using the Command Line Interface.

Change Workbook

You may also change the workbook to compare the benchmarked views with views in a different Workbook.

How to change the Workbook?

  1. In the Run page, click Change Workbook.
  2. In the page that opens, in the Run Views pane available on the right-hand side, click Change.
  3. In the Tableau Search dialog that opens, search and select the workbook, and then click Add. All the associated views are added
  4. In the Run Views pane, use the up and down arrows to change the sequential order of views according to the order of views in Benchmark Views pane. The order of views in Benchmark Views must match with the views in Run Views. 
  5. Click Save.

View Run History

The Run History page displays the past run details of the test plan, including the latest run. The run time stamps are shown on the left and by default always the latest test plan run details are shown when you navigate to the Run History. You can view the previous run by selecting a specific time stamp from the list. For each run, select and see which test cases have failed and passed, and which ones have errors and produced warnings. Clicking on the test case name opens the assertion results for View and Data. Alternatively, to open the assertion results, click the arrow menu button besides the Run History and select the desired test case.

Warning At this point, web-based reporting is unavailable for Functional Test Plan.

The Run History contains the following page:

Refresh

Click the  Refresh icon to refresh test plan results.

Delete

Select the time stamp and click the  Delete icon to remove the test run.

Delete All

Click the  Delete All icon to remove the complete test run history.

PreviousNext

© Datagaps. All rights reserved.
Send feedback on this topic to Datagaps Support