Welcome to XSTAF document’s documentation!

Contents:

XSTAF introduce

Page Status:Development
Last Reviewed:

XSTAF is an execution framework providing a bunch of features for testers’ convenience:

  • XSTAF is a disturbed execution framework based on STAF project, XSTAF can connect and run tests on multiple DUTs at same time
  • XSTAF provide test case management, and testers can have detail controls, down to atomic testcases
  • XSTAF provide a tool plugin mechanism, runtime data is exposed so user can develop plugins to control XSTAF runtime data
  • XSTAF provide some basic tools, like test report generator, test suite generator to handle common met tasks

XSTAF’s competition is other test manage and execution frameworks, like robot, spoon, testNG, or other test back-end of CI system.

XSTAF top view

Below image give an overview of XSTAF modules.

_images/XSTAF_top_view.png

XSTAF use scenarios

This chapter gives a preview of typical use scenarios XSTAF designed for:

PC like DUTs

Browser tests, desktop app tests, and PC function tests are always running on DUTs just like PC. STAF normally provides support for these platform, and these platform are power enough to support some extra tasks from STAF. So usually we deploy scripts, libraries and STAF client to DUTs directly, and use XSTAF to control the DUT directly.

The use scenario for this kind of tests, you can deploy XSTAF just like below:

_images/XSTAF_senarios_1.png

Note

For PC like DUTs, XSTAF run tests synchronously, which means these is only one test running on one DUT at some time

mobile DUTs (not ready yet)

Mobile app tests are running on mobile devices. STAF doesn’t support mobile device, and for tests on mobile devices, usually we need an extra PC to setup the test environment, like libraries, platform tools(adb). So XSTAF use a different structure to support tests on mobile devices, like below:

XSTAF server and client running on same machine, with multiple mobile devices

_images/XSTAF_senarios_2.png

XSTAF server control multiple machines, with multiple mobile devices connect with each machine

_images/XSTAF_senarios_3.png

Note

For mobile tests, XSTAF can run tests asynchronously, but for one subDUT, tests are running synchronously

XSTAF workspace

Page Status:Development
Last Reviewed:

XSTAF provide a mechanism to store a stand-alone test configuration all together (DUTs, testsuites, test results, test logs...). The benefit of this mechanism is you can have distinguishing workspace for different configurations, make your test plans clean and clear.

XSTAF workspace overview

Currently a workspace is a folder of pre-defined structure, like below:

_images/workspace_folder_structure.png

config.xml

config.xml is workspace config file, contains informations like DUTs info, testsuites info. XSTAF load the DUTs and testsuites according to infomations in this config file.

User need not create a config.xml file. When you new and save a workspace, a config.xml will be auto generated into the workspace folder, containing DUTs and testsuites info in the XSTAF runtime.

A sample config xml like below:

<XSTAF>
  <settings>
    <setting name="DefaultWorkspace">.default</setting>
    <setting name="WorkspaceLocation">c:\XSTAF\workspaces</setting>
  </settings>
  <DUTs>
    <DUT ip="10.239.36.123" name="">
      <testsuites>
        <testsuite name="testsuite1.xml" />
      </testsuites>
    </DUT>
    <DUT ip="10.239.36.124" name="">
      <testsuites>
        <testsuite name="testsuite2.xml" />
      </testsuites>
      <testsuites>
        <testsuite name="testsuite3.xml" />
      </testsuites>
    </DUT>
  </DUTs>
</XSTAF>

test results folder

test results folder contains testsuites info and test results for each DUT, test suites and test results are stored with a XML file, format like below:

<TestSuite>
  <TestCases>
    <TestCase>
      <ID>19730170-ced9-11e4-bd02-001b211bc956</ID>
      <Name>test_name</Name>
      <Command>C:/python27/python.exe python_script</Command>
      <Auto>False</Auto>
      <Timeout>6000</Timeout>
      <Description>demo test</Description>
      <Runs>
        <Run>
          <Start>1427087989.969</Start>
          <End>1427087995.672</End>
          <Result>fail</Result>
          <Status />
          <Log>C:\XSTAF\workspaces\workspace_name\test_logs\19730170-ced9-11e4-bd02-001b211bc956\1427087989.969</Log>
        </Run>
      </Runs>
    </TestCase>
  </TestCases>
</TestSuite>

test logs folder

test logs folder is used to store test logs for each run of test case. Logs location is like:

abs_workspaces_location/workspace_folder/test_logs/Test_ID/start_time

Note

Test logs include stdout/stderr of test scripts and all files in tmp log location.

XSTAF test management

Page Status:Development
Last Reviewed:

XSTAF manage test cases via a three level structure:

  • test suite, test suite is a test case container, could contain multiple test cases, is used to separate test cases into several categories, make test cases well organized.
  • test case, represent a basic test unit, contains information of how to run a test case
  • run, represent one execution of one test case, contains information of execution results

Test suite format

Test suite is a XML document with pre-defined format, like below:

<TestSuite>
  <TestCases>
    <TestCase>
      <ID>19730170-ced9-11e4-bd02-001b211bc956</ID>
      <Name>sample_test_case</Name>
      <Command>run_test_script_command</Command>
      <Auto>true</Auto>
      <Timeout>6000</Timeout>
      <Description>This is a sample</Description>
      <Runs>
        <Run>
          <Start>1427087989.969</Start>
          <End>1427087995.672</End>
          <Result>fail</Result>
          <Status />
          <Log>log_location</Log>
        </Run>
      </Runs>
    </TestCase>

  </TestCases>
</TestSuite>

A test suite can contain multiple test case, a test case can contain multiple runs.

Test case contains informations as below:

  • ID, a global unique GUID for this test case, recommend use a GUID generator to generate an unique GUID for this test case.
  • Name, tester friendly name to recognize this test case.
  • Command, command to run the test case, recommend to write test script using xUnit framework.
  • Auto, for auto case, XSTAF judges the pass/fail by command return value, return 0 is pass, other is fail. For manual case, XSTAF will prompt a dialog to let tester judge the pass/fail after command return.
  • Timeout, if test script running longer than timeout, XSTAF will terminate the test case, and continue run other test case
  • Description, test case description, you can fill in all related information useful.

Run contains informations as below:

  • Start run start time
  • End run end time
  • Result test result, pass/fail/no run
  • Status run status information, may contain informations about test issues
  • Log Log location, include stdout/stderr of test script, and other log files

Note

You can manually write your own test suite.

Recommend to write a translate tool, export test cases from test manage system (like testlink, HPQC...) and convert the these exported test cases to XSTAF test suite.

Test management under DUT view

After you add DUTs into one workspace, you can open DUT view, and add test suites to DUT.

_images/add_testsuite.png

After add test suites to DUT, you can select which tests you want to run, and add them to DUT task queue. XSTAF support add test case to task queue, or add whole test suite to task queue(all test cases in task queue will be added to task queue).

_images/task_queue.png

You can start the task queue if DUT status is normal, test case in task queue will be executed, and a new run will be added to each test case.

_images/start_task_queue.png

XSTAF tool management

Page Status:Development
Last Reviewed:

XSTAF provide a plugin mechanism to support extending XSTAF functions.

Tool manager

Tool manager is response to check the tool directory, and load the tools available. Tool manager supports load the tool dynamically, and provide tools runtime XSTAF window instance object, so tools can interact with XSTAF core.

_images/tool_manager.png

Add custom tools

To write custom tools, you just need implement an standard interface provided by tool manager, and place your tool to tools directory.

Below is code from tools/sample_tool, it demonstrates how to implement a sample interface:

import traceback
from XSTAF.core.logger import LOGGER
from PyQt4 import QtCore, QtGui
from ui.ui_sampleDialog import Ui_SampleDialog
import ui.resources_rc

###############################
#interface for tool manager
###############################
class Tool(object):
    _description = "Sample Tool"
    main_window = None

    #this method is for tool manager to pass in the XSTAF main window runtime object
    #so in this tool, we can interact with XSTAF main window
    @classmethod
    def set_main_window(cls, main_window):
        cls.main_window = main_window

    #method for tool manager to get icon of this tool
    @staticmethod
    def icon():
        tool_icon = QtGui.QIcon()
        tool_icon.addPixmap(QtGui.QPixmap(":icons/icons/sample.png"))
        return tool_icon

    #method for tool manager to launch this tool's dialog
    @classmethod
    def launch(cls):
        try:
            LOGGER.info("Launch sample tool")
            tool_dialog = SampleTool(cls.main_window)
            tool_dialog.exec_()
        except:
            LOGGER.error(traceback.format_exc())

    #method for tool manager to get description of this tool
    @classmethod
    def description(cls):
        return cls._description

################################
#dialog class for this tool
################################
class SampleTool(QtGui.QDialog, Ui_SampleDialog):
    def __init__(self, parent):
        QtGui.QDialog.__init__(self)
        self.setupUi(self)

Built-in Tools

XSTAF provide some built-in tools for common met test functions.

report_generator

report generator can generate a static html report for current workspace.

_images/test_report_generator.png

testsuite_generator

test suite generator can generate XSTAF test suite XML file from other format, like a CSV document.

Plan to support formats fro some common used QC system, like testlink...

workspace_merger

workspace merger supports merging multiple workspaces to one workspace.

_images/workspace_merger.png

workspace_spliter

workspace merger supports splitting one workspace with multiple DUTs into multiple workspaces with one DUT.

_images/workspace_spliter.png

Use XSTAF

Page Status:Development
Last Reviewed:

Prepare XSTAF environment

Prepare XSTAF server

  1. Install STAF
  2. Install XSTAF
  • For windows, you can download and use binary app
  • For other system, pip install XSTAF

Prepare XSTAF DUTs

  1. Install STAF
  2. Set proper trust level, add one line trust default level 5 in STAF config file STAF_dir/bin/staf.cfg

XSTAF UI preview

Server view

XSTAF has one server view per instance, server view is used to:

  • manage workspace, new/load/save workspace
  • manage DUTs for each workspace, add/delete DUT
  • settings
  • manage tools, like report generator, test suite generator
  • start STAF, check all DUTs status via STAF
_images/XSTAF_server_view.png

DUT view

XSTAF could have multiple DUT views per instance, DUT view is used to:

  • manage test suites for each DUT, add/delete test suite
  • run test case, run individual test case/run a whole test suite/rerun fail cases
_images/XSTAF_DUT_view.png

Use XSTAF step by step

settings

There are some setting you can set for your test environment

_images/Settings.png

new or load existing workspace

_images/new_load_workspace.png

add DUTs

After you have new or load a workspace, you can add/delete DUTs into this workspace

Note

When adding DUT, DUT ip is required, DUT name is optional

_images/add_DUT.png

add test suites for each DUT

Note

Double click/right click DUT raw, can open DUT view. On DUT view you can add/delete test suite.

_images/add_testsuite.png

add test case to DUT task queue

Each DUT has a task queue, tasks in task queue will be run on DUT one by one when task queue started.

Note

Right click at the test suite or test case, you can add test suite or test case to task queue

You can add same test suite/test case to task queue multiple times, then these test suite/test case will be run multiple times

_images/task_queue.png

start STAF

You can manage DUTs and test suites in workspace without enabling STAF, but if you want to execute test case on DUT, then you need to start STAF

You can start STAF by using STAF launcher, like startSTAFProc.bat on windows. Or you can use XSTAF to start STAF

_images/start_STAF.png

Check DUT status

After start STAF, you can check DUTs status

Note

If you start STAF not via XSTAF, you must check status in server view before run task

Note

Check status take longer time if you have a lot of DUTs in your workspace, to check specified DUT status, you can check status in DUT view

Check status take longer time if the DUT is not detected.

_images/check_status.png

start task queue

If you DUT status is normal, then you can start task queue of this DUT, test cases in task queue will run synchronously.

_images/start_task_queue.png

generate test report

XSTAF has a built-in test report tool, to generate html report for one workspace. You can just use this test report tool, or write a your own customized report tool.

XSTAF Trouble shooting

XSTAF could have bugs since it’s a new tool, not have been used and test thoroughly.

XSTAF logger

XSTAF provide a logger, with it you can see what is happening inside the XSTAF when bug happens. If you are a python developer, you maybe can fix it yourself and push the fix to github. Or you can just report a bug to me.

_images/XSTAF_logger.png

Indices and tables