TestRail From Gurock is a test management solution and accessible through the Syndeia Cloud REST API. In this notebook, we demonstrate how to navigate through the multilevel TestRail data model to obtain artifact data at any level.

• A TestRail Repository contains TestRail Projects, which map to Containers in the Syndeia data model
• A TestRail Project can contain Test Suites, Test Runs, Test Plans and Milestones, which are all Syndeia Artifacts.
• A TestRail Test Suite can contain Test Sections, which contain Test Cases, which describe how the individual tests are carried out. Sections and Test Cases are Artifacts.
• A TestRail Test Run can contain Tests, which hold the results for a specific execution of a Test Case. Tests are Artifacts.
• A Test Plan is a set of Test Runs, for example, sequential runs for a set of alternate configurations of a product.
• A Milestone is also a set of Test Runs, for example, runs leading up to an event like a product release

See the TestRail domain model description in the Syndeia Cloud Web Browser/Help/API Overview documentation for more information.

The Syndeia_Cloud_3.5_TestRail notebook illustrates request and response for several questions, drilling down through the TestRail data model to retrieve data for specified artifacts at any level.

• What Projects are in a TestRail Repository?
• What Milestones are in a TestRail Project?
• What Test Suites are in a TestRail Project?
• What Test Sections are in a TestRail Test Suite?
• What Test Cases are in a TestRail Test Section?
• What is a TestRail artifact (Test Case) connected to (in other models)?
• What Test Runs are in a TestRail Project?
• What Tests are in a TestRail Test Run?
• What Test Plans are in a TestRail Project?
• What Test Runs are in a TestRail Test Plan?

For each section, we will identify the question being asked and what the results look like. More detail about the API method/Python code patterns used to make these queries are described in the accompanying TechNote(s).

Getting Started

All Syndeia documentation can be accessed most easily through the Help icon on the Syndeia Web Dashboard once Syndeia Cloud has been installed. In terms of thinking about these queries, it may be especially helpful to review the Syndeia data model under API – Overview. Intercax periodically offers on-line live training classes that provide a comprehensive review of all things Syndeia. Click here for more details.

Parts 1 and 2

Figure 1 TestRail Repositories available to Syndeia Cloud

As in the previous notebooks, the initial cells are devoted to

• Importing Python libraries, including the Syndeia Cloud Python language client,
• Authenticating into Syndeia Cloud through its own REST API, and
• Retrieving a list of repositories (TestRail repositories in this example) that this instance of Syndeia Cloud is set up to access.

The results for this list are shown in Figure 1. This query was made to the Syndeia database, not the TestRail repository, but we will need a TestRail Repository Key, REPO150 in this example for TestRail @ Intercax, in our next TestRail-specific requests. For more information on the Syndeia-related Python code in this section, see the TechNote, “Queries to Syndeia Cloud Database”. Link shared below.

What TestRail Projects are in a Repository?

Figure 2 Projects in TestRail @ Intercax repository (partial)

The cells labelled Step 3.1 asks, “What TestRail projects are contained in the TestRail @ Intercax repository?” Before this request could be passed to TestRail, the user is asked at run-time to enter their TestRail password, which is required along with their username for all requests. For the request, we used the TestRailApi class with the get_test_rail_containers() method. The results are tabulated as in Figure 2. This report shows the project names and project keys (IDs). Which will be used for later requests. For more information on the Syndeia-related Python code in this section, see the TechNote, “TestRail Authentication and Projects”. Link shared below.

What TestRail Milestones are in a Project?

In Step 4.1, we ask, “What are the Milestones in a specific TestRail project, “Unmanned Ground Vehicle”?”. The results are tabulated in Figure 3, showing each artifact with name, ID, description, is_started, and is_completed Boolean values.

Figure 3 Milestones in the “Unmanned Ground Vehicle” Project

Step 4.1 uses the get_test_rail_artifacts() method with the optional parameters container_external_key to identify the TestRail project, type_external_key to restrict the results returned to Milestones only, page to index the pages of results returned and include_attributes to return all Milestone attribute values. It also introduces a pattern used in many of the subsequent cells to handle pagination when the number of results (e.g. Milestones) returned is large. For more information on the Syndeia-related Python code in this and following sections, see the TechNote, “Navigating TestRail Artifact Structures”. Link shared below.

What Test Suites are in a TestRail Project?

In Step 4.2, the notebook searches for Test Suites in “Unmanned Ground Vehicle” using the get_test_rail_artifacts() method with the type_external_key to restrict the results returned to Suites only. Figure 4 shows the results, including description and three additional test suite attributes that were returned.

Figure 4 Test Suites in the “Unmanned Ground Vehicle” Project (partial)

What Test Sections are in a Test Suite?

Figure 5 Test Sections in “UGV
Functional Tests” Test Suite

The response shown in Figure 5 for Step 4.3 digs one level deeper, using parameter other_suite set to “UGV Functional Tests”. This suite’s key was retrieved in the previous section.

What Test Cases are in a Test Section?

Figure 6 Test Cases in “General” Section in “UGV Functional Tests” Test Suite

One more level down, the query from Step 4.4 whose response is shown in Figure 6 uses the parameters other_section set to “General” and other_suite set to “UGV Functional Tests”. These keys were determined in the previous sections.

What is a TestRail Artifact (Test Case) connected to (in other models)?

Figure 7 List of inter-model relations to or from artifacts in Figure 6

In Figure 7, we display a list of the Syndeia inter-model relations in the Syndeia container (digital thread project) “UGV” to or from the TestRail artifacts from the list displayed in Figure 6. Unlike the previous requests described in this post, this request is directed to the Syndeia Cloud database, not the TestRail repository, and it is made in the form of a Gremlin language query.

The query iterates through the list of artifacts and prints the table in Figure 7 with the relation id (Syndeia key) and the source and target artifacts. Each of the artifacts is represented by a link to the element in its native repository, Teamwork Cloud and TestRail, respectively, in this example.

What Test Runs are in a TestRail Project?

The next example of the get_test_rail_artifacts() method is shown in Figure 8. The request uses the project key for “Unmanned Ground Vehicle” determined earlier and the optional parameter type = “RUN”. Test Runs record the results of testing and Syndeia has returned some of those results values for Passed_Count, Failed_Count , etc.

Figure 8 Test Runs in the “Unmanned Ground Vehicle” Project

What Tests are in a Test Run?

Figure 9 Tests in the “UGV Functional Tests”

In Step 6.2, the notebook searches for Tests in the “UGV Functional Tests” Test Run using the get_test_rail_artifacts() method with the optional parameters other_run to identify the Run and type_external_key to restrict the results returned to Tests only. Figure 9 shows the results, including the priority and the status for the individual Tests (passed/failed/blocked/retest). The color-coding by Status of the tests was specified by the highlight() function defined in Part 1 of the notebook.

What Test Plans are in a TestRail Project?

The next example of the get_test_rail_artifacts() method is shown in Figure 10. The request specifies type_external_key = “PLAN” to restrict the results returned to Test Plans only.

Figure 10 Test Plans in the “Unmanned Ground Vehicle” Project

What Test Runs are in a TestRail Test Plan?

The next example of the get_test_rail_artifacts() method is shown in Figure 11. The request specifies a particular Test Plan by name, “Test plan for UGV Release 2”, retrieves its key from results generated in the previous section and displayed in Figure 10, and uses the optional parameter other_plan.

Figure 11 Test Runs in “Test plan for UGV Release 2”

Conclusions

In this Jupyter notebook, we have demonstrated how to navigate through the relatively complex data model for TestRail and obtain artifact attributes for many different artifact types.

Download TechNote(s).

Click here to download our Technotes.

Next in this series

The next set of posts in this series will use Jupyter notebooks as the primary means for demonstrating the Syndeia API and will focus on the following topics.

• Part 1 –  Introducing the Syndeia API
• Part 2 – Hello Syndeia Cloud Notebook
• Part 3 – Syndeia Cloud 3.5 JIRA Notebook
• Part 4 – Syndeia Cloud 3.5 Analytics Notebook
• Part 5 – Syndeia Cloud 3.5 Jama Notebook
• Part 6 – Syndeia Cloud 3.5 DOORS-NG Regular Projects Notebook 
• Part 7– Syndeia Cloud 3.5 DOORS-NG CM-Enabled Projects Notebook
• Part 8– Syndeia Cloud 3.5 TestRail Notebook (This Part)

Contributions by

The notebook was created by Dr. Manas Bajaj, Chief Systems Officer at Intercax, who leads product research and development. He was a contributor to the SysML (Systems Modeling Language) 1.0 open standard and leads the SysML v2 API and Services standards development as part of the SysML v2 Submission Team (SST).