In Part 14 of this series, we developed a custom script to calculate project metrics for our Autonomous Vehicle digital thread project. The value of this information is greatest if it can be run regularly so that trends in the project can be tracked and potential problems identified early. This kind of repetitive work is familiar from the DevSecOps approach to software engineering and can be automated through the use of digital pipelines.
The Syndeia goal in creating and running digital pipelines is to generate and validate the pipelines in any of several standard DevOps tools (Jenkins, GitHub Actions, Airflow, etc.), then to run the pipelines from within Syndeia. The pipeline developer can use a tool with which they are familiar and then distribute the pipeline to a broader base of Syndeia users for routine application. In the upcoming Syndeia 3.7 release, this objective will be met for Airflow. Pipelines using Syndeia REST API endpoints generated on other platforms can be run on those platforms.
Figure 1 Syndeia Web Browser, Digital Pipelines. Project Metrics Jama-SysML, Task Structure Graph
A digital pipeline is composed of a series of linked tasks that together carry out the purpose of the pipeline. Figure 1 shows the task structure as a graph for the calculation of digital thread metrics for the Jama-SysML relations in the Autonomous Vehicle project, the same calculation carried out in a Jupyter notebook in Part 14. This screenshot is from the Syndeia Web Dashboard in the upcoming 3.7 release.
Figure 2 Airflow UI, Python Task Structure for Calculate_project_metrics Task
For this example, we have built the pipeline in Apache Airflow, a widely used open-source workflow platform. Figure 2 shows the task structure on the left and the Python code for a single task, Calculate_project_metrics, on the right. This Python code has been taken directly from the Jupyter notebook in Part 14 with only minor modifications to define Airflow task structure and the passing of variable between tasks. The Jupyter notebook provides a convenient interactive development and testing environment and the code transfer is made easy by the cell structure of the notebook. The modified code in the digital pipeline can be tested in the Airflow environment before loading into Syndeia. We can see green bars in columns indicating seven successful runs.
Figure 3 Syndeia Web Browser, Digital Pipelines. Project Metrics Jama-SysML, Pipeline Runs Log
Figure 3 shows the digital pipeline interface in the prototype Syndeia 3.7 release, where the Jama-SysML metrics pipeline has been uploaded to Syndeia Cloud. It has been scheduled to run daily and it can be run on demand as well. A log of the runs is shown, as in the Airflow interface and, at the bottom, a plot of the task durations. The longest task is extracting a list of Jama requirements in a hierarchical structure in the Jama repository, which takes about 35 seconds in this case. The Runs summary at top right shows that the total pipeline run time has been in the range 44-47 seconds for a relatively simple example, affirming the need for a way to run this analysis automatically in the background, rather than on-demand. The pipelines stored on Syndeia Cloud can also be executed through the Syndeia REST API.
Figure 4 Slack Metrics Notification for the Jama-SysML Connections in the Autonomous Vehicle Project
When reports are being generated in the background by digital pipelines, users need to be notified when a run is complete and to be able to access the run results. Airflow libraries make it easy to create a Slack notification or a Confluence page at the end of each run. Figure 4 shows project metrics for the latest run as a Slack notification.
In the next post, we show how some of the same Python code can be re-used again to create a custom dashboard combining real-time and historical product information from the project on demand.
Gregory Seeds, Senior Software Engineer at Intercax, was primarily responsible for the digital pipeline shown here.
In the following blog posts in this series, we will demonstrate some of these current capabilities and offer a few glimpses of the future
- Part 11 - Model-Based Systems Engineering for Autonomous Vehicles – Digital Threads
- Part 12 – An introduction to Digital Thread Projects, Relations, Baselines and the Syndeia Web Dashboard
- Part 13 – Reports, Graph Analysis and the Digital Thread Explorer
- Part 14 – Open REST API for scripts, Jupyter notebooks and project metrics
- Part 15 – Digital Pipelines (This Part)
- Part 16 – Custom Dashboards (To be released)
- Part 17 - Digital Threads for Agentic AI (To be released)
Missed the earlier parts? Check them out here.
To evaluate Syndeia with your own toolset, or just to discuss your requirements and use cases, send your questions and requests to www.intercax.com/help and let us help you adopt best practices in Digital Engineering.