Syndeia pipelines can be used to automate virtually any digital engineering workflow, not just those calling the Syndeia REST API. Users can observe run history and task durations inside the Syndeia Web Dashboard to evaluate performance and reliability.
In the accompanying video, Gregory Seeds, senior software development engineer at Intercax, demonstrates a digital pipeline that queries a database, generates a report and notifies the user on completion. Our example here includes two tools, Apache Hive and Microsoft Teams notifications, not shown in our earlier posts.
Hive is a data warehouse tool run in distributed environments for large-scale data analytics. The pipeline uses Hive’s own API and HiveQL query language to extract parts inventory data and Syndeia’s Confluence integration to create a report. On completion, the pipeline creates a notification in Teams, a widely used collaboration platform, using standard Airflow functionality, similar to the Slack notifications generated in previous examples.
Video Demonstration of Syndeia - Digital Pipelines with Apache Hive, Confluence, and MS Teams
Datalakes and big data repositories such as Apache Hive often consolidate data from disparate legacy tools that do not offer APIs or network connectivity. Syndeia makes it possible to connect datalakes as part of a federated digital thread in your digital engineering environment. Future blog posts will show advanced report generation with multiple HiveQL queries, historical evolution and comparison of data in Hive, and live digital threads between Hive and MBSE, PLM, ALM, and other digital engineering repositories.
This blog post is the 6th part in a series of posts and demonstrations that we will share, focusing on automating digital engineering workflows and use cases with Syndeia Pipelines.
Pipelines Part 6 - Applying Syndeia Digital Pipelines to Apache Hive-Confluence- MS Teams (This Post)