Published May 22, 2024
Git integrations and CI/CD pipeline in Celigo
In the dynamic field of software development, integrating DevOps practices such as continuous integration (CI) is crucial, especially for enterprises utilizing iPaaS solutions like the Celigo platform (integrator.io).
For builders working with the Celigo platform, integrating with external source code versioning solutions like GitHub serves as a robust framework for storing integration code in a centralized repository. This setup not only facilitates seamless collaboration across various teams, including development and QA, but also aligns with the company’s broader DevOps best practices.
By using various tools provided by Celigo, such as different platform APIs to interact with components, organizations can manage their integration processes more effectively. These tools enable an automated workflow where code can be consistently managed and updated, enhancing both transparency and efficiency in development processes.
This article will cover several key areas to provide a comprehensive understanding of effective integration management using Celigo.
We will discuss:
- An overview of the integration lifecycle management features within Celigo.
- The critical role of external code versioning systems and why they are essential for maintaining code integrity and facilitating team collaboration.
- A detailed example of a CI/CD pipeline, demonstrating how to use Celigo’s API for downloading integration packages, using a GitHub repository for code storage, and employing GitHub Actions to automate workflows.
- Specific workflows include the repository dispatch event workflow, which triggers actions based on external events, and the release workflow, which handles the building of the deployable package after a merge operation with the base code.
This guide will provide you with a practical example to help you improve your integration processes by effectively following the best DevOps practices.
Overview of Celigo’s integration lifecycle management
Integration lifecycle management (ILM) is a crucial feature on the Celigo platform, providing users with a Git-style method for managing the software development lifecycle of integrations through operations such as push, merge, and commit.
Many Celigo users are already familiar with the “revisions” section, where you can create “snapshots” that act as backups. Builders can pull a specific revision, revert to a particular revision, and more.
Why use external code versioning?
Adopting an external code versioning system such as Git offers several distinct advantages for developers working with integration components in the Celigo platform.
Let’s uncover some of them:
Centralized code storage
It serves as a single source of truth for all integration code. This repository can be version controlled and audited, ensuring that changes are tracked and managed systematically. Such centralization is particularly beneficial for companies that include highly technical teams. Those less familiar with source control might prefer to use only the out-of-the-box ILM feature mentioned previously.
Multi-environment / multi-accounts deployment
Using a versioning system facilitates consistent deployment across various environments—including sandbox and production—and multiple accounts. This ensures that every team member, regardless of their environment, works with the same code version, which enhances the reliability and predictability of deployments.
Collaboration and testing
External versioning systems significantly boost collaboration among cross-functional teams, such as development and QA. They act as a staging area where these teams can efficiently share, review, and test code. However, it is advisable to avoid making substantial changes to the integration components outside of Celigo. This is because any changes made externally cannot be tested within the platform, posing risks to the integrity and functionality of the code. Despite this, such systems still play a crucial role in enabling seamless teamwork and improving overall project transparency.
This approach not only aligns with DevOps best practices but also reinforces the robustness of the integration management process.
CI/CD pipeline example using Github and Github actions
NOTE: While the example provided utilizes flows within Celigo, similar functionality can be achieved using code, scripts, or any CI/CD pipeline building tools.
The code for the workflows described below is accessible at the following GitHub repository: Celigo Public GitHub Integration.
General assumptions:
- The primary function of this pipeline is to act as a robust repository for version-controlled code storage.
- Our strategy focuses on maintaining all our integration code in the main branch of our repository.
- We will utilize snapshots created in integration lifecycle management (ILM)
- Our main goal is to ensure a new deployable release is available on GitHub every time code is merged with the main branch.
Celigo APIs
The Celigo platform offers a large set of REST APIs that can be used for different automations.
In today’s use case we will need 2 APIs:
| Relative URI | Method | Success code | Description | Pipeline automation |
| /integrations/<_id>/revisions | GET | 200 | Get all revisions for a specific integration. | We will retrieve the latest snapshot for our integration. |
| /integrations/<_id>/template | GET | 200 | Download an integration. | We will retrieve a signedURL that will serve us in our automation to download the snapshot. |
GitHub and GitHub actions
GitHub is obviously a leading code hosting platform that facilitates project management, version control, and collaboration across software development projects through repositories.. In this example, we will leverage GitHub to store various versions of our integration component package. Integrated within GitHub, GitHub Actions offers a powerful automation framework that enables the execution of custom workflows directly within our repository.
In our configuration, we maintain a dedicated repository on GitHub named “snapshot,” which is designated to store the most current versions of our integration code. We utilize two custom workflows:
- Dispatch event type workflow: This workflow is tasked with pushing our code to the main branch through a series of successive actions.
- Release creation workflow: This workflow is responsible for generating a release from our base code, ensuring that updates are systematically and efficiently deployed.
Dispatch repository event – code push to main branch:
The workflow script linked to this repository is activated by a ‘repository_dispatch’ event, which is set to trigger when a new snapshot of our integration is ready. Upon activation, the script undertakes several crucial tasks:
- Fetch latest snapshot: The script retrieves the latest snapshot from an S3 signedURL, ensuring access to the most recent version of the integration.
- Unpack and clean: It unpacks the snapshot and cleans up residual files from previous operations to maintain a clean working environment.
- Configure Git: The script sets up Git with the necessary user credentials to allow for changes to be pushed back to the repository.
- Sync with main branch: It ensures the local repository is up-to-date with the main branch, preventing any conflicts.
- Update snapshot directory: The old snapshot directory is replaced with the new one, incorporating the latest updates.
- Create and populate new branch: A new branch is created, and the latest code changes are populated into this branch.
- Commit and push changes: These changes are then committed, and the new branch is pushed to GitHub, making the updates available for review.
- Create pull request: Utilizing the GitHub CLI tool, the script creates a pull request to merge the new branch into the main branch.
- Auto-merge and cleanup: The script attempts to automatically merge the pull request and, upon successful merge, deletes the branch to keep the repository organized.
This structured approach not only ensures that the repository is continuously updated with the latest integration snapshots but also adheres to DevOps best practices for efficient and collaborative software development.
On merge – build release:
In the “On Merge – Build Release” section of our workflow, we automate the crucial process of packaging and releasing updated integration code once it has merged into the main branch. Triggered by any push to the main branch, this workflow consists of two distinct jobs: prepare and release.
Prepare job:
- Checkout: The repository code is initially checked out.
- Compress: The contents of the ‘snapshot’ directory are compressed into a zip file, which is labeled with a timestamp to ensure uniqueness.
- Upload artifact: This zip file is then uploaded as an artifact, preserving it for the subsequent release phase.
Release job:
- Checkout again: The code is checked out once more.
- Download artifact: The previously uploaded zip artifact is downloaded.
- Create release: A release is created via a GitHub API request, which dynamically generates a release tag based on the current date and time.
- Upload asset: The zip file is uploaded as an asset to this release, making it readily available for download.
This workflow ensures that each merge into the main branch not only updates the repository with the latest code but also packages this code into a downloadable format. It automates the delivery process, making the latest version of the integration immediately available for use.
Running the pipeline:
- As discussed previously, it all starts with our flow that will:
- Get the Latest Revision
- Get the SignedURL of the Snapshot
- Trigger the events dispatcher in github Actions
We are configuring the payload of the trigger by sending details needed about the snapshot:
JSON
{
“event_type”: “continuous-integration-event”,
“client_payload”: {
“name”: “snapshot-18”,
“date_creation”: “2024-05-06T18:33:08.709Z”,
“id”: “663922641f6a2ca13e7dc527”,
“signedURL”:
“https://integrator-templates.s3.amazonaws.com/5f7e39a76bb8d813180a9549.zip?
AWSAccessKeyId=AKIA4QUUOLHQGHVAXGHN&Expires=1715020634&
Signature=ZUQlRoiUmuYaKAa4VVPNqVCNZSg%3D”
}
}
2. Next, the dispatcher event workflow will launch the tasks configured in job:
3. The merge of the pull-request will then trigger the second flow that will build the release.
This workflow has 2 Jobs {Prepare and Release}:
4. Our base code will be then updated in the repo:
5. And as a result of the second workflow, we will have a release with as assets, the zip of our integration, ready for deployment:
Celigo offers robust, developer-friendly features that enhance your code’s integrity and bolster collaboration within your teams. By integrating with external code versioning systems and leveraging comprehensive CI/CD pipelines, Celigo empowers you to streamline your integration processes effectively.
Following the practical guide and examples provided, such as utilizing Celigo’s API with GitHub for code storage and automating workflows with GitHub Actions, you can significantly improve your operations and adhere to best DevOps practices. This approach ensures your projects are scalable, maintainable, and aligned with the latest industry standards for continuous integration and deployment.
