Many clients, with the introduction of cloud services, still opt to deploy and maintain their own solutions. This choice is driven by various factors, including security requirements and cost-saving goals.
At first glance, self-maintenance of these solutions may appear relatively straightforward. For instance, one of our clients decided to implement their own NGINX node cluster to handle load balancing on their backend cluster.
The NGINX server cluster seemed like an uncomplicated solution to configure. However, as the workload increased, NGINX began struggling to process the incoming traffic effectively. Upon reviewing the log journal, an evident cause emerged: there was an insufficient number of available worker_connections, resulting in packet drops.
What surprised everyone was that the combined total of worker_connections multiplied by the number of worker processes was four times greater than the number of client requests, which should have been more than sufficient.
At this stage, the client turned to our company to diagnose this issue. Through our diagnostic process, we discovered that NGINX possesses architectural peculiarities that lead to an uneven distribution of user requests among worker processes. As a result, one particular worker process experienced a higher load, depleting the reserve of worker_connections.
The client was both astonished and appreciative of these findings, and they valued our technical investigations.
To achieve the desired goals, the following solutions were implemented:
Integration of Kustomize: The Kustomize templating engine was integrated into the project. Deployments were refactored, and basic deployments were created for each service. Moreover, specific overlays were developed for each environment. This restructuring improved the understanding of the project and simplified code maintenance.
Dynamic Secrets Management: Bash scripts were introduced to dynamically retrieve secrets from a Secret Manager. This approach replaced the hardcoded secrets in the deployment configurations, enhancing security and simplifying secret access.
Centralized Configuration: Code refactoring was performed to ensure that variables with the same purpose and value in services were consolidated into a single location. This centralization reduced complexity and improved maintainability.
Pipeline Transformation: The entire deployment pipeline was rewritten to utilize Kustomize. A Continuous Integration (CI) pipeline was established, including manifest validation and testing against the Kubernetes API server schema. This streamlined the integration of new code versions, ensuring compatibility and simplifying the deployment process.
Robust Continuous Deployment: Continuous Deployment (CD) was organized with post-deployment testing using the rollout status command. In case of failed deployments, an automatic rollback mechanism was triggered to revert to the previous version, increasing deployment reliability and ensuring seamless recovery from unexpected errors.
The customer was looking to provide a modern and user-friendly mobile banking experience for their customers.
The project team employed a DevOps approach to build the mobile banking solution. This included automating the deployment of the application, integrating automated testing into the build pipeline, and deploying the application to multiple cloud environments. The team also implemented a continuous delivery pipeline with automated QA to ensure the application was fully tested and ready for production.
The successful DevOps approach allowed the customer to rapidly iterate on the mobile banking solution, allowing them to get it to market quickly. This allowed them to launch the mobile banking solution with confidence that it was stable and secure. The customer was also able to gain insights into usage and performance, which enabled them to make improvements to the customer experience.
Reworked ansible scripts for creating Jenkins-master containers to organize multi-threaded configuration changes, which reduced the downtime for updating and maintaining 200+ Jenkins-slaves containers by 90%.
Using devops practices, we created template pipelines for builds using gradle, yarn, python and .net, integrating code tests and sonarqube checks into them, which made it possible to bring them to uniformity and reduce build errors for existing and new services and, accordingly, produce stable releases.