A Blind Spot in CI/CD: Database

Databases are at the core of many organizations’ IT applications. DevOps, GitOps, or other variations typically focus on software development and release controls. But what about our central databases?

Databases are the pioneers, housing the source of data. Therefore, they reside in a different place in the DevOps cycle compared to application code that is compiled and assigned a specific task. Test databases, for instance, cannot be merged with production environments in the same way as test code. This is because it would mean replacing real user or application data with test data, which the industry is not yet prepared for. When it comes to databases, the process works somewhat in reverse. We transition from testing to production by correlating live environment data and obtaining test environment data that closely resembles reality. This situation marks the first disconnect of the database from DevOps automation. Additionally, there are security risks and regulatory compliance concerns associated with accessing live environment data.

In the DevOps realm, where we have automated code control and review processes that do not consume much effort or time in the lifecycle of code, databases present a significant challenge. DBAs often manually review changes, which is a substantial time drain. Sometimes, to expedite the process or reduce effort, DevOps teams skip DBAs’ manual review and control steps. In a DevOps world where processes are becoming increasingly automated and accelerated, this may be somewhat understandable. However, unverified changes pose a serious integrity and continuity risk to the database. This situation represents another disconnect of the database from DevOps automation.

Let’s take a look at the lifecycle of an application. The project is defined in a version control system, and branches are distributed to teams. Developments are completed, and version information is uploaded to the source control system. Source code is added to the project, the project is compiled, a version is released, and testing begins. Unit tests, version tests, and user acceptance tests are carried out, and the application is released as an RC (Release Candidate). Now, let’s move on to deployment. The deployment phase, which is often the most challenging stage, is when the application comes to life in the live environment, and the probability of encountering an unexpected error is high. It’s often said that developers make some small mistakes. They say, “Kote opens every chest.” (A reference to “The Kingkiller Chronicle”) Our application is as robust as Ramston steel but equally fragile. Internal Server Error… It seems that there are some minor issues in the database after a new application or a new version is installed.

Conflicting queries, empty procedures, locks, cached tables, and performance issues due to configuration are possible database problems. Tables, schemas, views, procedures, functions, DDLs, DMLs pass before your eyes like a film strip…

Now, let’s think about what’s happening. In our source code, every change is recorded. Every branch update leaves a trace for every line of code. On the virtuous path to the master, every footprint is preserved. We can go back to the previous version, and the matter is closed. We forget everything and act like nothing ever happened. We express gratitude once again to the Holy Git. But will everything return to its previous state? What about the database, tables, schemas, procedures, what’s updated, what’s deleted, what’s new, what’s old, and the intricacies?

This is where the focus should shift to the central database. The sword of Damocles is over the application or code, but what about database changes? Reckless scripts running in the database? While the DevOps world meticulously tracks every line of code, every version, and every process, the database, unfortunately, has been left outside the DevOps realm. Despite organizations of all sizes adopting a CI/CD discipline, the database remains a blind spot in this process.

Unfortunately, databases continue to operate without enjoying many of the benefits of the modern DevOps world, such as real-world testing, version control, audit trails, reviews, or the ability to revert to a previous version. Version control for databases, aligning database changes with releases, and creating audit trails have become critically important for databases.

Therefore, databases, which are of paramount importance to our organizations, should find their place in the rapidly evolving and accelerating DevOps world. This is where the Database DevOps approach comes into play. Some organizations incorporate DBAs into DevOps, while others include databases in DevOps. Both approaches agree on the necessity of DevOps for the database. These include;

• • Database version control
  • Aligning database changes with application deployments
  • Maintaining records of database versions and changes
  • Live environment tests (Dry Run)
  • Ability to roll back to previous versions
  • More effective reviews and policies
  • Security

In today’s discussions on Platform Engineering, it seems increasingly difficult for databases to remain a blind spot. In this sense, organizations will need to adopt an approach that suits their needs for Database DevOps and bring these critical systems out of the dark corridors of CI/CD.

Amidst the debates between RDBMS and NoSQL, it’s said that self-managing ML-based DBMS is being discussed in the shadows of PARC temples. Indeed, does this technology ever tire?