DynamoDb schema migration made easy!
Introduction:
In this article, we will be exploring an open-source library “dynamo-data-migrations” which was built by our team in Technogise with the main purpose of automating and managing database schema migrations w.r.t AWS DynamoDb. We will kick off the article by first exploring what database migration is and why we need it.
Let’s get started!!!
What is database migration:
Initially the term “database migrations” was used to refer mainly to the process of migrating data from one or more databases(source) to another database(target).
However, over a period of time the term “database migrations” was also used to refer to “schema migrations”, “database schema migrations”, or simply “migrations”. All those terms refer to a controlled set of changes developed to modify the structure of objects within a database. Those changes help transition database schemas from their current state to a desired state, which could involve adding tables or columns, removing elements or changing types and constraints.
“Migrations” manage incremental changes to data structures in a programmatic way giving the ability to also rollback the changes if required.
Why do we need migration tools?
Today, “Agile ‘’ is fast becoming a de-facto standard for software development. With agile adoption, comes evolving requirements and continuous changes. Your database schema too evolves along with your application. There is a need to have a way to apply schema changes carefully along with the ability to maintain history, and perform rollback.
Any changes to your database schema will also in most cases involve changes to your application code, which implies the need to bundle schema changes along with application code base and to treat them as a single atomic unit.
Using migration tools or libraries we can manage schema changes programmatically and couple them with accompanying code changes, which provides us with various advantages, a few being:
- They help developers plan, validate, and safely apply schema changes to various environments.
- Schema changes are automated and integrate seamlessly with CI/CD setup.
- It helps construct a history of modifications to the database.
What are the tools available for database migrations?
There are many tools available in the market that help you address your schema migration concerns, the most popular of them being “Liquibase” and “Flyway”. Both of them have very good documentation and have been adopted on a large scale. Both of these tools and many other such similar tools provide a way to manage schema changes programmatically, have them executed as part of the build pipeline as a single build process.
Get Technogise’s stories in your inbox
Join Medium for free to get updates from this writer.
What is the problem that we are trying to address?
With so many third-party tools already available for database migration, why did we feel the need of a new library?
The main challenge that we faced is that most of those tools are designed to handle schema changes of a relational database, and not of a NoSQL database.
But then “NoSql databases” are supposed to be “schemaless”, right? Not exactly, NoSql databases are not “schemaless”, in fact, they are designed to have “flexible schema”. NoSql databases do not enforce rigid schema structure but allow your data to follow a certain schema “loosely”. Without a database migration tool for NoSql we are stuck with doing those changes manually, and in doing so we de-couple our db changes from our application changes when actually both are dependent on each other. Also creating and maintaining migration scripts becomes a pain when it is not integrated with version control.
Apart from schema migration, there could also be scenario(s) where you need to populate a table with some static data or move/copy data from one table to another within the same database. Doing all this manually makes it error-prone, and there is no place to track those changes.
There are open source libraries that provide a way to handle migrations specific to a type of NoSql db, like for MongoDb we have“migrate-mongo”, however there is no decent library that currently exists for AWS DynamoDb. This led us to create an open-source db migration library specifically for AWS Dynamodb, which would help us overcome our migration challenges. Let us go through some of the salient features of the library in the next section.
Features of “dynamo-data-migrations”
- The library is written in TypeScript, however, it also supports JS(both “commonjs” and “esm” type modules).
- It creates a template migration file with “up” and “down” commands where you need to specify your installation and rollback steps respectively.
- Allows you to rollback last applied migration or last “n” applied migrations.
- Gives the ability to execute the script against multiple profiles/environments.
- It creates a “MIGRATIONS_LOG” table in your DynamoDb account to keep track of what all files are installed and what are pending.
- Can be easily integrated into a NodeJS project and all migration files can be tracked and maintained in your code repository.
- Ensures a smooth schema and data migration process.
The library is released and available for download from npm, and the latest version is 4.1.3.
The GitHub repository is at: https://github.com/technogise/dynamo-data-migrations
You can browse the same in npm on this link:https://www.npmjs.com/package/dynamo-data-migrations
Feel free to give the library a try and in case you have any feedback, you can drop a note in the comments section. You can report any issue that you faced while integrating the library or any new feature that you would like to have added to the library in the below link:https://github.com/technogise/dynamo-data-migrations/issues
