Data Migrations
MUSE had a terrible database schema from it's inception.
I say this because
I say this because
- There was no relational mapping between the objects
- No foreign keys or cascading was used when entries were removed
- The entries for songs were "HARDCODED" as comma-separated string IDs so if a song entry was deleted then it would cause unforseen exceptions on fetching other data which was related to the ID
- There was no data redundancy measure, so every new request would just dump the songs in the db without checking if it already existed or not.
All of these issues were covered by writing altering the schema by writing migrations for models to develop a new schema that xovered all the issues mentioned above.
During the testing of the app almost 8000 songs and their corresponding lyrics were fetched and scraped so dropping all that was never an option. So I decided to write data migrations to get rid of redundant data and save data based on the new schema.
During the testing of the app almost 8000 songs and their corresponding lyrics were fetched and scraped so dropping all that was never an option. So I decided to write data migrations to get rid of redundant data and save data based on the new schema.
Setting up the CD pipelines
During the CD setup for production all the data from local development using SQLite had to be ported to PostgreSQL running inside a container which was achieved by dumping and loading data using Django management tools.