Executing stress free Strapi version upgrades : My experiences & attempts

Note: This post is specifically about minor version upgrades with Strapi v4 (eg: 4.2 -> 4.5) and not Strapi v3 -> Strapi v4 upgrade. That’s a different beast altogether. 😊

Upgrading the Strapi version for a setup is necessary not only to bring the latest Strapi features to it, but also to keep the setup secure from the latest known vulnerabilities. Upgrading regularly also ensures the CMS setup doesn’t fall behind too much. I have known a few setups that haven’t been upgraded for 2+ years leading to upgrade paths riddled with complications.

But, Strapi v4’s track record with respect to version upgrades has been patchy. This post details such experiences & my attempts to overcome issues to enable more confident Strapi version upgrades.

My goal here has been: easier upgrades leads to more frequent upgrades.

The problem with Strapi version upgrades

If we look at Strapi v4 upgrades in the last year or so, there have been multiple instances where upgrading the Strapi version introduced bugs in our setup:

• With Strapi 4.5.x, populating relation fields when duplicating rows stopped happening (details).
• With Strapi 4.5.x, items in draft mode added as relation to published items started to appear in the GraphQL response.
• With Strapi 4.6.x, the Graphql query response would return only 10 items for related records. It now required pagination parameters in place to return further records.
• With Strapi 4.10.5, sorting by the provided attribute in the GraphQL query stopped working (details).
• With Strapi 4.10.6, the long text fields became read-only (details).

None of the above regressions were known in prior via the changelog / release notes. As a result, upgrading Strapi demands a thorough regression testing to ensure existing functionalities do not break. The time and effort needed for this verification becomes a point of friction with respect to undertaking regular version upgrades.

Someone echoed my experiences with respect to Strapi version upgrade on Strapi's github issues

What can break during a Strapi version upgrade

Two kinds of functionalities can potentially break when undertaking a Strapi version upgrade:

• Strapi Admin UI functionalities
• Rest API and GraphQL responses

Issues in any of the above categories can joepardize functioning of a CMS setup. But, unknown regressions in Rest API / GraphQL request-response can immediately lead to a malfunctioning frontend.

For example:

• Upgrading to Strapi 4.10.6 made our long text fields read-only. This blocked our content team from updating descriptions but they were patient enough to let us have a window to downgrade to 4.10.5 to address the issue.
• In contrast, when the draft-mode items started to appear in the GraphQL response after the Strapi 4.5.0 upgrade, it resulted in our website displaying a lot of non-production ready records (panic!). This meant we had to release an emergency patch (sleepless night 😢) that added a filter criteria to all our GraphQL queries (publicationState: LIVE for the relationship fields).

Based on the above experiences, any regression in the Rest API / GraphQL request-response is what worries me more when undertaking a Strapi version upgrade.

Setting up automated regression testing for API request-responses

To safeguard against any unknown regression in the Rest API / GraphQL request-response, we undertook the following setup:

Setup logging of API requests and responses:

We wrote a Strapi middleware that could log the GraphQL requests & responses to files:

const fs = require('fs');
const crypto = require('crypto');
const path = require('path');

//CAUTION : DO NOT ENABLE IN PRODUCTION.
module.exports = (config, { strapi })=> {
  return async (ctx, next) => {
    if (process.env.UPGRADE_REQUEST_LOGGING_ENABLE &&
      process.env.UPGRADE_REQUEST_LOGGING_ENABLE === "true" &&
      process.env.UPGRADE_REQUEST_LOGGING_BASEPATH)
    {
      if (ctx?.request?.url === '/graphql' && ctx.request.method === 'POST')
      {
        await next();
        const req = String(ctx.request.body.operationName) +  JSON.stringify(ctx.request.body.variables) + ctx.request.body.query;
        const hash = crypto.createHash('md5').update(req).digest('hex');
        const reqFilePath = path.join(process.env.UPGRADE_REQUEST_LOGGING_BASEPATH, hash+'-request.json');
        const respFilePath = path.join(process.env.UPGRADE_REQUEST_LOGGING_BASEPATH, hash+'-response.json');
        fs.writeFileSync(reqFilePath, JSON.stringify(ctx.request.body, null, 2));
        fs.writeFileSync(respFilePath, JSON.stringify(JSON.parse(ctx.body), null, 2));
      }
      else
        await next();        
    }
    else
      await next();
  };
};

We enable this middleware (in a non-production environment) to run before and after undertaking the Strapi version upgrade with different folder locations specified in the UPGRADE_REQUEST_LOGGING_BASEPATH.

This middleware logs the GraphQL requests and responses. Each request and response gets logged into a separate file with a name like <hash>-request.json and <hash>-response.json. The hash is formulated from the contents of a request. This is necessary because when we run this logging before & after the Strapi version upgrade, we need the file names to remain the same.

Once the logs are generated, a comparison of the two log output folders (before and after version upgrade) looks like below (via meld):

In case of no regression, all the files in the two folders would be identical. But, in case if the shape of response changes for any request, the diff tool would highlight such files (like in the screenshot above). Such cases would require us to compare, analyze and address the differences.

In the near future, we plan to write a script to compare the two log output folders and notify the concerned stakeholders in case of differences.

Leverage the frontend to trigger the API requests

To actually generate the log files detailed in the previous section, we need something to make the API requests. The most reliable way to do so is via the frontend that consumes these APIs.

For statically built site pages, we achieve this by simply running our frontend build process twice - before and after the upgrade and then comparing the generated log files.

For website pages that make the API requests at run-time (when the website user loads these page), we execute our website UI automation testing scripts with Strapi middleware logging enabled. Here, the quality of our API regression testing is affected by the coverage of these automation UI scripts.

Areas of Improvement

I realize that the wider the coverage of our automated regression testing setup, more confident we will feel while executing Strapi version upgrades:

• For non-static web pages, we seek to find ways to capture the API requests that can be replayed later during the Strapi upgrade process. We seek to do this with a goal of 100% coverage of the API requests the website makes to our Strapi setup.
• For the admin UI functionality, we will have to write easy-to-maintain automated UI test scripts. However, we are iffy about this because UI verification scripts can be fragile. And, in this case, since our team doesn’t control the changes to the admin UI - such scripts may prove maintenance heavy.

Conclusion

A solid automated regression testing setup helps catch the regressions while executing Strapi version upgrades. This reduces the friction of doubts in regularly executing Strapi version upgrades.

Having an automated regression testing mechanism for API requests-responses is more critical and also easier to implement. However, writing and maintaining the Strapi admin UI verification scripts can prove to be maintenance heavy. In fact, a Strapi community driven or a Strapi team driven common admin UI test suite may be easier to maintain and leverage by Strapi consumers.

Feedback

I recognize that our testing setup is a work-in-progress. As a result, I’d be glad to learn if you would like to exchange ideas on how such a testing setup can be improved for:

• better coverage
• easier maintenance
• improved automation

Punit Sethi

My tryst with Strapi:

Back in mid-2021, one of my clients was having issues with their in-house CMS. Despite me being their frontend architect, they trusted me to build their CMS solution. At this point, evaluation of various frameworks, approaches and products led me to Strapi. And, I ended up implementing Strapi CMS for their data requirements.

Fast-forward to now, I have worked with multiple orgs on implementing, upgrading & customizing Strapi setups based on their requirements.

Read my other posts on customizing Strapi.