Blog

One of the key concepts to understand when using the WebsiteUpgradeTester.com tool when testing your website upgrade is the notion of runs.

The tool allows you to create three different types of runs: a Baseline Run, a Comparison Run and a Combined Run.

We just posted this new video tutorial, that outlines clearly the differences between each type and when it's appropriate to create a run of that type.

Video Transcript

Welcome to Website Upgrade Tester.com, the new cloud-based tool that helps you test any upgrade, service pack or security patch to your website.

In this short video, I'm going to walk you through performing Baseline, Comparison, and Combined runs, when testing your upgrade.

But before we jump right in and show you the process live, let me quickly set the stage.

There are three (3) types of runs you can perform:  Baseline, Comparison and Combined.

Baseline Run

Baseline Runs takes digital snapshots of all the pages specified in your Page Set, for the specified device sizes, against the 'before', or pre-upgrade version of your site.

A Baseline Run is always generated first. Its screenshot captures will be the baseline against which the comparison captures will occur.

Comparison Run

Comparison Runs are typically run against the 'after', or post-upgrade version of your site, and they take screenshots of all pages for all the device sizes specified in an associated Baseline run.

In addition, it performs pixel-by-pixel screenshot comparisons against the Baseline snapshots to see if the two versions are identical or different. The Comparison run reports those results to you.

It's worth noting, that you can run a Comparison Run against the pre-upgrade version of your site, to help identify dynamic content. This is recommended if you are upgrading in-place on a single server.

Combined Run

Lastly, Combined Runs are kind of a combination of running both a Baseline and Comparison run.  They run two snapshot captures and one snapshot comparison.

The snapshots are typically taken against the pre-upgrade and post-upgrade versions of your site, but it is possible to configure the Combined run to do both snapshots only against the pre-upgrade version, in order to find dynamic content.

Run a Baseline Run

OK, let's show you these in action.

Clicking the second icon in the left-side menu opens the Baseline Runs page.

Then click the 'New Baseline Run' button.

This opens the dialog where you:

• Simply give the run a name.
• Specify the Page Set to run against.
• Choose the screen sizes that you want to test.
• And specify the root URL for your server.

You can also specify a login to use under the Authentication tab if you want to test the site as a specific user.

You can also, optionally, schedule the Run to happen in the future. We’re going to run it right away, so we’ll skip the scheduling.

After a short period of time your run will start processing, and you will begin to see results coming back.

The Baseline Run is just performing a single task, capturing the screenshots of the pages before the upgrade, which is represented by the single doughnut chart.

Since we chose 4 device sizes, the tool will need to process over fourteen hundred (1,400) baseline captures.

Run a Comparison Run

To create a Comparison run, simply click on the third icon in the left side menu, and then choose 'New Comparison Run'.

In the resulting dialog you simply specify a name for the run and choose the Baseline to compare against.

The tool will use the same Page Set and list of device sizes from the baseline run.

You can also optionally specify a login and/or schedule the run to start in the future. We’ll run it now.

After a short time you will see the run beginning to progress. This time you will see two doughnut charts.

The first shows the status of the screenshot captures, and the second the status of the pixel-by-pixel image comparisons.

Any images that are identical are marked SAME, otherwise DIFFERENT.

Run a Combined Run

A combined run takes digital snapshots of each page specified in the Page Set, against both the 'before' and 'after' servers

The screen shots against the 'before' server are referred to as Baseline Captures, and the screen shots against the 'after' server are referred to as Comparison captures.

Here’s how you create a Combined Run.

The first two icons allow you to see the results for Baseline and Comparison Runs. Since a Combined Run is the combination of the two, you can open up either and click the 'New Combined Run' button.

This opens the dialog where you:

• Give the run a name
• Specify the Page Set to run against
• Choose the screen sizes that you want to test
• And specify the root URLs for your two servers

After a short period of time your run will start processing and you will begin to see results coming back.

The Combined Run is doing three tasks, represented by the three doughnut charts.

First it takes a Baseline capture of the page pre-upgrade, then a Comparison capture of the page post-upgrade, and then it digitally compares the two snapshot captures, pixel-by-pixel, to see if anything changed. If nothing is different it is marked as SAME, otherwise DIFFERENT.

The advantage of running a Combined Run, versus running a full Baseline and then a full Comparison, is that you can see the results page-by-page as the Combined run progresses. Any time along the way if you start to see lots of differences you can stop the run, inspect the differences, resolve any problems and then restart the run.

Run Results

After your run completes, you can view the results in one of two ways.

You can either access the Baseline or Comparison Results pages by clicking on either the second or third icon in the left-side menu, or you can open the Runs Dashboard page.

Baseline or Comparison Results pages show a similar listing of your runs.

Here you can see the Comparison Runs Results page.

It shows:

• The Date and Time the run was started
• The Name, and the page and device size counts
• And then a group of numbers representing the results for the Screenshot Captures:
  • Total Pages
  • Success
  • Failures
  • Pending
  • Modified

In the next column you see two counts:

• The total Number of JavaScript errors across all pages.
• And the total number of pages that have at least one JavaScript error.

Then the next column shows the stats for the pixel-by-pixel Screenshot Comparisons:

• Total Compares
• Success
• Failures
• and Blocked
• Then counts of pages that are Same and Different

The second to last column shows the number of run iterations that have taken place.  Each time you stop and restart a run, or re-run certain pages it is counted as a run iteration. You can drill into these to see the specifics.

And lastly, you have your action columns, where you can:

• Rerun or continue a stopped run
• Edit the properties
• View more detailed stats
• Or delete the run

If we drill into the Run Details you will see more specific counts for the different types of errors.  

For example, redirects, request timeouts, or other Internal errors, etc.

You can also quickly view your latest four (4) runs and their detailed results, on the 'Runs Dashboard' page.

To access this simply navigate to the Console Home Page and click the 'Open Dashboard' button under the Runs Dashboard pod.

This opens a new page showing a carousel of the last four runs with details below.

For example, you can see the results of a Combined run here side-by-side.

The first column has details of the Baseline Screenshot captures portion of the Run. The second column the Comparison Screenshot Captures and the last column the pixel-by-pixel Screenshot Comparison.

Clicking any of the detail items will drill into the results, showing just the pages for that item.

There you have it.  Those are the 3 types of runs you can perform in testing your upgrade.

Thanks for watching this video on Website Upgrade Tester.com. Be sure to check out our other videos for more detailed information.

---

WebsiteUpgradeTester.com is the only cloud-based, automated, visual testing tool designed to help you efficiently and affordably test your website upgrade. Whether it's a major version upgrade, service pack or security patch, you can quickly test your entire site, no matter how large, to uncover any lurking issues due to the upgrade.

Don't waste time manually testing your next Website upgrade. Visit WebsiteUpgradeTester.com and sign up for a free trial.

For more detailed information about best practices on upgrading your website and improving your website maintenance processes, we recommend reading The Ultimate Guide to Upgrading and Patching Your Website.