Toolkit Testing Checklist

Introduction

This page contains a checklist of things to test, and questions to answer, before releasing a perfSONAR Toolkit. It can be submitted as a bug report in the event of difficulties

Identifying Information

• Tester (Name, Email):
• Test Date (Date):
• Version Tested (x.y.z):
• Software Type (ISO Fullinstall or Netinstall):
• Bundle Type (CentOS6, CentOS7, Debian7, Debian8):
• Machine Type (VM e.g. VMWare, 1G Memory or Server Make/Model, e.g. Dell 1950):
• Architecture (x86 or x86_64):
• Testing Type (Upgrade from a.b.c or Clean Install):

Download and Installation Questions

• _____ Did the provided MD5 Sum Verify against the download?
• _____ Were any errors seen during OS Boot/Installation?

New Installation First Login

• _____ Could you log in as root on the console with a blank password?
• _____ Did addressing the host (e.g. DHCP, Static) work as expected?
• _____ Were you prompted to create a web administrator on first login?
• _____ Were you prompted to disable root SSH and create a privileged sudo user on first login?

Host Checklist

Upgraded Installation

• _____ Could you use existing accounts to access the machine?
• _____ Were services you enabled (including SSH and NTP) available after the upgrade?
• _____ Did host addressing work as expected (e.g. DHCP, Static, Hostname)?

GUI/Display

• _____ If you got to http://myhost are you taken to the new web interface?
• _____ Are services running as expected?
• _____ Are the versions under Host Information Details consistent with the release?
• _____ Is additional host status information displayed in the upper right after you login?
• _____ Were there any rendering or formatting issues in your browser? If so please list the browser type, version, and parent OS you are using.
• _____ If you click the "This page is also available as JSON" link at the bottom of the page, do you see the host details in JSON format?

Logging

• _____ Click View to see service logs under list of services. Did the page ask you to authenticate before viewing? Were you able to authenticate with root or other administrator credentials (perfSONAR web administrator account)?
• _____ Are all logs available in the page which opened?
• _____ Is the page using https?

Databases

• _____ Is cassandra running (systemctl status cassandra)?
• _____ Is postgresql 10 running (systemctl status postgresql-10)?

Launching Configuration GUI

• _____ Click Configuration. Did the page ask you to authenticate before entering information? Were you able to authenticate with root or other administrator credentials (perfSONAR web administrator account)?
• _____ Is the page using https?

Configuration - Administrative Information

• _____ Are the labels, prompts, and pop-ups clear and informative?
• _____ Does the Host Information (i.e. organization name, address, contacts, etc) match what it was before the upgrade?
• _____ Is the list of communities the host participates in the same as before the upgrade?

Configuration - Host

• _____ Did your previous auto update configuration get preserved?
• _____ Did your previous NTP configuration get preserved?
• _____ Did any custom NTP servers you had added get saved?
• _____ Was NTP restarted after saving and does an ntpq -p show the new hosts?

Configuration - Enabled Services

• _____ Are the labels, prompts, and pop-ups clear and informative?
• _____ Does the list of which services are enabled match what it was before the upgrade?

Configuration - Scheduled Tests

• _____ Are the labels, prompts, and pop-ups clear and informative?
• _____ Did all your previous tests get saved, and are they listed?
• _____ Do the tests in All Scheduled Tests have all the host members that were being tested to in the previous version (check details for each test)?
• _____ Do the tests in All Scheduled Tests have the same test parameters as in the previous version?

Graphs

• _____ Do the list of configured tests appear in the graphs? Is the information correct?
• _____ When you click on a test under Test Results in the main page, does the graph display?
• _____ Does the graph discover your bandwidth and interface speed?

Traceroute Visualization

• _____ Does the page load as expected?
• _____ Can you select a source and destination using the drop down menus.
• _____ Are there any errors that appear on load or on viewing of data?

New Install Checklist

Host Information

• _____ Is the information in Host Information and in Details consistent with your server hardware configuration?
• _____ When logged as website administrator user (or click Log in for more info in Host Information section), does Health display correct system information?

Configuration - Administrative Information

• _____ Fill in all administrative information fields.
• _____ Add Node Role under Metadata.
• _____ Select Node Access Policy from available policies.
• _____ Add a community via Add a Community.
• _____ Add multiple communities from existing communities drop-down lis.
• _____ Does the community just added get removed from the existing communities list?
• _____ Delete a community via the x symbol.
• _____ Does the community get deleted from the list?
• _____ If you deleted an existing community from your communities list, does it get added back to the existing communities drop-down list?
• _____ Save the administrative information.
• _____ Did all the information get saved?
• _____ Is you service registered in the sLS? Check http://stats.es.net/ServicesDirectory/ after 10 minutes or so. Also note that private addresses may be ignored.

Configuration - Host

• _____ Are auto updates set correctly? They should be enabled in a clean install, or retain previous settings on an upgraded host
• _____ Click this link: "Click here for important information regarding automatic updates." Do you see more info about auto updates displayed?
• _____ Click the X in the upper-right corner of the auto update info modal. Does the modal close?
• _____ Toggle the Auto Updates setting.
• _____ Click Save ... did the page save cleanly?
• _____ Check that auto-updates are enabled/disabled on the host. Does it match how you configured it through the GUI? (systemctl status yum-cron)

Configuration - NTP Servers

• _____ Select and deselect from known ntp servers from a drop-down list in the servers window area. Click Save and check that /etc/ntp.conf contains those servers and that ntpd has been restarted.
• _____ Add a new NTP Server using the Manage available NTP servers option. Click Save and check that /etc/ntp.conf contains the newly added server.
• _____ Try to re-add an existing server. Should fail.
• _____ Click Select closest servers. After it has finished, no more than 5 of the servers should be selected. If any servers cannot be contacted, they should be marked in red, and have a tooltip when you move your mouse over them. Click Save, and verify that the /etc/ntp.conf contains the selected hosts.
• _____ After entering information, did the page save cleanly and does an ntpq -p reflect changes?

Scheduled Tests

• _____ Are you able to add a new throughput test and see results within the test interval?
• _____ Are you able to add a new one-way latency test and see results within a few minutes of adding it?
• _____ Are you able to add a new ping test and see results within a few minutes of adding it?
• _____ Are you able to add a new traceroute test and see results within the test interval?
• _____ Are you able to remove a throughput test and see that it is no longer collecting new data?
• _____ Are you able to remove a one-way latency test and see that it is no longer collecting new data?
• _____ Are you able to remove a ping test and see that it is no longer collecting new data?
• _____ Are you able to remove a traceroute test and see that it is no longer collecting new data?
• _____ Are hostnames, IPv4 addresses, and IPv6 addresses displayed correctly?
• _____ After entering information, did the page save cleanly?

Reverse Clients (traceroute, tracepath, ping)

• _____ Do the web pages load as expected?
• _____ Are you able to perform tests to remote and local targets?
• _____ Were any errors observed in either step?

Graphs

• _____ Does the list of configured tests appear under Test Results in the main page?
• _____ When you click on a test under Test Results in the main page, does the graph display?
• _____ Does the graph discover your bandwidth and interface speed?
• _____ Are the labels, prompts, and pop-ups clear and informative?
• _____ Is the right source host data displayed in graph header?
• _____ Is the right destination data displayed in graph header?
• _____ Click Host info and see if data is available and correct?
• _____ Does clicking on Throughput (TCP), Throughput (UDP), Loss (Throughput), Loss (Latency), Latency or Ping buttons toggle the right graphs on and off?
• _____ Does clicking on Forward, Reverse or Failures buttons toggle the right graphs on and off?
• _____ Are you able to select different Report ranges and see changing graph results within different interval?
• _____ When you click arrows next to Report range, does the graph display shift the interval?
• _____ Does hover box appear when you point your pointer over a result line?
• _____ Are the labels and data in hover box clear and informative?
• _____ Do the labels and data in hover box correspond to the graph line in terms of value and time point?

Command Line Tools

• _____ Does OWAMP on the command line to a server function as expected? e.g. owping lbl-owamp.es.net
• _____ Does traceroute on the command line to a server (use one at Internet2 or ESnet) function as expected?
• _____ Does tracepath on the command line to a server (use one at Internet2 or ESnet) function as expected?
• _____ Does ping on the command line to a server (use one at Internet2 or ESnet) function as expected?

pScheduler

• _____ Does the command pscheduler troubleshoot complete without finding any problems?

pSConfig

• _____ Does the command psconfig remote list show the URLs that were configured prior to upgrade (or no URLs if this is a fresh install)?
• _____ Does the command psconfig pscheduler-stats complete successfully?

Documentation

• _____ Were you able to find the right documentation wrt tested functionalities?
• _____ Are the FAQ and documentation up-to-date wrt tested functionalities?