diff --git a/README b/README deleted file mode 100644 index 639d684a..00000000 --- a/README +++ /dev/null @@ -1,243 +0,0 @@ - - Testing Apache with the Perl Test Harness -Prerequisites -------------- -These two modules must first be installed; - -- perl-ExtUtils-MakeMaker -- perl-Test - -You'll need to install the CPAN modules listed in: -Apache-Test/lib/Bundle/ApacheTest.pm -All you have to do to install them all in one shot is: -perl -MCPAN -e 'install Bundle::ApacheTest' - -Which are also available in one tarball here: -http://perl.apache.org/~dougm/httpd-test-bundle-0.02.tar.gz - -Note: Crypt::SSLeay requires OpenSSL to be installed (only required -for t/TEST -ssl): http://www.openssl.org/ -More accurate results may be obtained by using the same openssl command -line and libraries as consumed by APR-util and mod_ssl, due to X509 -formatting behavior differences. - -For an extensive documentation see -http://perl.apache.org/docs/general/testing/testing.html - or -http://svn.apache.org/viewvc/perl/modperl/docs/trunk/src/docs/general/testing/testing.pod - -To run the tests for all Apache web server modules, some additional -CPAN modules will be required. If the tests don't work, make sure -that you have up to date versions of each of these perl modules: - -``` -cpan App::cpanminus -cpanm Bundle::ApacheTest \ - HTTP::DAV DateTime Time::HiRes \ - Test::Harness Crypt::SSLeay Net::SSLeay IO::Socket::SSL \ - IO::Socket::IP IO::Select LWP::Protocol::https AnyEvent \ - AnyEvent::WebSocket::Client LWP::Protocol::AnyEvent::http FCGI -``` - - -Quick Start ------------ - -If you don't care how it works and just want to run the tests, here's -how you go about doing that. - -1. You need an installation of Apache. (1.3.x thru trunk) -2. Any DSOs you wish to use should be configured in that Apache's - httpd.conf (the test harness will pick this configuration up) -3. Setup: - perl Makefile.PL -apxs /path/to/apache/bin/apxs -4. Run the tests: - t/TEST -5. Evaluate test output. - -Getting a little deeper ------------------------ - -The test harness will run every .t file under the t/ directory. So -let's say you only want to run the tests for PHP. Do this: - t/TEST -httpd /path/to/apache/bin/httpd t/php - -That will start the test server, run the .t tests under t/php and shut -down the test server. You can also control each of these steps. - -This will start the test server: - t/TEST -httpd /path/to/apache/bin/httpd -start - -This will run the PHP tests in the test environment: - t/TEST t/php - -This will stop the test server: - t/TEST -stop - -This will run the server under gdb (using -X): - t/TEST -d gdb - -Note: At this point, you have a working test environment. You can -look in t/conf for the test server configuration files. These are -generated by the test harness. Once you have a working test -environment, you do not need to specify 'httpd' on the t/TEST command -line. For instance, to start the server up again, the command - t/TEST -start -would be sufficient. - -Running Regression Tests ------------------------- -For a full regression test, you should have all modules loaded. Build the server -with - configure --enable-modules=reallyall --enable-load-all-modules ... -among other things. Edit the generated httpd.conf and comment all mpm modules -that you do not want. Run "t/TEST -clean" again. - -You will see some - skipped: cannot find module 'XXX' -as not all modules are in every apache release (but the tests run for all). - -All in all, some >4k tests will run and the result needs to be: PASS - -Trouble Shooting ----------------- -If you have a "PASS" at the end of "t/TEST", congratulations! If not, this -sections gives some advise in order to find the cause. Feel free to expand -this to make life easier for others. - -0. If your test startup hangs forever in "waiting for server to warm up", but - the test server is reachable under port 8529, you might be the victim of - ipv4/6 confusion. The default servername configured is "localhost" and - some operating systems define 127.0.0.1 *as well as* ::1 in /etc/hosts. - If the test server listens only on 0.0.0.0 it might not answer requests to - ::1 and that causes the test startup to hang. - Solution: comment the ::1 line in /etc/hosts and see if that improves matters. -1. Run "t/TEST -clean" every time you change something in your Apache - configuration. The test suite picks up certain things from your installed - httpd.conf (such as LoadModule statements) and will not see your changes - unless you clean it. -2. Failures in proxy.t may originate from the fact that the test script cannot - open the specified port. This happens on some machines if you abort a test - run and the socket is not properly shut down. Check if the error goes - away after a reboot. (proxy.t tests are slow, so chances you interrupt tests - at that point are good.) -3. Failures in access.t may result from reverse lookups not working or giving - other answers than expected. In the cause 0 above, if the test client - connects via 127.0.0.1, a "Grant for localhost" might resolve to "::1" - and therefore will not match the access rules of the tests. - Solution: check that your servername is 'localhost' (which is - the default) and that it *always* resolves to 127.0.0.1. -4. If some ssl test cases fail, especially when t/ssl/proxy.t fails, the - reason can be mismatches between your installed SSL library and the one - httpd uses. The "openssl" binary found in your $PATH is used to create - binary setup files by t/TEST. If another version of openssl then tries - to read these from your Apache server process, it might fail. - Try the following: - > t/TEST -clean - > PATH=:$PATH t/TEST - If a lot of ssl tests fail, check in the error log for the presence of - a certificate validation error. If you find it, check the expiration date - of the TLS/SSL certificates used by the tests, they might be expired. - Running TEST -clean should delete the old ssl certificates, so they'll be - regenerated during the next run. -5. If you see failures in the modules/h2.t test cases, please notify the dev - mailing list with a description of your setup. These tests are quite young, - currently only valid in 2.4.x and later and interact with quite some other - modules as well as Openssl versions installed. Some tests require mod_ssl - so make sure to load it in the httpd conf. -6. Segmentation faults and core dumps occurring while executing the test suite - might indicate a real problem but always run again the tests after - a clean make install to avoid inconsistencies from old objects. -7. If you see error messages like "Parse errors: Bad plan. - You planned X tests but ran Y." it usually means that you are missing - a perl module or the tested httpd module depends on another one - not loaded in the httpd config. -8. If you see SSL certificate errors, remove t/conf/ssl/ca prior to - t/TEST -clean -9. perl 5.28 in MacOS homebrew seems to hang the test suite. Invoking - /usr/bin/perl Makefile.PL -apxs ... will cause an older perl to be used. - -Smoking Tests -------------- - -Sometimes it's possible that the test is passing properly for the -first time, when it's run for the first time in the thread. But when -you run it again, the test might fail. It's important to run the -repetition smoke testing. For example to repeat the tests 5 times you -can run: - - t/SMOKE -times=5 - -It's also possible that a test will pass when it's run after a -particular test, but if moved to run after a different state it may -fail. For this reason by default the tests run in random order. - -Since it's important to be able to reproduce the problem with the -random testing, whenever -order=random is used, the used seed is -printed to STDERR. Which can be then fed into the future tests with: -via APACHE_TEST_SEED environment variable. - -By adding the option -order=repeat, the tests will be run in -alphabetical order. - -Combining these two important smoke testing techniques, one can run -tests with: - - t/SMOKE -times=N -order=(repeat|random) - -For example, to run the mod_rewrite tests 5 times, one would: - - t/SMOKE -times=5 -verbose t/modules/rewrite.t - -So the tests can be repeated N times, and run in the following three -modes: - -- randomize all tests -- repeat the whole tests suite N times - -For configuration options and default settings run: - - t/SMOKE -help - -For more information refer to the Apache::TestSmoke manpage. - - -Test Environment Configuration ------------------------------- - -The test server is configured with conf files like any normal Apache -server. The tricky part is those conf files are generated by the -harness just prior to starting the server. t/conf/httpd.conf is -generated by t/conf/httpd.conf.in. If that does not exist, the -harness will generate a working configuration and will include -LoadModule (and AddModule for Apache 1.3) directives from the -httpd.conf associated with the httpd binary you are using for testing. -If t/conf/extra.conf.in exists, t/conf/extra.conf will be generated -from that, and an Include directive for that file will be put in the -generated t/conf/httpd.conf. t/conf/apache_test_config.pm is -generated from the test configuration. It contains all the -information about the configuration of your test server. You can -access this information in test scripts by: - my $env = Apache::TestConfig->thaw; -Apache::TestConfig access apache_test_config.pm and returns a hash -reference with all the information. Look through -apache_test_config.pm, it's a lot of stuff. Once these conf files are -generated, you have a working test environment, and they must be -'cleaned' if you wish to make changes to them. To clean the -environment: - t/TEST -clean -(Now you will have to specify your httpd binary when starting back up -again.) - - -More Information ----------------- - -For more information on using the test harness and writing tests, see -the README in Apache-Test and the examples in Apache-Test/t. - -The test harness was originally written by Doug MacEachern and is -discussed on the httpd dev mailing list (dev@httpd.apache.org). - -It is also included in modperl-2.0 source along with tests for -modperl-2.0. diff --git a/README.md b/README.md new file mode 100644 index 00000000..92518dd0 --- /dev/null +++ b/README.md @@ -0,0 +1,278 @@ +# Testing Apache with the Perl Test Harness + +## Prerequisites + +These two modules must first be installed: + +- `perl-ExtUtils-MakeMaker` +- `perl-Test` + +You'll need to install the CPAN modules listed in +`Apache-Test/lib/Bundle/ApacheTest.pm`. All you have to do to install them +all in one shot is: + +``` +perl -MCPAN -e 'install Bundle::ApacheTest' +``` + +Which are also available in one tarball here: +http://perl.apache.org/~dougm/httpd-test-bundle-0.02.tar.gz + +> **Note:** `Crypt::SSLeay` requires OpenSSL to be installed (only required +> for `t/TEST -ssl`): http://www.openssl.org/ +> More accurate results may be obtained by using the same openssl command +> line and libraries as consumed by APR-util and mod_ssl, due to X509 +> formatting behavior differences. + +For an extensive documentation see +http://perl.apache.org/docs/general/testing/testing.html +or +http://svn.apache.org/viewvc/perl/modperl/docs/trunk/src/docs/general/testing/testing.pod + +To run the tests for all Apache web server modules, some additional +CPAN modules will be required. If the tests don't work, make sure +that you have up to date versions of each of these perl modules: + +``` +cpan App::cpanminus +cpanm Bundle::ApacheTest \ + HTTP::DAV DateTime Time::HiRes \ + Test::Harness Crypt::SSLeay Net::SSLeay IO::Socket::SSL \ + IO::Socket::IP IO::Select LWP::Protocol::https AnyEvent \ + AnyEvent::WebSocket::Client LWP::Protocol::AnyEvent::http FCGI +``` + +## Quick Start + +If you don't care how it works and just want to run the tests: + +1. You need an installation of Apache (1.3.x thru trunk). +2. Any DSOs you wish to use should be configured in that Apache's + `httpd.conf` (the test harness will pick this configuration up). +3. Setup: + ``` + perl Makefile.PL -apxs /path/to/apache/bin/apxs + ``` +4. Run the tests: + ``` + t/TEST + ``` +5. Evaluate test output. + +## Getting a Little Deeper + +The test harness will run every `.t` file under the `t/` directory. To run +only the tests for PHP: + +``` +t/TEST -httpd /path/to/apache/bin/httpd t/php +``` + +That will start the test server, run the `.t` tests under `t/php` and shut +down the test server. You can also control each of these steps individually. + +Start the test server: +``` +t/TEST -httpd /path/to/apache/bin/httpd -start +``` + +Run the PHP tests in the test environment: +``` +t/TEST t/php +``` + +Stop the test server: +``` +t/TEST -stop +``` + +Run the server under gdb (using `-X`): +``` +t/TEST -d gdb +``` + +> **Note:** At this point, you have a working test environment. You can look +> in `t/conf` for the test server configuration files. These are generated by +> the test harness. Once you have a working test environment, you do not need +> to specify `httpd` on the `t/TEST` command line. For instance, to start the +> server up again: +> ``` +> t/TEST -start +> ``` + +## Running Regression Tests + +For a full regression test, you should have all modules loaded. Build the +server with: + +``` +configure --enable-modules=reallyall --enable-load-all-modules ... +``` + +among other things. Edit the generated `httpd.conf` and comment all mpm +modules that you do not want. Run `t/TEST -clean` again. + +You will see some: +``` +skipped: cannot find module 'XXX' +``` +as not all modules are in every Apache release (but the tests run for all). + +All in all, some >4k tests will run and the result needs to be: **PASS** + +## Trouble Shooting + +If you have a "PASS" at the end of `t/TEST`, congratulations! If not, this +section gives some advice in order to find the cause. + +0. If your test startup hangs forever in "waiting for server to warm up", but + the test server is reachable under port 8529, you might be the victim of + IPv4/6 confusion. The default servername configured is `localhost` and + some operating systems define `127.0.0.1` *as well as* `::1` in + `/etc/hosts`. If the test server listens only on `0.0.0.0` it might not + answer requests to `::1` and that causes the test startup to hang. + **Solution:** comment the `::1` line in `/etc/hosts` and see if that + improves matters. + +1. Run `t/TEST -clean` every time you change something in your Apache + configuration. The test suite picks up certain things from your installed + `httpd.conf` (such as `LoadModule` statements) and will not see your + changes unless you clean it. + +2. Failures in `proxy.t` may originate from the fact that the test script + cannot open the specified port. This happens on some machines if you abort + a test run and the socket is not properly shut down. Check if the error + goes away after a reboot. (`proxy.t` tests are slow, so chances you + interrupt tests at that point are good.) + +3. Failures in `access.t` may result from reverse lookups not working or + giving other answers than expected. In the case 0 above, if the test + client connects via `127.0.0.1`, a "Grant for localhost" might resolve to + `::1` and therefore will not match the access rules of the tests. + **Solution:** check that your servername is `localhost` (which is the + default) and that it *always* resolves to `127.0.0.1`. + +4. If some ssl test cases fail, especially when `t/ssl/proxy.t` fails, the + reason can be mismatches between your installed SSL library and the one + httpd uses. The `openssl` binary found in your `$PATH` is used to create + binary setup files by `t/TEST`. If another version of openssl then tries + to read these from your Apache server process, it might fail. Try: + ``` + t/TEST -clean + PATH=:$PATH t/TEST + ``` + If a lot of ssl tests fail, check in the error log for the presence of a + certificate validation error. If you find it, check the expiration date of + the TLS/SSL certificates used by the tests; they might be expired. Running + `t/TEST -clean` should delete the old ssl certificates, so they'll be + regenerated during the next run. + +5. If you see failures in the `modules/h2.t` test cases, please notify the + dev mailing list with a description of your setup. These tests are quite + young, currently only valid in 2.4.x and later and interact with quite + some other modules as well as OpenSSL versions installed. Some tests + require `mod_ssl` so make sure to load it in the httpd conf. + +6. Segmentation faults and core dumps occurring while executing the test suite + might indicate a real problem but always run again the tests after a clean + `make install` to avoid inconsistencies from old objects. + +7. If you see error messages like "Parse errors: Bad plan. You planned X tests + but ran Y." it usually means that you are missing a perl module or the + tested httpd module depends on another one not loaded in the httpd config. + +8. If you see SSL certificate errors, remove `t/conf/ssl/ca` prior to + `t/TEST -clean`. + +9. perl 5.28 in MacOS homebrew seems to hang the test suite. Invoking + `/usr/bin/perl Makefile.PL -apxs ...` will cause an older perl to be used. + +## Smoking Tests + +Sometimes it's possible that the test is passing properly for the first time, +but when you run it again, the test might fail. It's important to run +repetition smoke testing. For example to repeat the tests 5 times: + +``` +t/SMOKE -times=5 +``` + +It's also possible that a test will pass when it's run after a particular +test, but if moved to run after a different state it may fail. For this reason +by default the tests run in random order. + +Since it's important to be able to reproduce the problem with random testing, +whenever `-order=random` is used, the used seed is printed to STDERR. It can +then be fed into future tests via the `APACHE_TEST_SEED` environment variable. + +By adding the option `-order=repeat`, the tests will be run in alphabetical +order. + +Combining these two important smoke testing techniques, one can run tests with: + +``` +t/SMOKE -times=N -order=(repeat|random) +``` + +For example, to run the mod_rewrite tests 5 times: + +``` +t/SMOKE -times=5 -verbose t/modules/rewrite.t +``` + +The tests can be repeated N times, and run in the following modes: + +- randomize all tests +- repeat the whole test suite N times + +For configuration options and default settings run: + +``` +t/SMOKE -help +``` + +For more information refer to the `Apache::TestSmoke` manpage. + +## Test Environment Configuration + +The test server is configured with conf files like any normal Apache server. +The tricky part is those conf files are generated by the harness just prior +to starting the server. `t/conf/httpd.conf` is generated by +`t/conf/httpd.conf.in`. If that does not exist, the harness will generate a +working configuration and will include `LoadModule` (and `AddModule` for +Apache 1.3) directives from the `httpd.conf` associated with the httpd binary +you are using for testing. + +If `t/conf/extra.conf.in` exists, `t/conf/extra.conf` will be generated from +that, and an `Include` directive for that file will be put in the generated +`t/conf/httpd.conf`. + +`t/conf/apache_test_config.pm` is generated from the test configuration. It +contains all the information about the configuration of your test server. You +can access this information in test scripts by: + +```perl +my $env = Apache::TestConfig->thaw; +``` + +`Apache::TestConfig` accesses `apache_test_config.pm` and returns a hash +reference with all the information. Look through `apache_test_config.pm`; +it's a lot of stuff. Once these conf files are generated, you have a working +test environment, and they must be 'cleaned' if you wish to make changes to +them: + +``` +t/TEST -clean +``` + +(Now you will have to specify your httpd binary when starting back up again.) + +## More Information + +For more information on using the test harness and writing tests, see the +README in `Apache-Test` and the examples in `Apache-Test/t`. + +The test harness was originally written by Doug MacEachern and is discussed +on the httpd dev mailing list (dev@httpd.apache.org). + +It is also included in modperl-2.0 source along with tests for modperl-2.0.