Testing: A Crash Course
Published on Nov 18, 2015
Specifically, testing in Python.
MORE DECKS TO EXPLORE
PRESENTATION OUTLINE
test breadth
- Unit: cover individual functions or classes
- Integration: cover calls between units
- System: test entire programs or collections thereof
Transparency
Photo by tim caynes
test transparency
- White box: test knows implementation details
- Black box: test operates on inputs and outputs
Popular combos
nose
- Unit + white box testing
- Low boilerplate, easy to add
- Fine granularity
A mildly-dated introduction to the "nose" tool:
http://ivory.idyll.org/articles/nose-intro.html
There's also py.test; this does mostly the same thing, but is newer, and is tailored towards a universal "assert" primitive instead of specialized "assert_X" functions.
See also:
https://www.reddit.com/r/Python/comments/17ia3q/nose_introduction_python_te...
http://ivory.idyll.org/articles/nose-intro.html
There's also py.test; this does mostly the same thing, but is newer, and is tailored towards a universal "assert" primitive instead of specialized "assert_X" functions.
See also:
https://www.reddit.com/r/Python/comments/17ia3q/nose_introduction_python_te...
behave
- Integration or system level
- Usually black box
- Behavior DSL (Gherkin)
- Moderate boilerplate / integration cost
- Coarse granularity
There are a lot of resources out there for reading about BDD and Behave. However, a lot of them are abusively windy and/or theoretical. To wit:
https://www.google.com/webhp?q=intro+behave+bdd
Here's the official docs:
http://jenisys.github.io/behave.example/
The original Gherkin language:
https://github.com/cucumber/cucumber/wiki/Gherkin
https://www.google.com/webhp?q=intro+behave+bdd
Here's the official docs:
http://jenisys.github.io/behave.example/
The original Gherkin language:
https://github.com/cucumber/cucumber/wiki/Gherkin
Common features
- Find tests based on naming conventions
- Output options (colorized, XML, etc.)
- Start a debugger
- Run a subset of tests
(1) Naming conventions:
- nose - http://nose.readthedocs.org/en/latest/doc_tests/test_issue145/imported_test...
- behave - https://pythonhosted.org/behave/gherkin.html
(2) output options: both support jUnit XML, plain text, and colorized text. See their command-line help for details.
(3) Debugging:
- nose - use the "--pdb" or "--pdb-failures" option. Don't forget to add "--nocapture", so you can see what PDB is writing to stdout.
- behave - requires a boilerplate environment hook to get post-mortem PDB launched; it's annoying, but it's the best way, for now.
See: https://pythonhosted.org/behave/tutorial.html#debug-on-error-in-case-of-ste...
(4) Test selection:
- nose - takes test fixtures (files) and names (functions) as command-lne arguments
See: http://nose.readthedocs.org/en/latest/usage.html#selecting-tests
- behave - use the "wip" tag and the "--wip" command-line argument.
See:
https://pythonhosted.org/behave/tutorial.html#controlling-things-with-tags
- nose - http://nose.readthedocs.org/en/latest/doc_tests/test_issue145/imported_test...
- behave - https://pythonhosted.org/behave/gherkin.html
(2) output options: both support jUnit XML, plain text, and colorized text. See their command-line help for details.
(3) Debugging:
- nose - use the "--pdb" or "--pdb-failures" option. Don't forget to add "--nocapture", so you can see what PDB is writing to stdout.
- behave - requires a boilerplate environment hook to get post-mortem PDB launched; it's annoying, but it's the best way, for now.
See: https://pythonhosted.org/behave/tutorial.html#debug-on-error-in-case-of-ste...
(4) Test selection:
- nose - takes test fixtures (files) and names (functions) as command-lne arguments
See: http://nose.readthedocs.org/en/latest/usage.html#selecting-tests
- behave - use the "wip" tag and the "--wip" command-line argument.
See:
https://pythonhosted.org/behave/tutorial.html#controlling-things-with-tags
Test goals
- Test the code (duh)
- Maximal flexibility over changes in non-test code
- Isolation / idempotence
- Speed (esp. for unit tests)
- No external data, services, or config
- Avoid unification / anti-DRY
coverage
Photo by The Eyes Have It!
Test coverage facts
- Proportion of your code run under test
- Higher coverage implies comprehensive testing
- ...but that's easily defeated
EASY 100% CoveragE
run everything, assert nothing
isolation & idempotence
- Primarily in unit tests
- Some units interact with files, services, etc.
- How do we make these tests idempotent?
isolation via mocks
- Create a mock object
- Monkey-patch at start of test
- Run the code under the test
- Revert the mock after
- Assert calls or args to mock
The original version that gained popularity in Python 2:
http://www.voidspace.org.uk/python/mock/
Python 3 adopted this into the standard library:
https://docs.python.org/3/library/unittest.mock.html
http://www.voidspace.org.uk/python/mock/
Python 3 adopted this into the standard library:
https://docs.python.org/3/library/unittest.mock.html
Some units have lots of external callees.
Mocking these leads to a "mock trainwreck"
Photo by dok1
counterfeit tests
Photo by Chris Yarzab
These aren't tests
- Routine use of a program
- Code in "if __name__ == '__main__':"
Problems
- Hard to determine coverage and gaps
- No reporting or output options
- No support for debugging
- Cannot selectively run portions of the test
LOTSA development drivers
- Test-Driven Development
- Behavior-DD
- Story-DD
- Acceptance-TDD
- ...whatever
Lookit dem acronyms:
https://en.wikipedia.org/wiki/Acceptance_test-driven_development
https://en.wikipedia.org/wiki/Acceptance_test-driven_development
Test first
Code second
Photo by CraftyGoat
debugging
- Add or amend a unit test to expose the bug
- It should fail before you change the code under test
- Fix the code
This is a unit test approach for debugging, and is not really applicable to behavior tests.
new feature: BDD
- Story or requirements
- Write behavior test from actor's point of view
- Implement tests
- Make steps pass, top to bottom
new UNIT TESTS
- Story or requirements
- Skeleton non-test code, top-to-bottom
- Write tests (first) and code (second), bottom-to-top
