"I have a mind like a steel... uh... thingy." Patrick Logan's weblog.

Search This Blog

Thursday, January 06, 2005

Testing Interfaces

Ian raises some concerns about testing interfaces in Python...

Unit tests aren't going to test across the boundaries where people are proposing interfaces -- between loosely coupled components.
I am not sure I understand this point. Say I give you version 1 of a component and the tests deemed "sufficient" for that component. Then I give you version 2 of the component and the tests. If you are able to run version 1 of the tests against version 2 of the component, that should be a reasonable indicator that your own use of version 1 will be compatible with version 2 of the component.

As the provider of the component, I need to know nothing about your own use of that component, for reasonable definitions of "sufficient tests".

Note also that interfaces are more explicit about contracts between components than tests will be. For instance, an interface might demand a method which is not yet being used in your code or your usage; the interface makes it explicit that this method should still be defined, and so predicts future developments, not just current developments (half-implemented dictionary interfaces are a common example in Python, where interfaces would motivate people to finish the work).
The original scenario was an upgrade to a component breaking backward compatibility. If an upgrade provides a new feature then your old code has not used that feature yet. I would not expect that to break backward compatibility.

If an upgrade requires an old feature to require a new method of its old consumers, then that *would* break backward compatibility. However I would expect a reasonable set of tests from the component provider to demonstrate this failure on their part to keep their old customers happy.

Or two interfaces can match methods and signatures, but include extra information about the semantic differences between those interfaces; because it doesn't use duck typing, you won't accidentally mix semantics (you have to explicitly say which, if any, of those interfaces you support). This is particularly a problem with interfaces that define only a small number of methods, or use common methods like __call__. The only way to test these is with system tests, and system tests are always incomplete, there's always some code path or combination you haven't tested.
I guess I would need a specific example that cannot be tested by a unit test. Certainly there are "big" tests that require a lot of "units" to execute. But if each of those units were developed and updated by a series of "unit" tests then I would expect those tests to represent the "differences between those interfaces" as they exist within each unit.

I am having trouble addressing this issue in the abstract. If it is a frequent problem, I would look for ways to avoid it or break it into smaller problems. Maybe this is a scenario where changing the language would solve the problem. I just don't know, but I suspect there are simpler ways to solve the problem on the basis that I have used Python and several languages like Python and I have not run into this scenario. That doesn't mean it isn't real, just that I have not recognized it.

1 comment:

Ian Bicking said...

Part of what I'm thinking about is a situation where the integrating programmer isn't the author of the things being integrated, where there's no producer/consumer situation. E.g., I'm writing code that uses an authentication library, and a web framework, and neither know about each other explicitly, but presumably there's some work to apply each to some common "standard" so they should work together.

There may be subtle aspects of the API in some case, like an attribute that shouldn't be changed. It's common in Python that you could change the attribute, but you just really shouldn't; like maybe cache consistency will be horribly lost, creating one of those awful it-works-but-the-data-is-wrong bugs. This is the kind of subtle thing that interfaces typically make explicit.

In that case, the interface provides an important piece of documentation. Unit tests aren't very good documentation; and when you propose that things like backward and forward compatibility be handled with tests, it sounds like you are treating them like documentation. Because something happens to work now doesn't mean it's right, or that it will work in the future, and I think it's totally impossible for a unit test to catch all those cases.

So, back to the example -- the thing you really want to know, do those two libraries work with each other? Do they even claim to work with each other? Here's where the ambiguity of duck typing also comes in -- they may seem to work with each other, because the objects that are exchanged share a sufficient API; it's still quite likely that they aren't really working to the same interface, there's things that aren't expressed just in the method signatures.

If you provide tests to confirm correctness, you are doing system tests, not unit tests (at least from my understanding of those terms). Not that system tests are bad, but they aren't as reliable as unit tests because of the combinatorial aspects of integration, and I don't think they are sufficient for this integration work.

Really -- a big part of what I want from testing is that I can start to forget about some things. If I have to test everything from scratch everytime I use a different combination of code, that just sucks. You might as well just hold off on testing until you have everything together, and test the whole ball. The point of unit testing in my mind is that you don't have to do that anymore.

Also, verifying tools in the language help me test more with less code. Consider error-ignoring languages like the shell -- if you're writing a test in that, you constantly have to explicitly ask "did this fail?" That sucks. Or it's like comparing a weakly typed language to a strongly typed; you can fix it all with tests, but those tests are going to suck to write.

I see these explicit interface requirements as that same sort of thing; I can stop asking "did this give me back something with methods X, Y, and Z?", and just assume it's right because the code says it a certain type (interface) of object.

Blog Archive

About Me

Portland, Oregon, United States
I'm usually writing from my favorite location on the planet, the pacific northwest of the u.s. I write for myself only and unless otherwise specified my posts here should not be taken as representing an official position of my employer. Contact me at my gee mail account, username patrickdlogan.