Wednesday, July 9, 2014

Property-based testing

Actually when I worked as a Java developer I didn't hear about the concept of that testing. We wrote small unit tests, some integration tests and a lot of end to end tests. But that kind of testing didn't come up somehow.

What is property-based testing?

Property-based testing is a good additional test when we have enough unit tests but we want to make sure that our functions or modules prepare for any type of incoming data possible. The main idea behind the two tools I know (PropEr, QuickCheck) that let us not to write test cases, let us generate them instead. From the function specifications one can easily guess what kind of inputs a function can receive. If we have an 'add/2' function and the specification says that it adds two numbers, we know that both parameters will be a number. So we can generate infinite number of test cases.

The problem is that we don't know the expected result since we don't know the input parameters. Ok, we can write that 'add(x, y) =:= x + y' but in that case we need to reimplement the function itself inside the test code. We don't want to do that. Instead we can find properties of those operations with which we can describe their nature.

Add is symmetrical, so 'add(x, y) =:= add(y, x)'. It is trivial here but testing an 'equals' method in Java that way is a very very useful test. Property-based tests become much more useful when we have the reverse operation at hand. Imagine that we implemented a 'sub/2' function which subtracts the second parameter from the first one. Great, we can test the two functions together we can write 'sub(add(x, y), y) =:= x'. If we implement a test that way, PropEr tool will generate 100 tests with random numbers, and it checks if the condition we have written is true. Sometimes we get surprising test fails because we didn't know about -0 or 0.0 or -0.0, things like that.

If the test fails PropEr will have the exact test case on which our test failed. It can be very complicated containing long lists or big float numbers and can be hard to understand the error if the tool just spit out those numbers. Instead those tools shrink the test case, they convert the test case to a simpler form and checks if it still fails. If the minimal failing test case found it is reported.

JSON name conversion

I implemented a simple framework which helps to convert Erlang records to JSON. Right now it can convert Erlang record to JSON but there is not way back. So I started to implement decoding too, and since encoding and decoding are two reverse operations we can use QuickCheck or Proper to implement both operations.

The project is here: ejson github repo and our first task is to convert an Erlang atom to json string. Unfortunately Erlang atom set is wider that json names and since we want to convert json values to Javascript object we need to make some restrictions on record field names.

A record field should look like this way: 'number_of_connections', and it should be converted into 'numberOfConnections'. The atom contains small letters and underscores (numbers as well), and the json name will be camel cased accordingly. If there is an underscore in the name the next character will be a capital letter. Those restrictions make it possible to convert the json names into Erlang atoms unambiguously.



all_test() ->
                                   [{to_file, user}])).

identifier_char() ->
        {$z - $a + 1, choose($a, $z)},
        {3, $_},
        {10, choose($0, $9)}

record_name() ->
    ?LET(Chars, list(identifier_char()),

camel_case_prop() ->
        ?SUCHTHAT(R, record_name(),
                CC = ejson_util:atom_to_binary_cc(Name),
                ejson_util:binary_to_atom_cc(CC) =:= Name

I am using Proper and eunit together. This module has an eunit unit test which is the main entry point. It is picked by eunit and executed. It calls Proper in order to check the 'camel_case_prop' test. The test basically says that for all Name generated if we convert the name to a binary (cc means the camel case) and that we convert that binary back, the converted atom and the generated atom should equal.

The FORALL macro is the executor which generates test cases (see the documentation). The test case will be put in the Name variable. The function record_name() is a generator which generates atom we specified above. It generates a list of identifier characters where those characters are generated by another generator. Inside identifier_char() there is a frequency (generator too) which generated weighted test cases. With 27 weight it will choose between 'a'-'z', with 3 weight it will be an underscore and with weight 10 it will be a decimal.

Obviously we need to filter out some names like '1st_step' or '_main' so we need to include SUCHTHAT macro to filter out all test cases which doesn't conform to 'is_convertable_atom/1' (which enforces those rules). So let us create an erlang module 'ejson_util' and start to put the functions there.

is_convertable_atom(Atom) ->
    %% true if the atom can be converted by the
    %% two functions unambiguously
    L = atom_to_list(Atom),
    start_with_char(L) andalso proper_underscore(L).

start_with_char([L|_]) when L >= $a andalso L =< $z ->
start_with_char(_) ->

%% If there is an underscore, it needs to
%% follow by a letter
proper_underscore([]) ->
proper_underscore([$_, L | T]) when L >= $a
                            andalso L =< $z ->
proper_underscore([$_ | T]) ->
proper_underscore([_ | T]) ->

Now Proper can generate test cases. Let us implement the atom-binary conversions during continuously running property tests. Tests can be run this way:

./rebar compile
./rebar eunit apps=ejson

Try to implement the functions by yourself, it is very useful experience. At first the test will fail with the empty atom '', and so on. Now I put here the final implementation of the two functions and the utility functions as well.

atom_to_binary_cc(Atom) ->
    CC = camel_case(atom_to_list(Atom), []),

binary_to_atom_cc(Binary) ->
    UScore = underscore(binary_to_list(Binary), []),

camel_case([], R) ->
camel_case([L], R) ->
camel_case([$_, L | T], R) ->
    camel_case(T, [string:to_upper(L) | R]);
camel_case([H | T], R) ->
    camel_case(T, [H | R]).

underscore([], R) ->
underscore([Cap | T], R) when Cap >= $A
                      andalso Cap =< $Z ->
    underscore(T, [Cap + 32, $_ | R]);
underscore([Low | T], R) ->
    underscore(T, [Low | R]).


  1. Selenium testers have great opportunities. Many new things and compatibility features are added every day. So, start to learn selenium course and get a bright career.
    Selenium Training in Chennai | Selenium Training | Selenium Training institute in Chennai

  2. Selenium is the mandatory certification to get professional career in software testing industry.
    Selenium Training|Selenium Course in Chennai

  3. The future of software testing is on positive note. It offers huge career prospects for talented professionals to be skilled software testers. Software testing training|Software training|Software testing training in Chennai

  4. Interesting article! Why still waiting to learn software testing certifcation. It provides a lot of career chances learn by today.
    Loadrunner Training in Chennai|Qtp training in Chennai

  5. Hi, thanks for sharing such an informative blog. I have read your blog and I gathered some needful information from your blog. Keep update your blog. Awaiting for your next update. Software Testing Training in Chennai | Software Testing Training in Chennai

  6. I really enjoyed while reading your article, the information you have delivered in this post was damn good. Keep sharing your post with efficient news.
    Software testing training|Software training|Software testing training in chennai

  7. The blog gave me idea about the property based testing my sincere thanks for sharing this post and please continue to share this kind of post
    Software Testing Training in Chennai

  8. hi admin...Thank you so much for explain property-based testing.I like it very much your blog.keep in blogging....
    Software Testing Training in Bangalore |
    Java Training in Bangalore


Richard Jonas. Powered by Blogger.

About me

My name is Richárd Jónás, live in Budapest, Hungary. In this blog I want to share my coding experiences in Erlang, Elixir and other languages I use. Some topics are simpler ones but you can use them as a reference. I also present some of my thoughts about developing distributed systems.