How to work around CrossHair Output Values

[thierryxdp]

Hello!

I'm adventuring myself using CrossHair coverage and notice that all output values are empty strings, 0's, empty lists, and stuff like that.

I want to get values accordingly with a given context (so they make more sense in a test). For example, given a list of strings that matches the context, it'll choose one string from that list instead of just using an empty string.

But I'm new to CrossHair and don't know where to begin. I tried to look a little bit into the code but didn't get too far. Can someone guide me?

[pschanely]

First, let me apologize for sitting on this for so long - it completely slipped my mind. In the future, don't hesitate to ping me if I don't respond in a few days.

Okay, so I completely understand your desire to generate meaningful test inputs - this is a pretty universal problem for unit test generators. Unfortunately, I'm not sure how we'd build a rational feature like you describe that would work for arbitrary data structures. If you have a more fleshed out proposal for how it might work, I'd be happy to consider it. (and happy to point you around the codebase once I understand more concretely how it should work)

Let me also propose a workaround: in theory, you could temporarily add constraints at the top of your function that immediately exits when the input that doesn't match the inputs (e.g. the strings) that you'd like to see. For example:

def busca_com_tipo(a, b):
  if a not in ("good string", "other good string"):
    return "not a nice input"
  ...

Afterwards, you can delete the test case(s) that relate to this temporary code block, and the remaining tests are hopefully the ones you want.

[thierryxdp]

Hello! Here's an example that explains my idea:

The method below counts the frequency of the words in a phrase:

def words_frequency(phrase: str) -> dict:
    dic = {}
    words_list = phrase.split()
    for word in words_list :
        if word in dic:
            dic[word] += 1
        else:
            dic[word] = 1
    return dic

running with crosshair:

!crosshair cover --example_output_format=pytest --coverage_type=solution.words_frequency --per_path_timeout=15

I get a bunch of tests like this:

def test_words_frequency():
assert words_frequency('\x00') == {'\x00': 1}

def test_words_frequency_2():
assert words_frequency('') == {}

def test_words_frequency_3():
assert words_frequency('\x01') == {'\x01': 1}

def test_words_frequency_4():
assert words_frequency('\x02') == {'\x02': 1}

def test_words_frequency_5():
assert words_frequency('\x03') == {'\x03': 1}

def test_freq_palavras_6():
assert freq_palavras('\x04') == {'\x04': 1}

def test_test_words_frequency_7():
assert test_words_frequency('\x05') == {'\x05': 1}

def test_test_words_frequency_8():
assert test_words_frequency('\x06') == {'\x06': 1}

def test_test_words_frequency_9():
assert test_words_frequency('\x07') == {'\x07': 1}

The case with an empty string, and the case with a single word are good. But crosshair only changes the first character to another one in the list of all characters.

Instead of trying all these characters, could it use values from another source? Let's say i give this list: ["another", "", " ", "value"]. Could it try to use each of these strings (as he already does, but using all available characters), and then use a combination of these strings?

It's currently a problem for me because, using this example, it never returns a dictionary with more than one occurrence of a word, and therefore it doesn't explore all possible paths.

[pschanely]

Oh! Okay, so let me isolate this into two distinct problems that you're having (and perhaps there's more that I'm not picking up on?):

1. crosshair cover is not able to cover all paths in your example.
2. You'd like your unit tests to use certain inputs to make them more clear/readable/etc.

Problem 1: I am very interested in examples where CrossHair cannot get coverage, and will investigate/act on them without reservation. In the example you have above, I've created an issue for str.split which is conspicuously missing symbolic support. I suspect I could get a fix for this released this week. Thanks for the example, and I will be very grateful if you continue to send such examples to me!

Problem 2: It's less clear to me how to develop a solution for this. Is this a command line option? A config file? How would it work? How could it support arbitrary types like classes? If this is a one-off issue, you can temporarily add a guard at the top of your function, like this:

def words_frequency(phrase: str) -> dict:
    if not re.fullmatch("((another)|(value)| )*", phrase):  # temporary addition
        return None.                                        # temporary addition
    dic = {}
    words_list = phrase.split()
    for word in words_list :
        if word in dic:
            dic[word] += 1
        else:
            dic[word] = 1
    return dic

CrossHair (with a long enough timeout) will produce this

def test_words_frequency():
    assert words_frequency('value  value') == {'value': 2}

def test_words_frequency_2():
    assert words_frequency('a\x00') == None

And then you can delete the second test case, along with the temporary lines in your source code.

At any rate, I'm still curious to hear how much of your problem is really just about coverage, and how much is about the nature of the inputs. Cheers!