How can I give the dataframe size as argument to an expectation?

[vasvyzas]

Hi all,

I am working locally, my datasource is a file on a filesystem and I am processing with Pandas. I have only one dataset (csv) that I need to validate which gets updated every month. My plan is to create the expectation suite with all the necessary expectations and run the checkpoint once a month when the data gets updated.

I want to write an expectation that needs the size of a Pandas dataframe as a parameter to the expect_column_sum_to_be_between expectation but I cannot figure out how to work with the evaluation parameters.

This is what I have so far

`df = validator.active_batch_data.dataframe

validator.set_evaluation_parameter("row_count", df.shape[0])

validator.expect_column_sum_to_be_between(
column='weights',
min_value={"$PARAMETER": "upstream_row_count * 0.8"},
max_value={"$PARAMETER": "upstream_row_count * 1.2"}
)`

How can I make sure every time the dataset gets updated the checkpoint will read the dataset and get the shape?