Add version field to data saver/ loader metadata

[skrawcz]

Is your feature request related to a problem? Please describe.
The data save and loaders return metadata. We should version this schema so that we can iterate and make changes-- people can then know how to version/write code against it.

Describe the solution you'd like
Add a __version__ field or something. It should follow semantic versioning. Major = breaking. Minor = new addition. Patch = bug fix.

Describe alternatives you've considered
N/A

Additional context
N/A

[roothsmz]

@skrawcz ,

One thought is to provide for an overridable Callable object.

def get_dataframe_metadata(df: pd.DataFrame, metadata_func: Optional[Callable[[pd.DataFrame], Dict[str, Any]]] = None) -> Dict[str, Any]:
    """Gives metadata from loading a dataframe.

    Note: we reserve the right to change this schema. So if you're using this come
    chat so that we can make sure we don't break your code.

    This will default to include:
    - the number of rows
    - the number of columns
    - the column names
    - the data types
    
    If you provide an override, then your DataFrame will be passed into that 
    Callable and the resulting Dictionary will include your custom metadata.
    """
    if metadata_func is not None:
        return {DATAFRAME_METADATA: metadata_func(df)}
    else:
        return {
            DATAFRAME_METADATA: {
                "rows": len(df),
                "columns": len(df.columns),
                "column_names": list(df.columns),
                "datatypes": [str(t) for t in list(df.dtypes)],  # for serialization purposes
            }
        }

Source code as is currently implemented:

hamilton/hamilton/io/utils.py

Lines 53 to 72 in 427f523

	def get_dataframe_metadata(df: pd.DataFrame) -> Dict[str, Any]:
	"""Gives metadata from loading a dataframe.
	Note: we reserve the right to change this schema. So if you're using this come
	chat so that we can make sure we don't break your code.
	This includes:
	- the number of rows
	- the number of columns
	- the column names
	- the data types
	"""
	return {
	DATAFRAME_METADATA: {
	"rows": len(df),
	"columns": len(df.columns),
	"column_names": list(df.columns),
	"datatypes": [str(t) for t in list(df.dtypes)], # for serialization purposes
	}
	}

[skrawcz]

Hmm -- this makes me think do we need schemas by type? Or can we have a general one on all dataframes, but where some fields might not be easily populated.