If complex type is present in schema (e.g. ArrayType), it could not be compared using symmetric_difference as done here
|
def _check_names_and_types(df1_schema, df2_schema): |
as the type is not hashable.
Need to rework this code part to compare only names and types (do not compare nullability) and be able to work with complex types.
Proposed: sort two lists with (name, DataType) as done now and compare columns one-by-one. Raise error on the first difference.
Add test showing the functionality works for simple and complex types.
If complex type is present in schema (e.g. ArrayType), it could not be compared using symmetric_difference as done here
Sim4Rec/simulator/modules/simulator.py
Line 99 in 215aeaa
Need to rework this code part to compare only names and types (do not compare nullability) and be able to work with complex types.
Proposed: sort two lists with (name, DataType) as done now and compare columns one-by-one. Raise error on the first difference.
Add test showing the functionality works for simple and complex types.