Import data source duplicate field name

[gnud]

Some of the fields in some of my CSV files start with symbols such as $, %, #; and when the data is cleaned then the symbols are removed automatically and I end up with duplicate fields.

Is there a way to override cleaning the column names before mapping starts?

[sheppard]

This normalization is handled in IterTable, so I transferred the issue to that repository. The relevant lines are here:

itertable/itertable/mappers.py

Lines 70 to 94 in fd2694f

	class TupleMapper(DictMapper):
	no_pickle_mapper = ['_tuple_class', '_tuple_prototype']
	@property
	def field_map(self):
	field_names = self.get_field_names()
	if not field_names and not getattr(self, 'data', None):
	raise NoData
	# FIXME: check for duplicates
	if not hasattr(self, '_field_map'):
	items = [
	(field, self.tuple_field_name(field))
	for field in field_names
	]
	self._field_map = OrderedDict(items)
	return self._field_map
	def tuple_field_name(self, field):
	field = self.clean_field_name(field)
	field = re.sub(r'\W', '', field.lower())
	# normalize identifiers for consistency with namedtuple
	# http://bugs.python.org/issue23091
	field = normalize('NFKC', field)
	return field

While this behavior can be overridden, I think it would make sense to fix this in IterTable directly (hence the FIXME). A simple solution for now would be to detect that a normalized name has already been used, and append e.g. _1, _2, _3 etc. to ensure uniqueness.

Eventually we could update Django Data Wizard to support DictMapper in addition to TupleMapper, but that will be a bigger effort.