Right now we consider an IMO number to be valid if it passes the checksum test and is the right length. Now we want to tighten this up and only use IMO in the vessel_id if it is a valid, registered IMO.
See https://github.com/GlobalFishingWatch/GFW-Tasks/issues/830
Proposal: See if we can do this entirely in bigquery within the segment_vessel_daily stage. Do this by simply filtering n_imo to only include IMO numbers on the published list from the vessel database, and the levenshtein distance of the name (normalized?) is within tolerance
Alternate: Do this in the segment dataflow process where we currently produce n_imo
Right now we consider an IMO number to be valid if it passes the checksum test and is the right length. Now we want to tighten this up and only use IMO in the vessel_id if it is a valid, registered IMO.
See https://github.com/GlobalFishingWatch/GFW-Tasks/issues/830
Proposal: See if we can do this entirely in bigquery within the segment_vessel_daily stage. Do this by simply filtering n_imo to only include IMO numbers on the published list from the vessel database, and the levenshtein distance of the name (normalized?) is within tolerance
Alternate: Do this in the
segmentdataflow process where we currently produce n_imo