Issues with loading data or Issues with the data

[philipjhj]

Either I have somehow pulled the data incorrectly, or there is an issue with the GPS location data.

After loading the data with the following code:

paths_raw=[]
for file in file_names:
    track = np.load(file)
    points=[(x[-1],x[-2]) for x in track['x']] # (longtitude,latitude)
    points=np.asarray(points)
    paths_raw.append(points)

where file_names are paths to these files

GFW_training_data/data/tracks/33272339789416.npz
GFW_training_data/data/tracks/278417339378522.npz
GFW_training_data/data/tracks/43454304520674.npz
GFW_training_data/data/tracks/238534759947303.npz
GFW_training_data/data/tracks/54820430995875.npz
GFW_training_data/data/tracks/71554660259352.npz
GFW_training_data/data/tracks/232400805249883.npz
GFW_training_data/data/tracks/148554964687863.npz
GFW_training_data/data/tracks/280574352919673.npz
GFW_training_data/data/tracks/187372144064677.npz

The data with basemap looks like

Not all contains "jumping" coordinates, but e.g. GFW_training_data/data/tracks/71554660259352.npz looks like

How do you suggest to clean this data?

Would you share your code for doing this, or update the data to a clean state?

[sakalu]

Either I have somehow pulled the data incorrectly, or there is an issue with the GPS location data.
After loading the data with the following code:
paths_raw=[]
for file in file_names:
track = np.load(file)
points=[(x[-1],x[-2]) for x in track['x']] # (longtitude,latitude)
points=np.asarray(points)
paths_raw.append(points)
where file_names are paths to these files
GFW_training_data/data/tracks/33272339789416.npz
GFW_training_data/data/tracks/278417339378522.npz
GFW_training_data/data/tracks/43454304520674.npz
GFW_training_data/data/tracks/238534759947303.npz
GFW_training_data/data/tracks/54820430995875.npz
GFW_training_data/data/tracks/71554660259352.npz
GFW_training_data/data/tracks/232400805249883.npz
GFW_training_data/data/tracks/148554964687863.npz
GFW_training_data/data/tracks/280574352919673.npz
GFW_training_data/data/tracks/187372144064677.npz

The data with basemap looks like

Not all contains "jumping" coordinates, but e.g. GFW_training_data/data/tracks/71554660259352.npz looks like

How do you suggest to clean this data?
Would you share your code for doing this, or update the data to a clean state?

@philipjhj

Hi philipjhj,

I would like to use this data for my project, so I am wondering if you have find the solution to your problem?

Saka

[philipjhj]

Hi Saka,

I believe we did not manage to solve the issue but simply cleaned the bad paths to the best of our abilities.

Unfortunately, it will be difficult for me to look into what we did exactly currently.

Bests,
Philip

[sakalu]

manage to
@philipjhj

Hi Philip,

Thank you very much for your prompt reply.

I would like to take this opportunity to ask if you have ever successfully generated the folder "/merged" with 6 files (alex_crowd_sourced.npz, false_positives.npz, kristina_longliner.npz, kristina_ps.npz, kristina_trawl.npz, pybossa_project_3.npz) according to the instruction ( run ./prepare.sh) in the following link?

"https://github.com/GlobalFishingWatch/training-data"

The reason why I ask is that all *.npz files I generated from my Window system but the results are incorrect as ALL labelled with "-1" for 'is_fishing' but no "1" for 'fishing' found.

Regards,
Saka