database: Add basic database validation and error description

[german77]

Some databases are corrupted like #11. This PR adds an explanation for parser errors and checks for new types that aren't handled by this program.

The new error message will contain the ID of the corrupted entry a short explanation and a dump of the entry blob. This will be helpful for users reporting issues with corrupted entries. However since the entry couldn't be parsed correctly it can contain sensitive data encoded within the blob.

Corrupted Entry id 10004: Unsupported Field type
        '4D320100FF8801000400000031751001307510090033751008FFFFFFFF0100FE081427000032751021001000FE210450696E67'

Update

The PR is more complete and has shown that a valid database can have invalid entries. It appears like the dude has some actual bugs. Like deleting images, services or notification types doesn't remove the reference in other objects.

The dude seems to handle these fairly well. For example an invalid image will show as no image. An invalid notification method will not show at all. However it will not remove these invalid references.

Most of these are fixable by removing the invalid reference or changing the value for the default one directly on the database without any data loss.

Reading database dude.db
Opened database successfully
Map 55910: Invalid notify id found 10275 in map xxx
Map 56071: Invalid notify id found 10272 in map xxa
Map 56071: Invalid notify id found 10275 in map xxa
Map 56235: Invalid notify id found 240167 in map xxb
Map 56235: Invalid notify id found 10272 in map xxb
Map 56235: Invalid notify id found 10275 in map xxb
Map 56235: Invalid notify id found 10271 in map xxb
Probe 392291802: Invalid snmp profile id found 0 in probe yya
Probe 392294366: Invalid snmp profile id found 0 in probe yyb
Device type 10268: Invalid allowed service found 10251 in device type zza
Device type 10269: Invalid allowed service found 10251 in device type zzb
Device type 10269: Invalid image file found 332219888 in device type zzb
Device type 16046646: Invalid allowed service found 10251 in device type zzc
Device type 388527808: Invalid allowed service found 10251 in device type zzd
Device type 396653748: Invalid allowed service found 10251 in device type zze
This database contains invalid object references.

[pmattia90]

Hello, now i can find this kind of description,

Reading database luglio25.db
Opened database successfully
Saving database luglio25.json
Invalid Entry expected 260936, found 0
Invalid Entry expected 261047, found 0
Invalid Entry expected 261049, found 0
Invalid Entry expected 261801, found 0
Invalid Entry expected 362679, found 0
Invalid Entry expected 580500, found 0
Invalid Entry expected 580506, found 0
Invalid Entry expected 580508, found 0
Invalid Entry expected 585445, found 0
Invalid Entry expected 856010, found 0
Invalid Entry expected 947641, found 0
Invalid Entry expected 947646, found 0
Invalid Entry expected 1183115, found 0
Invalid Entry expected 1301043, found 0
Invalid Entry expected 2134626, found 0
Invalid Entry expected 2160504, found 0
Invalid Entry expected 2162607, found 0
Invalid Entry expected 2253218, found 0

What i'm supposed to do? There are options to print details? If i search this entry and i delete them maybe i can fix the problem of the dude! I will try
Thanks

[german77]

I been working into adding ID validation for individual object references to produce outputs like.

Map 55910: Invalid notify id found 10275 in map BABA
Map 56071: Invalid notify id found 10272 in map KEKE
Map 56071: Invalid notify id found 10275 in map JIJI

However based on your log is quite interesting. No new data formats or types which is good. However your entries were parsed without errors but seem incomplete as the object id is zero. Give me some time to make the error a lot more verbose. I should be able to make it display all fields in plain text, hopefully this will allow us to identify what is wrong with these.

[german77]

@pmattia90 Give it a second try to this PR. I added a detailed description. If any error happens while the parser is running it will also display that error. The message should look as follows:

Invalid Entry expected 3516, found 0
Field data:  Format 0x39
        Category 0x10cb25, 0xa0: []
        Category 0x10cb20, 0x0: false
        Category 0x10cb23, 0x9: 0
        Category 0x10cb24, 0x9: 0
        Category 0xfe0001, 0x8: 3516
        Category 0x10cb22, 0x21: "Display Temperarure CPU"
        Category 0x10cb21, 0x21: "oid(\"iso.org.dod.internet.private.enterprises.mikrotik.mikrotikExperimentalModule.mtXRouterOs.mtxrHealth.11.0\")"
        Category 0xfe0010, 0x21: "Cpu_temp"

This is the same data but with text descriptions instead of hex values.

Invalid Entry expected 3516, found 0
Field data:  Format Function
        Category Function_ArgumentDescrs, StringArray: []
        Category Function_Builtin, Bool: false
        Category Function_MinArguments, Byte: 0
        Category Function_MaxArguments, Byte: 0
        Category SysId, Int: 3516
        Category Function_Descr, ShortString: "Display Temperarure CPU"
        Category Function_Code, ShortString: "oid(\"iso.org.dod.internet.private.enterprises.mikrotik.mikrotikExperimentalModule.mtXRouterOs.mtxrHealth.11.0\")"
        Category SysName, ShortString: "Cpu_temp"

[german77]

The text descriptions are based on the enums at dude_field_id.h and dude_types.h. 0xfe0001 is the sysId which actually matches with the expected id. So this is parser error an not actually corruption.

All your entries are of type service. I will try to find the mistake in the parser and upload a fix.

[german77]

@pmattia90 Seems like I was trying to read a byte on an int field. I promoted a bunch of fields to int. Now it should output no errors. I also added the -i argument to run an integrity test on the data but that's still WIP.

[pmattia90]

I can confirm that now no errors are reported, So i can't understand which node or field creates problem in dude, correct?

[german77]

That's great news for me. Not so much for you. Correct we don't know which node or field is causing your dude to crash. Maybe after finishing the integrity test we will have more clues. For now we don't have the slightest idea.