Data across different categories of IoT devices

[anupamme]

Hello

This is not really an issue and more of a Stackoverflow kind of question: currently the D3DB is organised under manufacturer hierarchy so I can look up data for a particular manufacturer. However, what I am interested in is data across categories e.g. security camera, printer.

I think there is a way to export the data into database but I'm not able to follow the instructions as this tutorial link is broken.

So can you please suggest how I can export this data into a database and then do sql queries on it?

[AshleySetter]

Hello, Sorry for the delay in responding, this issue has only just been brought to our attention. I think the notification must have got lost over the Christmas holiday period.

Sorry about the broken tutorial link also, I'll update the tutorial, we use the https://github.com/TechWorksHub/d3-cli CLI tool to lint, build and export D3 claims, which is pip installable with pip install d3-cli.

Currently this only supports building to a directory of JSON files or CSV files, or exporting a static html website of claims, which does not currently have search functionality. You could build the csv files for the D3DB database by using d3-cli --mode export --output <my_output_direcory> manufacturers, then importing the type.csv file inside <my_output_direcory> into a sql table, using the import from csv file functionality of https://sqlitebrowser.org for example:

You could then use SELECT DISTINCT tags FROM type to see the distinct category tags on types.

One such tag is #camera , so one could use SELECT * FROM type WHERE (tags LIKE '#camera') to get all camera types in the database.

Would it be beneficial for you to be able to export directly to CSV using the d3-cli tool?

[anupamme]

Hello @AshleySetter

Thanks for getting back, I was wondering if this issue got lost somewhere.

when I run this command:

d3-cli --mode export --output manu_temp manufacturers

I get the following error

PermissionError: [Errno 1] Operation not permitted: '/tmp/tmp5y1bzes4/maliciousUrls/mal-23c0c1c7f75a4e7bbc7752d95ed759d1.behaviour.d3.yaml'

when I check there is no folder with this name: tmp/tmp5y1bzes4/

Am I missing something here?

[AshleySetter]

@anupamme thanks for getting back to us.

Hmmm, it seems like there's a permission issue causing python to be unable to write to the temporary directory created to store the json claims here.

Could I ask what operating system you are using? If you are on Mac/Linux, could you run ls -ld /tmp and send me the output?

A workaround for this issue for now would be to build the json D3 claims first, then export from those claims, as this skips the step involving the temporary directory creation. You could do this with the command:

d3-cli --mode build --output manu_build manufacturers

Followed by:

d3-cli --mode export --output manu_temp --build-dir manu_build

[anupamme]

Hello @AshleySetter

I'm using Mac OS 13.3. The output of ls -ld /tmp

/tmp -> private/tmp

The output of ls /private/tmp is

com.apple.launchd.TDd1VTiJS4
mongodb-27017.sock

when I run this command: d3-cli --mode build --output manu_build manufacturers, I get following error:

PermissionError: [Errno 1] Operation not permitted: '/tmp/tmpyix2cp6j/maliciousUrls/mal-23c0c1c7f75a4e7bbc7752d95ed759d1.behaviour.d3.yaml'

I tried using sudo, there is no permission issue. when I look for folder tmpyix2cp6j within /tmp

ls /tmp/tmpyix2cp6j
/tmp/tmpyix2cp6j: No such file or directory

How do I trouble shoot this?

[anupamme]

Hello @AshleySetter

I ignored this error and ran the follow up command:

d3-cli --mode export --output manu_temp --build-dir manu_build

However, in the output file type.csv (in manu_temp/) file the column tags is empty for all rows. Am I missing something here?

[AshleySetter]

If it's the cli is producing the outputs correctly I suspect the you can ignore the message about no such file or directory, I suspect it is a race condition in the tempfile python library with deleting the temporary directory it creates after it goes out of scope.

There should be tags for the 5 of the types in the type.csv file, the rows with the following IDs:

ba3ba36d-be70-40f0-9c52-95b560066d11
3ec73667-68e0-4f84-98f6-2b795aff5e05
e22b0268-b6ca-42f8-9317-b629e12286ef
c4113d20-c378-43f6-bb1f-56f82c34b10f
74bfebf6-f497-4183-9c17-f89ef1bbd885

We still need to populate many of the types in this database with tags.

[anupamme]

In the type.csv file I got, non of these IDs are present.

Is it possible to share the type.csv you have?

[AshleySetter]

Hmm, that's odd, sure, I've attached it.

type.CSV

[AshleySetter]

I was looking into your issue some more and I wondered if this might be causing the permission error you are getting?

https://stackoverflow.com/questions/58479686/permissionerror-errno-1-operation-not-permitted-after-macos-catalina-update

[anupamme]

Hey thanks for the link, tried this but didn't work.

May be it is because: I'm using MacOS 13.3.1 (Ventura) and the link is for Catalina.

Nevertheless, is adding more tags/types for more data is on the roadmap?

[AshleySetter]

Yes, it's on the roadmap, but is probably a few weeks / months away at present. I'll aim to update you when the data is more complete.