Server side compilation

[amlwwalker]

Hi again,

So I have evolved the plugin somewhat - or atleast the server side part so that it compiles code that is sent to it from the console. Its quite neat actually!

Anyway, on my server I have Arduino, Java and Ino (terminal arduino compilation program). Then I have a golang server that receives the code as a POST request, calls some (pretty ropey currently) bash scripts that compile the code. and return the hex to the plugin which in turn uploads it to the arduino.

I'll put it up here at some point, once I have ironed a couple of things out, but I wondered if you might have a suggestion as to why its so slow to compile.
The arduino envrioment compiles blink in about 3 seconds. It takes the server about 12.

Part 2:
To complicate things the server (actually ino I think) cant handle two requests simultaneously. So I fired up docker containers so for each compilation request it is done in a seperate Docker container. This gets over the problem of concurrent compilations, however doesnt solve the speed issue.....

Any ideas - I just thought you might have a suggestion....

[paulobarcelos]

Have you considered http://platformio.org? Seems to be a more complete solution for what you are trying to achieve.

[durkie]

What is it you're building, and what hardware are you building on? I'm using a $13/year ubuntu server for server-side compilation experiments, and a sketch that calls 2 libraries and uses 15k of flash. Compilation here is 2.8 seconds (as reported by time ino build -m LilyPadUSB)

Can you run the bash scripts in the shell and see where it's getting hung up? 12s is indeed pretty long.

[mrjones]

Cool! I really like the idea of being able to do everything from the browser. Surely some people will want to ssh to a machine and compile themselves, but I bet a lot of other people would prefer something that is completely in-browser. I started doing something similar a little while back (https://github.com/mrjones/Chrome-Arduino/blob/master/buildservice/buildservice.go) but never got around to making it any good. In the long run, it would probably make sense to refactor the extension code in order to make it easier to integrate with some hosted/webapp environment that handles code editing and compilation.

But that's neither here nor there right now...

As for debugging performance issues: 12s is indeed pretty long. My linode instance (1x2.8GHz CPU, Ubuntu) can do a clean build of the sample-program in 2 seconds, and incremental builds in about 400ms. (Most of the difference between clean and incremental builds looks like all the Ardunio core libraries that get built once but never change.) I believe Ino and the Arduino IDE both use avr-g++ under the covers, so they should (theoretically) be able to compile programs in about the same amount of time.

Are you running Linux? Or some other OS.

Is it reproducibly this slow if you just log in and run the command yourself? Or only when you run it triggered by an HTTP request? (It'd be good to know for sure whether we're debugging slowness within Ino itself, or slowness in the environment that sets it up).

If you want to see where the time within Ino is going:

• One thing you can try is the -v flag on ino which will print all the sub-commands it runs. You should be able to get a feel for if it's one reaaaaaly slow command, or just lots of kind of slow ones.
• You can also try building a custom version of ino itself. Ino apears to be a bunch of python glue around avr-g++ (https://github.com/amperka/ino). I had some issues with it, so I cloned it, and built it ... and it seemed to work relatively well. You could try cloning the repo, filling it full of timing measurements, and seeing how long things take.

[amlwwalker]

Thanks for your responses. I will elaborate now, that there is interest in this!

I am running on Centos 7 64 bit. It is an m3.medium instance on AWS. My issues are based on that. I beefed it up to a C3.xlarge to see if that made a difference, and it did make a big difference, but I think there are other optimisations that can be fixed before doing that, so I reverted.
First it is slightly held together with sellotape - I am trying to get something working, and then will try and smooth it over (for instance with the Docker remote API rather than shell commands)

The server runs a docker container that the code will be run inside:
docker run -e USERID=$1 -t --volumes-from CODE --name $1 playduino:latest bash /build/buildapplication.sh
where USERID is set by the webserver -its a unique ID for the current build.
buildapplication.sh is:

mkdir ~/arduinocode
cd ~/arduinocode
ino init -t blink
cp -rf /srv/codefiles/$USERID.ino ~/arduinocode/src/sketch.ino
cp -rf /srv/codefiles/libraries ~/arduinocode/lib/
time ino build
cp ~/arduinocode/.build/uno/firmware.hex /srv/codefiles/$USERID.hex

It saves the hex to the volume container with the UID of the build request. The server then returns what is returned from the ino build to the user so the user can know the build was successful or not.
Its the time between the user sending their code and getting back a success/failure that takes so long. When I click compile I can see almost immediately the code appear on the server, however it takes AGES to actually finish the compilation process. However when I run a Docker container and do it manually (i.e go into the docker container and run buildapplication.sh) I get ino build time of:

real 0m4.423s
user 0m2.283s
sys 0m1.899s

[durkie]

I'm doing a similar process on my server, and while it won't probably won't make a huge difference in total build time, there are 2 steps you can eliminate:

• ino init -t blink -- do mkdir -p arduinocode/src and it will make the src directory and any missing parent directories. Put sketch.ino inside of it.
• no need to copy libraries over on a per-directory basis. Put them in a common place (i have mine in /usr/share/arduino/libraries -- not sure if ino creates /usr/share/arduino or if it was leftover from an Arduino.mk installation.

If, as you say, it is compiling pretty fast locally, but slow to be sent by the webserver, then it sounds like it might be a webserver issue...maybe it's slow to get a return status from ino? that is a bit more mysterious.

[amlwwalker]

Thanks guys. Does GCC (which is compiling the code - called from ino) have any capability to compile seperate programs in parallel? When I boost the server on AWS I can compile multiple programs from multiple laptops at once, but I wonder whether I am missing a trick with GCC?

[amlwwalker]

OK when I compile from two laptops on the server (I.e uploading to the server from 2 laptops) in parallel, this is what htop looks like:
Thats an m3.medium on AWS

you can see the docker container being run which is in turn running ino and buildapplication.sh

[durkie]

it seems like an ino issue, and i'm not sure why it's doing it.

i tried opening a screen session with one screen in one build directory and another in a different build dir. i started the build in dir 1 and it proceeded as normal. I quickly switched screens to start the build in dir 2 and ino simply returned instantly.

i definitely don't think it's a resource-constraint thing though -- you're compiling 32KB of really simple code. i can open up terminals all day and compile normal code and there's no inherent limitation to running a bunch of processes beyond what the system can handle.

[amlwwalker]

thats interesting.
What I noticed was it couldn't open boards.txt, which I thought was a limitation on GCC.
The above CPU usage is massive and very confusing!

So using docker gets over the ino issue, but not the insane processing resources that seem to be required

[amlwwalker]

Further research based on paulobarcelos suggestion.
I swapped out ino for platformio and I still run into the same issue that when I try two compilations in parallel one fails. I therefore feel I have no option but to use something like Docker for this situation. Really does feel like overkill, but perhaps I am missing something here.