[I posted this to the Arduino developer’s mailing list, but figured others might find it useful too]
When I first started with Arduino, I thought Serial.available() was a very loose wrapping of the RXC bit in the USCRA register, i.e. if I didn’t get data out of there fast, it’d be gone. That led to convoluted code like:
if( Serial.available() ) {
val1 = Serial.read();
while( !Serial.available() );
val2 = Serial.read();
// and so on
}
Yuck. So you end up designing protocols that are too terse. Or maybe you think you need to buffer so you don’t lose it:
while( Serial.available() ) {
commandbuffer[i++] = Serial.read();
}
Then parsing becomes a two step process: read serial, parse buffer. Confusing to newbies perhaps, but at least it allows for a better protocol down the line. (And descend into madness as you gaze into the maw of strtok())
Because Serial contains these big comfy buffers, we often don’t need a second buffer to let us easily implement good protocols. My current favorite is to do something like this:
// protocol is "CCaaaa", two bytes of command, four bytes of args
if( Serial.available() >= 6 ) { // command length is 6 bytes
cmd0 = Serial.read();
cmd1 = Serial.read();
arg0 = Serial.read();
arg1 = Serial.read();
// ...etc...
}
I don’t think I’ve seen any Serial examples that check for a specific number of bytes available. It’s really handy.
Implementing a human-friendly protocol like “command arg0 arg1 arg2”, where command and args are space-separated strings like “servo 12 0xff”, is currently hard with Serial. I do this right now with a 2nd buffer and lots of C hackery:
char* cmdbuf; char c; int i;
while( Serial.available() && c!= '\n' ) { // buffer up a line
c = Serial.read();
cmdbuf[i++] = c;
}
int i = 0;
while( cmdbuf[++i] != ' ' ) ; // find first space
cmdbuf[i] = 0; // null terminate command
char* cmd = cmdbuf; //
int cmdlen = i; // length of cmd
int args[5], a; // five args max, 'a' is arg counter
char* s; char* argbuf = cmdbuf+cmdlen+1;
while( (s = strtok(argbuf, " ")) != NULL && a < 5 ) {
argbuf = NULL;
args[a++] = (byte)strtol(s,NULL,0); // parse hex or decimal arg
}
int argcnt = a; // number of args read
This sort of functionality would be great in a library I think. Maybe not in Serial, but a core class.
Any other protocols people like to use and the Arduino code they use to do it?
Posted for Kasper, since there’s some weirdness with him posting this. He says:
I tried the method above and I got it working partly. It works almost perfect when I send a message with a keypress action in Processing. However when I trigger the sending code from the draw() function it fails. I tried several framerates ( 1 loop every second ) but that didn’t work out.
By the way, with lower framerates (<10) the keyPress function seems more unstable as well.
It could be Osx Snow Leopard in combination with Processing. Since they fixed serial communication with “duct-tape” for the 1.0.7 release.
Another thing could have been that I send DMX ( and use sei() and cli() to disable interrupts during that time ), however even turning the DMX sending of ( and let Arduino response with the received message ) didn’t help.
Maybe you see any flaws ? Code below is working good when I write the array from the keyPressed function.
Arduino code :
int intensity_Array[72]; int arraylength=72; byte ledNum; byte redVal; byte grnVal; byte bluVal; int arrayStartPos; void setup() { pinMode(11, OUTPUT); digitalWrite(13, HIGH); Serial.begin(57600); } void loop() { if( Serial.available() == 4 ) { ledNum = Serial.read(); redVal = Serial.read(); grnVal = Serial.read(); bluVal = Serial.read(); arrayStartPos = ledNum*3; intensity_Array[arrayStartPos] = redVal; intensity_Array[arrayStartPos+1] = grnVal; intensity_Array[arrayStartPos+2] = bluVal; } if(ledNum==22) { ledNum=0; // set the ledNum to the beginning for(int i=0; i<arraylength ; i++) { Serial.print(intensity_Array[i]); Serial.print(","); } Serial.println(13); /* send frame to led data */ //cli(); //sendFrame(); //sei(); } }Processing code :
import processing.serial.*; Serial port; String message = null; int counter = 0; void setup() { //frameRate(10); println("Opening: "+Serial.list()[0]); port = new Serial(this, Serial.list()[0], 57600); } void draw() { // running writeArray from here doesn't work //writeArray(); // print the reply from Arduino message = port.readStringUntil(13); if (message != null) println(message); } void keyPressed() { writeArray(); } void writeArray() { for( int i=0; i< 23; i++ ) { writeLed( i, 200, counter*20, 0 ); } counter++; if(counter>12) counter=0; } void writeLed( int ledNum, int r, int g, int b ) { port.write( (byte) ledNum ); port.write( (byte) r ); port.write( (byte) g ); port.write( (byte) b ); }Hi Kasper,
I’ve had good luck with the “if(Serial.available() == len)” before. I’m not sure why it doesn’t work for you, your example is light on details. Having the protocol consist of 4 bytes, one for the led number, and 3 bytes for the color, is a sound one. If I were to do this, it would look something like:
void loop() { if( Serial.available() == 4 ) { ledNum = Serial.read(); redVal = Serial.read(); grnVal = Serial.read(); bluVal = Serial.read(); set_led( ledNu, redVal, grnVal, bluVal ); } }(where “set_led()” is some function you have in your Arduino sketch that does the actual LED handling).
In Processing, the code to set a particular LED to a particular color would look something like:
void setLEDColor( int ledNum, color c ) { myPort.write( (byte) ledNum ); myPort.write( red(c) ); myPort.write( green(c) ); myPort.write( blue(c) ); } void draw() { for( int i=0; i< 22; i++ ) { // 22 == number of LEDs color c = color( random(), random(), random() ); setLedColor( i, c ); } }Because this very simple 4-byte protocol has no stop/start byte, it's possible for the Processing sketch and the Arduino sketch to get out of sync. In practice, this rarely happens.
I came across this post, because I’m trying to work a serial message out.
I want to update a string of 22 RGB led lights controlled by Arduino from processing.
I like to update the colors ( 3bytes ) from processing.
I tried several things. Like sending a really long string and using the messenger library ( http://www.arduino.cc/playground/Code/Messenger ) to put everything in an array when the carriage return is received.
That doesn’t really work.
I was thinking to make a small command.
1 address byte for the light
2 r byte
3 g byte
4 b byte
However the method above doesn’t seem to work
if( Serial.available() >= 6 ) { // command length is 6 bytes cmd0 = Serial.read(); cmd1 = Serial.read(); arg0 = Serial.read(); arg1 = Serial.read(); // ...etc... }This works for one light, but not for all. Do I need to use the while loop. And how do I make sure that I’ve received everything.
Or might the best solution by a call-response system with processing.
So arduino request light A value and processing reply’s with that value.
Fast updates are not really important, but of course I like to update as fast as possible :)
My main problem is the length of data. I worked out some things with shorter strings before.
Hope you can give me some hints, or code examples ( maybe there is already something out there on the net that I overlooked ).
Thanks in advance.
Hi ubi!
Yeah I often use a 2-3 byte protocol for most of my stuff too (which is usually computers talking to other computers). But sometimes I have to write code that talks to humans. :)
Things are well! I hope to make it back out to Amsterdam soon, later this year.
hey tod!
remember we met in amsterdam for lunch with Ben Cerveny.
how are things?
I was reading this article and I’m happy because I do the same when I have to write my protocols.
to be honest most of the times I write byte based protocols where in 2-3 bytes I can have all I need and use start/stop bytes to wrap my message.
this way I can have a flexible amount of bytes as a command to parse.
then depending on the number of bytes I decide where to route the action.
I do a lot of bit shifting/masking too :)
hope to see you again in amsterdam some day.
ciao.ubi
No worries about the formatting, I fixed it by wrapping in pre tags.
And yeah my code snippets were totally partial and didn’t include all the setup. Unfortunately, I’m an old-school C programmer so when I see “char* cmdbuf” I read that as “cmdbuf is a string defined elsewhere”. I should’ve just made it be “char cmdbuf[80]” or something.
In Arduino, people generally don’t use malloc(). This is mostly because malloc() can be confusing and you usually don’t need to dynamically allocate arrays.
As for the delay, I think that’s because there’s an error in the while() logic in what I originally wrote. I need to spend more a bit more time thinking about it, but a delay() will solve the problem as you’ve discovered.
Man, sorry about the poor formatting.
Hi there,
I tried your “human readable” code and I could only get it working with a few modifications. Mainly I had to allocate memory for the char pointers, and a delay was necessary (I’m not sure why this is). Also, it seems that char* will only hold a few bytes, so commands longer than about a dozen characters are truncated. Here’s the code:
(I’m using Arduino 15 and a Duemilanova with a 168 chip. Also I’m using the terminal app that’s built into Arduino).
——
#define STRING_SIZE 16 #define MAX_ARGS 5 void setup() { Serial.begin(9600); Serial.flush(); } void loop() { if( Serial.available() > 0 ) { // command length is 6 bytes delay(10); char* cmdbuf = (char*)malloc(sizeof(char) * STRING_SIZE); char c; int i = 0; while( Serial.available() && c != '\n' ) { // buffer up a line c = Serial.read(); cmdbuf[i++] = c; } i = 0; while( cmdbuf[++i] != ' ' ) ; // find first space cmdbuf[i] = 0; // null terminate command char* cmd = cmdbuf; int cmdlen = i; // length of cmd int args[5] = {0}, a; // five args max, 'a' is arg counter char* s; char* argbuf = cmdbuf+cmdlen+1; while( (s = strtok(argbuf, " ")) != NULL && a <= MAX_ARGS ) { argbuf = NULL; args[a++] = (byte)strtol(s, NULL, 0); // parse hex or decimal arg } int argcnt = a; // number of args read Serial.flush(); } }I finally rewrote the whole GPS reading code and used a switch statement, and now it works just the way I want it to. I wish I could implement it in a more compact way, but on the other hand this way works. My next task is to interpret and use the SiRF Binary data, and I think I am about halfway there. Pasting extensive code below (using a serial LCD adapter from Modern Devices):
#define gps Serial3 #define gpsBps 57600 byte gpsRxStage = 0; byte gpsRxPayload[1024]; int gpsRxPayloadLength = 0; int gpsRxPayloadChecksum = 0; byte gpsMsgStart[2] = {0xa0, 0xa2}; byte gpsMsgEnd[2] = {0xb0, 0xb3}; boolean gpsValidPackage = false; boolean gpsFix = false; void gpsRead() { int c, temp; byte bc; if(gps.available()) { c = gps.read(); if(c != 0xffff) { bc = byte(c); switch (gpsRxStage) { case 0: if(bc == gpsMsgStart[0]) { gpsRxStage = 1; gpsValidPackage = false; } break; case 1: if(bc == gpsMsgStart[1]) { gpsRxStage = 2; } break; case 2: gpsRxPayloadLength = (bc && 0x7f) <= gpsRxPayloadLength) { gpsRxPayload[i] = 0; gpsRxStage = 5; } break; case 5: gpsRxPayloadChecksum = bc << 8; gpsRxStage = 6; break; case 6: gpsRxPayloadChecksum += bc; temp = checksumCalc(gpsRxPayload, gpsRxPayloadLength); if(gpsRxPayloadChecksum == temp) { gpsRxStage = 7; } else { gpsRxStage = 0; } break; case 7: if(bc == gpsMsgEnd[0]) { gpsRxStage = 8; } else { gpsRxStage = 0; } break; case 8: gpsRxStage = 0; if(bc == gpsMsgEnd[1]) { gpsValidPackage = true; } break; default: lcd.print("?fUnknown error in switch statement, gpsRead"); for(;;); } } } }Hi Johan,
If you can get the device to output a fixed-length Sirf Binary datastream, that would be the easiest. You could use the third example I give above.
Otherwise, you’d probably be better reading the Payload length value right after the 0xa2,0xa0, and then just looping on that. Something like this:
byte buf[MAXSIZE]; // shouldn't be more than 512 probably if( Serial.available() == 4 ) { // we got start bytes + len int s0 = Serial.read(); int s1 = Serial.read(); int lenH = Serial.read(); int lenL = Serial.read(); if( s0 == 0xa0 && s1 == 0xa2 ) { int len = lenH << 8 + lenL; // make into 16-bit value for( int i = 0; i < len ; i++ ) { while( !Serial.available() ) { } // wait for it buf[i] = Serial.read(); } }The above has several issues if you want a fault-tolerant system, but it should work to test things out. The main problems with the above are:
– Initial Serial.read()s don’t take into account coming mid-way into the data stream.
– The “len” isn’t checked to see if it’s smaller than MAXSIZE, or if it’s the expected size for the payload you requested
– The “while(!Serial.avaialable())” can block indefinitely.
There are pretty easy solutions to all those however.
Nice tricks!
Right now I am thinking about how to implement a Sirf Binary GPS parser, reading serial strings from the GPS. The strings/containers have a two byte start sequence and another two byte stop sequence (and no “\n” etc). I would love to find a way to read this whole string without having to use multiple nested if statements.
Since you have worked with the serial communication more than I have, do you have any suggestions on how to do this?
The code should check for 0xa0 0xa2 and then read everything up to 0xb0 0xb3. Everything between these two two byte sequences is the useful and information carrying part of the message (payload length, payload and checksum).
On second thought my specific application might just use if statements or whatever is needed. Once the program is “in sync” with the GPS the stop sequence will always be followed by a new start sequence (though it can be up to one second later). As long as the program keeps up with the GPS transmissions the should be in sync.
Hehe, yeah I’ve done that too, when I was really pressed for RAM. For this post, I wanted to address things from a more normal Arduino user point of view, rather than an experienced embedded systems hacker. (and to be truthful, I really with there was a way to easily tune the size of HardwareSerial’s buffers or forgo them completely , so one could use as much RAM as possible and still gain the benefits of the Serial class)
A own (second) buffer can be avoided by directly accessing the input buffer of ther HardwareSerial library. I’ve done a hack which allows you to peek into the input buffer (read a byte without removing it from the buffer).
This is useful for ASCII based protocols where you have to check for a string command.
When you call serialPeek(index), you will get the character which is at position ‘index’ in the buffer or -1 when there are not enough characters.
The hack is very dependent of the definitions in HarwareSerial.cpp and has to be adapted when the buffer size or the structure ‘ring_buffer’ changes, so it would be much better to it directly in HardwareSerial.h/cpp
// originally defined in hardware\cores\arduino\HardwareSerial.cpp #define RX_BUFFER_SIZE 128 struct ring_buffer { unsigned char buffer[RX_BUFFER_SIZE]; int head; int tail; }; extern ring_buffer rx_buffer; // similar to Serial.read() but without removing the character from the buffer int serialPeek(int index) { // do we have enough characters? if (Serial.available() <= index) { return -1; } else { unsigned char c = rx_buffer.buffer[(rx_buffer.tail + index) % RX_BUFFER_SIZE]; return c; } }Good post, I thought that Arduino had a little buffer and if you don’t retrieve the character immediately it would be overrun like old uarts. I?l have to make a revision on my code…
I’ve been working on an iPod serial library for the Arduino at http://github.com/finsprings/arduinaap. So far I’ve gone with a callback approach, where the client sketch registers callback functions to be called when certain messages come in over serial from the iPod. The sketch makes a call to the library in loop() so the library can keep pulling data from serial, but otherwise the sketch is free to forget about serial stuff.
Not sure if it’s gonna pan out, but for the few cases I’ve implemented it seems to be pretty clean.
really good post,
i think it will help me a lot,
thanks a lot!
I designed a simple protocol which coded up nicely in C on the Arduino and Python inside MaxMSP on the host. It’s modelled after MIDI system exclusive: an ASCII character with the top bit set is the actual (single character) command, and subsequent 8-bit arguments are nybblised. (The number of arguments is known at both ends, according to the command character.)
I prefer this to argument-counting since the latter won’t recover if there’s any kind of framing error (caused by a dropped byte).