Sync data of 2 device using serial

ahbi82 · #1 11-Feb-2011, 03:50

Hi all, I'm sending data to and from 2 devices in the following format
<header byte><data><checksum byte><end byte>.

data can be anything, its basically flatten data from float and uint.

There`s chance that data may contain header or end or checksum.

Anyone know of a algorithm to so call "multiplex and de-multiplex" the data?

Any suggestion would be great.

Many thanks.

davekw7x · #2 11-Feb-2011, 09:28

Quote:

Originally Posted by ahbi82

Hi all, I'm sending data to and from 2 devices in the following format

Well, there is no way to send arbitrary binary data like this unless the entire packet is encapsulated in something else that has unambiguous ways of identifying packet begin and end. (For example feeding some data bytes to a UDP packet program that sends your stuff on an Ethernet channel. All of the encapsulation and transmission (and error detecting, etc.) stuff is taken out of the hands of the application program.

I mean if you just send packets like you are proposing over a simple serial channel, if there is no indicator that tells the length of the data stuff, how the heck could you ever know whether a given byte is the end byte or one of the data bytes?

If you include a <length> field, the receiver still can't know when a new packet begins unless there is a way of synchronizing transmitter and receiver. Even if there is some kind of (manual) synchronization scheme to start things off, what happens if a transmission error causes the receiver to lose its place?

If there is a non-zero probability of errors in the data transmission medium itself, how would you ever recover from an error that causes mis-identification of the <header byte> or the <end byte> (or the length field, if you decide to put one in)?

So, how could we accomplish this kind of transmission without going to UDP or Ethernet or other complicated protocol layers?

Well...

One possibility might be to specify that the data field and the checksum would contain the ASCII hexadecimal digits for the individual bytes. Use something other than an ASCII character for a hexadecimal digit for the <begin> byte. Put in a fixed number of pairs of ASCII hexadecimal digits in a <length> field right after the <begin> byte, and pairs of ASCII hexadecimal digits for the data field and then pair(s) of ASCII hexadecimal digits for the checksum.

Let's suppose that the maximum data size is no more than 255, so the length field can be represented as a single byte.

I'll use ':' for the <header byte>, and I will use a single checksum byte, which will be the 2's complement of the sum of the length byte and all of the data bytes. (Kind of like the Intel Hex file format, but without the "address" and "record type" fields.)

Here is the specification:

Code:

+-------------------------------------------------------------+
|Start | Byte Count|           Data              | Checksum   |
|Code  |     n     |   Sequence of n bytes       |            |
+=============================================================+
|ASCII | Two ASCII |   Each byte is represented  | Two ASCII  |
| ':'  | hex digits|   by two ASCII hex digits   | hex digits |
+-------------------------------------------------------------+

The checksum is equal to the least significant byte of
the two's complement of the sum of the byte count and
all of the data bytes.
Number of bytes in the packet is 5+2*n

For example, to send 0x01, 0x02, 0x03:

The length is 0x03, and will be sent as '0', '3'

Data byte 0x01 will be sent as '0','1'
Data byte 0x02 gets sent as '0', '2'.
Data byte 0x03 becomes '0','3'.

The checksum is the least significant byte of the two's
complement of the sum

0x03+0x01+0x02+0x03.

That sum is 0x09, and the least significant byte of its
two's complement is 0xf7. That will be sent as 'F','7'

So: here are the characters of the packet

:03010203F7

So, it takes 11 bytes to send the three data bytes. If the data length were, say 200 bytes, The packet length would be 405 characters.

Bottom line:

All characters are text (printable ASCII characters).
The only character that is not an ASCII hex digit is the start code
All packets have a checksum, so many transmission errors can be detected, in which case the receiver can reject non-valid packets.

The checksum method is very primitive, and is only recommended for really small transmission error probabilities. More robust techniques (such as use of Cyclic Redundancy Check) can decrease the probability that a bad packet will be mis-identified as valid, but at the expense of extra processing requirements. If transmission error probability is significant, and packet loss is not acceptable, a scheme involving "ack" and "nack" (acknowledge and non-acknowledge) packets from the receiver to indicate success or failure of each transmission might be appropriate.

Regards,

Dave

ahbi82 · #3 12-Feb-2011, 00:22

Thanks Dave. Initially my implementation was what you mentioned, However ASCII wasn't part of the choice as i'm dealing with floating points (for example 6 precision).

Device 1 will send 100 I/O readings to device 2. Its too slow in this sense. Thats why i'm thinking of packing the data. But i would lose sync.

I have an algorithm to actually pack the data with protected characters, it will replace the <data> with the protected chars and perform. But will fail on some checksum values. Basically, its sort of like a 2 bit error thingy.

With your experience, what could you suggest for me to go ahead. I've been cracking my head for few weeks.

davekw7x · #4 12-Feb-2011, 10:30

Quote:

Originally Posted by ahbi82

Thanks Dave. Initially my implementation was what you mentioned, However ASCII wasn't part of the choice as i'm dealing with floating points (for example 6 precision).

My suggestion was an example of a way to send arbitrary binary values from one end to the other using two ASCII hex digits for each binary byte.

If you are going to send text representing the decimal values of numbers, here's the deal: Use something that can never appear in a number to be a synchronization byte. For example ':' Then just send the ASCII digits (and decimal point and sign, if appropriate). Separate the numbers by something that can not appear in a number. Maybe an ASCII space.

You can use something like sprintf() to put the decimal digits (and decimal point and sign) of the numbers into a character buffer and transmit the buffer. (In C++ it's probably considered to be a little more "elegant" to use stringstreams, but the results can be the same).

Note that you can convert the checksum to its 1-3 digit decimal representation (range 0-255), so you wouldn't ever have to send a binary value of anything. It's all decimal digits, right?

Bottom line: Your packets can contain only the following characters:

The synchronizer byte, maybe ':'
ASCII decimal digits.
ASCII '.' for the decimal point
ASCII space to separate the numerical quantities
ASCII '-' if any of the numbers can be negative.

So I don't see what the problem would be with synchronization: The one and only place where you would transmit a ':' would be at the beginning of a new packet.

Now...

Let's look at some numbers:

If you have an 32-bit integer, whose value is, say 123, it takes four packet bytes (the three digits plus a space). If the value is, say 12345678, it takes nine packet characters (eight digits plus a space). So, packet sizes can vary, depending on the values of the numbers. (This might or might be a problem). If you want to make all of the packets the same size, than you can use spaces to pad each integer value to the maximum number of digits that you will ever encounter (plus an extra character for the '-' sign if the value is negative). It might take 10 or eleven packet bytes to send a given 32-bit integer value.

On the other hand, if you send the ASCII hex digits of the four-byte quantity, it always takes eight characters for each one. So, which is faster? It depends.

For floating point numbers, it's a little different, depending on your requirements. For example, if you have a floating point number that was calculated by dividing 1234 by 100, the result is not exactly representable as a binary floating point value. The compiler will do its best to store the number that is the nearest it can get. When you print out the number to 16 significant digits, then you may get something like 12.34000015258789 (or 1.2340000152587891e+01 in scientific notation).

The question is: What value do you want to convey to the other end? The exact internal machine representation, or the value rounded to some number of significant decimal digits? If you want to send 16 significant decimal digits (as text), it can take over 20 characters (16 significant digits, two or three digits for the exponent value, sign for number, sign for exponent). If the numbers are fixed precision and you don't have negative numbers, maybe you can save a couple of digits. What is the range of numbers that you want to send? Is there a downside to sending the rounded decimal approximation of the actual machine representation of the number?

On the other hand...
If you send the ASCII hex digits for the internal representation of a four-byte floating point number, it takes eight characters in the packet. Period.

Which is faster? Stop me if you have heard this before, but it depends.

In the scheme that I outlined, each packet will have only ASCII characters representing binary values.

Each numerical binary value will send two ASCII hex digits for each byte in the number.

Look at my example: The ':' is the only character that is not an ASCII hex digit, so that's the synchronizer.

You have to convert each numerical value to its ASCII hex digits. (And the receiver will convert them back.)

If you absolutely, positively, unequivocally, know that the transmitter and receiver are running on the same type architecture (both big-endian or both little-endian,), and the applications were compiled with the same version of the same compiler (so that ints are the same length, for example), then you can do something like the following to implement the transmitter:

CPP / C++ / C Code:

#include <iostream>
#include <iomanip>

using namespace std;

// For my example, I just use an int and a float for each packet
struct PacketData
{
    int ivalue;
    float fvalue;
};

unsigned char nibbleToASCIIHex(unsigned char x);

int main()
{

    // This is the number of data bytes that will be used to
    // form the packet
    const int packetDataSize = sizeof(PacketData);

    // Here's the packet to be transmitted
    // Start symbol 
    // Two ASCII hex digits for the length
    // Two ASCII hex digits for each data byte
    // Two ASCII hex digits for the checksum
    unsigned char buffer[1 + 2 + 2*packetDataSize + 2];
    int buffersize = sizeof(buffer);

    // Here are the index values of the components of the
    // packet to be transmitted

    // Start here
    const int startByteIndex = 0;

    // Here's where length digits will go in the packet
    const int lengthIndex    = startByteIndex + 1;

    // Start of the digits of packet data in the packet
    const int dataStartIndex = lengthIndex + 2;

    // Each packet data byte has two digits, so the checksum
    // digits will go here
    const int checksumIndex  = dataStartIndex + 2*packetDataSize;


    // Here's the struct that has the (binary) data values to be
    // transmitted:
    PacketData data;

    // Give it a couple of values to play with
    data.ivalue = 12345678;
    data.fvalue = 12.34;

    // Make a pointer to char that will be used to access the
    // packet data bytes one at a time
    unsigned char *source = reinterpret_cast<unsigned char *>(&data);

    // Summarize
    cout << "Number of binary data bytes = " << packetDataSize << endl;

    cout << hex << setfill('0');
    cout << "The binary data bytes: " << endl << "  ";
    for (int i = 0; i < packetDataSize; i++){
        cout << "0x" << setw(2)<< (int)source[i] << " ";
    }
    cout << endl;

    cout << dec;
    cout << "Total Number of ASCII chars in packet = " << buffersize << endl;
    cout << hex;
    // Now build the packet to be transmitted
    buffer[startByteIndex] = ':';
    buffer[lengthIndex] = nibbleToASCIIHex(packetDataSize >> 4);
    buffer[lengthIndex+1] = nibbleToASCIIHex(packetDataSize);

    // Step through the data bytes, updating the checksum as you go
    int i, j;
    unsigned char checksum = 0;
    for (i = dataStartIndex, j = 0; i < checksumIndex; i+= 2, j++) {
        checksum += source[j];
        buffer[i] = nibbleToASCIIHex(source[j] >> 4);
        buffer[i+1] = nibbleToASCIIHex(source[j]);
    }
    cout << "From calculation: Checksum = 0x" 
         << setw(2)<< (unsigned int)checksum << endl;
    // Put the two's complement of the calculated checksum
    // into the packet
    checksum = -checksum;
    cout << "Two's complement: Checksum = 0x"
         <<  setw(2) << (unsigned int)checksum << endl;

    // Store the checksum
    buffer[checksumIndex]   = nibbleToASCIIHex(checksum >> 4);
    buffer[checksumIndex+1] = nibbleToASCIIHex(checksum);

    cout << endl;

    cout <<"Finally, here's the packet:" << endl;
    for (i = 0; i < buffersize; i++) {
        cout << buffer[i];
    }
    cout << endl;

    return 0;
}

unsigned char nibbleToASCIIHex(unsigned char x)
{
    unsigned char digits[] = "0123456789ABCDEF";
    return digits[x & 0x0f];
}

Output:

Code:

Number of binary data bytes = 8
The binary data bytes: 
  0x4e 0x61 0xbc 0x00 0xa4 0x70 0x45 0x41 
Total Number of ASCII chars in packet = 21
From calculation: Checksum = 0x05
Two's complement: Checksum = 0xfb

Finally, here's the packet:
:084E61BC00A4704541FB

Instead of a single struct for the entire packet and stepping through the bytes, you can build the output packet, a number at a time, using a similar approach for the individual numbers.

Regards,

Dave

Footnote:
In general network applications, it is almost always specified that values of multi-byte numerical quantities will be sent most-significant byte first, so simply grabbing byte-at-a-time 32-bit ints or floats from memory won't work if the machine is little-endian. Even if the architectures are same-endian, not all compilers use the same internal representation of floats, so something special might have to be done to get the floating point representation to the other end also. (For example: I have run across some compilers for embedded processors that use non-standard floating point representation.)

Also, since lengths of ints may be different for different systems, a more robust protocol would specify a particular length of the integer values. Furthermore, not all compilers will pack structs in the same way (depending on the exact nature of the struct), so, depending on the struct, it might be better to build the transmitted packet a value at a time rather than use the scheme that I showed to grab the bytes from the struct.

Mexican Bob · #5 14-Mar-2011, 02:02

Quote:

Originally Posted by ahbi82

Thanks Dave. Initially my implementation was what you mentioned, However ASCII wasn't part of the choice as i'm dealing with floating points (for example 6 precision).

Device 1 will send 100 I/O readings to device 2. Its too slow in this sense. Thats why i'm thinking of packing the data. But i would lose sync.

I have an algorithm to actually pack the data with protected characters, it will replace the <data> with the protected chars and perform. But will fail on some checksum values. Basically, its sort of like a 2 bit error thingy.

With your experience, what could you suggest for me to go ahead. I've been cracking my head for few weeks.

The following are my random ramblings:

You may want to consider using a variant type to represent your data. Use an appropriate compiler pragma to pack the data using 1 byte packing.

An advantage of C is that the members of a struct must be located sequentially in memory beginning with the first member located at the address of the instance.

One should use the appropriate hton and ntoh macros when sending/receiving data from a "network" stream.

If the two devices are of the same endianness and if performance is paramount, you may elect to disregard converting to network byte ordering before sending and converting from network when receiving.

If you are on a serial link, many of which can be inherently unreliable, checksum the data and that "packet" separately. Approach it two-fold. If the packet checksum fails, either disregard it completely or request a retransmission.

If the data checksum fails, either error correct or request a retrans from your "networking layer."

As Dave indicated, variations among compilers and machine implementations will represent data types differently such that a binary sequence may not make any sense once received. If you can eliminate that as a concern, such as when using the same compiler and same hardware architecture between devices 1 and 2, the simplest approach is to use a structure and pack it. Use bit-field padding if you feel more comfortable having your data on even byte boundaries for things like 2, 4 or 8 byte fields. Using memcpy, which doesn't care about even or odd boundaries, will negate that issue, but on some architectures, performance can be dramatically improved when accessing memory on even byte boundaries. Depending on the architecture, even byte boundaries that are multiples of the sizeof(int) may also improve performance...which can lead to insight into how to approach packing and/or padding.

What I'd recommend is setting up an "echo" between your two devices. Use a ping-pong variable in the structure to indicate state between the devices.

Use a data pattern that is easily viewed on the scope or LA or even in a debug memory window. For example:

0x1111,1111
0x2222,2222
0x3333,3333
...
0xAAAA,AAAA
0xBBBB,BBBB

...and switch between sender and receiver "incrementing" the data value such that the next pattern is used. Once this works well, run them all day/night using pseudo random values or "real data" if available.

The problem with using real data is how do you tell if it is "right" or not? Particularly in "unattended mode." Obviously, checksums are the answer.

Another thing that you can do is to not reinvent the wheel. I don't know what "devices" you're using, but if you can use an existing, proven framework, you'll be light years ahead of the game.

Check out: state-machine.com for QP/C and/or QP/C++

This framework is amazingly excellent for embedded devices. It is great for a lot of uses, including device drivers and embedding hierarchical state machines and/or event processing into just about any architecture in a very small footprint.

What you'll want to do is to take a look at the QSpy piece. Miro implements a device independent serial protocol for displaying data to the user.

As an aside and not to be contrary to any other opinions/experiences offered, I tend not to use ASCII and prefer binary when reasonable. I don't like sending 8 or 10 bytes to represent 4: 0x12345678

...however, in cases where a relatively limited amount of data is to be sent on the communications link, it makes a lot of sense to use the simplest representation possible that also ensures reliable data transmission and reception.

What happens when you send ASCII hex characters is that both sides have to convert to AHex and back...not counting the extra bytes used in transmission.

A serial stream is guaranteed to be serially encoded regardless of byte order. IMO, it is a whole lot easier to shift bytes around and locate "packets" into structures than to decode ASCII hex into real byte data.

I once wrote an entire command and control communications protocol between a LE ARM9 and a BE ColdFire V2. It was all in binary. It was all tested and working. The EE on the project who designed the board ripped it all out and put in ASCII because he "could see it" using Hyperterminal.

MxB

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Rate This Thread
Rate This Thread:

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Using getline to read in data	krisbot	C Programming Language	4	24-Oct-2004 15:57
Transferring data between arrays	butters	C Programming Language	3	07-Jul-2004 22:55
Prob. w/ Initalizing Data Members to 0	BobbyMurcerFan	C++ Forum	5	26-Jun-2004 20:01
[CONTEST?]Data Structure Test	dsmith	C Programming Language	2	06-Jun-2004 15:13
Automate a data change php form	mjfmn	MySQL / PHP Forum	4	20-Oct-2003 09:37

Sync data of 2 device using serial

Re: sync data of 2 device using serial

Re: Sync data of 2 device using serial

Re: Sync data of 2 device using serial

Re: Sync data of 2 device using serial