Sync data of 2 device using serial
Hi all, I'm sending data to and from 2 devices in the following format
<header byte><data><checksum byte><end byte>.
data can be anything, its basically flatten data from float and uint.
There`s chance that data may contain header or end or checksum.
Anyone know of a algorithm to so call "multiplex and de-multiplex" the data?
Any suggestion would be great.
Re: sync data of 2 device using serial
I mean if you just send packets like you are proposing over a simple serial channel, if there is no indicator that tells the length of the data stuff, how the heck could you ever know whether a given byte is the end byte or one of the data bytes?
If you include a <length> field, the receiver still can't know when a new packet begins unless there is a way of synchronizing transmitter and receiver. Even if there is some kind of (manual) synchronization scheme to start things off, what happens if a transmission error causes the receiver to lose its place?
If there is a non-zero probability of errors in the data transmission medium itself, how would you ever recover from an error that causes mis-identification of the <header byte> or the <end byte> (or the length field, if you decide to put one in)?
So, how could we accomplish this kind of transmission without going to UDP or Ethernet or other complicated protocol layers?
One possibility might be to specify that the data field and the checksum would contain the ASCII hexadecimal digits for the individual bytes. Use something other than an ASCII character for a hexadecimal digit for the <begin> byte. Put in a fixed number of pairs of ASCII hexadecimal digits in a <length> field right after the <begin> byte, and pairs of ASCII hexadecimal digits for the data field and then pair(s) of ASCII hexadecimal digits for the checksum.
Let's suppose that the maximum data size is no more than 255, so the length field can be represented as a single byte.
I'll use ':' for the <header byte>, and I will use a single checksum byte, which will be the 2's complement of the sum of the length byte and all of the data bytes. (Kind of like the Intel Hex file format, but without the "address" and "record type" fields.)
Here is the specification:
For example, to send 0x01, 0x02, 0x03:
The length is 0x03, and will be sent as '0', '3'
Data byte 0x01 will be sent as '0','1'
Data byte 0x02 gets sent as '0', '2'.
Data byte 0x03 becomes '0','3'.
The checksum is the least significant byte of the two's
complement of the sum
That sum is 0x09, and the least significant byte of its
two's complement is 0xf7. That will be sent as 'F','7'
So: here are the characters of the packet
So, it takes 11 bytes to send the three data bytes. If the data length were, say 200 bytes, The packet length would be 405 characters.
The checksum method is very primitive, and is only recommended for really small transmission error probabilities. More robust techniques (such as use of Cyclic Redundancy Check) can decrease the probability that a bad packet will be mis-identified as valid, but at the expense of extra processing requirements. If transmission error probability is significant, and packet loss is not acceptable, a scheme involving "ack" and "nack" (acknowledge and non-acknowledge) packets from the receiver to indicate success or failure of each transmission might be appropriate.
Sometimes I just can't help myself...
Last edited by davekw7x : 11-Feb-2011 at 11:26.
Re: Sync data of 2 device using serial
Thanks Dave. Initially my implementation was what you mentioned, However ASCII wasn't part of the choice as i'm dealing with floating points (for example 6 precision).
Device 1 will send 100 I/O readings to device 2. Its too slow in this sense. Thats why i'm thinking of packing the data. But i would lose sync.
I have an algorithm to actually pack the data with protected characters, it will replace the <data> with the protected chars and perform. But will fail on some checksum values. Basically, its sort of like a 2 bit error thingy.
With your experience, what could you suggest for me to go ahead. I've been cracking my head for few weeks.
Re: Sync data of 2 device using serial
My suggestion was an example of a way to send arbitrary binary values from one end to the other using two ASCII hex digits for each binary byte.
If you are going to send text representing the decimal values of numbers, here's the deal: Use something that can never appear in a number to be a synchronization byte. For example ':' Then just send the ASCII digits (and decimal point and sign, if appropriate). Separate the numbers by something that can not appear in a number. Maybe an ASCII space.
You can use something like sprintf() to put the decimal digits (and decimal point and sign) of the numbers into a character buffer and transmit the buffer. (In C++ it's probably considered to be a little more "elegant" to use stringstreams, but the results can be the same).
Note that you can convert the checksum to its 1-3 digit decimal representation (range 0-255), so you wouldn't ever have to send a binary value of anything. It's all decimal digits, right?
Bottom line: Your packets can contain only the following characters:
So I don't see what the problem would be with synchronization: The one and only place where you would transmit a ':' would be at the beginning of a new packet.
Let's look at some numbers:
If you have an 32-bit integer, whose value is, say 123, it takes four packet bytes (the three digits plus a space). If the value is, say 12345678, it takes nine packet characters (eight digits plus a space). So, packet sizes can vary, depending on the values of the numbers. (This might or might be a problem). If you want to make all of the packets the same size, than you can use spaces to pad each integer value to the maximum number of digits that you will ever encounter (plus an extra character for the '-' sign if the value is negative). It might take 10 or eleven packet bytes to send a given 32-bit integer value.
On the other hand, if you send the ASCII hex digits of the four-byte quantity, it always takes eight characters for each one. So, which is faster? It depends.
For floating point numbers, it's a little different, depending on your requirements. For example, if you have a floating point number that was calculated by dividing 1234 by 100, the result is not exactly representable as a binary floating point value. The compiler will do its best to store the number that is the nearest it can get. When you print out the number to 16 significant digits, then you may get something like 12.34000015258789 (or 1.2340000152587891e+01 in scientific notation).
The question is: What value do you want to convey to the other end? The exact internal machine representation, or the value rounded to some number of significant decimal digits? If you want to send 16 significant decimal digits (as text), it can take over 20 characters (16 significant digits, two or three digits for the exponent value, sign for number, sign for exponent). If the numbers are fixed precision and you don't have negative numbers, maybe you can save a couple of digits. What is the range of numbers that you want to send? Is there a downside to sending the rounded decimal approximation of the actual machine representation of the number?
On the other hand...
If you send the ASCII hex digits for the internal representation of a four-byte floating point number, it takes eight characters in the packet. Period.
Which is faster? Stop me if you have heard this before, but it depends.
In the scheme that I outlined, each packet will have only ASCII characters representing binary values.
Each numerical binary value will send two ASCII hex digits for each byte in the number.
Look at my example: The ':' is the only character that is not an ASCII hex digit, so that's the synchronizer.
You have to convert each numerical value to its ASCII hex digits. (And the receiver will convert them back.)
If you absolutely, positively, unequivocally, know that the transmitter and receiver are running on the same type architecture (both big-endian or both little-endian,), and the applications were compiled with the same version of the same compiler (so that ints are the same length, for example), then you can do something like the following to implement the transmitter:
CPP / C++ / C Code:
Instead of a single struct for the entire packet and stepping through the bytes, you can build the output packet, a number at a time, using a similar approach for the individual numbers.
In general network applications, it is almost always specified that values of multi-byte numerical quantities will be sent most-significant byte first, so simply grabbing byte-at-a-time 32-bit ints or floats from memory won't work if the machine is little-endian. Even if the architectures are same-endian, not all compilers use the same internal representation of floats, so something special might have to be done to get the floating point representation to the other end also. (For example: I have run across some compilers for embedded processors that use non-standard floating point representation.)
Also, since lengths of ints may be different for different systems, a more robust protocol would specify a particular length of the integer values. Furthermore, not all compilers will pack structs in the same way (depending on the exact nature of the struct), so, depending on the struct, it might be better to build the transmitted packet a value at a time rather than use the scheme that I showed to grab the bytes from the struct.
Sometimes I just can't help myself...
Re: Sync data of 2 device using serial
The following are my random ramblings:
You may want to consider using a variant type to represent your data. Use an appropriate compiler pragma to pack the data using 1 byte packing.
An advantage of C is that the members of a struct must be located sequentially in memory beginning with the first member located at the address of the instance.
One should use the appropriate hton and ntoh macros when sending/receiving data from a "network" stream.
If the two devices are of the same endianness and if performance is paramount, you may elect to disregard converting to network byte ordering before sending and converting from network when receiving.
If you are on a serial link, many of which can be inherently unreliable, checksum the data and that "packet" separately. Approach it two-fold. If the packet checksum fails, either disregard it completely or request a retransmission.
If the data checksum fails, either error correct or request a retrans from your "networking layer."
As Dave indicated, variations among compilers and machine implementations will represent data types differently such that a binary sequence may not make any sense once received. If you can eliminate that as a concern, such as when using the same compiler and same hardware architecture between devices 1 and 2, the simplest approach is to use a structure and pack it. Use bit-field padding if you feel more comfortable having your data on even byte boundaries for things like 2, 4 or 8 byte fields. Using memcpy, which doesn't care about even or odd boundaries, will negate that issue, but on some architectures, performance can be dramatically improved when accessing memory on even byte boundaries. Depending on the architecture, even byte boundaries that are multiples of the sizeof(int) may also improve performance...which can lead to insight into how to approach packing and/or padding.
What I'd recommend is setting up an "echo" between your two devices. Use a ping-pong variable in the structure to indicate state between the devices.
Use a data pattern that is easily viewed on the scope or LA or even in a debug memory window. For example:
...and switch between sender and receiver "incrementing" the data value such that the next pattern is used. Once this works well, run them all day/night using pseudo random values or "real data" if available.
The problem with using real data is how do you tell if it is "right" or not? Particularly in "unattended mode." Obviously, checksums are the answer.
Another thing that you can do is to not reinvent the wheel. I don't know what "devices" you're using, but if you can use an existing, proven framework, you'll be light years ahead of the game.
Check out: state-machine.com for QP/C and/or QP/C++
This framework is amazingly excellent for embedded devices. It is great for a lot of uses, including device drivers and embedding hierarchical state machines and/or event processing into just about any architecture in a very small footprint.
What you'll want to do is to take a look at the QSpy piece. Miro implements a device independent serial protocol for displaying data to the user.
As an aside and not to be contrary to any other opinions/experiences offered, I tend not to use ASCII and prefer binary when reasonable. I don't like sending 8 or 10 bytes to represent 4: 0x12345678
...however, in cases where a relatively limited amount of data is to be sent on the communications link, it makes a lot of sense to use the simplest representation possible that also ensures reliable data transmission and reception.
What happens when you send ASCII hex characters is that both sides have to convert to AHex and back...not counting the extra bytes used in transmission.
A serial stream is guaranteed to be serially encoded regardless of byte order. IMO, it is a whole lot easier to shift bytes around and locate "packets" into structures than to decode ASCII hex into real byte data.
I once wrote an entire command and control communications protocol between a LE ARM9 and a BE ColdFire V2. It was all in binary. It was all tested and working. The EE on the project who designed the board ripped it all out and put in ASCII because he "could see it" using Hyperterminal.
|Thread Tools||Search this Thread|
|Rate This Thread|
|Thread||Thread Starter||Forum||Replies||Last Post|
|Using getline to read in data||krisbot||C Programming Language||4||24-Oct-2004 16:57|
|Transferring data between arrays||butters||C Programming Language||3||07-Jul-2004 23:55|
|Prob. w/ Initalizing Data Members to 0||BobbyMurcerFan||C++ Forum||5||26-Jun-2004 21:01|
|[CONTEST?]Data Structure Test||dsmith||C Programming Language||2||06-Jun-2004 16:13|
|Automate a data change php form||mjfmn||MySQL / PHP Forum||4||20-Oct-2003 10:37|