Fast WiFi Link For Binary Data

[Arctic_Eddie]

I have a project nearly complete that will eventually transfer 8192 bytes of binary sonar data from an RC boat to a shore station then into a laptop running a Lazarus data processing and 3D imaging program. It is running now with 1024 bytes. The data is captured by four dedicated Arduino Nanos then transferred to a Mega by I2C then to a shore station Mega by nRF24 then into the laptop by a serial/USB adapter. The sequential transfer of data has become too long and my 3D refresh rate is down to about 1Hz.

This is what I would like to do. The Nanos and Mega on the boat will be replaced by three Teensy 3.2 boards. One will fire the TX burst and do a few other functions. The other two Teensies will capture the data synchronously from the sonar sensors in a dual ADC fashion. All three will be connected to a Sparkfun ESP8266 Thing Dev board. They will all get the same instruction but only the one addressed, in the message packet, will respond. The data, 1024 items of two words in a record from each sonar pair, will go directly to a WiFi adapter connected to the laptop. The Synapse library will be used to send commands to the boat ESP/Teensies and receive the returned data. This eliminates many of the sequential operations and gets the data in the most direct fashion.

Here's my questions.

1. Does the concept sound feasible?

2. I envision the ESP as being a server of the simplest form. Is this the best way?

3. What protocol will be the fastest for binary data going both ways? Command packets going out will be a record of three words. Data returned will be two groups of 1024 records of two words in each record.

Note:
One Teensy is in the mail but the ESP8266 has not been ordered yet. All the libraries for Teensy and ESP are installed in the Arduino 1.6.5-rc5 IDE.

Your suggestions will be greatly appreciated.

[CaptBill]

I would try out ESP-Link :https://github.com/jeelabs/esp-link for a good "wifi---> UART bridge" and then you can talk directly from the Teensy over TCPIP or map it to a COM port. So 2) use it as a transparent bridge vs. your typical server arrangement. This way you just flash the bitfile and don't need to code the ESP yourself, unless you just have to.

You should also pass only hexadecimal values if you want compact transmission packets. You will need to manage the actual bits and design an efficient data packet. The easiest/best way to work with bits is using 'sets'. Encoder(Teensy)------>decoder(pc app).

[Arctic_Eddie]

I don't understand how a 'set' is going to do anything for my data as it's just raw binary. It's created by ADC digitizing an analogue sonar echo with 10 or 12 bit resolution. The Teensy ADC routines for two channel operation returns a structure of two 16 bit integers, channel 1 and channel 2. I want to pass this back to the shore station in that form as a continuous byte sequence. I'll parse it into sonarArray[0..1, 0..1023] where the first index is the sonar pair, front/back  and left/right pairs and the second is a time sequence in the echo. My command packet going out is also numeric as just three words, uint16_t. The meaning of the numbers will be handled at the receiving end with case/switch statements. I want to avoid any string data.

The transparent UART bridge looks like the best scheme as the three Teensies can listen and talk through the boat ESP and the laptop can do the same thing through it's FTDI USB/UART adapter and another ESP.  I already have the serial link running using LazSerial/Synaser talking to my shore station Mega2560 at 2Mbps. I hope the ESP-Link library works with the Teensy. If not, then I can still use the Mega controller as it only runs as a UART to ESP interface and the TX burst generator. The real value of the Teensies will be on the boat with the faster dual channel ADCs.

I was not aware of the ESP-Link package. Thank you for that info. I had even considered using a video TX/RX pair at each end on different frequencies with the UART TTL as the signal. The problem with that scheme is the unknown coupling of the input to the FSK modulator, AC or DC.

[CaptBill]

I don't understand how a 'set' is going to do anything for my data as it's just raw binary.

Sets are just a good language feature to deal with anything dealing with bit manipulation. Next, it would seem, is advanced records. These structures are light weight and close to the bare metal. I am speaking in general terms or 'best overall approach' when dealing at this level, it does appear. Sets are very powerful and should not be overlooked if you want top performance.

I hope the ESP-Link library works with the Teensy.

ESP-link is a direct tx/rx uart, actually 2 uarts. So no problems there.

Interesting project. Got any pics?

[Arctic_Eddie]

Here's a screen shot from a few days ago. The echoes in the fathometer on the right are fake and generated in the shore station Mega. A little noise was added with the random() function. The purpose of the boat is to find the bodies of drowning victims for local law enforcement. It's present name is Body Barge but that will change. I think I'll call the boat Lazarus due to the names historical significance. I used the same echo for both front/back and left/right data. The position is then on the diagonal depending on whether echo 1 or echo2 is used. Red is the forward sensor and yellow is the rear sensor. Two more will be added for left and right. Echo 0 is the transmitted pulse at the top, which is software controlled as to how many cycles in the burst. The TX is a 50W 40KHz ultrasonic cleaning tank module driven by an H-bridge.

My Teensy arrived today so now testing some of the ADC functions. Also looked at the ESP-Link docs and website but find it very confusing as to how I'll use this code with the Sparkfun ESP8266 Thing Dev board. Will do some searching for examples for a UART bridge. I'd like to connect the board directly to my FTDI adapter then to the USB port and read/write with SerialPort.SynSer.RecvBufferEx(..) and SerialPort.WriteBuffer(..) commands. The code will not likely change much as the shore station Mega is just a UART bi-directional pass-thru interfacing the laptop with an nRF24 transceiver. The Arm processor in the Thing can do the same task.

[CaptBill]

Neat!
The ESP-link enables you to deal directly with the generic read() and write() of the UARTs which is the beauty of it. Setting it up specifically for the 'Esp Thing' might need working out.

I'm a little confused with the setup you have. It seems things could be simplified. Do you really need FTDI? Can't you just go directly from the wifi--->COM port connection to pass the data? The ESP8266 is a full replacement for FTDI, when you need the same functionality wirelessly.

[CaptBill]

Forgot to ask are you running windows on the pc? If so, you will need to get a 'port redirector' app to map the tcp connection (telnet:23) to a COM port. I use HW_VirtualSerialPort from here:

http://www.hw-group.com/products/hw_vsp/index_en.html

Now you have a direct Teensy to pc COM port connection. No need for FTDI.

[Arctic_Eddie]

The PC is an Acer running W7-64. The FTDI is just an adapter to go from the PC USB port to UART pins. It's the same one I use to program Pro Minis using the Arduino IDE. I was going to connect the TX/RX pins to the Thing RX/TX hardware port and send the outgoing data from the PC using LazSerialOut commands over the bridge to the boat. The data coming back from the boat over the bridge will appear on the same Thing pins going into the PC and read by LazSerialIn. The Laz program will use the two LazSerial commands previously cited. I really only need the equivalent of a TX/RX pair of radio wires. The ESP-Link library appears to be intended for the bare ESP chip and a users Arduino but the Thing already has a 32bit 72MHz ARM on board. I just need to find the simplest way of creating the bridge. That's why I originally thought of the boat Thing being a server and the PC Thing the client. The serial code I already have would be directly reusable. The only difference is the nature of the sonar data due to Teensies being the source instead of Nanos and the absence of the nRF24 link. Primarily, it's just packet size coming back, 32 vs 128 bytes per packet.

[Arctic_Eddie]

PS
Were you thinking of using only one Thing on the boat and a PC WiFi adapter for the shore side? I had thought of that also using the Synapse library within the Laz program. I thought it would be easier to get two Things to talk to each other and go into the PC by serial(FTDI). All of my original serial code would work and i would not have to deal with new stuff like TFTPSend, etc.

[CaptBill]

Sounds to me like you can eliminate one of the ESP's , and FTDI altogether. You simply need  one ESP as the 'wifi uart' and a wifi router. You will need the "virtual COM port redirector" app running on Windows. Linux can map TCP--->COM port natively.

Here's how it works. (pseudo instructions)
1) Install/'flash' the binary, the 'esplink.bin' (or whatever it is)
2) Reset the unit and find the wifi signal on the pc. Connect to it.
3) Now you can view the web interface at 'http://192.168.xx.xx'. You have all the settings/configuration forms here.
4) Set up your pin mappings and uart settings etc.

Now you have the wireless equivalent to an FTDI cable with benefits of a more powerful MCU.

I can't find a datasheet on the ESP Thing or which version of the esp8266 it uses. The newer models have 4mb of sram, which may serve you well as a buffer. I use these:

http://www.ebay.com/itm/281811531729?_trksid=p2057872.m2749.l2649&ssPageName=STRK%3AMEBIDX%3AIT

[CaptBill]

PS
Were you thinking of using only one Thing on the boat and a PC WiFi adapter for the shore side? I had thought of that also using the Synapse library within the Laz program. I thought it would be easier to get two Things to talk to each other and go into the PC by serial(FTDI). All of my original serial code would work and i would not have to deal with new stuff like TFTPSend, etc.

Oh no, you just do your settings at the web interface. In your case, this will be a 'set and forget' thing anyway.

http://domoticx.com/wp-content/uploads/ESP266-esp-link-Firmware-screen-01.png

[Arctic_Eddie]

Ok, got it but it will take some reading to sort it out.

Thanks again for all of your help.

PS
The photo of the Thing Dev board shows the ESP chip as ESP8266EX.

[avra]

Most ESP8266 firmware uses 9600 baud with 64 byte buffer. You might want to change this, and adapt Arduinos to have the same baud rate.

[Arctic_Eddie]

My two ESP boards will arrive this week. I'll do a test by sending a group of 0x55 at 4Mbps. Will start at 32 byte groups and increase to 128 in steps of 32. The Laz serial buffer appears to be 128. However, for testing I'll use a terminal program and match the baud values with an FTDI adapter. I was able to get a Mega to run at 2Mbps but no higher.

My data size has increased from 512 words per sonar channel(4) to 1280 each. I have the Teensy 3.2 running now with timer interrupts and can capture data with dual synced ADCs at 100Ksps. Using three Teensies and ESP8266, I have replaced two Megas, four Nanos, and two nRF24 boards. I have also eliminated two intermediate storage points in the data stream. The data capture is also running 2.5x faster.

[avra]

If only ADC sampling is done on Teensies, then you can eliminate them completely and instead use Arduino IDE with ESP8266 to do ADC sampling directly on ESP8266. That would probably be the fastest way for your situation.

https://www.google.rs/search?q=esp8266+arduino+ide