Gesture Recognition: The Beginning

I’m been working on a debug mode for the IdeasX v0.3.1 sensor module to send back live accelerometer (XL) and gyroscope (G) from the LSM6DS3. The idea is to have the encoder send data back to a computer to be remotely processed. (The final design will have the module locally processing the data every 100ms.) The purpose of this is two fold:

  1. It will permit me to rapidly develop and deploy different gesture recognition / motion detection algorithms using a high level language such as Octave or Python before deploying them on the module.
  2. The methods developed for streaming data can later be used for training applications. Example: if a significant motion algorithm is selected to activate an actuator, an initial configuration dialog will stream the data back allowing the therapist to see the range in XL and G values. The therapist would then adjust the thresholds for significant motion according to the range in values of the data streamed back.

Design / Guesswork

The LSM6DS3 utilized by the V0.3.1 module contains a 8kB FIFO which can be used to store information generated by the device (accelerometer, gyroscope, and temperature). When raw IMU data is going to be used for adaptive switching, the FIFO will permit the ESP8266 to process the data in chunks freeing up processing time for other operations.

The normal order of operations would be some like what follows:

  1. Set the IMU to the desired sampling rate.
  2. Configure the FIFO to trigger an interrupt on INT1 when the desired number of samples is reached.
  3. When the interrupt is triggered, the ESP8266 will dump the specified number of samples used for the sense period.
  4. The ESP8266 will process the data by applying a detection algorithm (NN, DTW, etc)
  5. If the detection algorithm has detected an event, the ESP8266 sends a Data Message through the IdeasX system.

For streaming information, the following debug order of operations is proposed:

  1. Set the IMU to the desired sampling rate.
  2. Configure the FIFO to trigger an interrupt on INT1 when the desired number of samples is reached.
  3. When the interrupt is triggered, the ESP8266 will dump the specified number of samples used for the sense period.
  4. Repeat step 3 for X number of times then proceed to 4
  5. The ESP8266 sends the raw data via a Data Message through the IdeasX system.

Due to limited FIFO size and various sampling rates, a few plots were generated to determine the number of samples which would be accrued over 4 different proposed sampling periods (50ms, 100ms, 150ms, 200ms). “Sampling period” here refers the amount of time the LSM6DS is allowed to accrue samples before being dumped.

figure_1.png
Figure 1: Samples vs ODR

The first figure displays the number of samples accrued based on the sampling rate of the IMU or output data rate (ODR) and the length of time samples are allowed to accrue. It should be noted each sample is actually 12 bytes total. Each axis for the accelerometer or gyro is represented by two bytes. The resolution can be reduced to 1 byte per axis. As the response time of the module should be <200ms, a 200ms sampling period can be ruled out as too long.

figure_3.png
Figure 2: Time required to dump FIFO based on ODR.

Figure 2 displays the amount of time required to dump the FIFO. This was based on experiments which recorded the amount of time required to dump a single sample from the IMU and store that sample in RAM.

 

logic_analyer.png
Figure 3: It requires 8us to receive 32bits from the SPI port and to store it into an array.

Figure 4 displays the number of bytes the LSM6DS3 will attempt to store in the FIFO based on the 4 evaluation periods. The black horizontal line represents the limit of the 8kB FIFO. The simulation assumes the gyro and xl are both storing samples for each axis with full resolution (16 bits).

figure_2
Figure 4: Bytes!

Implementation

The v0.3.1 module required a tiny modification. A jumper needed to be soldered to attached INT1 to PBA. This also required a modification to the GPIO ISR routine to check if the interrupt was triggered by the LSM6DS3 or the pushbutton. Basically, the ESP12 has NO GPIO lines. I had to double up functionality.

3MoUFp.png
Figure 5: I soldered a 0R resistor to R33 shorting PB_Ax to INT1

The final settings had the LSM6DS3 sampling at 208Hz. The ESP8266 would dump the FIFO every 100ms and store it into a buffer. This would repeat 10 times and then I simply dumped the buffer into an MQTT message instead of first formatting it for a Protobuff. Note…this was a feasibility test. I modified the ideaX-sniffer tool to plot the data using Qt. You can see a short screen capture of me plotting the data here.

ideasX_sniffer_20160924.mp4 - VLC media player_071.png
Figure 6: Plotting XL data from v0.3.1 module over WeeeeFeeeee (Wi-Fi).

There is much that needs to be done to make this viable. The data needs to be packaged. There needs to be additional commands made to set the sampling rate of the XL and G remotely along with the threshold of the FIFO to trigger an interrupt. It might be easier to simply make a command to send byte X to address Y out of the SPI bus. The ideasX-sniffer tool also needs to be updated of course to support the new commands, but luckily that is super easy : D.

…then the fun with signal processing begins (muhaha, j.k. I have no idea what I’m doing.)

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s