Project Brief

  • Description: Socket streaming video-chat application, which could control sex toys at the same time
  • Platform: Android, STC12 singlechip
  • Award: First prize in Unique Hackday 2014
  • Relating Techniques: TCP Socket, Node.js, Camera API, Graphic API, Java, STC12C2052, Bluetooth 4.0
    Duration: 2014.2.23 - 2012.2.24
  • Team member: Hongkun Leng, Jialin Liang, Su Li

Background:

  • The rapid development of IoT(Internet of Things) influence our life impressively. Attracted by the ubiquitous trend, we were eager to take our part.
  • The rise of open source hardware, Raspberry PI Arduino and other existing ARM, 51 platform, made the development of new IoT device easier and more powerful. Under this circumstance, we chose STC12 as the hardware platform and connected it with Android application through Bluetooth 4.0
  • There had been some existing IoT toy products. However they lacked some essential features such as video chat and voice chat, which we considered critical for a IoT toy.

Components

1. Client side

The Android based application was responsible for video chatting and control of toys.
There were four functions of the client side

  • Toys control. the application was able to control the strength, length and vibrating mode of the toy through widgets on panel page.

Alt text

  • Square and Pair. In order to be paired with the right person, one could search on the “Square page” and pair with users he/she like.

Alt text

  • Video chat. After paring with a specific user, they can connect with each other and video-chat with each other. The upper and bottom part of the screen were face of relatively current user and current connected user.

Alt text

  • Remote control. Besides controlling their own toys, paired users could also control toys of each other through Socket Streaming.

In order to development these functions, I used these techniques:

  • TCP Socket. Although no direct connection can be set under mobile network, we could connect clients indirectly through servers. TCP Socket was able to exchange streaming data with servers. On Android platform, there was already existing Socket API for this technique, On Server side, there was Node.js Socket.io plugin to implement the same protocol.
  • Surface View. Displaying large quantity of images on mobile devices takes pretty much computing and memory space resources. In order to reduce potential influence on mobile device, I chose “”SurfaceView” widget, which could handle data on multiple threads, to display these images.
  • Bluetooth 4.0. Bluetooth was responsible for exchanging data between mobile and toys.
2. Hardware component

The hardware part was responsible for exchanging signals with mobile devices and driving toys. This part was consisted of three components:

  • Bluetooth component. This component was responsible for exchanging signals with mobile devices. These signals were then sent to microchip.
  • STC 12 microchip. This was processing center of this hardware device. It was in charge of exchanging and processing signals from mobile devices, controlling the micro motor component.
  • Micro Motor. This component was responsible for driving toys with received electronic signal.

Alt text
Alt text
Alt text

3. Server side

On the server side, we used Node.js framework and Socket.io framework to provide real-time streaming functionality.
There were some advantages of this platform:

  • Asynchronous operation. This feature made it possible for one server to exchange data with multiple clients at the same time with high efficiency.
  • Easy to deploy. All code could be modified and deployed within several lines of command.
4. Streaming optimization

Sending each complete frame on the stream is not a efficient strategy for video-chat. In order to reduce traffic pressure, we devised a streaming strategy:

  • We only send complete frame in the first connection, after that, only difference between two neighbor frames is sent.
  • In some case, when the connection is disrupted, and connected again, some differences would be lost. In order to maintain a stable streaming, we recorded time stamp for each frame, and compare the time stamp between two neighbor frames. If there the difference between two time stamps is bigger than threshold, a signal will be sent. And the server will send a complete frame.
  • Data is compressed in gzip format in order to make efficient transmission.

Relating pictures

Alt text
Alt text
Alt text