Streaming Data

•   What is Streaming Data
  

There are basically two types of web videos: progressive downloads and streaming video.

Progressive downloads are single files that are stored on a website; they will be downloaded via a browser to a computer or any mobile devices for viewing, and the downloads can be stored locally on the device for watching.

Streaming video is fed from a server and can be watched via a browser without transferring the file from the server to the viewer. For example, YouTube and Vimeo are providing streaming videos.

Streaming Data is the data generated continuously by thousands of data sources, which send in the data records simultaneously in small sizes (order of Kilobytes). Streaming data includes a wide range of data, such as log files, in-game player activity, information on social networks, ecommerce purchases… The data needs to be processed sequentially and incrementally on a record-by-record basis.

•   Benefits of Streaming Data
  

Streaming data is good because we can always track the new, dynamic (real-time) data continually. Its information provides us visibility into many aspects and emergencies because we can track changes happened under the cameras by continuously analyzing social media streams, and respond to them when necessary.

•   What is HTTP Live Streaming (filename extension: .m3u8)
  

This is an HTTP-based media streaming communications protocol. It works by breaking the overall stream into a sequence of small HTTP-based file downloads, each download loading one short chunk of an overall potentially unbounded transport stream. As the stream is played, the client may select from a number of different alternate streams containing the same material encoded at a variety of data rates, allowing the streaming session to adapt to the available data rate.

Architecture: HTTP Live Streaming uses a conventional web server to distribute audiovisual content and requires specific software to fit into the proper format transmission in real time. The service architecture comprises: Server, distributor and client.

*Note: HTTP Live Streaming does not work with video-only, or audio-only files, and there must be audio in the clips, even if it is silence; HTTP Live Streaming can also be used for live events.

•   Getting HTTP Live Streaming (HLS) Data
  

We know only two ways to get Live Streaming Data so far - either manually or semi-automatically.

Doing it manually using an add-on on websites – Flash Video Downloader by FVD Media: Add-on the Flash Video Downloader from https://addons.mozilla.org/en-US/firefox/addon/flash-video-downloader/

After getting this add-on on your browser, you will see the symbol like a blue arrow pointing downwards when you try to retrieve the streaming video from the original websites (i.e., the arrow showing blue means that there is a streaming video retrievable from the website). Then you click on the arrow-like symbol and you will find the streaming video(s) retrieved from the website there, and another blue symbol there meaning “copy to Clipboard” is the one you want to click on (as the image showing below), which will generate the desired streaming URL’s with “.m3u8” extension.

Semi-automatically: First you need to set up Streamlink as instructed here: https://github.com/PurdueCAM2Project/CAM2RetrieveData/tree/master/Retrieval/Streamlink

If you want to generate an HLS URL from only one URL, use the command: python fetchstreamsrc.py [WEBPAGE URL]

If you want to generate HLS URLs from a bunch of URLs, you need to use loop.py to go through a list of URLs being stored in an excel/text file and generate their HLS with “.m3u8” extension all at once.

Before running the script, you need to modify the line in loop.py showing below, by replacing the 'earthcamURL.txt' to your own file with URLs that needs to retrieve HLS from. with open('earthcamURL.txt') as f:

Command for running the script: python3 loop.py

*Note: some of the URLs might not work and will not generate anything. So we need to test the availability of the streaming data using specific methods as described below.

•   Testing the Streaming Data using FFMpeg
  

FFMpeg is a free software project that produces libraries and programs for handling multimedia data. It’s a command-line tool that converts audio or video formats; it can also capture and encode in real-time from various hardware and software sources such as a TV capture card. FFmpeg reads from an arbitrary number of input “files” (like regular files, pips, network streams, grabbing devices, etc.), specified by the –i option, and writes to an arbitrary number of output “files”, which are specified by a plain output url.

Using FFmpeg to check the availability of the HTTP Live Streaming urls (.m3u8) we have, and check the qualities of the real-time images provided by them with the specific frame rates we need (using “ffmreader.py”).

If you want to test only one HLS URL, use the command: ffmpeg –i [HLS url]

If you want to test a bunch of HLS URLs all at once is to first store them into a text file, you will need to run the script ffm_reader.py Before running the script, you need to modify a line in loop.py showing below, by replacing the “m3u8_usable.csv” to your own file with the HLS URLs that needs to be tested. filename = “m3u8_usable.csv”

Command for running the script: python ffm_reader.py

•   Issues that we have currently
  

1: The methods work only for EarthCam URLs from our testing currently. When dealing with mostly other types of URLs, they might be not able to create URLs with the “.m3u8” extension.

2: When we encounter some errors while generating URLs with “.m3u8” extension, we simply PASS all types of errors so that there might be some HTTP Live Streaming URLs being passed and generated while running the programs (though containing errors in any forms), and cause the unnecessary waste in testing afterwards.

3: Some cameras are intermittently unavailable for whatever reasons, and we do not have any solutions for that.

References

HTTP Live Streaming. (2018, February 17). Retrieved February 19, 2018, from https://en.wikipedia.org/wiki/HTTP_Live_Streaming

The Basics of HTTP Live Streaming. (n.d.). Retrieved February 19, 2018, from https://larryjordan.com/articles/basics-of-http-live-streaming/

What is Streaming Data? – Amazon Web Services (AWS). (n.d.). Retrieved February 19, 2018, from https://aws.amazon.com/streaming-data/