Streaming Server

What media encoding choose?

The Android platform supports many media formats.

Are there better choices in the context of the library? Here under are some hints, based on experience. Do not take them as references, but as an aid to make your own opinion.

The most important point is:

For 3GPP and MPEG-4 containers, the moov atom must precede any mdat atoms, but must succeed the ftyp atom.

In section Video Encoding Recommendations

The mdat atom is a section containing media frames and is the biggest in size. The moov atom is a section for metadata, i.e. properties, indexes, offsets and other structural information. For an encoder, the natural order is to write the mdat first, then the moov, after knowing what the offsets are. For a decoder, it's the opposite: the moov has to be fetched first to know how to interpret the mdat. Which one comes before the other is not that critical with a file on disk, thanks to a direct access. But it really matters for progressive streaming, and even more with contents encrypted with block-chaining algorithms where accessing any part supposes an entire reading from the very beginning.

An inadequate order does not prevent the play, it incurs a delay to the start, proportional to the file size. Some improvements over the native Android implementation have been made in the library (since version 1.3) in order to speed up this step, with a gain up to 5.6 times for AES/CBC and 8.2 times for RC4. But a delay remains: for a rough order of magnitude, count about 2.3 seconds per 10 megabytes for AES/CBC and 1 second per 10 megabytes for RC4 (on a Nexus 7 2012). In addition, in this case and for devices as of Android 4.3, special caution should be given to the selected Security Provider.

How to check the order?

You can use any of the following ways to check the correct order of the atoms:
  • A basic text editor. Look for the two 4-characters patterns.
  • MediaInfo software. For the GUI version, check Debug / Advanced mode ; For the CLI version, use the --full option or this specific option: --inform=General;%IsStreamable%. In both cases, check that the IsStreamable property is Yes.
  • MP4Box software. Use the --info option and check the presence of File suitable for progressive download (moov before mdat). When the file is not suitable, there is no negative mention.

How to generate a correct order?

When possible, set an option or a preference in the encoder software. For instance, with ffmpeg, use this option: -movflags +faststart

Otherwise, use another tool. Here are some examples:

  • MP4Box software. Run it as: mp4box -inter 500 in.mp4 -out out.mp4

What security provider choose?

When you request a cipher with Cipher.getInstance(…) without specifying the provider, one is chosen among the registered security providers, in accordance with an order of preference.

Android 4.3 introduced AndroidOpenSSL as the preferred default provider, before BouncyCastle ("BC"). Initially, this new implementation didn't work as expected when the cipher is used a second time. This is particularly true with the atom order issue detailed in Media Encoding.

The bug has been partially fixed in 5.0. There is still a problem with algorithms having a padding. It may help to have a fix if you star the issue.

So it seems safer to specify "BC" as the provider, for devices running version:
  • 4.3 and 4.4, in any case
  • 4.3 and more, for padded algorithms without the use of a necessary CipherFactory (case of the ECB mode)

    Unfortunately, the BC provider is deprecated and can't no more be imposed as of Android P (an exception is thrown). So this fallback must be restricted to versions before 9.

See also below some code samples.

New in v3.2.5 In the above mentioned situations, an alternative to the BC provider is to implement a CipherFactory, even if it doesn't seem mandatory otherwise. But it becomes the only solution for padded algorithms as of Android P (anyway, padded algorithms are discouraged, see next section).

What transformation (algorithm/mode/padding) choose?

In case of no need for forward/rewind seeking

You should prefer a stream cipher to a block cipher. Even if the library is able to deal with both families, stream ciphers are naturally more suitable in the context of streaming a video flow. Because block ciphers may use some padding data to respect a fixed block length, the file size is not a valid piece of information and so the real payload length cannot be known before the decryption completion. This lack of information is unfavorable for an HTTP server, which SHOULD (RFC 2616) serve a Content-Length header.

However, you can use a block algorithm and select a mode that does not require padding to effectively use a block cipher as a stream cipher.

Here are some examples:
  • AES/CFB/NoPadding
  • AES/CTR/NoPadding
  • AES/CTS/NoPadding
  • AES/CBC/WithCTS
  • AES/OFB/NoPadding

In case you don't have any criteria or legacy/compatibility constraints, the RC4 stream cipher is a good candidate, it's a widely used stream cipher and it's less processing intensive than the block ciphers.

The problem of forward/rewind seeking

There is a big constraint with seeking: for most of the encryption algorithms, seeking is done by reading - from the start of the file. So even to go around the end of the video, one needs to read all the data from the very beginning, so it takes some time, proportional to the amount of data. It may be not noticeable for small files (say, less than 10 MB), but for larger files it becomes a real delay. The trouble with a seek on a progress bar is that it falls somewhere in the stream. The MediaPlayer asks for data at this index, and tries to synchronize. If it can't, it asks again at another index, nearby. It does this several times, until a successful synchronization. After too much failures it gives up.

There is no obvious solution to this problem. However, among the various block cipher modes of operation, some allow a random read access during decryption.

It doesn't change anything for the repeated hits from the MediaPlayer but at least the response comes quicker.

To benefit from this advantage, you have to:
  • Choose a mode allowing a random read access. The library supports these modes:
    • CTR. For example: AES/CTR/NoPadding
    • CFB (v3.2+)
    • CBC (v3.2+)
  • Instead of a single Cipher, give to the LocalSingleHttpServer an instance of your implementation of a CipherFactory (see a code sample below)

Additionnal information on other modes:

OFB: This mode does not allow a random read access.

ECB: Although suitable for random access, it does not hide data patterns well and is not recommended in cryptographic.
Nevertheless, the library supports (v3.2+) this mode as well, and there is no need for a CipherFactory.

Data Sources

The server is able to deliver a data flow from a variety of sources:

Basic Local File

File file = new File(
  Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_MOVIES),
  "path/to/video.mp4");
String path = mServer.getURL(file.getPath());

Entry in a Zip File

String path = mServer.getURL("/path/to/container.zip", "path/to/video.mp4");

Entry in an APK Expansion Zip File

The library may use but doesn't embed the Google Play APK Expansion Library. Get it with the SDK Manager and follow the instructions at the Developers site.
int mainVersion = 1;
int patchVersion = 0;
String path = mServer.getURL(mainVersion, patchVersion, "path/to/video.mp4");

Raw Asset File

Note that the "asset://" is not a standard scheme like "http://" or "file://" but a convention of the library.
String path = mServer.getURL("asset://hd/video.mp4");

SMB Server

The library may use but doesn't embed the jCIFS Library. Get it at www.jcifs.org.
String path = mServer.getURL("smb://server/share/path/to/video.mp4");

HTTP or HTTPS Server

String path = mServer.getURL("http://media.mydomain.com/path/to/video.mp4");

FTP Server

String path = mServer.getURL("ftp://ftp.mydomain.com/path/to/video.mp4");

Code Samples

Playing <video> in WebView

You can design your application in two basic ways (or even a mix of both):

  • Host-driven: The host application is the master and controls the video source.
  • JavaScript-driven: The HTML page has some embedded interaction widgets to allow the user to select a media.
Host-driven case:
val webview = WebView(this)
webview.settings.javaScriptEnabled = true
// mServer = ...
var path = "/some/path"
path = mServer.getURL(path)
webview.webViewClient = object : WebViewClient() {
  override fun onPageFinished(view: WebView, url: String) {
    view.loadUrl("javascript:setURL('$path'); void(0);")
  }
}
val page = """
<html><body>
 <script>
  function setURL(url) {
   document.getElementById('v_id').src = url; }
 </script>
 <video id="v_id" controls></video>
</body></html>
"""
webview.loadData(page, "text/html", null)
final WebView webview = new WebView(this);
webview.getSettings().setJavaScriptEnabled(true);
// mServer = ...
String path = '/some/path';
path = mServer.getURL(path);
final String mediaPath = path;  // just because 'final' is mandatory
webview.setWebViewClient(new WebViewClient() {
  @Override
  public void onPageFinished(WebView view, String url) {
    view.loadUrl("javascript:setURL('" + mediaPath + "'); void(0);");
  }
});
String page = "<html><body>" +
    "<script>" +
    "function setURL(url) {" +
    "  document.getElementById('v_id').src = url; }" +
    "</script>" +
    "<video id=\"v_id\" controls></video>" +
    "</body></html>";
webview.loadData(page, "text/html", null);
JavaScript-driven case:
val webview = WebView(this)
webview.settings.javaScriptEnabled = true
// mServer = ...
webview.addJavascriptInterface(
    mServer.jsInterfaceObject(),
    "serverObject")
val page = """
<html><body>
 <script>
  function setURL(url) {
   document.getElementById('v_id').src = url; }
  function setPath(path) {
   setURL(serverObject.getURL(path)); }
  // ... logic for:
  // setPath('/some/path');
 </script>
 <video id="v_id" controls></video>
</body></html>
"""
webview.loadData(page, "text/html", null);
final WebView webview = new WebView(this);
webview.getSettings().setJavaScriptEnabled(true);
// mServer = ...
webview.addJavascriptInterface(
    mServer.getJsInterfaceObject(),
    "serverObject");
String page = "<html><body>" +
    "<script>" +
    "function setURL(url) {" +
    "  document.getElementById('v_id').src = url; }" +
    "function setPath(path) {" +
    "  setURL(serverObject.getURL(path)); }" +
    "// ... logic for:" +
    "// setPath('/some/path');" +
    "</script>" +
    "<video id=\"v_id\" controls></video>" +
    "</body></html>";
webview.loadData(page, "text/html", null);

Playing a basic encrypted video

A partial sample of how to play an encrypted video in a VideoView widget:

private fun myPlay(path: String) {
    mServer = LocalSingleHttpServer()
    mServer!!.setCipher(cipher)
    mServer!!.start()
    path = mServer!!.getURL(path)
    mVideoView.setVideoPath(path)
    mVideoView.start()
}
override fun onCompletion(mp: MediaPlayer) {  // MediaPlayer.OnCompletionListener interface
    mServer!!.stop()
    mServer = null
}
private val cipher: Cipher?
    @Throws(GeneralSecurityException::class)
    get() {
        val algorithm = "AES"
        val transformation = "AES/CTR/NoPadding"
        val key = "1234567890123456"
        val iv = ByteArray(16)
        val provider = selectProvider(transformation)
        val c = if (provider != null) Cipher.getInstance(transformation, provider)
                else Cipher.getInstance(transformation)
        c.init(Cipher.DECRYPT_MODE, SecretKeySpec(key.toByteArray(), algorithm), IvParameterSpec(iv)))
        return c
   }
companion object {
    private fun selectProvider(transformation: String): String? {
        /* Avoid the default security provider "AndroidOpenSSL" in some cases between Android 4.3 and 9
        http://libeasy.alwaysdata.net/network/#provider 
        */
        val ucName = transformation.toUpperCase()
        return if (Build.VERSION_CODES.JELLY_BEAN_MR2 <= Build.VERSION.SDK_INT
                    && (Build.VERSION.SDK_INT < Build.VERSION_CODES.LOLLIPOP
                        || (ucName.endsWith("PADDING") && !ucName.endsWith("/NOPADDING")
                            && Build.VERSION.SDK_INT < Build.VERSION_CODES.P))) "BC" else null
    }
}
private void myPlay(String path) {
    mServer = new LocalSingleHttpServer();
    mServer.setCipher(myGetCipher());
    mServer.start();
    path = mServer.getURL(path);
    mVideoView.setVideoPath(path);
    mVideoView.start();
}
public void onCompletion(MediaPlayer mp) { // MediaPlayer.OnCompletionListener interface
    mServer.stop();
    mServer = null;
}
private Cipher myGetCipher() {
    final String algorithm = "AES";
    final String transformation = "AES/CTR/NoPadding";
    final String key = "1234567890123456";
    final byte[] iv = new byte[16];
    final String provider = selectProvider(transformation);
    final Cipher c = provider != null ?
            Cipher.getInstance(transformation, provider) :
            Cipher.getInstance(transformation);
    c.init(Cipher.DECRYPT_MODE,
            new SecretKeySpec(key.getBytes(), algorithm),
            new IvParameterSpec(iv);
    return c;
}
private static String selectProvider(final String transformation) {
    /* Avoid the default security provider "AndroidOpenSSL" in some cases between Android 4.3 and 9
    http://libeasy.alwaysdata.net/network/#provider 
    */
    final String ucName = transformation.toUpperCase();
    return (Build.VERSION_CODES.JELLY_BEAN_MR2 <= Build.VERSION.SDK_INT
            && (Build.VERSION.SDK_INT < Build.VERSION_CODES.LOLLIPOP
                || (ucName.endsWith("PADDING") && !ucName.endsWith("/NOPADDING")
                    && Build.VERSION.SDK_INT < Build.VERSION_CODES.P))) ?
            "BC" : null;
}

Playing an encrypted video, with tolerance to forward/rewind seeks

A partial sample of how to implement a CipherFactory:

private fun myPlay(path: String) {
    mServer = LocalSingleHttpServer()
    mServer!!.setCipherFactory(MyCipherFactory())
    mServer!!.start()
    path = mServer!!.getURL(path)
    mVideoView.setVideoPath(path)
    mVideoView.start()
}
override fun onCompletion(mp: MediaPlayer) {  // MediaPlayer.OnCompletionListener interface
    mServer!!.stop()
    mServer = null
}
private void myPlay(String path) {
    mServer = new LocalSingleHttpServer();
    mServer.setCipherFactory(new MyCipherFactory());
    mServer.start();
    path = mServer.getURL(path);
    mVideoView.setVideoPath(path);
    mVideoView.start();
}
public void onCompletion(MediaPlayer mp) { // MediaPlayer.OnCompletionListener interface
    mServer.stop();
    mServer = null;
}
private class MyCipherFactory : CipherFactory {
    @Throws(GeneralSecurityException::class)
    override fun getCipher(): Cipher {
        // you are free to choose your own Initialization Vector
        val initialIV = ByteArray(16)
        return rebaseCipher(initialIV)
    }
    @Throws(GeneralSecurityException::class)
    override fun rebaseCipher(iv: ByteArray): Cipher {
        val c = Cipher.getInstance("AES/CTR/NoPadding")
        c.init(Cipher.DECRYPT_MODE,
                SecretKeySpec("1234567890123456".toByteArray(), "AES"),
                IvParameterSpec(iv))
        return c
    }
}
private class MyCipherFactory implements CipherFactory {
  @Override
  public Cipher getCipher() throws GeneralSecurityException {
    // you are free to choose your own Initialization Vector
    byte[] initialIV = new byte[16];
    return rebaseCipher(initialIV);
  }
  @Override
  public Cipher rebaseCipher(byte[] iv) throws GeneralSecurityException {
    final Cipher c = Cipher.getInstance("AES/CTR/NoPadding");
    c.init(Cipher.DECRYPT_MODE,
      new SecretKeySpec("1234567890123456".getBytes(), "AES"),
      new IvParameterSpec(iv));
    return c;
  }
}
private class MyCipherFactory implements LocalSingleHttpServer.CipherFactory {
  // ... same as v3
private class MyCipherFactory implements LocalSingleHttpServer.CipherFactory {
  @Override
  public Cipher getCipher() throws IOException {
    // you are free to choose your own Initialization Vector
    byte[] initialIV = new byte[16];
    return rebaseCipher(initialIV);
  }
  @Override
  public Cipher rebaseCipher(byte[] iv) throws IOException {
    Cipher c = null;
    try {
      // avoid the default security provider "AndroidOpenSSL" in Android 4.3+ (http://libeasy.alwaysdata.net/network/#provider)
      c = Cipher.getInstance("AES/CTR/NoPadding", "BC");
      c.init(Cipher.DECRYPT_MODE,
        new SecretKeySpec("1234567890123456".getBytes(), "AES"),
        new IvParameterSpec(iv));
    } catch (GeneralSecurityException e) {
      throw new IOException("Unable to create a cipher", e);
    }
    return c;
  }
}

How to generate an encrypted file?

There are many ways to produce an encrypted copy of a video file. As a starter for your implementation, here are some code samples.

Java code

  1. public static void encrypt() throws Exception {
  2. final byte[] buf = new byte[8192];
  3. final Cipher c = Cipher.getInstance("AES/CTR/NoPadding");
  4. c.init(Cipher.ENCRYPT_MODE, new SecretKeySpec("1234567890123456".getBytes(), "AES"), new IvParameterSpec(new byte[16]));
  5. final InputStream is = new FileInputStream("/path/to/src/video.mp4");
  6. final OutputStream os = new CipherOutputStream(new FileOutputStream("/path/to/dst/video.mp4"), c);
  7. while (true) {
  8. int n = is.read(buf);
  9. if (n == -1) break;
  10. os.write(buf, 0, n);
  11. }
  12. os.close(); is.close();
  13. }

node.js code

  1. var crypto = require('crypto');
  2. var algorithm = 'aes-128-ctr';
  3. var key = new Buffer('1234567890123456');
  4. var iv = new Buffer('\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0');
  5. var c = crypto.createCipheriv(algorithm, key, iv);
  6. var fs = require('fs');
  7. var r = fs.createReadStream('/path/to/src/video.mp4');
  8. var w = fs.createWriteStream('/path/to/dst/video.mp4');
  9. r.pipe(c).pipe(w);

Python3 code

This sample uses the M2Crypto library [pypi.python.org/pypi/M2Crypto: A Python crypto and SSL toolkit]. Unfortunately you need to adapt and build the library by yourself for the support of AES CTR modes.

Note: code checked with M2Crypto version 0.35.2.

  1. from M2Crypto.RC4 import RC4
  2. key = bytes('BrianIsInTheKitchen', 'utf-8')
  3. cipher = RC4(key)
  4. src_path = '/path/to/src/video.mp4'
  5. dst_path = '/path/to/dst/video.mp4'
  6. with open(src_path, 'rb') as input, open(dst_path, 'wb') as output:
  7. output.write(cipher.update(input.read())) # no need of cipher.final() for RC4

Command Line Tools

openssl enc -aes-128-ctr -in /path/to/src/video.mp4 -out /path/to/dst/video.mp4 -K 000102030405060708090A0B0C0D0E0F -iv 0

FAQ

Do you propose a similar library for iOS or Windows Phone?

No. We are focused on Android only. A React Native module is also available (again, for Android build only).


Is it usable for anything other than VideoView?

Yes. The decrypted stream can be delivered to a <video> or <audio> tag in a WebView, or to a Chromecast device.


I see these Error lines in LogCat:
"D/MediaPlayer: setDataSource IOException happend :"
" java.io.FileNotFoundException: No content provider: http://127.0.0.1: ..."

Note that it is not a thrown exception but a Debug message from the MediaPlayer, detailed with a stacktrace.
It's a normal behavior of MediaPlayer: whatever the path content is, it first tries it as a local resource and if it fails it will fallback to a remote resource. You see that on the next Debug message:
"D/MediaPlayer: Couldn't open file on client side, trying server side"


I see these Error lines in LogCat:
"Could not find class 'jcifs.smb.SmbFile', referenced ..."
"Could not find class 'com.android.vending.expansion.zipfile.ZipResourceFile', referenced ..."
together with a bunch of Warning or Debug lines prefixed with "VFY:".

Do you intend to use any of these optional data sources: SMB server or APK Expansion Zip file?
If not, just ignore the error, it doesn't prevent the app to work.
Otherwise, you forgot to embed an additional library in your app, so expect the dedicated getURL() methods to return null.


It works fine on a Wifi network, but on a mobile network it fails with:
"Server responded with http status 503", or 502.

A possible explanation is the setting of a proxy on the network interface, which routes all traffic to outside, even for localhost requests. On the Wifi interface, there is a "Bypass proxy for" option where "127.0.0.1" can be entered. Unfortunately this is not the case for mobile interface: in the Edit screen of an Access Point, you can find a Proxy and Port but no bypass input field.

The solution is to unset the Proxy setting on the mobile network interface.