So back in February this year I ordered some Blurays on amazon.com and imported them from the US. Even with custom fees they were way cheaper than buying them in Germany and thus I thought: "Well, I'm sure a friend of mine will be nice enough to create proper copies of the discs."
A few months later I happened to acquire a notebook with a Bluray drive, so that was out of question and I was excited for my Blurays to arrive.
Fast forward, two days ago:
I received my package with the DVD/BD combo pack of Spice and Wolf released by FUNimation in the US, and I was immediately let down when I found out that it is not possible to play back Blurays out of the box. To that day I thought the big companies had figured out that using DRM is essentially useless and only sets the hurdle a little higher. I never bothered reading about Bluray DRM before, neither had I ever really heard about it. So I started reading the Archlinux Bluray Wiki-Entry, which already helped a lot, but I wasn't able to play my legally purchased Blurays until after I ran into many errors and compiled software many, many times.
The first keyword to drop is AACS. With it goes aacskeys and libaacs. After trying to mess with aacskeys I gave up a little and used DVDFab9 to rip one of the Blurays and I was shocked how easily it decrypted the disc. By this time I already spend quite some hours on this, but I was somewhat happy that I could play the decrypted files. The next day I started reading about this again and learned about libaacs. I downloaded the sources, compiled it and with aacs_info I was able to gain the VID, I don't know how it works, but feel free to dig through the code. Now, with the VID I could use aacskeys with the included Host Certificates and Private Keys to get the VUK (see: http://en.wikipedia.org/wiki/Advanced_Access_Content_System#Decryption_process). Now once you have the VUK you don't need anything else to finally decrypt your disc, so better save it somewhere, in fact you are needed to save the VUK for libaacs and libaacskeys in the directory: ~/.cache/aacs/vuk/. This should happen automagically, and it did once I messed around long enough with it on my laptop with the Bluray drive. You can use VLC, mplayer or even mplayer2 to play back your Bluray. I took this one step further and mounted the mounted Bluray (yes mounted twice!) via sshfs on my notebook without a Bluray drive and was able to play it, too. I installed libaacs and libaacskeys and copied the files in ~/.cache/aacs/vuk/ in the same directory. The files in that directory are named after the DiscID, which is also output by aacskeys and contain the VUK.
So after messing with this for almost two days I was finally able to watch my Blurays. Once again I spent a lot of time on something that shouldn't be. The bad thing is the big companies don't even care. I made my purchase, they made money, I gained nothing. At least now I can say I really own these discs, because before I was able to decrypt them, I felt betrayed and robbed. I felt like I wasted my money on unreadable discs.
Wednesday, May 8, 2013
Friday, March 15, 2013
mp3server
A few weeks ago I started writing a server to re-stream an mp3-stream. It all started on IRC (like everything on the internet does):
So I thought of ways how to do it:
Later on Sean_McG started to contribute code, at first it was only to make the server IPv6 compatible, but later he added autotools support and provided many fixes for my poorly written code.
What motivated me to do this was the fact that most streaming software I used until now (shoutcast and mpd) are resource clogging monsters. What I wanted was an extremely lightweight application, which did nothing but data redirection at its core, and this is what evolved from it.
Today and yesterday I did some benchmarks with people from IRC (Thanks Kabaka, LordV, Phase4
Yesterday's test results were basically:
A bit insight on the technical details:
The server opens a port (currently hardcoded port 8080) and waits for a client to connect. This client is taken as the source for the mp3-stream (There is no authentication method or anything similar as of now). Each client that connects now will be treated as a "real" client, which wants to listen to our stream.
Now this is where it gets interesting: Whereas mpd uses a buffer for every client and uses some multi-threaded model to push out the stream over HTTP, I implemented a ringbuffer for the stream which every client can access. "Client" in this case is a struct storing the client's filedescriptor, a counter of how many bytes have already been written other data for various things and pointers to the previous and next client struct. In the main-loop select() checks whether or not a new frame from the source can be read, new clients want to connect, or existing clients can be written. In the first case we just remove the oldest frame from the ringbuffer and replace it with the new one. In the second case a new client struct is created and added at the end of the doubly linked client list. In the third case we traverse the client list and write as many frames as possible, in most cases this should be one frame. It is possible that a client lags behind and new frames got added to the ringbuffer without new frames being written to this particular client. In that case upon the next iteration in the main-loop the server will try to write all frames to the client that are needed to be up-to-date with the stream. If clients lag behind one round in the ringbuffer they will experience skips in the stream, if they lag behind two rounds in the ringbuffer they are declared dead and dropped.
My future plans entail:
* klaxa is listening to millie - Dreaming Forest
<Sean_McG> klaxa: you play good music -- do you stream it at all?
<klaxa> no, should i?
* Sean_McG would listen to it
<klaxa> it wouldn't be up all the time, but i guess i can set something up
So I thought of ways how to do it:
Well, turns out mp3 isn't robust against starting "just anywhere in the bytestream" AT ALL. Therefore I read the mp3 header specification and started implementing a simple program to split an mp3 stream into frames. This alone took some hours of coding, but it was totally worth it. After about a week in total I had written some naive code that could re-stream an mp3-stream to some clients. However, because of the design at the time, all the sockets were blocking and not multiplexed. One night I went to the library (I love working late in the evening) and started to rewrite my code to use select() to multiplex through the clients. Shit worked and we did some first tests which were kinda passed.
<klaxa> i have mpd running, i'd try to add some output that i grab locally via http, have something connect to my server which then encodes to mulitple formats and streams
<klaxa> the hardest part i see right now is the streaming part
<klaxa> the rest shouldn't be too hard, i could even do it with netcat
<klaxa> if i knew how robust mp3 is against starting to stream from just anywhere in the bytestream i could write my own server
Later on Sean_McG started to contribute code, at first it was only to make the server IPv6 compatible, but later he added autotools support and provided many fixes for my poorly written code.
What motivated me to do this was the fact that most streaming software I used until now (shoutcast and mpd) are resource clogging monsters. What I wanted was an extremely lightweight application, which did nothing but data redirection at its core, and this is what evolved from it.
Today and yesterday I did some benchmarks with people from IRC (Thanks Kabaka, LordV, Phase4
, Hans_Henrik and Reiuji) to see how well it scales.Yesterday's test results were basically:
<klaxa> Sean_McG: i got some people benchmark the stream with me and it's at least able to saturate 100 mbit/sToday I decided to test my server on a different box with more bandwidth. The test results can be boiled down to:
<klaxa> okay so what broke it was probably the filedescriptor limit of 1024The bandwidth usage of todays test maxed out at about 40 MB/s so circa 320 Mbit/s, serving almost 1000 clients.
A bit insight on the technical details:
The server opens a port (currently hardcoded port 8080) and waits for a client to connect. This client is taken as the source for the mp3-stream (There is no authentication method or anything similar as of now). Each client that connects now will be treated as a "real" client, which wants to listen to our stream.
Now this is where it gets interesting: Whereas mpd uses a buffer for every client and uses some multi-threaded model to push out the stream over HTTP, I implemented a ringbuffer for the stream which every client can access. "Client" in this case is a struct storing the client's filedescriptor, a counter of how many bytes have already been written other data for various things and pointers to the previous and next client struct. In the main-loop select() checks whether or not a new frame from the source can be read, new clients want to connect, or existing clients can be written. In the first case we just remove the oldest frame from the ringbuffer and replace it with the new one. In the second case a new client struct is created and added at the end of the doubly linked client list. In the third case we traverse the client list and write as many frames as possible, in most cases this should be one frame. It is possible that a client lags behind and new frames got added to the ringbuffer without new frames being written to this particular client. In that case upon the next iteration in the main-loop the server will try to write all frames to the client that are needed to be up-to-date with the stream. If clients lag behind one round in the ringbuffer they will experience skips in the stream, if they lag behind two rounds in the ringbuffer they are declared dead and dropped.
My future plans entail:
- Implement icy-metadata
- Fork project and rewrite it to work with the most popular audio codecs.
- Become a real low-resource alternative to icecast.
Tuesday, September 18, 2012
Streaming videos over http while encoding and hardsubbing at the same time
So a few days ago I bought the Google Nexus 7. I also have to say that I like to watch Anime. That being said you might already see where I am going, given that you read the title. As you know the Nexus 7 has very limited storage capacity, I even bought the 8 GB version because I didn't see why I should pay 50€ just for 8 GB more storage. Anyways, to understand why I'm encoding the Anime I stream to my tablet, you have to know that recently most Anime Fansub-Groups started releasing their videos in the High 10P H.264 Profile, which the Nexus 7 (and most other devices, including Desktop PC's) can't decode in hardware. Either you try to decode it in software, which is okay on the Desktop, but lags horribly on the Nexus 7, or you go and try to decode it in hardware, which messes up color decoding.
My first approach to tackle this was re-encoding in the High Profile of the H.264 Codec, that, however, would have taken about 3 times the playtime of an episode, while reducing quality.
My second approach, since this long waiting period just isn't worth it, was re-encoding to MPEG4 Part 2, which is a lot easier to encode. The filesize increase was huge though, more than twice the size of the input file, also visual quality is reduced due to the way the colors are encoded (I think?).
These two options given, I first encoded the files in MPEG4 Part 2 and put them on my tablet, with 8 GB of storage, however, it was clear that this was not a solution, so I started playing around with streaming.
The first problem was, that while I was able to encode video- and audio-stream separately if encoding to my harddrive, this was not the case for streaming. For simply encoding and hardsubbing the video-stream I used MPlayer and a fifo pipe which then was encoded by FFmpeg.
The MPlayer line looks like this:
and the FFmpeg line like this:
I had to get both streams, audio and video to FFmpeg so it could mux the streams together. At first I tried using only one MPlayer instance, it turned out, that this:
in combination with this:
produced a deadlock, MPlayer would not decode the video-stream, because it was waiting for the audio-stream to be read by FFmpeg, which itself was waiting for the video input by MPlayer. So I rearranged the order in the FFmpeg line, but that didn't fix it, MPlayer proceeded to decoding the video-stream, but FFmpeg would still lock up.
The solution was to use two MPlayer instances, one for the audio-stream, one for the video-stream. Now I could write the encoded video to a file, while muxing it with the audio stream. That was not what I was looking for though. What I wanted was streaming the video over the network, so I piped the output of FFmpeg to stdout, which greeted me with the fact that I need to specify a container format to be able to stream to stdout. Therefore I tweaked the FFmpeg line a bit:
If I played /tmp/stream now with MPlayer, I could see the encoded video while it was being encoded. This was quite the achievement for me and the only thing left was streaming it over the network.
At first I tried serving the fifo pipe with lighttpd, that obviously failed and lighttpd produced a 404 error. Next thing I tried was using ffserver, but I never really understood how the configuration file is supposed to work. Then I remembered that I once created a fifo pipe on the tablet's internal storage and managed to pipe a data-stream over ssh to that fifo pipe. That however showed some weird behavior, such as the media player not playing the file until the stream was terminated. The solution to my problem was netcat. I just learned about netcat a few days ago and already use it to stream some log-files over telnet, so I figured it should be able to do the job, but how? After quite some research I came across a page that described how to use netcat to imitate a web-server, which is exactly what I needed. So tried using netcat to stream the fifo pipe over some port:
In general that did work when opened the socket, piped the data to stdout and used another pipe to let MPlayer play it like this:
MPlayer itself would fail to play the stream though, this was because it didn't get a proper HTTP reply, so after a little research I found myself typing this:
MPlayer accepted that little hack and so did MX Player on my Nexus. So after a few hours of testing and research I was able to hardsub, encode and stream any video file whatsoever. In the end I put everything neatly in a little script so I won't have to remember everything:
TL;DR part:
This script also includes another functionality: you can skip to a given time in the stream, so you don't have to watch the whole stream over and over again.
To open the stream now, direct whatever player you are using at http://<ip of the machine running this>:8080/
My first approach to tackle this was re-encoding in the High Profile of the H.264 Codec, that, however, would have taken about 3 times the playtime of an episode, while reducing quality.
My second approach, since this long waiting period just isn't worth it, was re-encoding to MPEG4 Part 2, which is a lot easier to encode. The filesize increase was huge though, more than twice the size of the input file, also visual quality is reduced due to the way the colors are encoded (I think?).
These two options given, I first encoded the files in MPEG4 Part 2 and put them on my tablet, with 8 GB of storage, however, it was clear that this was not a solution, so I started playing around with streaming.
The first problem was, that while I was able to encode video- and audio-stream separately if encoding to my harddrive, this was not the case for streaming. For simply encoding and hardsubbing the video-stream I used MPlayer and a fifo pipe which then was encoded by FFmpeg.
The MPlayer line looks like this:
1 | mplayer -vo yuv4mpeg:file=/tmp/foo -ao null
|
and the FFmpeg line like this:
1 | ffmpeg -i /tmp/foo -vcodec mpeg4 -sameq /tmp/tmp.mp4 |
I had to get both streams, audio and video to FFmpeg so it could mux the streams together. At first I tried using only one MPlayer instance, it turned out, that this:
1 | mplayer -vo yuv4mpeg:file=/tmp/foo -ao pcm:file=/tmp/bar |
in combination with this:
1 | ffmpeg -i /tmp/foo -vcodec mpeg4 -sameq -i /tmp/bar -acodec aac -ab 192k /tmp/tmp.mp4 |
produced a deadlock, MPlayer would not decode the video-stream, because it was waiting for the audio-stream to be read by FFmpeg, which itself was waiting for the video input by MPlayer. So I rearranged the order in the FFmpeg line, but that didn't fix it, MPlayer proceeded to decoding the video-stream, but FFmpeg would still lock up.
The solution was to use two MPlayer instances, one for the audio-stream, one for the video-stream. Now I could write the encoded video to a file, while muxing it with the audio stream. That was not what I was looking for though. What I wanted was streaming the video over the network, so I piped the output of FFmpeg to stdout, which greeted me with the fact that I need to specify a container format to be able to stream to stdout. Therefore I tweaked the FFmpeg line a bit:
1 | ffmpeg -i /tmp/foo -vcodec mpeg4 -sameq -i /tmp/bar -acodec aac -ab 192k -f matroska pipe: > /tmp/stream |
If I played /tmp/stream now with MPlayer, I could see the encoded video while it was being encoded. This was quite the achievement for me and the only thing left was streaming it over the network.
At first I tried serving the fifo pipe with lighttpd, that obviously failed and lighttpd produced a 404 error. Next thing I tried was using ffserver, but I never really understood how the configuration file is supposed to work. Then I remembered that I once created a fifo pipe on the tablet's internal storage and managed to pipe a data-stream over ssh to that fifo pipe. That however showed some weird behavior, such as the media player not playing the file until the stream was terminated. The solution to my problem was netcat. I just learned about netcat a few days ago and already use it to stream some log-files over telnet, so I figured it should be able to do the job, but how? After quite some research I came across a page that described how to use netcat to imitate a web-server, which is exactly what I needed. So tried using netcat to stream the fifo pipe over some port:
1 | nc -l -p 8080 < /tmp/stream |
In general that did work when opened the socket, piped the data to stdout and used another pipe to let MPlayer play it like this:
1 | wget -O - http://localhost:8080/stream | mplayer - |
MPlayer itself would fail to play the stream though, this was because it didn't get a proper HTTP reply, so after a little research I found myself typing this:
1 | (echo -e 'HTTP/1.1 200 OK\r\n'; cat /tmp/stream) | nc -l -p 8080 |
MPlayer accepted that little hack and so did MX Player on my Nexus. So after a few hours of testing and research I was able to hardsub, encode and stream any video file whatsoever. In the end I put everything neatly in a little script so I won't have to remember everything:
TL;DR part:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | #!/bin/bash if [[ "$2" = "" ]] then skip=0 else skip=$2 fi rm /tmp/bar /tmp/foo /tmp/stream mkfifo /tmp/foo mkfifo /tmp/bar mkfifo /tmp/stream # dump raw streams to fifo pipes echo Started audio mplayer screen -d -m mplayer -ss $skip "$1" -ao pcm:file=/tmp/bar -vo null echo Started video mplayer screen -d -m mplayer -ss $skip "$1" -vo yuv4mpeg:file=/tmp/foo -ao null # encode and mux raw fifo pipe streams echo Starting ffmpeg pipe ffmpeg -i /tmp/bar -acodec aac -ab 192k -i /tmp/foo -vcodec mpeg4 -sameq -f matroska pipe: > /tmp/stream 2> /dev/null & echo Started ffmpeg pipe # server that bitch some http echo Starting netcat httpd (echo -e 'HTTP/1.1 200 OK\r\n'; cat /tmp/stream) | nc -l -p 8080 |
This script also includes another functionality: you can skip to a given time in the stream, so you don't have to watch the whole stream over and over again.
To open the stream now, direct whatever player you are using at http://<ip of the machine running this>:8080/
Subscribe to:
Posts (Atom)