UPDATE (06/04/2012): Part II has just been published; it's HERE.
(UPDATE (some hours later): 720p 60 fps mode now works just great! Scroll down to the first update!) While the camera of the iPhone 4S is pretty decent for taking videos, it has some major problems – as had the iPhone 4. The default 1080p mode the stock Camera application records in:
1. has quite bad low-light performance
2. has a pretty narrow (around 43-44mm equivalent) field-of-view (FoV), which is even a bit narrower than most consumer video cameras
(3. there is no way of setting the frame rate, which is always around 30 frames per second (FPS))
Let's take a look at how each of these problems can be (quite easily!) fixed!
Want to jailbreak?
First and foremost, if you don't want / aren't able to jailbreak your iDevice, the only way to go is FiLMiC Pro (AppStore link). It's, if you can learn to live with the problems and shortcomings caused by the “app sandbox” (and to fix them manually, if necessary), pretty much fixes all the problems. However, it isn't free and the indirect access to the camera roll, topped by non-existing error correction, can make the life of its users pretty hard.
If you do want to jailbreak, it's far easier to achieve the same – or, for that matter, in many respects even (far) more! You'll not only have direct access to the camera roll (the advantages of which will be shortly explained) and can use the stock Camera app, but also have access to a special “binned” mode at 720p (that is, at a fairly good resolution), which makes the low-light performance of the sensor excellent while not trading in (much) resolution.
First, let's directly compare the two approaches. Let's start with what problems can be fixed by both solutions equally well.
1. Weak low-light performance
Compare the following two direct framegrabs from video streams (click on them to make them larger!):
What do you notice? Yes, it's almost impossible to recognize anything in the first shot, while the second, while has much lower resolution, is, recognition-wise, perfect. When there is so little light that shooting video would result in a totally dark image without anything recognizable, you'll want to prefer the second mode, even at the expense of the lower resolution.
1.1 Effects of the frame capture frequency on the noise
If you don't need to take 30 frames in a second, but can live with, say, 24 or even 15, the low-light performance can be somewhat enhanced even at higher resolutions. Just compare the above 1080p grab to the following one, taken at 15 fps:
While definitely not the same low-light performance than that of the VGA mode (or the jailbroken “binned” 720p mode – see 1.2), it's still somewhat better than the default 30 fps mode.
Note that there is almost no visible difference, low-light performance-wise, between the 24p and 30p modes of either 720p or 1080p. This is certainly bad news, given that with several cameras, slightly (from 30 to 24) decreasing the FPS dramatically enhances the low-light performance. (One of the current cameras I've tested in this regard is the well-known “Extreme sports camera”.) Two framegrabs demonstrating this (720p first, 1080p second):
1.2 “Binned” 720p mode (jailbreak only!)
(Note that the following section doesn't apply to FiLMiC Pro as it requires jailbreaking.)
During my testing, I've thoroughly checked the effects of the “binned” boolean, now, officially enabled and configurable on the 4S (not so on the previous model, the iPhone 4). It allows for making use four times as many sensor pixels as the non-binned version to render one final pixel, practically making the final “superpixel” four times more sensitive to light. Unfortunately, it only works in 720p mode but not the higher-resolution 1080p one. Nevertheless, 720p itself represents a quantum leap in quality from the VGA resolution available, via FiLMiC Pro, for low-light shooting without jailbreaking.
Compare the following two 720p shots (feel also free to compare them to the 1080p/VGA shots in the previous, introductory section):
There is quite a huge difference, isn't there?
Fortunately, using binned mode didn't result in any kind of problems like decreased FoV (it's exactly the same as with non-binned modes) or decrease in frame recording speed.
2. Narrow field-of-view
The narrow field-of-view has been a major problem with both the iPhone 4 and the 4S. (Actually, it was the main reason I've started enhancing the video mode of these models.) The hardware (lens + sensor combination) would be able to take videos of a much wider (28 mm equivalent at best) area with some restrictions (more on them later). Let's see how the different modes compare to each other. The following series of framegrabs shows exactly the same area in a controlled shooting environment. Note how the FoV widens as we go down with the resolution!
(1080p)
(720p)
(960*540)
(VGA)
(360*480)
Basically, this all (720p has a slightly wider FoV than 1080p) can be easily explained. 720p uses pixels on the sensor two times its native resolution. That is, it uses 2*1280 (= 2560) x 2*720 (=1440) pixels (of the 3264 × 2448). When not binned (without hacking), only one of the four hardware pixels contribute to the final image; in binned mode, however, all the four.
You can easily understand why 1080p has a somewhat narrower FoV than 720p: as two times 1080p exceeds the physical size of the sensor (1920*2 = 3840 > 3264), you can't use any multipliers to widen the FoV – you can only use the center area of the sensor, which is slightly larger than the half of the width (and slightly smaller than that of the height). As the 720p, as opposed to 1080p, can use an integer multiplier of 2, it effectively uses a slightly larger sensor area (2560 vs. 1920 horizontally and 1440 vs. 1080 vertically). Hence the wider FoV.
Lower resolutions can use higher integer multipliers, which make it possible to reach the physical sensor sizes better than with 720p, let alone 1080p. This is why FoV gradually widens when you decrease the resolution.
2.1 Side note: why not use the entire surface of the light sensor?
Note that, in the most ideal case, all pixels of the sensors would be used in movie mode. This would have resulted in far better low-light performance and a FoV equal to the stills mode. The processing required for runtime resizing, however, have cased Apple's engineers' not to use this approach in either the iPhone 4 or the 4S. Given that the iPhone 4 really struggled with runtime resizing from its considerably smaller (5 Mpixel only) sensor to around 1080*900 in the hack I developed resulting in a net framerate of about 15 fps only (instead of the default 30), it's easily understandable the 4S wouldn't be able to deliver 30 fps either. After all, while it does have a much faster CPU, it would also need to resize much more pixels. The net effective framerate would be hardly over 20 fps.
Why they use twice the resolution of the sensor area in 720p, and not just the inner rectangle sized 720p, you may ask. The answer is simple: then, the FoV would have been even worse – much-much narrower (something around 60-70mm equiv.). And why don't they use three times as many to have a much wider FoV? Because 3 times 1280 is 3840, while the sensor's width is 3264 pixels only. Next time maybe with a 12+ Mpixel sensor that is at least 3840 pixels wide. (Note that 3*720 is 2160, which is smaller than the physical pixel height count of the sensor (2448); that is, it's the width that was insufficient).
Finally, why not using, say, 2.5 as a multiplier (by, say, reading the second, third, second, third etc. pixels and mapping them to one final pixel in the output video stream)? Then, 3200 pixels would be used, which is only 64 pixels less than the physical size, resulting in almost as wide video as with the stills. The reason for this would be some kind of a visual distortion, particularly visible with jagged edges and where moire can occur. Moire would be much more present in such a configuration. (It is also present in anything with pixel skip technology and is only completely eliminated when using pixel binning, assuming a stills-specific low-pass filter in front of the sensor.) That is, it's perfectly understandable why the Apple folks haven't gone the non-Integer-multiplier way. As with all high-resolution digital cameras except for a very few like the Panasonic GH2, the “king” of all DSLR's when it comes to producing decent video.
3. Using different frame recording speed
In cases, you may want to decrease the number of frames. For example, you target your video as a movie, requiring 24p (you won't really want this, the 4S' camera being pretty poor at both dynamic range and noise, really restricting its use compared to top phone cameras like the Nokia N8, let alone “real” cameras) or, for PAL TV, 25 fps.
It's, unfortunately, in no way possible to set the recording framerate with the stock Camera app without jailbreaking. Alternatively, if you in no way want to jailbreak, you can use FiLMiC Pro for lower-FPS recordings.
Actually, FiLMiC Pro can use speeds not achievable with the jailbreaking approach; for example, 1 FPS. Using this has resulted in an invalid MOV file with jailbreaking, while FiLMiC Pro has produced excellent results. Nevertheless, at common speeds like 15, 24 or 25 fps, both approaches work.
3.1 Recording speed larger than 30 fps (UPDATE: note that this section is no longer true - scroll down to the first update for more info!)
Tons of dedicated cameras offer recording speed larger than 30 fps; some even offer framerates in the hundreds or even thousands – at lower resolution, of course. Even the one-year-old Nikon P300 has a 120 fps mode at VGA resolution (excellent for graphical benchmarking purposes – I use this mode very often!)
Unfortunately, the 4S in no way can be hacked to achieve anything over 30 fps, not even at lower resolutions. Interestingly, there's a 720p 60 fps mode buried in the configuration plist file, but it just doesn't work: the video it produces has an effective framerate of around 18-20 fps and stutters heavily. Unfortunately, I haven't been able to make it work flawlessly and, therefore, it's best to be left alone.
All in all, you can forget using the 4S as a high(er)-speed video camera. It's plain not possible.
4. FiLMiC Pro
I don't have the time to write a complete introduction to FiLMiC Pro, the only solution for the non-jailbreaker folks that still want to have a solution for the low-light / FoV / non-30-fps problems. That is, I'll only elaborate on its (major) problems and how they can be fixed. I recommend googling around for reivews. I don't think you should read previous user comments (particularly not in the AppStore) on the app as some of them are plainly based on some of the users' not knowing sufficiently the recording modes and/or simply refer to problems fixed in later versions.
Both resolution and frame rate can be set from inside Settings. If you change the frame rate, you'll also be able to select from four different encoding bitrate. (With the default 30 fps mode, only the stock one is available. However, it's, quality-wise, very good in my opinion – far better than, say, Panasonic's H.264 encoder in the famous ZS3 / TZ7, which is a 2009 model. I haven't tested the quality of the encoder used by FiLMiC Pro. Hope it isn't worse than that of Apple. Note that if you jailbreak your phone and use a non-standard framerate, the system will always use Apple's own – and very good! - encoder.)
Note that, on the iPad 2 and iPod touch 4th gen, there isn't a separate exposure and focus bracket, only exposure setting. On all video-capable iPhone models (even the 3G S), there is.
4.1 What you should really be aware of...
Compared to the stock Camera app, the biggest problem with FiLMiC Pro is that it records to its own (temp inside its home) directory and only transfers the video to the standard Camera roll when it's finished shooting (or, if you change the default setting of always exporting videos, when you manually instruct it to do so). With (compared to the size of the videos you shoot) infinitely large free storage and relatively small videos, this won't be a problem. On storage-restricted iDevices, however, you'll run into severe problems if you stick with the (default) Camera Roll target mode.
Unfortunately, if there's any problem (the storage runs out at either recording or during the final copying) during recording and the video isn't copied to the Camera Roll, you'll completely lose everything you've done – it remains inaccessible from inside the app. All you can do is looking for the file “output.mov” in the above-mentioned “temp” directory and copying it to the desktop. Do it before starting to make another video, as the next recording overwrites the previous ones without warning – even ones that were not transferred to the Camera Roll!
The app doesn't give any visual feedback when it's copying the videos to the Camera Roll either. You can only see this from the device's slowing down and/or the video's inaccessibility inside Camera Roll. This is a VERY bad approach as there will surely be users that kill the app and/or restart the iDevice as they don't know there is a copy in progress in the background.
If you do switch to its own storage, there won't be such problems. Then, however, you'll need to manually copy your videos to the Camera Roll. Needless to say, you can't directly access these videos through iTunes' Document Sharing either – you MUST copy them to the Camera Roll or use a desktop tool allowing for accessing the document directories of apps; for example, my personal favorite, Phone Disk (or, if you don't want to pay, iExplorer).
All in all, the first thing you do after installing FiLMiC Pro is going to Settings and setting “Save To...” to FiLMiC Library instead of the default Camera Roll!
5. Jailbreaking – what should be kept in mind?
If you don't want to go the FiLMiC Pro way as you want to directly record to the camera roll, want to use the binned 720p mode etc., you'll need to jailbreak and overwrite a file, /System/Library/PrivateFrameworks/MediaToolbox.framework/N94/AVCaptureSession.plist, with any file transfer tool. (Again, I recommend Phone Disk or, if you don't want to pay, iExplorer).
As I've found two additional modes you'll want to use, based on the circumstances (do you need to shoot in low light? Do you need as wide a FoV as possible?), I've made three files available for now, before releasing a Cydia-based complete converter, as I've done for the iPhone 3G S and the iPhone 4 back in June-July 2011 (you'll need to wait for the Cydia app a bit, though. Dunno when I'll be able to release it. Maybe in April only, given that I'll be traveling and skiing during the entire March, which means I simply won't have any energy – not to mention lust! - for coding “only” for fun). They're as follows:
VGA mode - if you want absolutely the widest FoV, excellent low-light performance and low storage usage and aren't bothered by the lowish resolution
720p standard mode - if you want a somewhat wider FoV and/or, compared to the 1080p mode, significantly lower storage usage and the resolution disadvantage (which is in no way as pronounced as between the 1080p and VGA modes!) isn't that big a problem
720p binned mode - if you need all the advantages explained in the previous bullet and, in addition, much better low-light performance.
Note that you'll need to rename these files to AVCaptureSession.plist before transferring them to /System/Library/PrivateFrameworks/MediaToolbox.framework/N94/. I've kept them under a different name so that you can easily see what they're for.
Note that I haven't created a separate file for non-standard encoding speeds. You can play with the “AVCaptureDevices / AVCaptureSessionPresetHigh / VideoCompressionProperties / AverageDataRate” dictionary sub-entry (see next section), should you want to lower/raise the encoding rate. Also, I haven't provided 24p/25p versions either. You can just include the following two dictionary entries right under AVCaptureDevices / AVCaptureSessionPresetHigh / LiveSourceOptions:
<key>MaxFrameRate</key>
<integer>24</integer>
<key>MinFrameRate</key>
<integer>24</integer>
(Change 24 to your desired fps if you need to shoot at a different framerate. Remember NOT to supply ANYTHING over 30!)
Remember to always kill the Camera app after overwriting the original plist file so that the app re-reads the new ones!
5.1 What's in the AVCaptureSession.plist file? (Advanced users only!)
As has already been pointed out in my iPad 2-specific article, iOS 5 has relocated the camera configuration files. Now, they're located at /System/Library/PrivateFrameworks/MediaToolbox.framework/N94. (N94 is model-specific; for example, the iPad2 has the files in K83). As with the iPad 2, you'll only need to edit one file (AVCaptureSession.plist) and, inside, only one dictionary entry (AVCaptureDevices / AVCaptureSessionPresetHigh) to modify.
Basically, this entry is similar to those under iOS4. They have two dictionary sub-entries:
- LiveSourceOptions: this allows for setting the number of pixels in the target video file (Capture), the sensor pixels utilized (Sensor) and, finally, the size of the preview area in pixels (Preview). All these have both height and width parameters. It's by overwriting the Sensor parameters that I was able to gain access to the full sensor area in the iPhone 4 to wildly widen its FoV.
- VideoCompressionProperties sets the properties of the video compression. The most important dictionary sub-entry is AverageDataRate in this section, which tells the video encoder the encoding speed. The higher the value (given in bytes/sec), the more storage your video footage takes – but at higher quality. Should you want to record mostly static stuff and/or want to take as long recordings as possible, you will want to decrease this value.
In dual-core models like the 4S, some other parameters have been introduced to the VideoCompressionProperties entry, which tell the system to utilize both cores (NumberOfParallelCores and, probably, NumberOfSlices). However, in real-world problems like recording video at 60 fps, I haven't found them resulting in vastly superior recording speed. Actually, removing them from the default 60 fps entry didn't result in any kind of visible performance decrease. I wonder if those two cores of the A5 CPU are indeed utilized, or, is this new dictionary entry just a placeholder...
6. More info?
In this writeup, I tried to compare the two main approaches to non-1080p / low-light / wide-FoV video recording on the 4S, which isn't possible with the stock Camera app.
The above should be sufficient to get you started in both the non-jailbroken and jailbroken case. I've even provided ready-made config files, should you want to use my hacks right away. Remember: my late-February (MWC in Barcelona) and March (traveling & skiing) will be extremely busy; I don't think I'll be able to release a 4S-specific version of my previous, 3GS/4-specific camera hacking Cydia tool. In the meantime, you'll need to stick to directly overwriting the plist file. (This, incidentally, can also be done using iFile on the iDevice itself. Then, you won't need to connect your iDevice to a desktop for overwriting the AVCaptureSession.plist file.)
UPDATE (some hours later): upon a reader's comment, I've checked out THIS. Indeed, by not letting the iPhone 4S do any kind of noise suppression (this is what nulling out TemporalNoiseReductionMode does), the frame rate can be greatly enhanced, on the expense of the low-light performance. (Nevertheless, the latter won't be that bad, either.)
With my standard 60 fps FPS checker video (warning: 125 Mbytes!), I've shot a demo video showing this. As you can see HERE (warning: 50 Mbyte video file!), indeed all frames are recorded when shooting a MacBook Pro screen playing back the video at 60 fps. (Hint: to indeed see every single frame is recorded, pause the video and advance video frames one by one using the Right arrow key.)
As you can guess, the noise is higher than in the default binned case – but is still considerably smaller than in the 1080p case. A comparison:
(1080p)
(720p 60fps, no noise reduction, binned mode)
Note that the binned mode delivers some kind of a resolution hit, as can be seen in the following ISO chart framegrab:
I've made the new config file available HERE. As with the previous, recommended config files, it needs to be renamed to AVCaptureSession.plist before overwriting the original one (/System/Library/PrivateFrameworks/MediaToolbox.framework/N94/AVCaptureSession.plist).
