Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions docs/ios27/CAMERA.md
Original file line number Diff line number Diff line change
Expand Up @@ -309,6 +309,66 @@ The still image output wanting a 10-bit pixel format (`stillImageOutputRequires1
come in a 10-bit pool ('x420' or the like), which `camera_device.c` does not draw into (`StillImage: ... buffers are not for this
ISP`): the next gap if the shutter gives nothing.

### The fifth cause: no pixel format for the still image output (18 November)

N104's `AVCaptureSession.plist` says `UsesPacked10BitFirmwareStillImageOutputPixelFormat` for the photo formats of both
cameras ("12MP", the front camera's "7MP") and for 1080p, 720p and 1920 x 1440: the A13's ISP gives its photographs in a
packed 10-bit format. iOS 27.0.1's CMCapture (24A446's shared cache, its logic only) then sets the source node's
`stillImageOutputRequires10BitPixelFormat`, and `-[BWMultiStreamCameraSourceNode _supportedPixelFormatsForOutputID:]`
keeps, of each output's `SupportedPixelFormats` in the stream's format, the ones whose range, bit depth, 4:2:2-ness and raw-ness
match what the output wants: for the still image output full range, 10-bit exactly when that flag is set, not 4:2:2, not raw,
and no more lossy compression than the session allows. The other outputs follow the session's client video format (420f
for Photo mode). The S9's streams offered 420f, 420v and BGRA, so with the flag the still image output was left with no
pixel format at all, and the session that builds its pools for every output could not start.

Dylib `-17`: `camera_refuse.c` answers NO to `-[FigCaptureSourceVideoFormat usesPacked10BitFirmwareStillImageOutputPixelFormat]`
(`still: ... replaced (NO: the still image output takes 420f)`, and at a session `still: the format wants a 10-bit still
image ... answered NO`): the still image output takes 420f, which `camera_device.c` draws. `cam_still10=1` on the boot line
keeps the plist's answer; for that case `camera_formats.c` also offers `xf20` (10-bit full range, two planes, each sample 16
bits with its 10 bits at the top) on the StillImage output, and `camera_tenbit.c` draws the picture in 8 bits into a
scratch 420f buffer and widens it into the pool (`StillImage: the graph's pool is 10-bit (xf20): ...`). The preview does
not need 10 bits: its outputs take 420f, a two-plane IOSurface that the Metal bridge samples (`newTextureWithDescriptor:iosurface:plane:`
with its YCbCr planes, `tools/metal/bridge/NativeMetalObjects.rs`). **Not run on a phone yet.**

### The sixth cause: outputs the graph configures that the stream did not keep (18 November)

Phone 1 ran dylibs `-17` to `-20` on 18 November (front and back camera, `cam_trace=1`). With `-17` the still image output had
its 420f, and cameracaptured still aborted at every session: `Internal inconsistency, output pixel format () doesn't match the
pixelBufferPool's pixel format (420f)`. The throw is in `-[BWMultiStreamCameraSourceNode allocateOrReuseBufferPoolsFromSourceNode:]`
(24A446, its logic only): after setting `VideoOutputConfigurations`, the source node reads the property back from the stream
and wants each output's `PixelFormat` to equal its pool's. Two things made it differ:

1. The graph configures `IntermediateTap` and `Depth`, which `camera_device.c` did not keep (`stream: an output not here`), so
they were missing from the read-back. On iOS 27 `IntermediateTap` is now a fourth output, drawn as the scalers are, and the
configurations of outputs that are not drawn are given back as they were set (dylib `-18`).
2. The graph's own configuration of `PrimaryScaler` (1504 x 1128 in Photo mode) carries `PixelFormat` 0. The source node takes a
streaming output's pixel format from its node output's video format, which nothing downstream resolves on the S9 (the Live
Photo movie that would use it is left out, above), while the output's pool, shared with the other streaming outputs, is
420f. `camera_outputs.c` reads a configuration whose `PixelFormat` is missing or 0 back as 420f (dylib `-20`;
`-19` filled only a missing one). Each set of configurations, enables and handlers is said on one line: `outputs: Back
Camera: VideoOutputConfigurations: StillImage (format 420f, 4032x3024, ...), PrimaryScaler (...)`.

With `-20` (11:00) the graph ran for the first time on iOS 27: no abort, `the graph starts the stream`, `PrimaryScaler: the first
picture given (1504x1128 420f)`, the source node's `Preview(PrimaryScaler)` output emitting about eight pictures a second
(`camera_isp.c` on the CPU), the shutter giving a 4032 x 3024 picture to the still image coordinator, and the flip button
changing to the front camera. Still missing: the screen stayed black (no node after the source node emitted anything; the
viewfinder was open, so the frames stop inside cameracaptured's graph), and the photograph failed (`-11800`, underlying
`-16808`, 10 ms after the shutter, before the picture was made) and was not saved. Dylib `-21` follows each output to the node
it leads to (`camera_flownext.c`: `trace: flow: ... leads to <class> "<name>"`, `... renders a sample ...`).

### SpringBoard's main-display layout (18 November)

`camera_sbtrace.c` on phone 1 (dylib `-16`, `-17`) shows that SpringBoard does publish its layout: the home screen, the lock
screen and then `com.apple.camera` as elements, and the backlight's state 2 (on) once it is on. Two publishers carry it. The
one FrontBoard's shell makes for the main display gets the elements and the backlight state, flushes once at 23 s (no
elements, backlight 0) and never again. The second, made by SpringBoard and set as the primary publisher at 43 s
(`setPrimaryPublisher` in SpringBoard's log), flushes every change with its elements but without the backlight. A third,
for the continuity display, is held by a suppression (`startListener`, `-[SBContinuityDisplayLayoutCoordinator start]`) that
never ends, which is harmless. cameracaptured's monitor timed out waiting for the first layout at 21 s, before SpringBoard
published anything, and afterwards sees no application in front: `camera_layout.c`'s stand-in is still what lets the
camera run. Why the first publisher stops flushing (the shell's `%@ Loading` transition, `FBSystemShellInitialize`, ends
at 55 s with the other milestones, so it is not that) is not known yet.

### Where the pictures stop: the flow trace

With `cam_trace=1`, `tools/ios27/camera/camera_flowtrace.c` follows the frames through cameracaptured's graph: which of the
Expand Down
2 changes: 1 addition & 1 deletion tools/arch-size.allow
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ tools/qmi-test/test_qmi.c 1562
userland/src/HakiosMetal.m 1534
userland/src/HakiosMetalCA.m 8747
userland/src/audio_plugin.c 1100
userland/src/camera_device.c 1989
userland/src/camera_device.c 2043
userland/src/camera_isp.c 1230
userland/src/eagl.c 4781
userland/src/gfx/gfx_gl.c 1083
Expand Down
2 changes: 1 addition & 1 deletion tools/ios27/camera/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ xcrun --sdk iphoneos clang -arch arm64 -isysroot "$SDK" -miphoneos-version-min=1
-dynamiclib -install_name /usr/lib/cam.dylib \
-DHAKIOS_CAMERA_N104 -DHAKIOS_CAMERA_CONSOLE '-DHAKIOS_CAMERA_FW="/private/var/hakios/camera"' -I"$SRC" -I"$HERE" \
-framework CoreFoundation -framework CoreMedia -framework CoreVideo -lobjc \
-o "$OUT" "$HERE/camera27.c" "$HERE/camera_trace.c" "$HERE/camera_oslog.c" "$HERE/camera_formats.c" "$HERE/camera_apptrace.c" "$HERE/camera_statetrace.c" "$HERE/camera_layout.c" "$HERE/camera_refuse.c" "$HERE/camera_flowtrace.c" "$HERE/camera_colour.c" "$HERE/camera_sbtrace.c" \
-o "$OUT" "$HERE/camera27.c" "$HERE/camera_trace.c" "$HERE/camera_oslog.c" "$HERE/camera_formats.c" "$HERE/camera_apptrace.c" "$HERE/camera_statetrace.c" "$HERE/camera_layout.c" "$HERE/camera_refuse.c" "$HERE/camera_flowtrace.c" "$HERE/camera_colour.c" "$HERE/camera_sbtrace.c" "$HERE/camera_tenbit.c" "$HERE/camera_outputs.c" "$HERE/camera_flownext.c" \
"$SRC/camera_device.c" "$SRC/camera_say.c" "$SRC/camera_isp.c"
codesign -f -s - "$OUT" 2>/dev/null
[ "$(lipo -archs "$OUT")" = arm64 ] || { echo "build.sh: $(basename "$OUT") is not plain arm64" >&2; exit 1; }
Expand Down
184 changes: 184 additions & 0 deletions tools/ios27/camera/camera_flownext.c
Original file line number Diff line number Diff line change
@@ -0,0 +1,184 @@
/*
* SPDX-License-Identifier: MIT-0
*
* The flow trace's next step (cam_trace=1, with camera_flowtrace.c): where each node output's samples go, and whether the
* node there takes them.
*
* Why: phone 1 with dylib -20 (18 November) had the graph running, the source node's Preview(PrimaryScaler) output
* emitting about eight pictures a second, and the screen black: no node after the source node emitted anything, and
* neither the preview stitcher nor the image queue sink camera_flowtrace.c wraps was given a sample. The preview's nodes
* on iOS 27 are not the ones that file names.
*
* At the first sample a node output emits, scioncam_flow_next() says the node its connection leads to
* (-[BWNodeOutput connection], -[BWNodeConnection input], -[BWNodeInput node], its class and -name), or that it has none,
* and wraps that node class's own -renderSampleBuffer:forInput: and -handleDroppedSample:forInput: (each class once, at
* most 24): their first call, then every 300th, is said with the input's name and the sample. A node that renders and
* never emits is where the pictures stop; one that is never called is behind a connection that does not deliver.
* Names and types are iOS 27.0.1's CMCapture (24A446's shared cache: names only). NOT RUN ON A PHONE.
*/
#include <CoreFoundation/CoreFoundation.h>
#include <objc/message.h>
#include <objc/runtime.h>
#include <pthread.h>
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

void hakios_camera_say(const char *format, ...) __attribute__((format(printf, 1, 2))); /* camera_say.c */
#define say hakios_camera_say

static id
call0(id o, const char *sel)
{
SEL s = sel_registerName(sel);

return o == nil || !class_respondsToSelector(object_getClass(o), s) ? nil : ((id (*)(id, SEL))objc_msgSend)(o, s);
}

static void
name_of(id o, char *out, size_t cap)
{
snprintf(out, cap, "?");
if (o != nil && CFGetTypeID((CFTypeRef)o) == CFStringGetTypeID()) {
CFStringGetCString((CFStringRef)o, out, (CFIndex)cap, kCFStringEncodingUTF8);
}
}

#define WRAPPED 24
static struct wrapped {
Class c;
IMP render, drop;
unsigned long long renders, drops;
} wrapped[WRAPPED];
static unsigned nwrapped;
static pthread_mutex_t wrap_lock = PTHREAD_MUTEX_INITIALIZER;

/* the entry of self's class or the nearest wrapped superclass (the class whose own method was replaced) */
static struct wrapped *
entry_of(id self)
{
for (Class c = object_getClass(self); c != Nil; c = class_getSuperclass(c)) {
for (unsigned i = 0; i < nwrapped; i++) {
if (wrapped[i].c == c) {
return &wrapped[i];
}
}
}
return NULL;
}

static void
said(id self, struct wrapped *w, bool drop, id input)
{
unsigned long long n = drop ? ++w->drops : ++w->renders;
char in[48], node[64];

if (n == 1 || n % 300 == 0) {
name_of(call0(input, "name"), in, sizeof(in));
name_of(call0(self, "name"), node, sizeof(node));
say("trace: flow: %s \"%s\" %s on its input %s (%llu so far)", class_getName(object_getClass(self)), node,
drop ? "is given a DROPPED sample" : "renders a sample", in, n);
}
}

static void
w_render(id self, SEL sel, void *sb, id input)
{
struct wrapped *w = entry_of(self);

if (w != NULL) {
said(self, w, false, input);
((void (*)(id, SEL, void *, id))w->render)(self, sel, sb, input);
}
}

static void
w_drop(id self, SEL sel, id dropped, id input)
{
struct wrapped *w = entry_of(self);

if (w != NULL) {
said(self, w, true, input);
if (w->drop != NULL) {
((void (*)(id, SEL, id, id))w->drop)(self, sel, dropped, input);
}
}
}

/* the class's own method of that name, or NULL */
static Method
own(Class c, SEL s)
{
unsigned n = 0;
Method *all = class_copyMethodList(c, &n), found = NULL;

for (unsigned i = 0; all != NULL && i < n && found == NULL; i++) {
if (method_getName(all[i]) == s) {
found = all[i];
}
}
free(all);
return found;
}

static void
wrap_class(Class c)
{
SEL rs = sel_registerName("renderSampleBuffer:forInput:"), ds = sel_registerName("handleDroppedSample:forInput:");
Class k = c;
Method rm = NULL, dm;

/* the class in c's line that has its own render method (the one a call on c ends in) */
while (k != Nil && (rm = own(k, rs)) == NULL) {
k = class_getSuperclass(k);
}
if (rm == NULL || strcmp(class_getName(k), "BWNode") == 0 || method_getNumberOfArguments(rm) != 4) {
say("trace: flow: %s renders with no method of its own: not traced", class_getName(c));
return;
}
pthread_mutex_lock(&wrap_lock);
for (unsigned i = 0; i < nwrapped; i++) {
if (wrapped[i].c == k) {
pthread_mutex_unlock(&wrap_lock);
return;
}
}
if (nwrapped == WRAPPED) {
pthread_mutex_unlock(&wrap_lock);
return;
}
struct wrapped *w = &wrapped[nwrapped];

w->c = k;
w->render = method_getImplementation(rm);
dm = own(k, ds);
w->drop = dm != NULL && method_getNumberOfArguments(dm) == 4 ? method_getImplementation(dm) : NULL;
nwrapped++;
method_setImplementation(rm, (IMP)w_render);
if (w->drop != NULL) {
method_setImplementation(dm, (IMP)w_drop);
}
pthread_mutex_unlock(&wrap_lock);
say("trace: flow: %s's renderSampleBuffer:forInput: traced%s", class_getName(k), w->drop != NULL ? " (and its drops)" : "");
}

/* called once for each node output, at its first emitted sample (camera_flowtrace.c) */
void
scioncam_flow_next(void *output_object, const char *node_class, const char *output_name)
{
id output = (id)output_object, connection = call0(output, "connection"), input = call0(connection, "input");
id node = call0(input, "node");
char in[48], name[64];

if (node == nil) {
say("trace: flow: %s's output %s leads nowhere (connection %s, consumer %s)", node_class, output_name,
connection != nil ? "there" : "nil", call0(output, "consumer") != nil ? "there" : "nil");
return;
}
name_of(call0(input, "name"), in, sizeof(in));
name_of(call0(node, "name"), name, sizeof(name));
say("trace: flow: %s's output %s leads to %s \"%s\" (its input %s)", node_class, output_name,
class_getName(object_getClass(node)), name, in);
wrap_class(object_getClass(node));
}
1 change: 1 addition & 0 deletions tools/ios27/camera/camera_flowtrace.c
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,7 @@ scioncam_flow_emit(void *output_object, void *sample_buffer)
if (fresh) {
sample_of(sample_buffer, what, sizeof(what));
say("trace: flow: %s's output %s emits its first sample: %s", f->node ? class_getName(f->node) : "?", f->output, what);
scioncam_flow_next(output, f->node ? class_getName(f->node) : "?", f->output); /* camera_flownext.c */
}
summary(mach_absolute_time());
pthread_mutex_unlock(&flow_lock);
Expand Down
32 changes: 32 additions & 0 deletions tools/ios27/camera/camera_formats.c
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,37 @@ add_intermediate_tap(CFMutableDictionaryRef format, int width, int height)
CFRelease(all);
}

/*
* The StillImage output also offers 'xf20' (10-bit full range, biplanar 4:2:0): N104's photo formats say
* UsesPacked10BitFirmwareStillImageOutputPixelFormat, and with it the source node keeps only 10-bit full-range formats for
* the still image output (camera_tenbit.c, which draws it). camera_refuse.c answers that key NO, so a session normally
* takes 420f; this is what one that still asks for 10 bits finds.
*/
static void
add_still_tenbit(CFMutableDictionaryRef format)
{
CFDictionaryRef outputs = CFDictionaryGetValue(format, CFSTR("SupportedOutputs"));
CFDictionaryRef still = outputs != NULL ? CFDictionaryGetValue(outputs, CFSTR("StillImage")) : NULL;
CFArrayRef pfs = still != NULL ? CFDictionaryGetValue(still, CFSTR("SupportedPixelFormats")) : NULL;
int xf20 = 0x78663230; /* 'xf20' */

if (pfs == NULL) {
return;
}
CFMutableDictionaryRef all = CFDictionaryCreateMutableCopy(NULL, 0, outputs), one = CFDictionaryCreateMutableCopy(NULL, 0, still);
CFMutableArrayRef more = CFArrayCreateMutableCopy(NULL, 0, pfs);
CFNumberRef n = CFNumberCreate(NULL, kCFNumberIntType, &xf20);

CFArrayAppendValue(more, n);
CFDictionarySetValue(one, CFSTR("SupportedPixelFormats"), more);
CFDictionarySetValue(all, CFSTR("StillImage"), one);
CFDictionarySetValue(format, CFSTR("SupportedOutputs"), all);
CFRelease(n);
CFRelease(more);
CFRelease(one);
CFRelease(all);
}

CFArrayRef
scioncam_n104_formats(uint32_t sensor, const char *camera, double fov, scioncam_format_fn make)
{
Expand Down Expand Up @@ -133,6 +164,7 @@ scioncam_n104_formats(uint32_t sensor, const char *camera, double fov, scioncam_
CFRelease(base);
CFDictionarySetValue(f, CFSTR("SIFREnabled"), e->sifr ? kCFBooleanTrue : kCFBooleanFalse);
add_intermediate_tap(f, e->width, e->height);
add_still_tenbit(f);
if (e->fes_width > 0) {
put_number(f, "FrontEndScalerOutputWidth", e->fes_width);
put_number(f, "FrontEndScalerOutputHeight", e->fes_height);
Expand Down
13 changes: 13 additions & 0 deletions tools/ios27/camera/camera_formats.h
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,17 @@ typedef CFDictionaryRef (*scioncam_format_fn)(int sensor_width, int sensor_heigh
/* the SupportedFormatsArray of the stream of the kernel's SENSOR (1 the back camera, 2 the front one); NULL for another */
CFArrayRef scioncam_n104_formats(uint32_t sensor, const char *camera, double fov, scioncam_format_fn make);

/* camera_tenbit.c: the still image output's 10-bit ('xf20') pool: an 8-bit 420f buffer of pb's size to draw into (NULL:
none), and the picture widened from it into pb (both locked) */
#include <CoreVideo/CoreVideo.h>
CVPixelBufferRef scioncam_tenbit_scratch(CVPixelBufferRef pb);
void scioncam_tenbit_widen(CVPixelBufferRef from, CVPixelBufferRef pb);

/* camera_outputs.c: one line for each output a VideoOutputConfigurations / VideoOutputsEnabled / VideoOutputHandlers set
names (its pixel format, size and pool), so that the console shows which outputs the graph uses */
void scioncam_say_outputs(const char *camera, CFStringRef key, CFDictionaryRef value);
/* camera_outputs.c: an output's configuration as VideoOutputConfigurations is read back (retained): with 420f put in as
its PixelFormat when the graph set none */
CFDictionaryRef scioncam_config_read_back(const char *camera, CFStringRef output, CFDictionaryRef config);

#endif
Loading
Loading