CGDSeg

Formal Definition

Challenge Factors

• Substantial intra-class variance: illumination, gaze angle, ethnicity-dependent scleral chromaticity
• Inter-class boundary ambiguity: specular reflections mimic scleral whiteness
• Occlusion by eyelids, eyelashes, and contact lenses
• Synthetic-to-real domain shift between training and test distributions
• Class imbalance: sclera occupies 15–30% of the eye image area

Evaluation Metrics

Competition Context

SSBC 2026 Foundation Model Track mandates the use of a large pre-trained model backbone. Test evaluation is performed on two held-out real-world datasets (MASD, SBVPI) from which the training pipeline has zero samples in the synthetic-only condition.

Architectural Philosophy

Parameter efficiency: The two frozen backbones (DINOv2 + CLIP) account for 84.9M of 88.3M total parameters. Trainable capacity is confined to LoRA adapters (0.59M), the CLIP projection and cross-attention (≈0.6M), the FPN projection heads, decoder, and output head (≈2.0M).

Multi-scale decoding: The FPN provides feature representations at 16×16, 32×32, and 64×64 spatial resolutions, enabling the decoder to simultaneously exploit global context and fine-grained boundary information.

Language-guided segmentation: CLIP-based text grounding biases the patch tokens toward semantically relevant sclera features prior to the spatial pyramid, improving disambiguation between sclera and specular highlights.

Parameter Budget Summary

torch.compile() applied at runtime for kernel fusion and XLA optimisation.

Novel Architectural Contributions

DINOv2-ViT-S/14 Specifications

Why DINOv2 over Supervised ViT?

DINOv2 was trained via self-supervised knowledge distillation (DINO) and masked image modelling (iBOT) on 142M curated images. This yields semantically structured patch representations — patches belonging to the same semantic region exhibit high cosine similarity — without label supervision. Empirically, DINOv2 features generalise to novel tasks with minimal fine-tuning, a critical property given the small sclera segmentation training set.

Position Embedding Resizing

The model's native positional encodings were trained for a 37×37 patch grid (518×518 input). At 256×256 input, the grid reduces to 18×18. The pretrained positional embeddings are bilinearly interpolated from (37,37) to (18,18) grid positions at model load time, as confirmed in the training log:

Rationale for Freezing

Fine-tuning 21M parameters on ≤11,000 training pairs risks catastrophic forgetting of the rich general-purpose representations acquired during large-scale pre-training. The LoRA approach preserves these representations while adapting the model to sclera-specific features through a low-rank perturbation of the QKV projection weights.

Scaled Dot-Product Attention

Residual Block Structure

Global Receptive Field

Each patch token can attend to all 324 other tokens simultaneously. This is architecturally distinct from CNNs, where receptive field grows logarithmically with depth. For sclera segmentation, this enables a patch at the medial canthus to attend to a patch at the lateral limbus in a single layer — critical for establishing the full extent of the scleral region.

DINOv2 Self-Supervised Pre-training Objectives

These objectives collectively encourage patch tokens to encode both local texture and global semantic context — properties directly beneficial to segmentation tasks.

Core Formulation (Hu et al., 2022)

Parameter Efficiency

Rank Selection Rationale

Rank r=8 provides sufficient expressivity for domain adaptation from natural images to ocular photographs, consistent with recommendations in the LoRA literature for moderate distribution shift scenarios. Higher ranks (r=16,32) showed marginal improvement in preliminary experiments but substantially increased memory footprint. The rank-8 constraint is not a limitation — it is a deliberate inductive bias: if the difference between DINOv2 features optimised for ImageNet and features optimised for sclera segmentation can be expressed as a rank-8 perturbation, the model is compelled to find the most compact, generalising adaptation.

Alpha Scaling — Why 2.0?

The hyperparameter α=16 with r=8 gives a scale of α/r=2.0. This has a specific interpretation: at initialisation, ΔW=0 (due to B=0), so the first gradient step on A and B is scaled by 2.0 relative to a naive unit-scale LoRA. This accelerates early convergence without destabilising the frozen backbone representations. The choice of α=2r (a common convention) ensures that as r increases, the learning rate contribution per rank dimension remains constant — a desirable invariance property when comparing models across ranks.

Why LoRA Over Other PEFT Methods?

The key insight: The pre-trained DINOv2 weight matrix W₀ encodes a high-dimensional manifold of natural image representations. The adaptation required for sclera segmentation — essentially learning to recognise a specific anatomical region — corresponds to a low-dimensional perturbation of this manifold. LoRA's low-rank constraint directly captures this hypothesis: if the task-specific shift ΔW truly has intrinsically low rank, LoRA will find it exactly; if ΔW is not low-rank, LoRA finds the best low-rank approximation, which is empirically sufficient for most transfer tasks.

LoRA vs. Alternatives

LoRALinear Module

Freeze Protocol

Injection Logic

During forward pass, the frozen linear projection and the LoRA bypass are evaluated in parallel and summed. Backpropagation flows only through A and B; W₀ accumulates zero gradient.

During forward pass, the frozen linear projection (W₀·x) and the LoRA bypass path (B·A·x·α/r) run in parallel and are summed at the output. Only A and B receive gradients — W₀ accumulates zero gradient throughout all 70 training epochs.

CLIP ViT-B/32 Text Encoder

Prompt Ensemble

Prompt Ensembling Justification

CLIP text embeddings are sensitive to surface-form prompt variation. The empirical technique of prompt ensembling, introduced alongside CLIP (Radford et al., 2021) and formalised in CoOp (Zhou et al., 2022), averages L2-normalised embeddings of semantically equivalent prompts to produce a more robust, less template-dependent semantic anchor. For sclera, this mitigates the sensitivity to whether the encoder was trained with medical or colloquial descriptions of the structure.

Why CLIP Grounding? — Architectural Innovation

Problem it solves: DINOv2 patch tokens are purely vision-derived — they encode texture, colour, and shape statistics without any semantic alignment to human-defined categories. For sclera segmentation, this means the network must discover the concept "sclera-ness" entirely from visual patterns in the training data, which is particularly challenging given the visual similarity between sclera and specular highlights, or conjunctival vessels near the limbus.

What CLIP contributes: CLIP was trained via contrastive alignment on 400M image-text pairs. Its text encoder maps "sclera white of the eye" to a point in a 512-dimensional semantic space where visually similar concepts cluster together. By projecting this semantic anchor into the patch token space and fusing it via cross-attention, we impose a learned semantic bias on the patch token sequence — patches representing the scleral region are pulled toward this anchor, while patches representing non-sclera structures are not.

Zero-cost at inference: The text embedding is computed once and cached. At training and inference time, CLIP contributes zero FLOPs beyond the single cached matrix multiply in the projection layer and the cross-attention operation. This is fundamentally different from approaches that require joint vision-language processing per image.

Integration into Forward Pass

Self- vs. Cross-Attention

Self-attention (used in DINOv2 internally) computes queries, keys, and values from the same token sequence. Cross-attention draws queries from one sequence (patch tokens) and keys/values from another (text embedding), enabling inter-modal information fusion without altering the query sequence's positional structure.

Attention Score Interpretation

Motivation for Multi-Scale Representation

A single-resolution feature map at 18×18 pixels (the DINOv2 output) is insufficient for high-quality boundary delineation at 256×256 target resolution. The FPN constructs a set of feature maps at progressively finer spatial resolutions while reducing channel depth, providing the decoder with both high-level semantic information and spatial precision at each upsampling stage.

Scale Construction

Design Decisions

Channel halving: Channel widths {384, 192, 96} follow a 2× reduction per scale level, balancing computational cost against feature capacity. Finer scales carry lower channel depth because spatial information compensates for reduced representational dimensionality.

GELU activation: GELU is used in the FPN projection heads for consistency with DINOv2 and CLIP's internal activations. Activation consistency across stages reduces representational mismatch at the feature boundaries.

Bilinear interpolation: Bilinear upsampling from 18×18 to each target resolution is artifact-free and parameter-free. The Conv1×1 projection following the upsample provides the learnable channel mixing necessary for scale-appropriate feature extraction.

Why FPN Over U-Net Encoder?

The architectural insight: A classical U-Net requires a dedicated encoder path that progressively downsamples the input image, storing skip connections at each resolution. With DINOv2 as the backbone, this entire encoder is already provided — the 12-block ViT outputs patch tokens that already encode multi-level semantic information through its depth. Building a separate encoder on top of DINOv2 would be redundant and computationally expensive.

What the FPN does instead: Rather than encoding multiple input resolutions, the FPN decodes a single, rich 18×18 representation into multiple spatial scales via cheap bilinear interpolation and Conv1×1 channel projection. This is O(C) cost per scale level, versus O(H²·C) for a full convolutional encoder path — an enormous computational saving that contributes directly to the 3.61% trainable parameter fraction.

The bottleneck asymmetry: Notably, f2 (the coarsest FPN scale) performs a slight downsampling from 18×18 to 16×16. This produces a clean 2× spatial relationship across all three scales (16→32→64), simplifying the decoder upsampling arithmetic and avoiding misaligned concatenation at decoder stages.

DenseNet Layer (Huang et al., 2017)

In a DenseBlock with L=4 layers and growth rate k=32, the input to layer ℓ is the concatenation of all prior layer outputs:

Dense Connectivity — Full Skip Connection Map

Advantages of Dense Connectivity

• Implicit deep supervision: Loss gradients propagate directly to all preceding layers, mitigating vanishing gradient in the decoder. Each layer receives direct gradient signal from the final loss, effectively acting as if it were the last layer — this is particularly valuable in narrow decoder pathways where gradient flow might otherwise degrade.
• Feature reuse without redundancy: Each layer has direct access to all antecedent feature maps. Unlike ResNets (which add features), DenseBlocks concatenate them — the network can learn to select from the full feature history, exploiting low-level edge detectors and high-level semantic activations simultaneously at each stage.
• Regularisation through diversity: Concatenating L prior feature maps at each layer input forces the subsequent convolutional kernel to operate on a highly heterogeneous input manifold. This implicit diversity serves as a regulariser analogous to dropout — overfitting to any single feature map is structurally penalised.
• Parameter efficiency: Dense connectivity achieves high effective depth with fewer parameters per layer. A standard 4-layer block with growth rate k=32 adds only 4×32=128 channels while providing 10 total connection paths — far richer connectivity than a plain 4-layer stack of equivalent width.
• Bottleneck + transition compression: The 4k bottleneck (Conv1×1→Conv3×3) pattern and final transition Conv1×1 provide channel compression that prevents quadratic channel explosion, keeping memory cost tractable even with dense concatenation.

Why DenseBlock in This Decoder?

The sclera boundary is defined by extremely subtle texture gradients — at 256×256, the sclera-iris boundary is often 1–2 pixels wide. Standard decoder layers (simple Conv→BN→ReLU stacks) risk discarding fine-grained boundary evidence as features propagate through the upsampling stack. DenseBlocks preserve this evidence by keeping all intermediate feature maps accessible at every layer, making it particularly well-suited to boundary-sensitive segmentation tasks. The growth rate k=32 is deliberately conservative — it allows the network to accumulate boundary-relevant features without overwhelming the semantic context features carried from the FPN.

Channel Accounting per Decoder Stage

Channel Squeeze-Excitation (CSE) — In Depth

Spatial Squeeze-Excitation (SSE) — In Depth

Concurrent Fusion — Design Rationale

CSE vs. SSE: Complementary Roles

CSE addresses which features are informative: the global average pooling aggregates spatial context into a channel descriptor, and the bottleneck FC network learns channel interdependencies. For sclera segmentation, CSE can suppress chrominance features and amplify edge-sensitive channels.

SSE addresses where in the feature map to focus: the spatial attention map identifies regions of high relevance, providing a form of spatial gating independent of channel composition.

Head Architecture

Design Rationale

Bilinear head upsample: Restores the 128×128 decoder output to the 256×256 input resolution. Bilinear interpolation is preferred over transposed convolution to avoid checkerboard artifacts at this final stage.

Conv3×3 before Conv1×1: The 3×3 convolution aggregates a local spatial context prior to the binary classification projection, providing a spatially-aware feature vector at each pixel location as opposed to a purely point-wise prediction.

Logit loss vs. probability loss: The binary cross-entropy is computed on raw logits via F.binary_cross_entropy_with_logits, which computes the sigmoid internally using the numerically stable log-sum-exp formulation, avoiding floating-point underflow at extreme probability values.

Binary Cross-Entropy Component

Soft Dice Loss Component

AdamW Optimizer

Cosine Annealing Schedule

LR Trajectory Chart

Rationale for Differential LR

LoRA parameters require a lower learning rate than the task-specific modules (FPN, decoder, head) because they modify the representations of a frozen, pre-trained backbone. A larger LoRA learning rate risks over-steering the QKV projection perturbations, degrading the DINOv2 representations.

Geometric Transforms

Photometric Transforms

Occlusion Augmentation

Implementation Notes

Geometric consistency: All geometric transforms are applied identically to the image and its corresponding binary mask using the same random parameters (enforced via shared random.random() seeds per sample), preserving pixel-level annotation validity.

Specular highlight simulation: The specular augmentation adds Gaussian-shaped intensity blobs to the image only (not the mask), specifically simulating corneal and scleral specular reflections that constitute a primary source of false positives for sclera detection.

Dataset Summary

Cache Statistics (from log)

Dataset Bar Chart

SynCROI Train/Val Split

Rationale

In the synthetic-only condition, the model trains and validates on SynCROI, exhibiting strong in-domain performance (val F1 = 0.9771) but reduced cross-domain generalisation (MASD test F1 = 0.8795). The synthetic-to-real domain gap arises from differences in illumination distribution, lens characteristics, and scleral texture between rendered and photographic images.

The mixed strategy incorporates real-world images during training, directly exposing the model to the target domain distribution and substantially improving real-world test performance.

Mixed Dataset Composition

Oversampling Design

Real images are oversampled at 3× to counteract the numerical dominance of the synthetic dataset. Without oversampling, real images would constitute only ≈30% of each epoch's training distribution, insufficient to overcome the domain prior established by the synthetic data. The 3× factor balances domain representation while retaining the synthetic data's annotation density advantage.

Mixed vs. Synthetic Comparison

The mixed model incurs a marginal 0.42pp drop on in-domain validation, which is expected due to the increased distribution complexity. The gains on real-world test sets (+7.5pp, +6.7pp) substantially outweigh this trade-off.

Convergence Characteristics

The mixed model exhibits rapid initial convergence (F1 = 0.9376 → 0.9598 in 5 epochs), consistent with the frozen backbone providing strong prior features. The learning trajectory enters a plateau phase around epoch 25–30 (F1 ≈ 0.9700), followed by a slower asymptotic approach to the final 0.9729 over the remaining 40 epochs as the cosine LR schedule anneals to η_min.

Epoch Timing (from log)

Synthetic Model Observations

The synthetic-only model shows faster per-epoch training (≈95s vs. ≈171s for mixed, due to 312 vs. 686 batches). Convergence is monotonically improving across 70 epochs with no plateau or regression, confirming the absence of overfitting to the 10,000-sample synthetic training set. The final val F1 of 0.9771 represents the upper bound on same-domain performance.

Epoch Timing (Synthetic)

Synthetic Model — Selected Epochs

Mixed Model — Selected Epochs

Cross-Domain Test Results

F1 Bar Chart — Test Datasets

Validation vs. Test Performance

Domain Gap Quantification

Analysis

The synthetic-only model demonstrates a 9.17pp F1 gap between in-domain validation (0.9771) and cross-domain MASD test (0.8795), and an 11.07pp gap to SBVPI. This is attributable to:

• Texture statistics: Synthetic sclera textures lack the vessel network complexity and conjunctival detail of real photographs
• Lighting distribution: SynCROI uses constrained illumination models vs. unconstrained natural and flash lighting in MASD/SBVPI
• Specular reflection characteristics: Corneal reflections in synthetic images follow physically-simplified models
• Lens/sensor artifacts: Real cameras introduce chromatic aberration, sensor noise, and compression artifacts absent from synthetic data

Mixed Training Efficacy

The mixed training strategy reduces the domain gap from 9.17pp to 1.83pp for MASD (an 80% reduction) and from 11.07pp to 3.93pp for SBVPI (a 64% reduction). The 3× real oversampling ensures sufficient gradient signal from real images despite their numerical minority in the mixed dataset.

Remaining Gap Discussion

The residual 1.83pp gap on MASD and 3.93pp gap on SBVPI may be attributable to:

• SBVPI-specific imaging conditions not represented in MASD/SynCROI training data
• Test set distribution shift relative to the MASD/SBVPI training subsets
• Fixed threshold (≥0.5) applied uniformly; per-dataset threshold calibration could reduce this

Inference Procedure

Submission Format

Output Resolution

Inference Performance

With AMP and torch.compile(), inference runs at approximately 4ms/image on a single CUDA device, well within the SSBC evaluation latency budget. The text embedding is cached on model initialisation, contributing zero per-image overhead.