GDIP: Object-Detection in Adverse Weather Conditions Using Gated Differentiable Image Processing

Sanket Kalwar1*
Dhruv Patel1*
Aakash Aanegola1
Krishna Reddy Konda3
Sourav Garg2
K Madhava Krishna1

Accepted at ICRA 2023
* denotes equal contribution
1 Robotics Research Centre, IIIT Hyderabad, India.
2 Australian Centre for Robotic Vision at the Queensland University of Technology (QUT), Brisbane, Australia.
3 ZF Friedrichshafen TCI, Hyderabad, India
[Paper]
[GitHub]

Motivation

    Object detection is challenging under adverse weather conditions (such as fog or low-lighting) because the objects are partially or not visible, resulting in missed detections.

    We tackle this problem by proposing a Gated Differentiable Image Processing (GDIP) block, which enhances the image by performing weighted Image Processing (IP) operations concurrently on the adverse input image. This enhanced output is sent to the downstream object detection network, leading to superior detection performance to other state-of-the-art methods.

    real-world foggy (RTTS dataset) and dark (ExDark dataset) images with annotations

Overview

  • We present Gated Differentiable Image Processing (GDIP) block, a domain-agnostic architecture which adaptively enhances adverse images for object detection by performing concurrent gated-weighing of image processing operations.
  • GDIP provides flexible concurrent gated weighting of the individual IP operations that result in superior detection performance.
  • GDIP block can be integrated as single or multi-level enhancement module with an encoder
  • In addition, GDIP block has a unique advantage with its utility as a training regularizer, which directly improves object detection training for adverse conditions. This eliminates any image enhancement overhead during inference, unlike other state-of-the-art works, thus resulting in higher throughput.

Modules

Using our GDIP block, we propose novel variants as below:

  • GDIP-Yolo: a single GDIP block (fed with an adverse input and latent embeddings from an encoder) can be plugged into existing object detection networks (e.g., Yolo) and trained end-to-end with adverse condition images (fog and low-lighting).
    GDIP-Yolo Forward Pass

  • Multi-level GDIP-Yolo (MGDIP-Yolo): a multi-level version of GDIP where an image is progressively enhanced through multiple GDIP blocks, each guided by a different layer of the image encoder. Providing access to such multiple feature scales helps it utlilize the local/global properties to selectively apply Image Processing operations.
    MGDIP-Yolo Forward Pass

  • GDIP as regularizer: an adaptation of GDIP as training regularizer, which directly improves object detection training by learning weather-invariant features. It can be removed during inference, thus saving compute time with improved performance in adverse conditions.

    GDIP as regularizer

Quantitative Analysis

    Foggy setting

    Quantitative results for foggy conditions on the V_N_Ts (VOCNormal Test set), V_F_Ts (VOCFoggy Test set) and realworld RTTS dataset. Best and second best mAP scores are bold and italicized, respectively

    Low-lighting setting

    Quantitative results for low-lighting conditions on the V_N_Ts (VOCNormal Test set), V_D_Ts (VOCDark Test set) and real-world ExDark dataset. Best and second best mAP scores are bold and italicized, respectively.

    NOTE: IA-Yolo uses a prior (i.e. IA-Yolo (with prior)) for low-lighting setting by removing the defogging and white balance filters. Hence, we evaluate IA-Yolo (w/o prior) by incorporating the defogging and white balance filters (their same pipeline as in case of foggy setting) to do fair comparison with our approach.

GDIP in the Wild

    GDIP runs seamlessly on video sequences in the wild without any need for retraining or fine tuning, showcasing its ability to detect objects by composing Image Processing operations through gating by training on only synthetic adverse data.


    Note: how as the car moves from complete darkness to natural lighting, the gate firing changes from only gamma to gamma, defog and tone.


    Our GDIP-Yolo does robust detection with defog gate firing, which is consistent with the foggy weather setting.

Contact

If you have any questions, please reach out to any of the above mentioned authors.