Camera

Cameras are one of the main sensors used in autonomous vehicles (AVs). They are good at high resolution tasks like:

Classification
Scene understanding
Color perception
Traffic light or sign recognition

Cameras are relatively cheap compared to Radar and Lidar. They capture images or video of the environment which can be used to:

Detect and identify objects, such as pedestrians, other vehicles, traffic lights, and road signs
Gain real-time road image data of the driving direction
Extract the needed lane line position data and to determine whether the present vehicle has strayed from the lane
Estimate depth, which is very necessary for the self-driving car to understand its surroundings accurately
Detect, classify and measure the distance between objects on the road and the vehicle itself

The image is processed by the photosensitive component, circuit, and control component in the vehicle camera and converted into a digital signal that can be processed by the computer

Where is camera located in vehicles ?

Front camera

Front cameras are usually mounted in the front of a car, behind the rear view mirror. They are designed to face forward and help drivers avoid hitting obstructions like parking blocks and curbs

Surround view camera

Rear cameras, mounted near the license plate
Side cameras, mounted next to the mirrors
360-degree surround view cameras, typically located in the front grille, rear plate area, and side mirrors

Driver monitoring camera

Steering column, Dashboard, Instrument cluster, Interior mirror, Front bumper, Door mirrors, Liftgate.
The camera uses infrared light-emitting diodes (LEDs) to see the driver's face, even at night. The LEDs can even see the driver's eyes through sunglasses

Infrared Camera (Night vision)

Front grille

In cabin monitoring camera

Facing inside cable on rear view mirror

How is the Front view camera connected to the ECU?

The connection between a front view camera and the Engine Control Unit (ECU) in a vehicle can vary depending on the make and model of the car and the specific camera system used. However, I can provide a general overview of how this connection typically works in modern vehicles.

Camera Installation: The front view camera is usually mounted at the front of the vehicle, typically near the grille or the front bumper. It is positioned to capture a clear view of the road ahead.

Wiring: The camera is connected to the vehicle's wiring harness. There are typically several wires, including power, ground, and video signal wires.

Video Signal: The video signal wire from the camera carries the video feed captured by the camera and sends it to a display unit or interface module. This video feed shows the live view of what the camera is recording in front of the vehicle.

Display Unit or Interface Module: In many modern vehicles, there is a dedicated display unit or interface module that handles the video feed from the front view camera. This unit is often connected to the vehicle's ECU.

ECU Connection: The ECU (Engine Control Unit) may be connected to the display unit or interface module, typically via a data bus or communication network within the vehicle. This connection allows the ECU to receive information from the front view camera system, such as images and data related to the vehicle's surroundings.

Integration: The ECU can use the information from the front view camera for various purposes, including advanced driver-assistance systems (ADAS) like lane departure warning, adaptive cruise control, and collision avoidance. It may also use this data for parking assistance and other safety features.

Power and Ground: The front view camera also requires a power source and a ground connection, which are typically provided through the vehicle's electrical system. The power source ensures that the camera has a constant supply of power for operation.

Please note that the specific wiring and connections may vary between different vehicle manufacturers and models. Some vehicles may have more advanced camera systems that include multiple cameras (front, rear, side) and may require additional wiring and control modules to manage these cameras and their connections to the ECU. Always refer to the vehicle's service manual or consult with a qualified technician for detailed information on the camera system's installation and connection to the ECU in a specific vehicle.

Connection of front view camera and ECU

The connection between a rearview camera and the Engine Control Unit (ECU) in a vehicle also varies depending on the specific make and model of the vehicle and the camera system used. However, I can provide a general overview of how this connection typically works in modern vehicles with rearview cameras.

Camera Installation: The rearview camera is typically mounted at the rear of the vehicle, often near the license plate area or on the rear bumper. It is positioned to capture a clear view of the area behind the vehicle.

Wiring: The camera is connected to the vehicle's wiring harness. The wiring typically includes power, ground, video signal, and sometimes additional wires for features like parking guidelines.

Video Signal: The video signal wire from the camera carries the video feed captured by the rearview camera and sends it to a display unit or interface module located in the vehicle's interior.

Display Unit or Interface Module: In many modern vehicles, there is a dedicated display unit or interface module inside the vehicle, often located on the dashboard or in the center console. This unit is responsible for receiving and displaying the video feed from the rearview camera.

ECU Connection: The connection between the rearview camera system and the ECU can vary. In some vehicles, the ECU may not be directly connected to the camera system. Instead, the ECU may receive information from the camera system through the vehicle's data bus or communication network. This allows the ECU to access the video feed and use it for various purposes.

Integration: The ECU can use the information from the rearview camera for safety and driver-assistance features, such as backup collision warnings, parking assistance, and to improve overall situational awareness when reversing the vehicle.

Power and Ground: Like the front view camera, the rearview camera also requires a power source and a ground connection, typically provided through the vehicle's electrical system. The power source ensures that the camera has a constant supply of power for operation.

It's important to note that the specific wiring and connections can vary between different vehicle manufacturers and models. Some vehicles may have more advanced rearview camera systems that include additional features like cross-traffic alerts, and these may involve more complex wiring and connections.

As with any vehicle-specific installation or modification, it's essential to consult the vehicle's service manual or seek the assistance of a qualified technician for detailed information on how the rearview camera system is connected to the ECU in a specific vehicle.

Rear and surround view camera

What is inside the Camera

Inside an integrated circuit (IC) for the front camera in autonomous driving, several key components and functionalities enable the camera to capture and process visual information. Here are some common components found in ICs for front cameras in autonomous driving:

Image Sensor: The front camera IC incorporates an image sensor, typically a Complementary Metal-Oxide-Semiconductor (CMOS) sensor or a Charge-Coupled Device (CCD) sensor. The image sensor captures light and converts it into electrical signals, forming the basis of the visual data.

Analog-to-Digital Converter (ADC): The analog electrical signals from the image sensor need to be converted into digital format for further processing. An ADC within the IC performs this conversion, allowing the captured image data to be processed digitally.

Preprocessing and Filtering: The front camera IC may include preprocessing and filtering capabilities to enhance the image quality and reduce noise. This can involve operations like noise reduction, color correction, gamma correction, and white balance adjustment, ensuring optimal image quality for subsequent processing steps.

Feature Extraction and Analysis: Deep learning-based algorithms may be implemented within the IC to extract and analyze relevant features from the captured images. These algorithms can detect and classify objects, recognize traffic signs, estimate distances, and perform other visual perception tasks critical for autonomous driving.

Communication Interfaces: The front camera IC may include communication interfaces, such as Serial Peripheral Interface (SPI), I2C (Inter-Integrated Circuit), or MIPI (Mobile Industry Processor Interface), to facilitate data transfer between the camera module and other components of the autonomous driving system, such as the central processing unit (CPU) or a dedicated perception module.

Hardware Acceleration: To handle the computational requirements of real-time image processing, some front camera ICs may incorporate hardware acceleration units, such as specialized digital signal processors (DSPs) or dedicated hardware for convolutional neural network (CNN) computations. These accelerators enable efficient and fast processing of visual data, supporting the real-time demands of autonomous driving applications.

It's important to note that the specific components and functionalities inside a front camera IC may vary depending on the design and manufacturer. Additionally, advancements in technology and the ongoing development of autonomous driving systems may introduce new features and capabilities to enhance the performance and functionality of front-camera ICs.

IC's and CMOS For camera

Let's have a look at different ICs for the CMOS Camera

There are several common integrated circuits (ICs) used for cameras in automotive applications, specifically in ADAS (Advanced Driver Assistance Systems) and autonomous driving. These ICs are designed to process and analyze the visual data captured by the front camera. Here are some examples of commonly used ICs for front cameras:

Image Signal Processors (ISPs): ISPs are dedicated ICs designed to process the raw image data from the image sensor. They perform various functions such as noise reduction, color correction, gamma correction, white balance adjustment, and other image enhancement techniques. ISPs help improve the image quality and prepare the data for further processing and analysis.

System-on-Chip (SoC) Solutions: SoCs designed specifically for automotive applications often include integrated image processing capabilities. These SoCs may combine multiple functions, such as image signal processing, feature extraction, object recognition, and communication interfaces, into a single chip. They provide a compact and efficient solution for front camera systems.

Application-Specific Integrated Circuits (ASICs): ASICs are custom-designed ICs tailored for specific applications. In the context of front cameras, ASICs can be developed to address the specific needs of the camera system, such as real-time processing, power efficiency, and integration with other components of the ADAS or autonomous driving system.

Digital Signal Processors (DSPs): DSPs are ICs optimized for performing digital signal processing tasks. They are commonly used in front camera systems to handle computationally intensive image processing algorithms, such as feature extraction, object detection, and tracking. DSPs can efficiently execute these algorithms, providing real-time processing capabilities.

Field-Programmable Gate Arrays (FPGAs): FPGAs offer flexibility in implementing custom logic and algorithms for front camera systems. They can be programmed to perform specific image processing tasks, enabling customization and optimization for different camera system requirements.

It's important to note that specific ICs used for front cameras can vary depending on the system design, manufacturer preferences, and the desired functionality of the front camera. Different automotive companies and camera suppliers may use different ICs based on their specific requirements, performance targets, and integration capabilities.

Let's have a look at the common terms for the camera

Frame rate

The frame rate of a camera refers to the number of individual images or frames that the camera can capture in one second. It is typically measured in frames per second (fps) or Hertz (Hz).

The frame rate of a camera can vary widely depending on the type of camera and its intended use. Here are some common frame rates for different types of cameras:

Standard Video Cameras:

Consumer video cameras: 30 fps is a common frame rate for consumer-grade video cameras used for everyday video recording.
Professional video cameras: Professional-grade video cameras often support multiple frame rates, including 24 fps (common for cinematic content), 30 fps, and higher frame rates for slow-motion recording (e.g., 60 fps, 120 fps, or even higher).

Resolution

The resolution of a camera refers to the level of detail and clarity that the camera can capture in an image or video. It is typically expressed in terms of the number of pixels in the image, often represented as width x height (e.g., 1920 x 1080), which indicates the number of horizontal and vertical pixels.

Here's what the components of camera resolution mean:

Width and Height: The resolution is specified as two numbers, with the first number representing the width or the number of horizontal pixels and the second number representing the height or the number of vertical pixels. For example, in the resolution "1920 x 1080," 1920 pixels represent the width, and 1080 pixels represent the height.
Total Pixel Count: To determine the total pixel count or resolution of the camera, you can multiply the width and height values together. In the example above (1920 x 1080), the camera has a total pixel count of 2,073,600 pixels, or approximately 2.1 megapixels. This is often rounded to 2MP

Higher-resolution cameras can capture more detail and produce sharper images because they have a greater number of pixels to represent the scene. This can be particularly important for tasks like photography, video recording, and surveillance where image quality and detail are crucial.

Common camera resolutions include:

Standard Definition (SD): Common SD resolutions include 640 x 480 pixels (0.3 megapixels) and 720 x 480 pixels (DVD resolution).
High Definition (HD): HD r
esolutions include 1280 x 720 pixels (720p) and 1920 x 1080 pixels (1080p or Full HD).
4K Ultra High Definition (UHD): 4K resolutions typically have a width of 3840 pixels and a height of 2160 pixels (3840 x 2160), resulting in approximately 8.3 megapixels.
8K Ultra High Definition (UHD): 8K resolutions have a width of 7680 pixels and a height of 4320 pixels (7680 x 4320), resulting in approximately 33.2 megapixels.

The choice of camera resolution depends on the intended use. Higher resolutions are ideal for applications where fine details matter, such as professional photography, video production, and large-screen displays. Lower resolutions may be sufficient for webcams, video conferencing, and some surveillance applications, where image quality can be balanced with other factors like bandwidth and storage requirements.

Field of view

The field of view (FOV) in camera-based Advanced Driver Assistance Systems (ADAS) refers to the extent of the visual area that the camera can capture or monitor. In the context of ADAS, these cameras are typically used for various purposes, including lane-keeping assistance, adaptive cruise control, pedestrian detection, collision avoidance, and more.

The FOV of an ADAS camera is essential for its functionality because it determines what the camera can "see" and analyze in its surroundings. A wider FOV allows the camera to capture more of the environment, which can be beneficial for detecting potential hazards, pedestrians, vehicles, and road markings. Conversely, a narrower FOV may be used for specific tasks that require a more focused view, such as parking assistance or reading traffic signs.

The FOV can vary from one ADAS camera to another, depending on its intended purpose and placement on the vehicle. Wide-angle lenses are often used to provide a broader FOV, while narrower lenses offer a more focused view. Manufacturers design ADAS systems to use multiple cameras with varying FOVs to cover a wider range of scenarios and improve overall safety.

It's worth noting that FOV is just one of the factors to consider when designing an ADAS system. Other factors include camera resolution, image processing capabilities, and the integration of data from multiple sensors (e.g., radar and LiDAR) to provide a comprehensive view of the vehicle's surroundings and enable advanced driver assistance features and autonomous driving capabilities.

How did the image from the front camera help ?

The image from the front camera in a vehicle's Advanced Driver Assistance System (ADAS) plays a crucial role in enhancing safety and aiding the driver in various ways. Here are some of the ways in which the image from the front camera helps:

Lane-Keeping Assistance: The front camera can monitor the lane markings on the road. It helps the vehicle stay within its lane by providing input to a lane-keeping assist system. If the vehicle starts to drift out of its lane without the turn signal activated, the system can provide visual and audible warnings or even gently steer the vehicle back into the lane.
Adaptive Cruise Control: The front camera can detect the distance and relative speed of the vehicle in front of you. This information is used by adaptive cruise control systems to adjust your vehicle's speed to maintain a safe following distance automatically. If the vehicle in front slows down or stops, the camera helps your vehicle respond accordingly.
Forward Collision Warning and Automatic Emergency Braking: The front camera can identify potential obstacles or vehicles in your path. It provides data for forward collision warning systems that can alert the driver if a collision is imminent. In some cases, it can even trigger automatic emergency braking to prevent or mitigate a collision.
Pedestrian Detection: The front camera can recognize pedestrians in the vehicle's path. This feature is especially important in urban environments and helps warn the driver or activate emergency braking if a pedestrian is at risk of being hit.
Traffic Sign Recognition: The front camera can read and interpret traffic signs, such as speed limits, stop signs, and yield signs. This information can be displayed on the vehicle's dashboard or head-up display, helping the driver stay aware of relevant road signs.
Parking Assistance: When parking or maneuvering in tight spaces, the front camera can provide a visual representation of the vehicle's surroundings. It helps the driver avoid obstacles and park more accurately.
Headlight Control: Some vehicles use the front camera to control the high-beam headlights automatically. The camera can detect oncoming vehicles or vehicles in front and adjust the headlights to avoid blinding other drivers while maintaining optimal visibility.
Traffic Jam Assistance: In some advanced systems, the front camera can assist with steering, acceleration, and braking in traffic jams or stop-and-go traffic. It helps maintain a safe following distance and reduces driver fatigue.

Overall, the front camera in an ADAS is a critical component that provides real-time information about the road and traffic conditions. It enables various safety features and driver assistance functions to enhance road safety and driving comfort.

In general, the Mpixel (MP) of the imaging sensor decides the furthest distance to which a given object can be detected as well as identified. Figure 3 shows the typical relationship of distance to pedestrian vs. Mpixel for multiple configurations.

In general, the frame rate of the imaging sensor decides the maximum stopping distance to avoid object collision. Figure 4 shows the typical relationship of the Stopping Distance of Car Vs. The frame rate for multiple configurations of the car’s speed.

The stopping distance is an addition of the distance covered during the detection of the pedestrian before applying the break and the distance covered during braking. The assumptions of plots are seven frames of latency during pedestrian detection, which consists of approximately three frames of processing latency and about four frames of tracking latency to improve the quality of detection. The other assumption is breaking distance as per British Highway Code. As per Figure 4, assuming car speed of 80 Km/Hr, the stopping distance reduces from 55m to 45m as frame-rate goes from 10fps to 15fps (for systems as of today) to 30 fps (for system in future).

Table II shows the impact of increasing car speed (30Km/Hr to 250Km/Hr) and the impact on stopping distance (8m to 390m) along with various parameters and assumptions.

For a given speed of the car (e.g. 80 Km/Hr), there are multiple ways to reduce stopping distance as follows.

Usage of Higher frame rate: This results in lower latency (750ms to 284ms) as the frame rate increases (10 to 30 fps). This also improves the performance of object (e.g. pedestrian) tracking but it comes at the expense of higher processing power (Row No. 4 & 5).

Usage of streaming and slice-based architecture: This reduces latency (750ms to 210ms) as communication across multiple sub-components happens at smaller granularity e.g. lines/subpicture (Row No 6). This kind of sophisticated system requires more time to develop and possible algorithmic frame-level dependency challenges.

The next level of details in terms of accuracy of detection, computation processing needs, power and cost constraints, robustness, and reliability implications makes the selection of actual parameters an “art” rather than “science”.

Let's discuss an ECUS in the ADAS/AV

BMW’s AV platform architecture clarifies the company’s AV design priorities. They are focused on scalability and reusability of software and hardware.

Across all BMW passenger vehicles, from current Level 2 to Level 4/5 cars, both hardware and software are being reused as much as possible across ECUs and cameras. BMW’s base platform is built on the AUTOSAR (Automotive Open System Architecture), using classic microcontrollers (BMW is using Infineon’s Aurix).

As it increases levels of automated driving and their features, BMW addresses those needs by deploying additional sensor systems and high-end microprocessors. The platform’s baseline uses Infineon’s Aurix and Renesas’ R-CAR SoCs to optimize its application in stereo front cameras.

For Level 3 models, BMW is adding two Mobileye EyeQ5, two Intel Denverton CPUs and another Aurix. For Level 4/5 vehicles, BMW expands the configuration to three EyeQ5, one Xeon 24C and Aurix.

ADCAM (low, Mid) ADCAM" in the context of BMW typically refers to "Active Driving Assistant Camera." This is a part of BMW's driver assistance and safety systems. The Active Driving Assistant Camera, often comprised of multiple cameras strategically placed around the vehicle, helps enable various advanced driver-assistance features, such as:

Lane Departure Warning: The camera can monitor lane markings on the road and provide a warning if the vehicle begins to drift out of its lane without using a turn signal.
Lane Keeping Assist: This system can assist the driver by gently steering the vehicle to keep it centered within the lane.
Adaptive Cruise Control: The camera can help maintain a safe following distance from the vehicle in front by adjusting the vehicle's speed as necessary.
Traffic Sign Recognition: It can identify and display traffic signs, including speed limit signs, on the vehicle's instrument panel.
Pedestrian and Object Detection: The camera can detect pedestrians and other objects in or near the vehicle's path and trigger warnings or automatic braking if necessary.
Collision Avoidance and Emergency Braking: The system can provide alerts and, in some cases, automatically apply the brakes to prevent or mitigate collisions.
High-Beam Assist: The camera can automatically switch between high and low beams to improve nighttime visibility without blinding other drivers.

Please note that the specific features and capabilities of the ADCAM system can vary depending on the BMW model and its equipped options. BMW continuously updates and enhances its driver assistance systems, so there may be newer features and technologies available in more recent models.

UCAP

An Electronic Control Unit (ECU) for a surround view camera system, often referred to as the Surround View Monitor (SVM) ECU or UCAP, is a critical component in modern vehicles equipped with advanced driver assistance systems (ADAS). The surround view camera system is designed to provide drivers with a 360-degree bird's-eye view of their vehicle's surroundings, making parking and maneuvering in tight spaces easier and safer. Here's how the ECU for the surround view camera system works:

Camera Inputs: The surround view camera system typically consists of multiple cameras placed around the vehicle, including front, rear, and side cameras. These cameras capture video footage of the vehicle's surroundings.
Image Processing: The ECU processes the video feeds from these cameras in real-time. It performs various image processing tasks, such as stitching together the individual camera feeds to create a composite bird's-eye view image of the vehicle's surroundings.
Display: The processed image is then sent to the vehicle's infotainment screen or dedicated display, where the driver can see the composite view. The image often includes dynamic overlays, such as guidelines to assist with parking.
User Interaction: In many systems, the driver can interact with the surround view display, such as changing the view mode (e.g., switching between front, rear, or side views) or turning the system on and off.
Object Detection and Alerts: Some SVM ECUs are equipped with object detection algorithms. They can identify obstacles, pedestrians, or other vehicles in the camera's field of view. If an object is detected, the ECU may provide visual and audible alerts to warn the driver.
Integration with Other ADAS: The SVM ECU can also integrate with other ADAS features, such as parking sensors and automatic parking assistance. It can use data from these sensors to enhance the surround view display and assist with parking maneuvers.
Vehicle Data: The ECU may also receive data from other vehicle sensors and systems, such as the steering angle, speed, and gear position. This information can help the ECU tailor the surround view display to provide the most relevant information based on the vehicle's current state.
Safety and Security: The SVM ECU is designed with safety and security in mind. It must reliably process video feeds and communicate with other vehicle systems while protecting against unauthorized access.

In summary, the ECU for a surround view camera system is a critical component that enables the creation of a comprehensive and intuitive view of a vehicle's surroundings. It enhances safety and convenience for drivers, especially during parking and low-speed maneuvers, by providing a clear visual representation of potential obstacles and blind spots.

mPAD

In automotive applications, a domain controller is a computer that controls a set of vehicle functions related to a specific area, or domain. Functional domains that require a domain controller are typically compute-intensive and connect to a large number of input/output (I/O) devices. Centralization of functions into domain controllers is the first step in vehicles’ evolution toward advanced electrical/electronic architectures, such as Aptiv’s Smart Vehicle Architecture™.

An active safety domain controller receives inputs from sensors around the vehicle, such as radars and cameras, and uses that input to create a model of the surrounding environment. Software applications in the domain controller then make “policy and planning” decisions about what actions the vehicle should take, based on what the model shows. For example, the software might interpret images sent by the sensors as a pedestrian about to step onto the road ahead and, based on predetermined policies, cause the vehicle to either alert the driver or apply the brakes.

In the area of user experience, a domain controller typically controls multiple elements of the in-cabin experience — for example, providing the software and computing power needed to run the infotainment system, driver cluster and other vehicle interfaces for the user. These interfaces are increasingly accomplished through dynamic, reconfigurable displays, such as a touchscreen that can provide navigation, audio controls and climate functions.

A step toward the future

Domain controllers represent an important milestone toward more software-defined vehicles and centralization.

Functions that have previously been handled through individual electronic control units ( ECUs) can be consolidated, or up-integrated, into domain controllers. For example, a radar might previously have had its processing done in a self-contained ECU; however, that processing could move to an active safety domain controller through a Satellite Architecture approach.

Domain controllers are further complemented by zone controllers. Zone controllers are nodes in the vehicle that serve as hubs for power distribution and data connection. They handle the I/O with sensors, actuators and peripherals, which abstracts the I/O from the compute and frees a domain controller to focus on software that performs higher-level functions.

Eventually, domain controllers will consolidate further into “serverized” controllers. With I/O abstracted from compute and a high-speed network in place, it makes sense to consolidate the software in the domain controllers onto fewer computers capable of dynamically sharing workloads among the different domains. This centralization will further reduce cost and space, unlock new functionality such as driver-out-of-the-loop automation, and make it easier to perform over-the-air updates

The use of their L2 system (mPAD) (EyeQ5 mid + Denverton) as the backup system for L3 and higher.The sensors and algorithms used in autonomous driving above L3 level have very high requirements for real-time calculation and synchronous communication. Without a central computing unit (domain controller) with excellent performance, only relying on traditional discrete ECU and more centralized MCU will not effectively reduce the complexity and cost of E / E architecture. The emergence of centralized domain controller computing units can integrate different processors, accelerate various redundant algorithms,meet the needs of multi-sensor data fusion, and effectively reduce the system complexity and cost. L3 to L4+, a new generation of cloud computing simulation platform based on AI intelligent and DNN, supports large scale parallel verification and training, and provides a fast lane for go-to-market autonomous vehicle.

Camera

SUBSCRIBE