Application of Image Recognition and Identificationin ADAS2013.06.19PLK TechnologiesCompany BriefPLK TechnologiesHistory•PLK started as a venture TFT in Hyundai Motor Company (2000)•PLK Technologies span off in July 2003•Developed and successfully launched ADAS vision products–First LDWS for commercial vehicles in Korea (HMC Trago)–First LDWS for passenger vehicles in Korea (HMC Equus)–First LDWS+HBA in Korea (KMC K9)Status•42 Employees (20 Developers/Engineers)•Products for 12 vehicles (passenger car, bus and trucks)•Verified in many regions–Korea, China, Japan, Europe, US, Middle East, Australia •TS16949, ISO9001, HKMC SQOEM ADAS Vision Products of PLKFactoryProduction(Test) FacilityDevelopment HistoryEarly Development HistoryAvailable OEM ProductsEarly Development History•Developed prototypes and product in the early years of PLKAvailable OEM Products•Recent Products available for OEM applicationFCWHBALDWMIPSFPSResolutionSensor CPU FCWHBALDW1st Passenger Car Model(EQUUS)MIPS800FPS15Resolution640 * 480Sensor MT9V125CPU BF5392009FCWHBA LDWDual FunctionMIPSFPS ResolutionSensor CPU 2012Target Schedule•Development : Dec. 2012•Production : Oct. 2013LDWS onlyLDWS HBALDWS HBA FCWFunctionsStatus & RoadmapLane RecognitionLight RecognitionVehicle RecognitionTraffic Sign RecognitionTraffic Light Recognition Pedestrian RecognitionADAS Vision FunctionsTraffic SignFront VehiclePedestrianLinesRegulation/Market RequirementsADAS vision functionalitiesImplemented in single ADAS cameraPLK ADAS Vision platformSpeed Assist SystemAEB CityAEB InterurbanAEB PedestrianLDWS/LKASHBA/DHBLightsEuro NCAPRecommendation•Regulation and Market needs drives ADAS vision function requirements•(LED) Traffic Signs, Road Lines and Lights are only detectable by image recognition •Objects (Vehicles and Pedestrians) are detectable by image recognition and other sensors–the sensors have pros and cons–sensor fusion, if well implemented, can produce more reliable sensor system–image recognition is the better way to ‘identify’ an object, detected by any collaborative sensorFunction RoadmapLDW Production HBA Production LKA Development (for curvature modeling)FCWDevelopment (for OEM)ALC / DHB Ready for implementationFCWA/M Production TSRCamera Tuning TLRCamera Tuning PDDevelopmentTSRDevelopmentPDDevelopment for Heavy Load VehicleTLRDevelopmentPDDevelopment for AEB-PedestrianLane RecognitionLDW (Lane Departure Warning)•A vision-based ASV system, which gives drivers warning in any media including sound, vibration, haptic, etc. , when the vehicle departs from its driving lane•Importance of LDW–Vehicles should run between road lines ÆLine detection is thebasic feature of all ADAS technology–Image Recognition is the only way to detect the road lines ahead •Application History & Schedule–(2009) Equus–1st launch for passenger car–(2010) Genesis, K7–(2011) Grandeur HG–(2012) Ce’ed, Santa Fe, Sorento, RP–(2013) QZ Truck–Will be applied to Class A and Class B vehicles•Validated Regions–Korea, China, US, Europe, Middle EastCore Technology•LEF (Line Enhancement Filter), a unique image filter designed by PLK–robust in noisy environment –sensitive to vague lines in shadow –color enhancement •Logic Flow–image filtering –segment grouping–linear regression (least square method)–comparison of competing candidate lines –decision of dominant line pairOriginal ImageFiltered Image Segment GroupingCompeting CandidatesDominant Line PairCompetitiveness•Color enhancement makes difference in recognition percentage–Yellow lines on concrete road (and faded asphalt road)–Blue lines on dark asphalt road•Post-processing color analysis can distinguish the line color–based on absolute value comparison –also based on relative value comparison •Benchmark test result–HMC test team to benchmark before launching EQUUS in US –tested during Nov. 2009 in California and Nevada –HMC EQUUS vs. BMW 750 LIColor ImageBlack and white ImagePLK’s Filter ImageConcreteAsphaltRoadClassificationLineClassificationHMC Genesis VS.BMW 750 LI AsphaltBott’s Dots92%>90%Dotted line 99%>76%Solid line 99%=99%ConcreteBott’s Dots92%>>>29%Dotted Line 99%>>87%Solid Line98%>>80%Lane Parameter Prediction for LKA (Lane Keeping Assist)Deployment •Predicts the road parameters ahead–lane width–lane offset–head angle–curvature (up to 1/250m)•Send information to MDPS controller to control steering in unintendedlane departure•Needs dynamic model for vehicle driving based on Kalman filter–state model, to describe geographical change in time–measurement model, to related the road parameters with imagerecognition•Development status–Implementation finished–under test and can be deployed in 3 months•Future Plan–will be developed for SCC sensor fusionHBA (High Beam Assist)•Detect and Discriminate the light of the following, to disable high beam–On-coming vehicles (up to 600m)to deactivate high beam–Front vehicles (up to 400m)to deactivate high beam–Street light (at least one)to deactivate high beamHigh Beam Assistant –Traffic light to keep high beam–Road reflectors to keep high beam•Application History and Schedule–(2012) K9 --1st launch for passenger car–(2013) Genesis F/L, K7 F/L•Proven Area–Korea–China, Europe (2013)•False operation rate in the road condition of Korea(in an hour)–Less than 1 time in highway (similar to Mobileye)–Less than 3 times in local road (Mobileye: ~5 times)–Less than 5 times in mountainous road (Mobileye: ~40 times)(heavily-curved road with lots of reflectors)Core Technology•Multiple level exposure control, to make the detection easier–Bright frame, to detect in long distance –Dark frame, to detect in short distance •Color space processing–Absolute magnitude to detect light sources –Relative magnitude to classify light sources •Also considering the followings of light source–shape –consistency•Robust in harsh conditions of mountainous area (Kangwon-do)DHB (Dynamic High Beam)•Detect and Discriminate the light to control head lamp dynamically •Need to combine with active head light unit •Transfer the light control information over CAN •Ready for applicationBright red reflectorsReflecting Signs causing false operation (Capable of elimination by PLK Technology)Lots of bright reflectorsConsecutive reflectorsFCW (Forward Collision Warning)•Detect the vehicles ahead, to alert the driver –up to 5 vehicles in the 3 lanes•one in the driving lane•two in the left lane•two un the right lane–up to 100m ahead•Application schedule–(2012) Development Verification–(2013~2014) Production –model not yet determined•Robust in adverse conditions, such as shadow, reverse light, etc.–Less than 1 false operation in an hour•Due to the combination of HBA,almost the same performance between day and night1. Recognizing2. Road Surface3. CalculationForward Collision WarningDeploymentCore Technology•Combination of shadow-oriented method and feature-oriented method–Threshold for shadow detection–Edge and template matching for feature detection–combine to track consistent reference point•Kalman filtering–to track each vehicle consistently–estimates the distance and relative speedInverse Filtering Threshold Edge Filter andTemplate matchingVehicle ConfirmationTSR (Traffic Sign Recognition)•Find the roadside markings and recognize the signs–mainly speed limit signs including•Plate signs (light reflecting)•LED signs (light emitting)•Schedule: Demonstration possible in 2013•Target performanceTraffic Sign RecognitionLED signsPlate signsSigns Day Night Plate signs Up to 160 kph Up to 130 kph (with degradation)LED signsUp to 160 kphUp to 130 kphTLR (Traffic Light Recognition)•Find the traffic light and recognize the direction–green light–red light–direction light•Daytime TSR in preliminary development stage•Application Schedule–no concrete schedule yetTraffic Light RecognitionStop Left Turn TemporaryStop Go Ahead Go ahead andLeft turnPedestrian DetectionPedestrian Detection•A project is underway with Hyundai Heavy Industry (SOP: 2014.04)–A VM (around view monitor) system –with PD functionality on all 4 cameras •Functional requirement–should overcome (1) translational movement, (2) rotational movement and (3) vibration –operating day and night •Technical approach–moving object detection with optical flow –classification based on human body shape–light illumination for better performance during nightUnder Development4 cameras4-way image &Pedestrian detectionAdvanced around viewProduct ScalabilityAF ED CBT a r g e t V e h i c l e C l a s s34567+FunctionsLDWS HBA/DHB FCW AHBBlackfin DSPAptina VGA sensorOperating temperature -40 ~ 105 (ambient)LDWS/ LKAS HBA/DHB FCW AHB TSR TLR PDAdditional functionsPlatform APlatform BPlatform C2x Devonshire DSP Aptina HD sensor Operating temperature -40 ~ 105 (ambient)LDWS/ LKAS HBA/DHB FCW AHB TSR TLRDevonshire DSP Aptina HD sensorOperating temperature -40 ~ 105 (ambient)AFE D CBClass2013201420152016LDWS HBA/DHB FCW AHBBlackfin DSPAptina VGA sensor Operating temperature -40 ~ 105 (ambient)Platform A•Most cost-optimized ADAS vision system•Applicable to low class vehiclesAF E D CBClass2013201420152016Platform BLDWS/ LKAS HBA/DHB FCW AHB TSR TLRDevonshire DSP Aptina HD sensor Operating temperature -40 ~ 105 (ambient)•Best product for performance and cost •Aiming to fulfill most Euro NCAP requirementsAF E D CBClass2013201420152016LDWS/ LKAS HBA/DHB FCW AHB TSR TLR PDAdditional functionsPlatform C2x Devonshire DSP Aptina HD sensor Operating temperature -40 ~ 105 (ambient)•High performance product•Aiming to fulfill ALL Euro NCAP requirementsThank youAll members of PLK R&D Center appreciate the visit and interest in our technology。