Appendix V: Development of Data Acquisition
In evaluating the data acquisition needs, an evaluation of existing DAS‟s was conducted. It was determined through the study that no commercial or government developed system (including DASCAR) was available that would meet all the performance requirements. PATH therefore designed a system that is composed of two distinct systems - one system records engineering data and the other records video data. The engineering data is recorded with a PC based computer. The computer used is an Industrial Computer Systems 9300? series bench top computer using ISA/PCI architecture. This computer records the output from a variety of sensors. The sensors selected by PATH to capture the environment around the bus include commercially available mono-pulse millimeter-wave radars and scanning infrared lasers. Both the radar and scanning laser measure distance and azimuth angle for multiple targets. The radar units are mounted on the front bumper, one on each end, pointing forward. Ultrasonic sensors were originally used as corner sensors, however they did not work well for two reasons. Firstly, the ground was being picked up as a target as the sensitivity was adjusted to a high level. Secondly, as ultrasound transceiver surface was not water proof it was decided that they were not appropriate as corner sensors. It was then decided that Denso LIDAR sensors would be better for this role, so several of these were acquired from Denso. Three lidar units are mounted on the bumper. The units mounted at each end of the bumper are pointing out 20 degrees and the one mounted near the center is pointing straight ahead. Other sensors record the driver inputs to the bus, such as steering wheel angle, brake line pressure, throttle position, and turn signal activation. Other sensors include an accelerometer and a GPS system. The radars, lidars, and GPS data are recorded using RS232 communication protocol. The remaining sensors are recorded using an analog to digital board and anti-aliasing filters.
- 1 -
Fig. 1 Sensors installed on a bus
Video data is recorded using a commercially available digital video system. The first digital video recording system implemented saved the video as a series of still images in an encrypted proprietary format. This limited the level of compression and allowed only three days of data to be collected before the removable hard disks had to be changed. This also required that the video data first be converted to a standard still-picture format, and then be converted to a standard moving-picture format (MPEG-1). This was a very time consuming manual process. The video recorder was not reliable such that it crashed the flash-ROM system several times. A Loronix?
video system was found that offered several improvements over the previous system. This system records video in a standard still format (AVI) and allows for automated conversion to MPEG-1 format. Much less time is required to convert the video data now that the process is automated. The system also has greater storage capacity than the previous one, allowing one week of data collection before the removable hard disks need to be changed. This system was retrofitted on the first bus and has proven to be much more reliable and easier to use. The video cameras in the originally developed system were too obtrusive, and easily damaged or moved by passengers. A different style of video camera was selected to replace them. These cameras have a form factor that allowed them to be installed in the destination window of the bus. This makes them less obtrusive and prevents them from being tampered with. This system records up to six cameras in AVI format onto a PC hard drive. Four miniature “board cameras” capture video images around
the bus. The cameras capture the front road scene, the left and right front corner road scene, and the passenger compartment of the bus. The video streams from the four cameras are combined into one video stream by a quad image combiner to extend the hard drive storage capacity.
Synchronization between engineering and video data is very important for later playback. The first item of information for synchronization is the time stamp recorded in the video frame as a title. This time stamp is generated by a title generator which receives the clock time from the engineering computer. This title allows for manual synchronization. The engineering computer
- 2 -
also sends three synchronization signals to the video recorder through the alarm inputs. These
signals and their triggering time stamps are recorded separately by both the engineering computer
and the video recorder. The signals are triggered every one minute, 15 minutes and 60 minutes
respectively. By matching the signal records in the engineering data with the records of alarms in
the video recorder, time difference between the two computers can be determined. Once the
computer time difference is matched, the video clips can be synchronized with the engineering
data streams. The synchronization occurs as part of the process of transferring the data from the
removable hard disks to a permanent data base storage system. The permanent data base storage
system is composed of a Redundant Array of Inexpensive Disks (RAID). Once the data base has
been synchronized and broken into small data clips each set of data clips is saved in one folder for
Lidar Computer Enclosure
Fig. 2 System layout on the bus
The data acquisition system has been installed on three buses in the SamTrans fleet. A fourth
system has been prepared for installation on a yet to be determined bus from another agency in the
Bay Area. The first system started collecting data in August 2000. The second system started
collecting data in April 2001. After the second system started running, the first system was updated
with the new design. The third bus started collecting data in January 2002.
Calibration of DAS
The location and direction of some sensors will influence the system performance. Before running
the bus out to collect data, the sensors and the entire system must be calibrated. The calibration
process involves the following three tasks: 1) measure the location and direction of the sensors, 2)
correct the location and direction of some sensors, and 3) examine the system alignment.
This section describes the calibration process of the first DAS on the first bus and gives the results. stndThe 1 section gives the measurements of location and sensor direction. The 2 section describes rdthe laser radar calibration procedure and results. The 3 section describes the calibration thapproaches for cameras. Calibration of system alignment is given in the 4 section. Calibration of thother sensors is given in the 5 section. The DAS design was changed after the first DAS was
calibrated. However, the calibration process and the techniques presented in this document were
conducted to calibrate all the systems. For convenience, the following abbreviations are used.
- 3 -
Table 1 DAS calibration abbreviations
Sensor Name Abbreviation
passenger side corner camera P-CAM
front-looking camera F-CAM
driver side corner camera D-CAM
passenger side upper ultra-sensor UP-SONAR
passenger side lower ultra-sensor LP-SONAR
passenger side radar P-RADAR
laser radar LIDAR
front-looking ultra-sensor F-SONAR
driver side radar D-RADAR
driver side upper ultra-sensor UD-SONAR
driver side lower ultra-sensor LD-SONAR
Interior-looking camera I-CAM
rear-looking camera R-CAM
rear radar R-RADAR
global positioning system GPS
To locate the sensors, two reference frames were built on the bus. One is the Front Coordinate
System (FCS) and the other is the Rear Coordinate System (RCS). Locations of front sensors,
including P-CAM, F-CAM, D-CAM, UP-SONAR, LP-SONAR, P-RADAR, LIDAR, F-SONAR,
D-RADAR, UD-SONAR, LD-SONAR and I-CAM, are measured in the FCS. Locations of rear
sensors, including R-CAM, R-RADAR and GPS are measured in the RCS. The reference points of
the coordinates and the positions of the sensors are illustrated in the following figures. The positive
x-axis is horizontally to the left, the positive y-axis is vertically upward, and the positive z-axis is
horizontally to forward. The basic dimensions of the bus are: length = 12200 mm, width = 2750
- 4 -
The reference point of the FCS and the locations of the front sensors are illustrated in Fig. 3.
Fig. 3 FCS and front sensors
The reference point is on the front center of the bus. The height of the reference point from the
ground is 585mm. The coordinates of the front sensors are listed in the following table.
Table 2 Front sensor locations
1. Sensors 2. x 3. y 4. z 5. Angle (Deg)
(mm) (mm) (mm) 16. LIDAR 7. -836 8. -195 9. 78 10. N.A.
11. P-RADAR 12. -1050 13. -132 14. 70 15. N.A. 216. UP-SONAR 17. -1201 18. -97 19. 64 20. -36 221. LP-SONAR 22. -1201 23. -176 24. 64 25. -26
26. D-RADAR 27. 985 28. -135 29. 67 30. N.A. 231. UD-SONAR 32. 1190 33. -95 34. 64 35. 35 236. LD-SONAR 37. 1190 38. -175 39. 64 40. 26
41. F-SONAR 42. 790 43. -161 44. 61 45. N.A. 346. D-CAM 47. 396 48. 991 49. -80 50. 14 351. F-CAM 52. -69 53. 1653 54. -61 55. 13 356. P-CAM 57. -109 58. 1563 59. -95 60. 25
61. I-CAM 62. -409 63. 2186 64. -365 65. N.A.
66. 1: N.A. = Not available;
67. 2: These are azimuth angles;
68. 3: These are tilting angles.
- 5 -
The reference point of the RCS and the locations of the rear sensors are illustrated in Fig. 4.
Fig. 4 RCS and rear sensors
The reference point is on the rear center of the bus. The height of the reference point to the ground
is 790mm. The coordinates of the rear sensors are listed in the following table.
Table 3 Rear sensor locations
69. Sensors 70. x 71. y 72. z 73. Angle (Deg)
(mm) (mm) (mm) 174. R-RADAR 75. 950 76. -154 77. -39 78. N.A.
79. GPS 80. 590 81. 2220 82. 800 83. N.A. 284. R-CAM 85. 500 86. 1500 87. 140 88. 16
89. 1: N.A. = Not available;
90. 2: Tilting angle.
Optical axis orientation
The LIDAR beam is scanning in 2D by rotating a hexagon mirror. The equivalent detection scope
is 16 degrees in horizontal and 4.4 degrees in the vertical direction. The equivalent optical axis is
defined to originate from the LIDAR lens extending to the center of the detection scope, i.e. eight
- 6 -
degrees to both the left and the right margins and 2.2 degrees to both the top and the bottom
margins. There are two adjustable screws on the front face of the LIDAR, which can be rotated to
adjust the optical axis in 2D (both horizontal and vertical directions). As the LIDAR has been stmounted on the passenger side on the 1 bus, to calibrate the LIDAR, we must first adjust the
optical axis to an appropriate direction .
The LIDAR optical axis is set horizontally to the point on the bus‟s longitudinal center line, 50
meters away from the bus front reference point, and vertically 2.2 degrees up with respect to the
horizontal plane. The geometric relationship is illustrated in Fig. 5.
2.2? longitudinal center
Reflector LIDAR R=50m 4.4? Detection Scope Fig. 5 LIDAR calibration geometry
LIDAR calibration procedure
91. LIDAR calibration was done by the following procedure.
1. Measure LIDAR lens vertical position (height to ground) H =_0.425__(m).
2. Measure R=_50m_ from bus front reference point along the longitudinal direction.
3. Set the reflector at R=50m with vertical position = H.
4. Adjust both the lower and the higher screws simultaneously, make reported “lateral position” = __0__. Change lateral position to check the adjustment.
Table 4 LIDAR lateral position test
thActual lateral position Expected report number LIDAR report (5 col)
6m Left -60 *.1m _____-61___
3m Left -30 *.1m _____-30___
3m Right 30 *.1m _____30___
6m Right 60 *.1m _____61___
5. Adjust the lower screw, make reported “Vertical Position” changing from smaller to larger numbers thru
6. Adjust the lower screw to “ – direction” __0.3-0.5__ rev, make sure that the LIDAR keeps detecting the
7. Change distance to check the adjustment:
Table 5 LIDAR range test
stndActual distance Expected report number LIDAR report (1-2 col)
40m 31*1.28m 32*.01m _31__*1.28m _98__*.01m
30m 23*1.28m 56*.01m _24__*1.28m _14__*.01m
20m 15*1.28m 80*.01m _16__*1.28m _48__*.01m
10m 7*1.28m 104*.01m _8___*1.28m _46__*.01m
8. Put the reflector at R=10m, with vertical position changing, check the adjustment:
- 7 -
Table 6 LIDAR vertical position test
thActual vertical position Expected report number LIDAR report (9 col)
H+0.76m 2 __2_____
H+0.57m 3-4 __4_____
H+0.38m 6-7 __5_____
H+0.19m 9-10 __6_____
H+0m 12 __8_____
Three different options of focal length are available: 3mm, 4mm, and 7.5mm. Lenses with
different focal length were fitted on the camera heads. Comparing the field of view and selecting
the one list that best matches the area of interest around the bus, the optimal fitted focal length was
chosen for each camera, as in the following table.
Table 7 Focal length of cameras
Camera Focal length
Image plane rotation and optical axis direction of each camera was roughly adjusted by monitoring
the video output. The factors of interest while adjusting are: range coverage, azimuthal direction of
interest, and consistency between adjacent cameras. The tilting angle of each camera was
measured with a level and an angle measure.
Intrinsic and extrinsic parameters calculation
To calibrate the cameras, 20 control points arranged in 4 lines with 5 points in each line were made
on a vertically standing black screen. The adjacent lines are 50 centimeters apart. The distance
between adjacent points in each line is also 50 centimeters. The screen was put in front of each
camera with the points facing the camera. A picture was taken and stored in the computer. The
screen was then moved 25 centimeters (for F-CAM and R-CAM) or 20 centimeters (for D-CAM
and P-CAM) closer to the camera. This process was repeated until five pictures were taken for
each camera. Every time a picture was taken, the position of the screen in the bus coordinate
system was marked on the ground and measured later to calculate the control point coordinates.
- 8 -
The pictures were opened in Microsoft Photo Editor? to read the image coordinates of the control
points. We get the coordinates of the control points in the bus coordinate system and their
corresponding image coordinates in the picture. Each control point and its image are called a
calibration pair. By substituting the coordinates of the calibration pairs in the camera model
described below, two equations for each pair were obtained. We can solve the unknown camera
parameters from the equations for all pairs in the sense of Least Square Error (LSE).
T??P?X,Y,Z represent the coordinates of a point in the bus coordinate system (FCS or RCS), Let
T represent the coordinates of the point in the camera coordinate system, ??P?X,Y,ZCCCC(x,y)(x,y) and represent the undistorted and distorted image coordinates of the point UUDDrespectively, and represent the coordinate read in Microsoft Photo Editor?, i.e. the pixel (i,j)location with respect to the top-left corner in the image, viz. the computer image coordinate. The
relationship between the bus coordinate system and the camera coordinate system is given by :
P?RP?T (1) C
??R?rwhere is a 3?3 ortho-normal rotation matrix defining the camera orientation and ij
T is a translation vector defining the camera position. The camera coordinate system ??T?t,t,t123
is transformed to the undistorted image coordinate (2D) system according to the pin-hole model:
X?Cx?fU?Z?C (2) ?YC?y?fU?ZC?
where f is the focal length. The distortion of image coordinates can be modeled by :
222?????2pxy?pr?2x?kxr?xUUUU121 (3) ?222???pr2y2pxykyr?????yUUUU121?
222r?x?yp,pkwhere , are coefficients of tangential distortion, and is the coefficient of UU121
radial distortion. The distorted image coordinates are then obtained:
x?x???DUx (4.1) ?yy???DUy?
222???x?x?2pxy?pr?2x?kxrDU1UU2U1U (4.2) ?222??yypr2y2pxykyr?????DU1U2UU1U?
The relationship between the distorted image coordinates and the computer image coordinates is
- 9 -
??x??i?i?Dx0 (5) ???y?jj??Dy0?
?,?where are the distance between the adjacent imaging sensor elements in rows and columns, xy
??i,jrespectively, represents the computer image coordinate of the principal point of the image 00coordinate system.
The model itself is a nonlinear one. The unknown parameters can be categorized into intrinsic and
extrinsic, or linear and non-linear parameters, as follows:
Table 8 Parameter table
Intrinsic ?,???i,j,, k,p,p fxy00112
TExtrinsic , ??R?r??T?t,t,tij123i,j?1,2,3
It is hard to solve all the parameters simultaneously from the complete nonlinear camera model.
However, if the nonlinear distortion can be neglected, the model becomes linear. Once the linear
parameters are known, the nonlinear parameters can be solved from linear equations (3). These
properties of the camera model help us to simplify the calibration procedure into the following
Step 1: Assume no distortion, calculate linear model parameters Step 2: Calculate distortion using the linear parameters estimated in Step 1
Step 3: Calculate nonlinear parameters using the distortion and linear parameters estimated in Step
Step 4: Calculate distortion using the linear and nonlinear parameters estimated in Step 2 and 3
Step 5: Subtract the distortion estimated in Step 4 from the image coordinates, loop to Step 1 or
The procedure is terminated when it is convergent. As noise exists in the calibration pair
coordinates, the distortion used in Step 5 was multiplied with a positive fraction to confirm
convergence. The positive fraction used in our calculation is 0.999.
Control point images
Control point image coordinates estimated with linear-only and nonlinear-plus models together
with the actual image coordinates read in Photo Editor? are illustrated in the following plots,
where the „o‟ signs represent the actual images read in Photo Editor?, the „+‟ signs represent the
- 10 -