2019 IEEE 8th Joint International Information Technology and Artificial Intelligence Conference (ITAIC 2019)
Research and Application of Artificial Technology for Substation Environment Surveillance System Haipeng Wang1 ,Xu Zhang1*, Yang Sun3,Jianxiang Li1,Yong Li 2
1. Shandong Luneng Intelligence Technology Co.,Ltd 2. State Grid Shandong Electric Power Company, 3. State Grid Corporation of China Jinan, China
[email protected],
[email protected],
[email protected],
[email protected],
[email protected] position of switching transformer oil surface. Liu et al.[4] developed an image-based state recognition approach, and they propose extraction of texture features using a Gabor transformation, then the state of the isolator is classified by the SVM. Kong [5] adopted layered model and different classification algorithm to realize abnormal behavior analysis.
Abstract—In order to satisfy the requirements of power operation scene safety monitoring and lessen the human accidents, a novel framework of video surveillance in the substation based on artificial technology was presented. Firstly, a multi-class detector based on region-based fully convolutional networks (R-FCN) was proposed to realize the detection and localization of multiple targets. Then types of violation and defect were discriminated by secondary analysis for the workers or equipments based on detection results. In order to implement auto-alarm of flames and smoke, an algorithm for automatic smoke and flame detection is presented, using different methods to extract candidate regions and a unified CNN to recognition. In addition, a virtual electronic fence was designed to prevent workers from entering dangerous areas and other no-working areas. The experimental results demonstrate that the approach is effective for detecting illegal operations of works automatically and substation security monitoring, and it will improve the automation and intelligentization level of unmanned substation.
Above-mentioned researches mainly focus on the equipment fault detection and state recognition in substation. Aside equipment safety, intelligent surveillance system still needs to monitor operator safety work and alarms at the unusual states. In [6], Yang et al. proposed a method on analyzing environmental monitoring video data in substation, but the robustness and real-time performance of the algorithm need to be further improved. The main purpose of this paper is to develop an innovative and practical intelligent surveillance system for abnormal patterns detection in power substation, based on computer vision, deep learning and image processing. Toward this objective, R-FCN [7] is employed to detect multitargets in substation, which leads to the improvement of detecting efficiency. Furthermore, further analysis can be done to identify behavioural of workers and state of equipments based on the person or equipment location. A novel framework is applied in our system to detect smoke and flame in real time using a small region-based CNN. According to the substation code of conduct, a virtual electronic fence was designed to judge peoples whether to cross the safety lines without permission.
Keywords—video surveillance; object detection; behavior analysis; flame detection; electric fence; R-FCN
I.
INTRODUCTION
Power substations plays a crucial role in delivering electricity to consumers by converting transmission voltage to the lower voltage used in homes and businesses. With the improvement of substation automatic level, video surveillance systems are widely used in substations. Current substation video monitoring system is still mainly worked as traditional video surveillance, only provides simple functions as video capture, storage and playback, which lacks the ability to effectively analyze information. Besides, the monitor of substation workers or equipments is still relying on artificial observation and patrol, which did not take full advantage of the potential of intelligent video surveillance technology.
The rest of this paper is organized as follows. Section 2 is a brief introduction for automatic multi-object detection based on R-FCN. Section 3 presents the proposed flame and smoke detection algorithm for substation environmental monitoring. A new virtual electronic fence with no need for manual arrangement is described in section 4. Section 5 discusses our experiments and empirical results. Section 6 concludes this work.
At present, intelligent video surveillance technology have been studied and applied in the power system [1-5]. In [1], a recognition method of abnormal patterns for video surveillance in unmanned substation was proposed according to the technical characteristics of pattern recognition. Song et al. [2] prospered a new intelligent video surveillance consisted of proprocessing, improved ViBe and post-processing methods, which is more effective and suitable for moving object detection in substation. In [3], image recognition technology was used to identify signal lights, 7-segment digital and the
978-1-5386-8178-7/19/$31.00 ©2019 IEEE
II.
MULTI-TARGETS DETECTION BASED ON R-FCN
Pedestrian and equipment detection in power substation is a premise for intelligent monitoring of substation. In the substation surveillance application, the detection results can be used to locate the target object and segment out from whole image for next analysis. In order to improve detection accuracy and efficiency, many object detection schemes have been applied for target detection like Haar+Adaboost[8,9] and
901
HOG+SVM[10]. Recent advances in object detection are driven by the success of deep learning algorithm, such as region-based convolutional networks (R-CNN) [11] and its extension[12-14]. Compared with R-CNN based methods, RFCN proposes much fewer region-wise layers to balance the learning of classification and detection for naturally combining fully convolutional network with region-based module. In this report, we develop a multi-targets detector based on R-FCN which achieves more decent performance than the R-CNN detectors. Firstly, the data sets which consist of training images and testing images are collected, including workers and essential equipments in substation such as breaker, transformer, arrester and so on. Data augmentation based on image scaling, translation, and rotation applied here to solve the case of samples with unequal numbers in the different varieties. Then, our framework based on the R-FCN was implemented to detect different targets in substation environment. Since our framework is based on the R-FCN, we refer the reader to [7] for more technical details.
(a)
(b)
(c)
(d)
Fig.1. Safety helmet wearing detection results.
Based on R-FCN, we make several effective modifications for improving detection performance. For better describing tiny targets, we introduce more anchors with smaller scales. These smaller anchors are very helpful for sufficiently capturing the extremely tiny targets. Besides, small pooling size for position sensitive ROI pooling was set to reduce redundant information. Finally, we apply atrous convolution in the last stage of ResNet to keep the scale of feature maps without losing the contextual information in larger receptive field. Inspired by [15], we perform multi-scale training and testing strategy to improve performance. In the training phase, we resize the shortest side of the input to 1024 pixels. This training strategy keeps our model being robust on detecting the target at the different scale. In the testing phase, we build an image pyramid for each test image. Each scale in the pyramid is independently tested. The results from various scales are eventually merged together as the final result of the image. After object detection in substation, further analysis can be done for behavioural analysis of workers and status detection of equipments. Safety helmet wearing detection for perambulatory workers is very essential in substation. According to the result of workers detection, the safety helmet wearing detection is implemented using the color feature discrimination. Usually, the top region of one-fifth in the bounding box is our interest of region for head location. This value is chosen empirically. In order to use the most important color information of helmet to determine human whether wearing helmet or not, color space transformation and color feature discrimination are executed. Since the image in HSV color space is more adaptive to color segmentation, so we convert RGB to HSV. It is worth noting that setting fix threshold for Hue and Saturation channels can segment various colors, and then discriminating human with or without safety helmet. Whereas we only set the Hue channel threshold and not define the Saturation channel threshold. Instead, we use OSTU method on the channel of Saturation to obtain automatically threshold and segment color. As shown in Fig.1, pedestrian safety helmet can be detected pretty well.
Furthermore, substation equipments state identification can be realized according to location results of different equipments such as instrument automatic reading recognition and state identification for disconnector. Research on the status identification of substation equipment is beyond the scope of this paper, it will be addressed in our future work. III.
SMOKE AND FLAME DETECTION
Fire detection technology is the major part of unmanned substation surveillance system. Flame and smoke detection methods via video processing and the related research work can be found in [16-20]. Existing approaches have limited utility in practical application because the main features used to detect flame and smoke was set experimentally. Here, we developed a flame and smoke detection framework which can be divided into two steps: (i) region proposal, and (ii) region classification with a small CNN. Fig.3 shows a flow chart of our approach to implement flame and smoke detection. Generic object detection methods are moving from dense sliding window approaches to sparse region proposal framework. High-quality and category-independent object proposals reduce the number of windows each classifier needs to consider. In order to achieve the purpose of smoke and flame detection and reduce the computational cost, it is first necessary to acquire candidate area which called “region proposal” algorithm. For object proposals detection, different approaches were designed to extract region proposal of smoke and flame images respectively. We implemented region proposal generation for smoke with a fully convolutional network [21]. The detection of flame candidate regions is carried out using image color segmentations. The details of the image color segmentations are described as follows. For each pixel in flame blob value of Red channel is greater than the Green channel, and the value of Green channel is greater than the value of Blue channel and mostly of flame color is the color with high saturation in Red channel. RGB represents not only the chrominance but also the luminance of
902
Object proposal
Result
Color Segmentation
CNN
...
FCN
Fig. 2. Overall architecture of smoke and flame detection.
dropout of 0.5 in the two fully connected layers to avoid overfitting. The network was trained for roughly 100 cycles.
the pixels. The luminance component of the RGB color space can be eliminated by color channel ratios. In our application RG, R-B, and G-B ratios were used for color based flame candidate area extraction. So for a pixel located at spatial location of (x,y), the condition can be as follows:
R ( x, y ) > G ( x, y ) > B ( x, y ) R ( x, y ) > Rmean 0.2 ≤ G ( x, y ) /( R ( x, y ) + 1) ≤ 0.7 0.05 ≤ B ( x, y ) /( R( x, y ) + 1) ≤ 0.5
IV.
VIRTUAL ELECTRONIC FENCE
So far, most of the existing security fences are put up by artificial to avoid workers entering charged regions or the other non-working areas. This approach has the disadvantage of complex operation and lack of effective supervision. The electronic fence based on electronic fence and infrared have the demerit of cost high. Based on surveillance system, we proposed a virtual electronic fence to decide workers whether to cross the safety lines without permission. The followings are the procedure:
(1)
0.15 ≤ B( x, y ) /(G ( x, y ) + 1) ≤ 0.65 where R(x,y), G(x,y), and B(x,y) are Red, Green and Blue values for a pixel located spatially at (x,y). Rmean is the mean of Red channel of pixels which defined as , and K is the total number of pixel in image. The upper and lower bounds for inequalities are estimated by using the hand labeled data set. The rule defined in (1) is used to generate an overall binary map. Then morphological operation and connection areas calculation are implemented according to the binary map. Finally, candidate boxes with high probability to be really the flame regions were generated by color segmentation.
(1) According to the information of work ticket, area of virtual electronic fence and alert rules are set. (2) After the module of virtual electronic fence start, information of workers and working area is received by the system. Then, the function of virtual electronic fence comes into operation. Besides, real electronic fence and alert area is determined by object detection based on R-FCN in section 2.
Our classification architecture is classical for convolutional neural network, combining convolution and Max pooling. However, to get a fast classification we choose a small network. Here, we use the nine layers CNN. A color image goes through successively two convolutional operations with kernel of size 3×3. The same structure is applied after layer three. A max pooling 3×3 with stride 2 follows the convolutional layer two and five. The layers one to four have 16 feature maps. The layers five and six have only one feature map. The layer seven and eight are fully connected. The output of the last fully connected layer is fed to softmax which produces a distribution over 3 class labels. Similarly to [22], we choose for convolution and fully connected layers a Leaky Relu activation function with coefficient a=1/3. The training set is composed of 9600 labeled images of size 64×64 pixels, 2000 for smoke, 2000 for flame and 5600 negative. The weight in the network is initialized randomly. The initial learning rate is 0.01 and momentum 0.9. The learning rate decreases by a factor 0.95 each epoch. We use
(3) The system will automatically alarm and remind the illegal workers, when recognizing that filed personnel breaking alert rules in real time by position analysis of personnel location and the fence boundary. As shown in Fig.3(a), there is a break-in in the alert area was detected by the system. It also automatically alerts workers when abnormal behavior identification is implemented, as shown in Fig.3(b). (4) The virtual electronic fence will be close, when an end message of work ticket was received.
（a)
（b)
Fig.3. Virtual electronic fence system. (a) region boundary automatic generation in video; (b) real security fence detection.
903
V.
EXPERIMENTS RESULT
To evaluate the proposed safety detection framework, extensive experiments were conducted on surveillance video sequences of 220kV Xusi substation with the format of pixel resolutions of 1920×1080. We capture 20 videos with different scenes under unfixed view and run our detection algorithm. All experiments are evaluated on a single NVIDIA 1080TI GPU. A. Multi-targets Detection In order to test the performance of our multi-targets detection algorithm in substation, we will be comparing the proposed algorithm with Faster R-CNN and SSD. The results of test in Table 1 show that R-FCN achieved better mAP compared two other methods.
Fig. 4. Flame and smoke detection result: images in the 1stand 2nd line were the results of flame detection, and the 3rd and 4th line show the smoke detection results.
VI.
TABLE I.
RESULTS OF POLE-TOP DETECTION IN THREE POWER DISTRIBUTION LINES USING DIFFERENT METHODS Faster rcnn
SSD
R-FCN
mAP of worker(%)
94.6
94.2
95.3
mAP of blade(%)
83.2
79.8
82.9
mAP of insulator(%)
83.6
81.9
84.1
Detect time(sec/img)
0.86
0.18
0.25
Here, we have labeled data into wearing safety helmet and no wearing safety helmet manually. The performance of helmet detection is evaluated by detection correctness and completeness, as defined in (2):
TP Correctness = , TP + FP TP Completeness = TP + FN
CONCLUTION
Surveillance is very essential for the safety of substation. In this paper, a substation safety monitoring platform was presented. Multi-targets in substation environment were detected by integrating R-FCN and several sophisticated techniques for better detecting targets and boosting overall performance. On the basis of workers’ location, further analysis is implemented for safety helmet wearing detection. Besides, a novel framework is applied in our system to detect smoke and flame in real time using a small region-based CNN. In order to avoid workers entering danger area, we proposed a virtual electronic fence to judge workers whether to cross the safety lines without permission. Extensive experimental results have illustrated the effectiveness and efficiency of this substation environment surveillance system. ACKNOWLEDGMENT Thanks to Jinan power supply companies to provide testing data and testing site. This work was supported by the science and technology project named “Research and application of key technologies for automatic inspection” supported by the State Grid Corporation of China (Grant No.520600180003).
(2)
where TP is the number of “true” detected no wearing helmet; FP is the number of “false” detected objects which are no wearing helmet; and FN is the number of objects missed. The mean correctness of helmet detection in 20 videos is around 92%, and completeness is around 94%. Not only single safety helmet but also multiple safety helmets can be detected pretty well. In our opinion, the main reason of missed examination is that the target is far from the surveillance camera.
REFERENCES [1] [2] [3]
B. Flame and Smoke Detection Because pyrotechnic experiments are not allowed in substations, it is difficult to obtain ideal experimental video data. In view of the experiment of fire detection, we set up a fire data set, and collected 3000 smoke and flame images in different environments and scenarios. Here, the performance of smoke and flame detection is also evaluated by detection correctness and completeness. The mean correctness of smoke and flame detection is around 94% and completeness is around 95%. The average detection time for per image is about 80ms, which can meet the real-time requirements of substation fire detection. Fig.4 shows some results of smoke and flame detection.
[4] [5]
[6]
904
S.M. Liu, X.J. Xu, “Substation intelligent monitoring system based on pattern recognition,” International conference on applied informatics and communication, pp. 473-478, 2011. W. Song, J.Q. Zhang, W.J. Zhang, “Research of intelligent video surveillance in substation based on ViBe,” Chinese Journal of Scientific Instrument, vol. 35, issue 2,pp. 36-41, 2014. F.J. Sun, W.X. Cui, J.B. Zhang, X.D. Zhang, X.D. Xiao, “Application of remote digital video monitoring and image recognition technology in power system,” Power system technology, vol. 29, issue 5, pp. 81-84, 2005. J.Z. Liu, X.Y. Meng, “Research of isolator’s status image recognition based on Gabor wavelet and SVM,” 2010 3rd International conference on advanced computer theory and engineering, 2010. Y.H. Kong, M.L. Jing, “A recognition method of abnormal patterns for video surveillance in unmanned substation,” Journal of north China electric power university(Natural Science), vol. 38, issue 6, pp. 11-16, 2011. J. Yang, X. Ai, X.F. Jia, “Study of image recognition used for unattended substation,” Proceedings of international conference on power system technology, pp. 1-6, 2006.
[7] [8] [9] [10] [11]
[12]
[13] [14]
[15] A. Shrivastava, A. Gupta, R. Girshick, “Training Region-based Object Detectors with Online Hard example mining,” In conference on computer vision and pattern recognition, 2016. [16] T.H. Chen, C.L. Kao, S.M. Chang, “An intelligent real-time firedetection method based on video processing,” Proc. IEEE 37th Ann. 2003 Int. Carnahan Conf. on SecurityTechnology, pp. 104–11. [17] B.U. Toreyin, Y. Dedeoglu, U. Gudukbay, A.E. Cetin, “Computer vision-based method for real-time fire and flame detection,” Pattern Recognit. Lett. vol. 27, issue 1, pp. 49–58. [18] T.H. Chen, Y.H. Yin, S.F. Huang, Y.T. Ye, “The smoke detection for early fire-alarming system base on video processing,” Proc. Intelligent Information Hiding and Multimedia Signal Processing, pp. 427–30. [19] C.L. Lai, J. C. Yang , Y. H. Chen, “A real-time video processing-based surveillance system for early fire and flood detection,” Proc. Instrumentation and Measurement Technology Conf., pp. 1–6. [20] K. Poobalan, S. Liew, “Fire detection algorithm using image processing techniques,” Proceedings of the 3rd International Conference on Artificial Intelligence and Computer Science , pp. 160-168, 2015. [21] H. Wang, Z.F. Li, X. Ji, Y.T. Wang, “Face R-CNN,” In arXiv preprint arXiv:1706.01061, 2017. [22] A.L. Maas, A. Y. Hannun, A. Y. Ng, “Rectify nonlinearities improve neural network acoustic model,” ICML 2013 Workshop on Deep Learning for Audio, Speech, and Language Processing, 2013.
J.F. Dai, Y. Li, K.M. He, J. Sun, , “R-FCN: object detection via regionbased fully convolutional networks,” In Advances in Neural Information Processing Systems (NIPS), 2016.. Y. Freund, R. Schapire, “ A decision-theoretic generalization of on-line learning and an application to boosting,” Journal of Computer and System Sciences, vol. 55, issue 1, pp.119-139, 1997. R. Lienhart, J. Maydt, “An extended set of haar-like features for object detection,” in Proc. Int, Conf. Image Processing , pp. 900-903, 2002. N. Dalal, B. Triggs, “Histograms of oriented gradients for human detection,” IEEE International Conference on Computer Vision, pp. 824-831, 2005. R. Girshick, J. Donahue, T. Darrell, et al. “Region-Based Convolutional Networks for Accurate Object Detection and Segmentation,” IEEE Transaction on Pattern Analysis & Machine Intelligence, vol. 38, issue 1, pp. 1-1, 2015. K.M. He, X. Zhang, S. Ren, et al., “Spatial Pyramid Pooling in Deep Convolutional Networks for visual recognition,” IEEE Transaction on Pattern Analysis & Machine Intelligence, vol.37, issue 9, pp. 1904-1916, 2014. R. Girshick, “Fast R-CNN,” 2015 IEEE International Conference on Computer Vision. pp. 1440-1448, 2015. S.Q. Ren, K.M. He, R. Girshick, J. Sun, “Faster R-CNN: Towards Realtime Object Detection with Region Proposal Network,” International Conference on Neural Information Processing Systems, pp. 1-14, 2015.
905
