Liu Y.,CAS Institute of Computing Technology |
Liu Y.,University of Chinese Academy of Sciences |
Hu S.,CAS Institute of Computing Technology |
Rabl T.,University of Toronto |
And 5 more authors.
Proceedings of the VLDB Endowment | Year: 2014
In Smart Grid applications, as the number of deployed electric s- mart meters increases, massive amounts of valuable meter data is generated and collected every day. To enable reliable data collec- tion and make business decisions fast, high throughput storage and high-performance analysis of massive meter data become crucial for grid companies. Considering the advantage of high efficiency, fault tolerance, and price-performance of Hadoop and Hive system- s, they are frequently deployed as underlying platform for big data processing. However, in real business use cases, these data anal- ysis applications typically involve multidimensional range queries (MDRQ) as well as batch reading and statistics on the meter data. While Hive is high-performance at complex data batch reading and analysis, it lacks efficient indexing techniques for MDRQ. In this paper, we propose DGFIndex, an index structure for Hive that efficiently supports MDRQ for massive meter data. DGFInd- ex divides the data space into cubes using the grid file technique. Unlike the existing indexes in Hive, which stores all combination- s of multiple dimensions, DGFIndex only stores the information of cubes. This leads to smaller index size and faster query pro- cessing. Furthermore, with pre-computing user-defined aggrega- tions of each cube, DGFIndex only needs to access the bound- ary region for aggregation query. Our comprehensive experiments show that DGFIndex can save significant disk space in comparison with the existing indexes in Hive and the query performance with DGFIndex is 2-50 times faster than existing indexes in Hive and HadoopDB for aggregation query, 2-5 times faster than both for non-aggregation query, 2-75 times faster than scanning the whole table in different query selectivity. © 2014 VLDB Endowment 2150-8097/14/08.
Zhang X.,Dalian Maritime University |
An J.,Dalian Maritime University |
Chen F.,Zhejiang Electrical Power Corporation
Proceedings - 2010 2nd WRI Global Congress on Intelligent Systems, GCIS 2010 | Year: 2010
In this paper, a new method of insulator fault detection by texture feature sequence is proposed. Morphology, Hough transform line detection and statistic texture feature are applied to this method. Because there are some noises during image shooting, it is necessary that the insulator image should be preprocessed and corrected. The preprocessing includes image grayness, image enhancement and morphological processing. The insulator tilt correction is based on Hough transform line detection. Then, the processed insulator image is separated into ten parts. Each part of ten parts is featured by seven texture values. The matrix is constructed by ten columns which represent ten parts and seven rows which represent seven texture features. Each row of the matrix is defined as a sequence. According to the analysis of seven feature sequence curves, three feature sequence curves are selected because they are active. A fault feature formula that is named CMV is made of the three features. Insulator fault is detected by CMV curve. The position of fault is located by the biggest vale of CMV curve. Experimental results indicate the proper morphological processing improves the precision of Hough transform line detection and the insulator fault is detected and the position of fault is located by the proposed method accurately. © 2010 IEEE.
Liu X.,Huazhong University of Science and Technology |
Tao L.,Zhejiang Electrical Power Corporation |
Jiang S.,Huazhong University of Science and Technology |
Cheng Y.,Hubei Electric Power Company
Huazhong Keji Daxue Xuebao (Ziran Kexue Ban)/Journal of Huazhong University of Science and Technology (Natural Science Edition) | Year: 2014
The large impulse current generator was delivered to the site, and large simulation lightning current was injected into actual grounding devices of transmission line tower. The potential rise and impulse grounding resistance were measured, and the impulse characteristic parameters of the transmission line tower grounding were obtained. A simulation model of grounding was presented, and the simulation was taken by the ATP-EMTP (ATP-electromagnetic transients program) following the field tests. Comparison between the results of tests and simulation also is completed, and the consistency is good.
Wang Y.,CAS Institute of Computing Technology |
Wang Y.,University of Chinese Academy of Sciences |
Xu Y.,CAS Institute of Computing Technology |
Xu Y.,University of Chinese Academy of Sciences |
And 5 more authors.
Proceedings of the ACM SIGMOD International Conference on Management of Data | Year: 2015
Apache Hive has been widely used by Internet companies for big data analytics applications. It can provide the capability of compiling high-level languages into efficient MapReduce workflows, which frees users from complicated and time consuming programming. The popularity of Hive and its HiveQL-compatible systems like Impala and Shark attracts attentions from traditional enterprises as well. However, enterprise big data processing systems such as Smart Grid applications often have to migrate their RDBMS-based legacy applications to Hive rather than directly writing new logic in HiveQL. Considering their differences in syntax and cost model, manual translation from SQL in RDBMS to HiveQL is very difficult, error-prone, and often leads to poor performance. In this paper, we propose QMapper, a tool for automatically translating SQL into proper HiveQL. QMapper consists of a rule-based rewriter and a cost-based optimizer. The experiments based on the TPC-H benchmark demonstrate that, compared to manually rewritten Hive queries provided by Hive contributors, QMapper dramatically reduces the query latency on average. Our real world Smart Grid application also shows its efficiency. Copyright © 2015 ACM.
Zhou Q.,Accenture |
Hou F.,Accenture |
Huang Y.,Zhejiang Electrical Power Corporation
Proceedings of the 2013 IEEE 7th International Power Engineering and Optimization Conference, PEOCO 2013 | Year: 2013
This paper presents a new solution for load management of a power grid to address the ever growing supply shortage problem. It profiles customers' load behavior data provided by the advanced metering infrastructure and divides the power customers into groups based on their usage characteristics. The possible load management strategy for each of the group(s) was developed to guide the customer's participation in the system load shaving process. An optimization model is developed to perform the load shaving analysis subject to the constraints of the customer participation strategies for each of the group(s) and other load management constraints. The optimization solution provides a recommended load management decision for each of the power customers in order to optimally reduce the system peak load and increase the valley load such that the predicted system peak hour supply shortage could be avoided. The developed load shaving optimization model was solved utilizing the Mixed Integer Linear Programming technique. © 2013 IEEE.