• Dalavai Durga

In Real-time, digital data is increasing in all aspects like banking, healthcare, education and, social science, etc. A Large amount of data will be stored in the form of text. For effective data analysis representation, the compression process is used which reduces the data size, storage space and transmission cost of data. In an existing system, the Content-aware Partial Compression scheme (CaPC) was developed which improves the performance and reduces the resources for textual big data analysis in Hadoop. But this scheme should not offer better compression ratio and performance. To get the maximum value of the compression for data analysis, the two-layered architecture of the Record-aware Partial Compression scheme (RaPC) is developed in Hadoop. But the main disadvantage with Hadoop is, it supports only batch processing and it can't support stream processing. The batch processing is efficient to process a large amount of data, but it depends on the data size and computational power of the system, so there will be a delay in output and overall performance will be slow. The proposed work includes the replacement of Hadoop with Spark which supports stream processing. This stream processing will be worked as continuous input and output, which improves the speed of the data and the data will be processed in less amount of time. Spark uses in-memory processing to process the data. The memory is still considered to be an expensive hardware resource, so this memory consumption will be reduced by using CaPC or RaPC Layer-1 scheme. In CaPC or RaPC Layer-1 compression, the compressed data can be processed directly without decompression.

Big data,Data storage,Compression,Spark,In-memory processing,RaPC scheme,Huffman encoding

"Record-aware Partial Compression scheme in Spark", International Journal of Emerging Technologies and Innovative Research (www.jetir.org), ISSN:2349-5162, Vol.6, Issue 6, page no.35-42, June-2019, Available :http://www.jetir.org/papers/JETIR1906589.pdf

"Record-aware Partial Compression scheme in Spark", International Journal of Emerging Technologies and Innovative Research (www.jetir.org | UGC and issn Approved), ISSN:2349-5162, Vol.6, Issue 6, page no. pp35-42, June-2019, Available at : http://www.jetir.org/papers/JETIR1906589.pdf