• Mr. S. S. Shinde

Data is now getting generated with tremendous use of distributed network and sensor network. This data is processed in terms of streams. In distributed knowledge stream process, a program with multiple queries parallelized by partitioning input streams in keeping with the values of specific attributes, or partitioning keys. By applying completely different partitioning keys to different queries needs re-partitioning. It cause further communication and reduce throughput. Re-partitioning may be avoided by detecting dependencies between the partitioning keys applicable to every question. Existing partitioning optimization ways analyze question query syntax at compile-time to observe inter-key dependencies hence avoid re-partitioning. This work extends those compile-time ways by adding Auto-parallelization. Auto-parallelization technique involves locating regions in the application’s data flow graph that can be replicated at run-time to apply data partitioning, to achieve scale. This work, propose an elastic auto-parallelization solution that can dynamically adjust the number of channels used to achieve high throughput without unnecessarily wasting resources. A runtime re-optimization step supported the mining of temporal approximate dependencies (TADs) between partitioning keys. A small indefinite amount is outlined during this work as a sort of dependency which will be roughly valid over a moving time window. This addresses the profitability problem associated with auto-parallelization of general-purpose distributed DataStream processing applications.

Data-Stream Partitioning, Auto-parallelization, Runtime Optimization, TAD

"USING RUNTIME DEPENDENCY MINING APPROACH FOR RE-OPTIMIZATION OF DATA STREAM PARTITIONING", International Journal of Emerging Technologies and Innovative Research (www.jetir.org), ISSN:2349-5162, Vol.6, Issue 3, page no.272-275, March-2019, Available :http://www.jetir.org/papers/JETIRAL06058.pdf

"USING RUNTIME DEPENDENCY MINING APPROACH FOR RE-OPTIMIZATION OF DATA STREAM PARTITIONING", International Journal of Emerging Technologies and Innovative Research (www.jetir.org | UGC and issn Approved), ISSN:2349-5162, Vol.6, Issue 3, page no. pp272-275, March-2019, Available at : http://www.jetir.org/papers/JETIRAL06058.pdf