Optimizing Spark DataFrame Processing: A Deep Dive into Memory Management and Pipeline Optimization Strategies for Better Performance