DSA7002 WRIT1 Big Data Technologies Assessment Brief 2024-25| CMU

Task/assessment brief:

Scalability is one of the most critical challenges in the field of Big Data, especially when working with distributed systems designed to process massive and continuously growing datasets. This essay provides an opportunity to critically examine scalability issues in Big Data applications, propose advanced solutions, and reflect on their extension to real-time streaming data scenarios.
As part of this assessment, you will:

• Critically analyse the challenges associated with scaling up Big Data applications in distributed environments, with a particular focus on performance, fault tolerance, resource optimisation, and efficiency.
• Propose solutions to these scalability challenges by evaluating Spark’s optimisation techniques, such as caching, partitioning, and broadcast variables, and explain their practical implications in distributed computing.
• Reflect on the complexities of handling real-time streaming data and propose strategies for extending your scalability solutions to real-time problems such as traffic analytics, drawing on frameworks like Spark Streaming or similar tools.
• Integrate insights from contemporary research, including peer-reviewed journal articles, industry white papers, and academic case studies, to support your arguments and recommendations.

Word count (or equivalent):    1000 words equivalent

Learning outcomes assessed

•  [LO5] Review and address scalability challenges in big data applications and recommend solutions to scaling up to large data sets, including real-time streaming data.        

Other skills/attributes developed

This includes elements of the Cardiff Met EDGE (Ethical, Digital, Global and Entrepreneurial skills) and other attributes developed in students through the completion of the module and assessment. These will also be highlighted in the module guidance, which should be read by all students completing the module. Assessments are not just a way of auditing student knowledge. They are a process that provides additional learning and development through the preparation for and completion of the assessment. 

Ethical    

Students must evaluate the ethical implications of handling large datasets and real-time streaming data. This includes considerations such as:

• Privacy concerns in the use of real-world traffic data or similar sensitive datasets.
• Accountability and transparency in developing scalable Big Data solutions, ensuring compliance with data protection laws (e.g., GDPR).

Entrepreneurial    

Students will showcase entrepreneurial competencies by:

• Worked independently to formulate recommendations for solving an authentic, complex problem related to Big Data scalability.
• Proactively identifying and addressing gaps in existing solutions, demonstrating creativity and critical thinking.