Real Time Spark Project for Beginners: Hadoop, Spark, Docker

Building Real Time Data Pipeline Using Apache Kafka, Apache Spark, Hadoop, PostgreSQL, Django and Flexmonster on Docker

Created by : Rated :4.2 54800 views

Last Updated: 2022-05-03 10:05:58 English Course Type: Self Learning

Complete Development of Real Time Streaming Data Pipeline using Hadoop and Spark Cluster on Docker
Setting up Single Node Hadoop and Spark Cluster on Docker
Features of Spark Structured Streaming using Spark with Scala
Features of Spark Structured Streaming using Spark with Python(PySpark)
How to use PostgreSQL with Spark Structured Streaming
Basic understanding of Apache Kafka
How to build Data Visualisation using Django Web Framework and Flexmonster
Fundamentals of Docker and Containerization

In many data centers, different type of servers generate large amount of data(events, Event in this case is status of the server in the data center) in real-time.

There is always a need to process these data in real-time and generate insights which will be used by the server/data center monitoring people and they have to track these server's status regularly and find the resolution in case of issues occurring, for better server stability.

Since the data is huge and coming in real-time, we need to choose the right architecture with scalable storage and computation frameworks/technologies.

Hence we want to build the Real Time Data Pipeline Using Apache Kafka, Apache Spark, Hadoop, PostgreSQL, Django and Flexmonster on Docker to generate insights out of this data.

The Spark Project/Data Pipeline is built using Apache Spark with Scala and PySpark on Apache Hadoop Cluster which is on top of Docker.

Data Visualization is built using Django Web Framework and Flexmonster.

Windows 10 Operating System
Laptop or Desktop with 8 GB of RAM(minimum) and 100 GB of free disk space
Any Text Editor
Java 1.8

Basic understanding of Programming Language
Basic understanding of Apache Hadoop
Basic understanding of Apache Spark

Section 1 : Introduction 2 Lectures 00:40:10
Lecture 1 :
Introduction to Apache Spark Preview
Lecture 2 :
Real Time Spark Project Overview | Building End to End Streaming Data Pipeline
Section 2 : Environment Setup 6 Lectures 01:18:31
Lecture 1 :
Setting up Docker Environment
Lecture 2 :
Create Single Node Kafka Cluster on Docker
Lecture 3 :
Create Single Node Apache Hadoop and Spark Cluster on Docker
Lecture 4 :
Setting up IntelliJ IDEA Community Edition(IDE)
Lecture 5 :
Setting up PyCharm Community Edition(IDE)
Lecture 6 :
Setting up Django Web Framework
Section 3 : Development | Project Code Walk-through 5 Lectures 01:43:56
Lecture 1 :
Building Streaming Data Pipeline using Scala | Spark Structured Streaming
Lecture 2 :
Building Streaming Data Pipeline using PySpark | Spark Structured Streaming
Lecture 3 :
Event Simulator using Python(Server Status Detail
Lecture 4 :
Setting up PostgreSQL Database(Events Database)
Lecture 5 :
Building Dashboard using Django Web Framework and Flexmonster | Visualization
Section 4 : Complete Project Demo 2 Lectures 00:21:49
Lecture 1 :
Real Time Spark Project Demo
Lecture 2 :
Running Real Time Streaming Data Pipeline using Spark Cluster On Docker
Section 5 : Bonus Tutorial - Docker Tutorial for Beginners 9 Lectures 02:03:55
Lecture 1 :
Introduction to Docker
Lecture 2 :
Install Docker on Ubuntu 18.04
Lecture 3 :
Docker Commands | Commonly Used
Lecture 4 :
Create First Docker Image and Container
Lecture 5 :
Create MySQL Docker Container
Lecture 6 :
Cassandra on Docker Container
Lecture 7 :
MongoDB on Docker Container
Lecture 8 :
Setting up Docker Compose
Lecture 9 :
How to create Docker Volume

How do i access the course after purchase?
It's simple. When you sign up, you'll immediately have unlimited viewing of thousands of expert courses, paths to guide your learning, tools to measure your skills and hands-on resources like exercise files. There’s no limit on what you can learn and you can cancel at any time.
Are these video based online self-learning courses?
Yes. All of the courses comes with online video based lectures created by certified instructors. Instructors have crafted these courses with a blend of high quality interactive videos, lectures, quizzes & real world projects to give you an indepth knowledge about the topic.
Can i play & pause the course as per my convenience?
Yes absolutely & thats one of the advantage of self-paced courses. You can anytime pause or resume the course & come back & forth from one lecture to another lecture, play the videos mulitple times & so on.
How do i contact the instructor for any doubts or questions?
Most of these courses have general questions & answers already covered within the course lectures. However, if you need any further help from the instructor, you can use the inbuilt Chat with Instructor option to send a message to an instructor & they will reply you within 24 hours. You can ask as many questions as you want.
Do i need a pc to access the course or can i do it on mobile & tablet as well?
Brilliant question? Isn't it? You can access the courses on any device like PC, Mobile, Tablet & even on a smart tv. For mobile & a tablet you can download the Learnfly android or an iOS app. If mobile app is not available in your country, you can access the course directly by visting our website, its fully mobile friendly.
Do i get any certificate for the courses?
Yes. Once you complete any course on our platform along with provided assessments by the instructor, you will be eligble to get certificate of course completion.
For how long can i access my course on the platform?
You require an active subscription to access courses on our platform. If your subscription is active, you can access any course on our platform with no restrictions.
Is there any free trial?
Currently, we do not offer any free trial.
Can i cancel anytime?
Yes, you can cancel your subscription at any time. Your subscription will auto-renew until you cancel, but why would you want to?

Pari Margu,

54800 Course Views

1 Courses

Data Engineer(Big Data/Hadoop, Apache Spark, Python) cum Freelance Consultant, YouTube Creator. Having 12+ years of experience in implementing solutions to the enterprise clients and having strong Framework skills to implement complex business solutions. Worked on, Web, Windows, Mobile and Hadoop/Big Data, Apache Spark applications. Having 6+ years of experience with Hadoop/Big Data, Apache Spark Framework. Worked on Hadoop distributions like Cloudera CDH, Apache Hadoop.