Tech & IT
Business
Coding & Developer
Finance & Accounting
Academics
Office Applications
Art & Design
Marketing
Health & Wellness
Sounds & Music
Lifestyle
Photography
Free
3 days left at this price!
This plan includes
- Limited free courses access
- Play & Pause Course Videos
- Video Recorded Lectures
- Learn on Mobile/PC/Tablet
- Quizzes and Real Projects
- Lifetime Course Certificate
- Email & Chat Support
Get Unlimited Learning Access
$29
3 days left at this price!
30-Day Money-Back Guarantee
This plan includes
- Access to 11,000+ Courses
- Ads free experience Courses
- Play & Pause Course Videos
HD Video Recorded Lectures- Learn on Mobile/PC/Tablet
- Quizzes and Real Projects
- Lifetime Course Certificate
- Instructor Direct Support
- Email & Chat Support
- Cancel Anytime
$29
$29
What you'll learn?
- Complete Development of Real Time Streaming Data Pipeline using Hadoop and Spark Cluster on Docker
- Setting up Single Node Hadoop and Spark Cluster on Docker
- Features of Spark Structured Streaming using Spark with Scala
- Features of Spark Structured Streaming using Spark with Python(PySpark)
- How to use PostgreSQL with Spark Structured Streaming
- Basic understanding of Apache Kafka
- How to build Data Visualisation using Django Web Framework and Flexmonster
- Fundamentals of Docker and Containerization
Course Overview
In many data centers, different type of servers generate large amount of data(events, Event in this case is status of the server in the data center) in real-time.
There is always a need to process these data in real-time and generate insights which will be used by the server/data center monitoring people and they have to track these server's status regularly and find the resolution in case of issues occurring, for better server stability.
Since the data is huge and coming in real-time, we need to choose the right architecture with scalable storage and computation frameworks/technologies.
Hence we want to build the Real Time Data Pipeline Using Apache Kafka, Apache Spark, Hadoop, PostgreSQL, Django and Flexmonster on Docker to generate insights out of this data.
The Spark Project/Data Pipeline is built using Apache Spark with Scala and PySpark on Apache Hadoop Cluster which is on top of Docker.
Data Visualization is built using Django Web Framework and Flexmonster.
Pre-requisites
- Windows 10 Operating System
- Laptop or Desktop with 8 GB of RAM(minimum) and 100 GB of free disk space
- Any Text Editor
- Java 1.8
Target Audience
- Basic understanding of Programming Language
- Basic understanding of Apache Hadoop
- Basic understanding of Apache Spark
Curriculum 24 Lectures 06:08:21
-
Section 1 : Introduction
- Lecture 2 :
- Real Time Spark Project Overview | Building End to End Streaming Data Pipeline
-
Section 2 : Environment Setup
- Lecture 1 :
- Setting up Docker Environment
- Lecture 2 :
- Create Single Node Kafka Cluster on Docker
- Lecture 3 :
- Create Single Node Apache Hadoop and Spark Cluster on Docker
- Lecture 4 :
- Setting up IntelliJ IDEA Community Edition(IDE)
- Lecture 5 :
- Setting up PyCharm Community Edition(IDE)
- Lecture 6 :
- Setting up Django Web Framework
-
Section 3 : Development | Project Code Walk-through
- Lecture 1 :
- Building Streaming Data Pipeline using Scala | Spark Structured Streaming
- Lecture 2 :
- Building Streaming Data Pipeline using PySpark | Spark Structured Streaming
- Lecture 3 :
- Event Simulator using Python(Server Status Detail
- Lecture 4 :
- Setting up PostgreSQL Database(Events Database)
- Lecture 5 :
- Building Dashboard using Django Web Framework and Flexmonster | Visualization
-
Section 4 : Complete Project Demo
- Lecture 1 :
- Real Time Spark Project Demo
- Lecture 2 :
- Running Real Time Streaming Data Pipeline using Spark Cluster On Docker
-
Section 5 : Bonus Tutorial - Docker Tutorial for Beginners
- Lecture 1 :
- Introduction to Docker
- Lecture 2 :
- Install Docker on Ubuntu 18.04
- Lecture 3 :
- Docker Commands | Commonly Used
- Lecture 4 :
- Create First Docker Image and Container
- Lecture 5 :
- Create MySQL Docker Container
- Lecture 6 :
- Cassandra on Docker Container
- Lecture 7 :
- MongoDB on Docker Container
- Lecture 8 :
- Setting up Docker Compose
- Lecture 9 :
- How to create Docker Volume
Our learners work at
Frequently Asked Questions
-
How do i access the course after purchase?
It's simple. When you sign up, you'll immediately have unlimited viewing of thousands of expert courses, paths to guide your learning, tools to measure your skills and hands-on resources like exercise files. There’s no limit on what you can learn and you can cancel at any time. -
Are these video based online self-learning courses?
Yes. All of the courses comes with online video based lectures created by certified instructors. Instructors have crafted these courses with a blend of high quality interactive videos, lectures, quizzes & real world projects to give you an indepth knowledge about the topic. -
Can i play & pause the course as per my convenience?
Yes absolutely & thats one of the advantage of self-paced courses. You can anytime pause or resume the course & come back & forth from one lecture to another lecture, play the videos mulitple times & so on. -
How do i contact the instructor for any doubts or questions?
Most of these courses have general questions & answers already covered within the course lectures. However, if you need any further help from the instructor, you can use the inbuilt Chat with Instructor option to send a message to an instructor & they will reply you within 24 hours. You can ask as many questions as you want. -
Do i need a pc to access the course or can i do it on mobile & tablet as well?
Brilliant question? Isn't it? You can access the courses on any device like PC, Mobile, Tablet & even on a smart tv. For mobile & a tablet you can download the Learnfly android or an iOS app. If mobile app is not available in your country, you can access the course directly by visting our website, its fully mobile friendly. -
Do i get any certificate for the courses?
Yes. Once you complete any course on our platform along with provided assessments by the instructor, you will be eligble to get certificate of course completion.
-
For how long can i access my course on the platform?
You require an active subscription to access courses on our platform. If your subscription is active, you can access any course on our platform with no restrictions. -
Is there any free trial?
Currently, we do not offer any free trial. -
Can i cancel anytime?
Yes, you can cancel your subscription at any time. Your subscription will auto-renew until you cancel, but why would you want to?
Instructor
54783 Course Views
1 Courses
Data Engineer(Big Data/Hadoop, Apache Spark, Python) cum Freelance Consultant, YouTube Creator. Having 12+ years of experience in implementing solutions to the enterprise clients and having strong Framework skills to implement complex business solutions. Worked on, Web, Windows, Mobile and Hadoop/Big Data, Apache Spark applications. Having 6+ years of experience with Hadoop/Big Data, Apache Spark Framework. Worked on Hadoop distributions like Cloudera CDH, Apache Hadoop.
View More...
