Master Big Data with Sqoop, Flume, Hive & Spark

Master Big Data with Sqoop,Flume,Hive and Spark

Complete course on Sqoop, Flume & Hive- Great for CCA175 & Hortonwork Spark Certification preparation.

Created by : Rated :4.3 83272 views

Last Updated: 2024-11-21 08:11:49 English Course Type: Self Learning

Hadoop distributed File system and commands. Lifecycle of sqoop command. Sqoop import command to migrate data from Mysql to HDFS. Sqoop import command to migrate data from Mysql to Hive. Working with various file formats, compressions, file delimeter,where clause and queries while importing the data. Understand split-by and boundary queries. Use incremental mode to migrate the data from Mysql to HDFS. Using sqoop export, migrate data from HDFS to Mysql. Using sqoop export, migrate data from Hive to Mysql. Understand Flume Architecture. Using flume, Ingest data from Twitter and save to HDFS. Using flume, Ingest data from netcat and save to HDFS. Using flume, Ingest data from exec and show on console. Flume Interceptors.

In this course, you will start by learning what is hadoop distributed file system and most common hadoop commands required to work with Hadoop File system.

Then you will be introduced to Sqoop Import

Understand lifecycle of sqoop command.

Use sqoop import command to migrate data from Mysql to HDFS.

Use sqoop import command to migrate data from Mysql to Hive.

Use various file formats, compressions, file delimeter,where clause and queries while importing the data.

Understand split-by and boundary queries.

Use incremental mode to migrate the data from Mysql to HDFS.

Further, you will learn Sqoop Export to migrate data.

What is sqoop export

Using sqoop export, migrate data from HDFS to Mysql.

Using sqoop export, migrate data from Hive to Mysql.

Further, you will learn about Apache Flume

Understand Flume Architecture.

Using flume, Ingest data from Twitter and save to HDFS.

Using flume, Ingest data from netcat and save to HDFS.

Using flume, Ingest data from exec and show on console.

Describe flume interceptors and see examples of using interceptors.

Flume multiple agents

Flume Consolidation.

Finally, we will start with our last section about Apache Hive

Hive Intro

External & Managed Tables

Working with Different Files - Parquet,Avro

Compressions

Hive Analysis

Hive String Functions

Hive Date Functions

Partitioning

Bucketing

Cloudera vm installation if you want to run examples.

Who want to learn sqoop and flume or who are preparing for CCA and HDPCD certifications

Section 1 : Hadoop Introduction 3 Lectures 00:16:57
Lecture 1 :
HDFS and Hadoop Commands Preview
Lecture 2 :
Mapreduce Yarn
Lecture 3 :
Cloudera VM Machine Setup Guide
Section 2 : Apache Sqoop Import 13 Lectures 01:01:22
Lecture 1 :
Managing Target Directories
Lecture 2 :
Working with Different File Formats
Lecture 3 :
Working with Different Compressions
Lecture 4 :
Conditional Imports
Lecture 5 :
Split-by and Boundary Queries
Lecture 6 :
Fiield delimeters
Lecture 7 :
Incremental Appends
Lecture 8 :
Sqoop Hive Import
Lecture 9 :
Sqoop List Tables/Database
Lecture 10 :
Sqoop Import Practice 1
Lecture 11 :
Sqoop Import Practice 2
Lecture 12 :
Sqoop Import Practice 3
Lecture 13 :
Sqoop intro
Section 3 : Sqoop Export 2 Lectures 00:06:09
Lecture 1 :
Export from Hdfs to Mysql
Lecture 2 :
Export from Hive to Mysql
Section 4 : Apache Flume 8 Lectures 00:40:06
Lecture 1 :
Flume Introduction & Architecture
Lecture 2 :
Exec Source and Logger Sink
Lecture 3 :
Moving data from Twitter to HDFS
Lecture 4 :
Moving data from NetCat to HDFS
Lecture 5 :
Flume Interceptors
Lecture 6 :
Flume Interceptor Example
Lecture 7 :
Flume Multi-Agent Flow
Lecture 8 :
Fume Consolidation
Section 5 : Apache Hive 14 Lectures 01:05:37
Lecture 1 :
Hive Introduction
Lecture 2 :
Hive Database
Lecture 3 :
Hive Managed Tables
Lecture 4 :
Hive External Tables
Lecture 5 :
Hive Inserts
Lecture 6 :
Hive Analytics
Lecture 7 :
Hive Parquet
Lecture 8 :
Parquet Snappy
Lecture 9 :
Working with Fixed File Format
Lecture 10 :
Alter Command
Lecture 11 :
Hive String Functions
Lecture 12 :
Hive Partitioning
Lecture 13 :
Hive Bucketing
Lecture 14 :
Hive Date Functions
Section 6 : Spark Introduction 4 Lectures 00:23:35
Lecture 1 :
Spark Intro
Lecture 2 :
Resilient Distributed Dataset
Lecture 3 :
Cluster Overview
Lecture 4 :
DAG Stages
Section 7 : Spark Transformation Actions 10 Lectures 00:48:36
Lecture 1 :
Map
Lecture 2 :
Filter Intersect
Lecture 3 :
Union Distinct
Lecture 4 :
Group By Key
Lecture 5 :
Reduce By Key
Lecture 6 :
Sort By Key
Lecture 7 :
Map Partition
Lecture 8 :
Coleasce Repartition
Lecture 9 :
Join
Lecture 10 :
Spark Actions
Section 8 : Spark RDD Practice 6 Lectures 00:42:19
Lecture 1 :
Scala Tuples
Lecture 2 :
Extract Error
Lecture 3 :
Word Frequency
Lecture 4 :
Customer Orders
Lecture 5 :
City Population
Lecture 6 :
Movie Rating
Section 9 : Spark Dataframe SparkSQL 10 Lectures 00:42:36
Lecture 1 :
Dataframe Intro
Lecture 2 :
Working With Json
Lecture 3 :
Working With Parquet
Lecture 4 :
Working With CSV
Lecture 5 :
Working With Avro XML
Lecture 6 :
Compressions
Lecture 7 :
Dataframe API 1
Lecture 8 :
Dataframe API 2
Lecture 9 :
Spark SQL
Lecture 10 :
Hive Context

How do i access the course after purchase?
It's simple. When you sign up, you'll immediately have unlimited viewing of thousands of expert courses, paths to guide your learning, tools to measure your skills and hands-on resources like exercise files. There’s no limit on what you can learn and you can cancel at any time.
Are these video based online self-learning courses?
Yes. All of the courses comes with online video based lectures created by certified instructors. Instructors have crafted these courses with a blend of high quality interactive videos, lectures, quizzes & real world projects to give you an indepth knowledge about the topic.
Can i play & pause the course as per my convenience?
Yes absolutely & thats one of the advantage of self-paced courses. You can anytime pause or resume the course & come back & forth from one lecture to another lecture, play the videos mulitple times & so on.
How do i contact the instructor for any doubts or questions?
Most of these courses have general questions & answers already covered within the course lectures. However, if you need any further help from the instructor, you can use the inbuilt Chat with Instructor option to send a message to an instructor & they will reply you within 24 hours. You can ask as many questions as you want.
Do i need a pc to access the course or can i do it on mobile & tablet as well?
Brilliant question? Isn't it? You can access the courses on any device like PC, Mobile, Tablet & even on a smart tv. For mobile & a tablet you can download the Learnfly android or an iOS app. If mobile app is not available in your country, you can access the course directly by visting our website, its fully mobile friendly.
Do i get any certificate for the courses?
Yes. Once you complete any course on our platform along with provided assessments by the instructor, you will be eligble to get certificate of course completion.
For how long can i access my course on the platform?
You require an active subscription to access courses on our platform. If your subscription is active, you can access any course on our platform with no restrictions.
Is there any free trial?
Currently, we do not offer any free trial.
Can i cancel anytime?
Yes, you can cancel your subscription at any time. Your subscription will auto-renew until you cancel, but why would you want to?

Navdeep Kaur,

364784 Course Views

3 Courses

I am a big data professional with 11 years of industry experience in different technologies and domains. I have keen interest in providing training in new technologies. I have received CCA175 Hadoop and Spark developer certification and AWS solution architect certification. I love guiding people and helping them achieve new goals.