• Understanding the basics of big data and its characteristics
  • Definition of big data
  • Sources of big data
  • Types of big data
  • Big data processing technologies
  • Big data analytics
  • Challenges of big data
  • Applications of big data:
• Exploring the challenges of storing, processing, and analyzing big data
• Introduction to big data technologies and tools

• Understanding different types of data storage systems
• Relational and non-relational databases

Hadoop:

• Introduction
• Configuring HDFS parameters and tuning its performance
• Performing file read and write operations on HDFS
• Managing block replication and storage space
• Implementing HDFS security and access control policies
•  Setting up a Hadoop cluster and HDFS
• Configuring HDFS parameters and tuning its performance
• Performing file read and write operations on HDFS
• Managing block replication and storage space
• Implementing HDFS security and access control policies
• HDFS Overview
• HDFS Operations
• Command Reference
• MapReduce
• Streaming
•  Multi-Node Cluster

HIVE

• What Is Hive?
• Data Units, Type System, Built In Operators and Functions, Creating, Showing,   Altering, and Dropping Tables
• Loading Data, Querying, and Inserting Data
• Hive SQL Language Manual:  Commands, CLIs, Data Types,
• File Formats and Compression:  RCFile, Avro, ORC, Parquet; Compression, LZO
• Procedural Language:  Hive HPL/SQL
• Hive Web Interface
• Hive SerDes:  Avro SerDe, Parquet SerDe, CSV SerDe, JSON SerDe   
• Hive Accumulo Integration
• Using TiDB as the Hive Metastore database
• Installing Hive
• Configuring Hive
• Setting Up Metastore
• Hive Schema Tool
• Setting Up Hive Web Interface
• Setting Up Hive Server (JDBC, ODBC, Thrift, HiveServer2)
• Hive Replication
• Hive on Spark: Getting Started

Spark(3.3.2)

• Quick Start: a quick introduction to the Spark API; start here!
• RDD Programming: an overview of Spark basics – RDDs (core but old API)
  • Spark SQL, Datasets, and DataFrames: processing structured data with relational queries (newer API than RDDs)
• Structured Streaming: processing structured data streams with related queries .
• Spark Streaming: processing data streams using DStreams (old API)
• MLlib: applying machine learning algorithms
• PySpark: processing data with Spark in Python
• Spark SQL CLI: processing data with SQL on the command line.

HBase

• Introduction
• Apache HBase Configuration: Standalone 
• Managing HBase
• HBase Replication
• HBase High Availability
• Troubleshooting HBase
• Upstream Information for HBase
• Creating HBase tables and performing CRUD operations
• Using HBase shell commands and API operations
• Implementing schema design and modeling in HBase
• Processing HBase data with MapReduce and Apache Spark
• Integrating HBase with other Hadoop ecosystem components like HDFS and yarn
• Tuning HBase performance and optimizing HBase queries
• Implementing security and access control policies in HBase

Map-Reduce

• User Interfaces
• Payload, Mapper, Reducer, Partitioner, Counter
• Job Configuration
• Task Execution & Environment
• Memory Management
• Map Parameters
• Shuffle/Reduce Parameters
• Configured Parameters
• Task Logs
• Distributing Libraries
• Job Submission and Monitoring, Job Control, Job Input
• InputSplit
• RecordReader, Job Output
• OutputCommitter
• Task Side-Effect Files
• RecordWriter
• Other Useful Features
• Submitting Jobs to Queues
• Counters
• DistributedCache
• Profiling
• Debugging
• Data Compression
• Skipping Bad Records

Sqoop

• Installation
• Import
• Import-All-Tables
• Export
• Sqoop Job
• Codegen
• Eval
• List Databases
• List Tables

NoSQL 

• What is a NoSQL Database?
• Brief History of NoSQL Databases
• NoSQL Database Features
• Types of NoSQL Database
• Difference between RDBMS and NoSQL
• Why NoSQL?
• When should NoSQL be Used?
• NoSQL Database Misconceptions
• NoSQL Query
• NoSQL clusters
• NoSQL data modeling
•  NoSQL databases like Hadoop MongoDB Cassandra

MongoDB 

• Setting up a MongoDB instance and configuring it
• Create an Atlas Free Tier Cluster
• Creating databases and collections in MongoDB
• Documents
• MongoDB Query API
• BSON Types
• Installation
• MongoDB Shell (mongosh)
• Performing CRUD operations and queries in MongoDB
• Implementing data modeling and schema design in MongoDB
• Using indexing and performance optimization techniques in MongoDB
• Implementing data aggregation and map-reduce operations in MongoDB
• Setting up and configuring MongoDB sharding and replication
• Implementing security and access control policies in MongoDB
• Using MongoDB drivers for various programming languages
• Administering and monitoring MongoDB instances
• Replication, Sharding, Change Streams, Time Series, Transactions, Administration, Storage.

Yarn

• Getting Started
• The Yarn Workflow
• CLI Commands
• Migrating from npm client
• Creating a Package
• Dependencies & Versions
• Configuration
• Offline Mirror
• Workspaces
• Plug’n’Play

PIG:

• Introduction
• Environment
• Installation
• Execution
• Grunt Shell
• Basics
• Load & Store Operators
• Reading Data
• Storing Data
• Describe Operator
• Union Operator
• Split Operator
• Filter Operator
• Distinct Operator
• Foreach Operator
• Order By
• Limit Operator
• Eval Functions
• Bag & Tuple Functions
• String Functions

Impala:

• Concepts and Architecture
• Deployment Planning
• Administration
• SQL
• Resource Management
• Performance Tuning
• Scalability Considerations
• Partitioning
• File Formats
• Using Impala to Query Kudu Tables
• HBase Tables
• S3 Tables
• ADLS Tables
• Logging
• Impala Client Access
• Troubleshooting Impala

AWS

• Services: EC2, EMR, S3, Redshift, Athena, AWS ECS

• Data processing using Apache Spark and MapReduce
  • Data analysis using R and Python
  • Data visualization using Tableau and other tools
  • Connecting to and extracting data from various Big Data sources
  • Transforming and cleaning data in preparation for analysis in Tableau
  • Creating interactive dashboards and visualizations in Tableau for Big Data
  • Optimising Tableau performance for large-scale data analysis
  • Collaborating and sharing Tableau dashboards with others
  • Integrating Tableau with other Big Data tools and technologies
• Batch processing vs. stream processing
• Data processing using Apache Flink

• Understanding big data infrastructure and architecture
• Cluster computing with Hadoop and Apache Spark
• Containerization with Docker and Kubernetes
• Infrastructure as a service (IaaS) and platform as a service (PaaS) on the cloud

• Apache Kafka for real-time data streaming
  • Introduction to Apache Kafka
  • Setting up a Kafka Cluster
  • Producers and Consumers
  • Kafka APIs
  • Configuration 
  • Connectors and Kafka Connect
  • Replication
  • Monitoring and Operations
  • OPERATIONS
  • Security

• Splunk Introduction:
  • Introduction to search language
  • Overview of Splunk and its capabilities
  • Understanding Splunk data models
  • Splunk Architecture
• Installation and Configuration:
  • Installing and configuring Splunk
  • Splunk configuration files and their purpose
  • Deployment Server configuration
  • Managing Splunk instances
  • Upgrading Splunk instances
  • Best practices for Splunk deployment
• Data Inputs:
  • Understanding various data input sources
  • Configuring data inputs
  • Parsing and indexing data
• Searching and Reporting:
  • Search fundamentals
  • Creating search queries
  • Visualizing search results
  • Understanding reporting options
  • Creating reports and dashboards
• Splunk Administration:
  • User management and authentication
  • Configuring roles and permissions
  • Monitoring and troubleshooting Splunk instances
  • Backup and recovery of Splunk instances
• Splunk Apps and Add-ons:
  • Understanding Splunk apps and add-ons
  • Installing and configuring apps and add-ons
  • Managing app and add-on configurations
• Monitoring and Alerting:
  • Setting up monitoring and alerting
  • Creating alerts and triggers
  • Configuring notifications
• Advanced Topics:
  • Advanced search techniques
  • Performance tuning and optimization
  • Advanced deployment and scalability
  • Integrating Splunk with other systems

• Working on a real-world project in big data
• Applying the knowledge and skills learned in the course
• Presenting the project work to the class