Introduction
Protocols are standardized rules that govern how data is transmitted, formatted, and processed across systems.
In Big Data, protocols are essential for:
- Data ingestion (getting data in)
- Inter-node communication in clusters
- Remote access to APIs/services
- Serialization of structured data
- Security and authorization
| Protocol | Layer | Use Case Example |
|---|---|---|
| HTTP/HTTPS | Application | REST API for ingesting external data |
| Kafka | Messaging | Stream processing with Spark or Flink |
| gRPC | RPC | Microservices in ML model serving |
| MQTT | Messaging | IoT data push to cloud |
| Avro/Proto | Serialization | Binary data for logs and schema |
| OAuth/Kerberos | Security | Secure access to data lakes |