Command Line Data Analysis

58mAdvanced2023-04-27

Authors

Miki Tebeka

CEO at 353Solutions

Course details

This course is integrated with GitHub Codespaces, an instant cloud developer environment that offers all the functionality of your favorite IDE without the need for any local machine setup. With GitHub Codespaces, you can get hands-on practice from any machine, at any time—all while using a tool that you’ll likely encounter in the workplace. Check out the “Using GitHub Codespaces with this course” video to learn how to get started.

In this course, Miki Tebeka presents the information and instructions you need to perform data analysis using the many tools that the command line offers for quick data analysis and visualization. Learn how to use the command line and command line tools for analyzing and displaying data. Go over Unix shell basics, and then use tools like grep, jq, sed, awk, curl, gnuplot, and others to analyze and display data from various sources, such as CSV files, APIs, databases, and more.

Skills covered

UnixHelp Desk SkillsIT Help DeskWeb Development ToolsData AnalysisWeb DevelopmentData ScienceBusiness Analysis and StrategyBusiness Software and ToolsOpen SourceDeep Dive (X:Y)

Concepts

Introduction

Command-line data analysis
What you should know
Setting up

The Command Line

The shell
Running programs
Shell expansion
Pipes and redirects
Files and directories
Installing commands

Data Ingestion

Cat and less
curl
SCP
gsutil and AWS CLI
Databases
Challenge - ETL
Solution - ETL

Selecting Data

head, tail, and shuf
grep and regular expressions
cut
jq
Databases
Challenge - Station status
Solution - Station status

Data Transformations

tr
sed
AWK
Challenge - Normalized items
Solution - Normalized items

Calculations

sort and uniq
bc
AWK
Challenge - Quantity sold
Solution - Quantity sold

Visualization

printf
gnuplot - Introduction
gnuplot - Box plots
gnuplot - Bar charts
Challenge - Weather
Solution - Weather

Conclusion

Next steps