1 pig는 언제 쓰는게 좋을까?

2 샘플 데이터 얻어서 HDFS에 넣기

3 pig 스크립트 실행과 다중 쿼리 실행과 store

4 스키마를 가지지 않는 릴레이션의 위치 참조

5 sample data input

6 참고자료

• ETL 할 때
  
• 데이터 흐름 언어이므로 사용자들이 데이터 분석 할 때 --> 어려울까? ㅇㅇ 어렵겠다.
  
• 음.. 암턴.. 좋은거 같음..
  
  

wget http://hanb.co.kr/exam/1746/htdg-examples-0.1.1.tar.gz
tar htdg-examples-0.1.1.tar.gz
mv htdg-examples-0.1.1 samples
cd samples
hadoop fs -put input input

--store_test.pig
rmf output/1949
rmf output/1950

A = LOAD 'input/ncdc/micro-tab/sample.txt' AS (year:chararray, temperature:int, quality:int);
B = FILTER A BY year == '1949';
C = FILTER A BY year == '1950';
STORE B INTO 'output/1949';
STORE C INTO 'output/1950';

cat output/1949/part-m-00000
cat output/1950/part-m-00000

pig store_test.pig

grunt> exec store_test.pig

또는

grunt> run store_test.pig

B = FOREACH A GENERATE $0; --$0은 릴레이션A의 첫번째 어트리뷰트를 참조함을 의미
DUMP B;

wget http://databaser.net/moniwiki/pds/Hive_ec_98_88_ec_a0_9c_ed_8c_8c_ec_9d_bc/data.zip
unzip data.zip
hadoop fs -mkdir scott
hadoop fs -put dept.csv scott
hadoop fs -put emp.csv scott
hadoop fs -put salgrade.csv scott
pig

emp = load 'scott/emp.csv' using PigStorage(',') as (empno, ename, job, mgr, hiredate, sal, comm, deptno:int);
grouped = group emp by deptno;
total = foreach grouped generate group, SUM(emp.sal) as total_sal;
--total = foreach grouped generate emp.depno, SUM(emp.sal) as total_sal;

dept = load 'scott/dept.csv' using PigStorage(',') as (dname, loc, deptno:int);
join_data = join total by group left, dept by deptno;
view = foreach join_data generate $0, $3, $1;
dump view;

emp = load 'scott/emp.csv' using PigStorage(',') as (empno, ename, job, mgr, hiredate, sal, comm, deptno:int);
emp = foreach emp generate ename, sal;
filtered_set = filter emp by sal >= 2000;
sorted_set = order filtered_set by sal desc;
top3 = limit sorted_set 3;
dump top3;

• https://cwiki.apache.org/confluence/display/PIG/PigTutorial
  
• http://julingks.tistory.com/143 --> 한들 메뉴얼 ver 0.7
  
• Pig Latin Reference Manual 2
  
• Programming Pig 다운로드
  
• DataFu for Pig and Hadoop
  
• https://github.com/linkedin/datafu