Pig Interpreter for Apache Zeppelin
Overview
Apache Pig is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. The salient property of Pig programs is that their structure is amenable to substantial parallelization, which in turns enables them to handle very large data sets.
Supported interpreter type
%pig.script
(default)All the pig script can run in this type of interpreter, and display type is plain text.
%pig.query
Almost the same as
%pig.script
. The only difference is that you don't need to add alias in the last statement. And the display type is table.
Supported runtime mode
- Local
- MapReduce
- Tez_Local (Only Tez 0.7 is supported)
- Tez (Only Tez 0.7 is supported)
How to use
How to setup Pig
Local Mode
Nothing needs to be done for local mode
MapReduce Mode
HADOOP_CONF_DIR needs to be specified in
ZEPPELIN_HOME/conf/zeppelin-env.sh
.Tez Local Mode
Nothing needs to be done for tez local mode
Tez Mode
HADOOP_CONF_DIR and TEZ_CONF_DIR needs to be specified in
ZEPPELIN_HOME/conf/zeppelin-env.sh
.
How to configure interpreter
At the Interpreters menu, you have to create a new Pig interpreter. Pig interpreter has below properties by default. And you can set any pig properties here which will be passed to pig engine. (like tez.queue.name & mapred.job.queue.name). Besides, we use paragraph title as job name if it exists, else use the last line of pig script. So you can use that to find app running in YARN RM UI.
Property | Default | Description |
---|---|---|
zeppelin.pig.execType | mapreduce | Execution mode for pig runtime. local | mapreduce | tez_local | tez |
zeppelin.pig.includeJobStats | false | whether display jobStats info in %pig.script |
zeppelin.pig.maxResult | 1000 | max row number displayed in %pig.query |
tez.queue.name | default | queue name for tez engine |
mapred.job.queue.name | default | queue name for mapreduce engine |
Example
pig
%pig
raw_data = load 'dataset/sf_crime/train.csv' using PigStorage(',') as (Dates,Category,Descript,DayOfWeek,PdDistrict,Resolution,Address,X,Y);
b = group raw_data all;
c = foreach b generate COUNT($1);
dump c;
pig.query
%pig.query
b = foreach raw_data generate Category;
c = group b by Category;
foreach c generate group as category, COUNT($1) as count;
Data is shared between %pig
and %pig.query
, so that you can do some common work in %pig
, and do different kinds of query based on the data of %pig
.
Besides, we recommend you to specify alias explicitly so that the visualization can display the column name correctly. Here, we name COUNT($1)
as count
, if you don't do this,
then we will name it using position, here we will use col_1
to represent COUNT($1)
if you don't specify alias for it. There's one pig tutorial note in zeppelin for your reference.