HDFS File System Interpreter for Apache Zeppelin
Overview
Hadoop File System is a distributed, fault tolerant file system part of the hadoop project and is often used as storage for distributed processing engines like Hadoop MapReduce and Apache Spark or underlying file systems like Alluxio.
Configuration
| Property | Default | Description |
|---|---|---|
| hdfs.url | http://localhost:50070/webhdfs/v1/ | The URL for WebHDFS |
| hdfs.user | hdfs | The WebHDFS user |
| hdfs.maxlength | 1000 | Maximum number of lines of results fetched |
This interpreter connects to HDFS using the HTTP WebHDFS interface.
It supports the basic shell file commands applied to HDFS, it currently only supports browsing.
- You can use ls [PATH] and ls -l [PATH] to list a directory. If the path is missing, then the current directory is listed. ls supports a -h flag for human readable file sizes.
- You can use cd [PATH] to change your current directory by giving a relative or an absolute path.
- You can invoke pwd to see your current directory.
Tip : Use ( Ctrl + . ) for autocompletion.
Create Interpreter
In a notebook, to enable the HDFS interpreter, click the Gear icon and select HDFS.
WebHDFS REST API
You can confirm that you're able to access the WebHDFS API by running a curl command against the WebHDFS end point provided to the interpreter.
Here is an example:
$> curl "http://localhost:50070/webhdfs/v1/?op=LISTSTATUS"