Alluxio Interpreter for Apache Zeppelin
Overview
Alluxio is a memory-centric distributed storage system enabling reliable data sharing at memory-speed across cluster frameworks.
Configuration
| Name | Class | Description |
|---|---|---|
| alluxio.master.hostname | localhost | Alluxio master hostname |
| alluxio.master.port | 19998 | Alluxio master port |
Enabling Alluxio Interpreter
In a notebook, to enable the Alluxio interpreter, click on the Gear icon and select Alluxio.
Using the Alluxio Interpreter
In a paragraph, use %alluxio to select the Alluxio interpreter and then input all commands.
%alluxio
help
Tip : Use ( Ctrl + . ) for autocompletion.
Interpreter Commands
The Alluxio interpreter accepts the following commands.
| Operation | Syntax | Description |
|---|---|---|
| cat | cat "path" | Print the content of the file to the console. |
| chgrp | chgrp "group" "path" | Change the group of the directory or file. |
| chmod | chmod "permission" "path" | Change the permission of the directory or file. |
| chown | chown "owner" "path" | Change the owner of the directory or file. |
| copyFromLocal | copyFromLocal "source path" "remote path" | Copy the specified file specified by "source path" to the path specified by "remote path". This command will fail if "remote path" already exists. |
| copyToLocal | copyToLocal "remote path" "local path" | Copy the specified file from the path specified by "remote path" to a local destination. |
| count | count "path" | Display the number of folders and files matching the specified prefix in "path". |
| du | du "path" | Display the size of a file or a directory specified by the input path. |
| fileInfo | fileInfo "path" | Print the information of the blocks of a specified file. |
| free | free "path" | Free a file or all files under a directory from Alluxio. If the file/directory is also in under storage, it will still be available there. |
| getCapacityBytes | getCapacityBytes | Get the capacity of the AlluxioFS. |
| getUsedBytes | getUsedBytes | Get number of bytes used in the AlluxioFS. |
| load | load "path" | Load the data of a file or a directory from under storage into Alluxio. |
| loadMetadata | loadMetadata "path" | Load the metadata of a file or a directory from under storage into Alluxio. |
| location | location "path" | Display a list of hosts that have the file data. |
| ls | ls "path" | List all the files and directories directly under the given path with information such as size. |
| mkdir | mkdir "path1" ... "pathn" | Create directory(ies) under the given paths, along with any necessary parent directories. Multiple paths separated by spaces or tabs. This command will fail if any of the given paths already exist. |
| mount | mount "path" "uri" | Mount the underlying file system path "uri" into the Alluxio namespace as "path". The "path" is assumed not to exist and is created by the operation. No data or metadata is loaded from under storage into Alluxio. After a path is mounted, operations on objects under the mounted path are mirror to the mounted under storage. |
| mv | mv "source" "destination" | Move a file or directory specified by "source" to a new location "destination". This command will fail if "destination" already exists. |
| persist | persist "path" | Persist a file or directory currently stored only in Alluxio to the underlying file system. |
| pin | pin "path" | Pin the given file to avoid evicting it from memory. If the given path is a directory, it recursively pins all the files contained and any new files created within this directory. |
| report | report "path" | Report to the master that a file is lost. |
| rm | rm "path" | Remove a file. This command will fail if the given path is a directory rather than a file. |
| setTtl | setTtl "time" | Set the TTL (time to live) in milliseconds to a file. |
| tail | tail "path" | Print the last 1KB of the specified file to the console. |
| touch | touch "path" | Create a 0-byte file at the specified location. |
| unmount | unmount "path" | Unmount the underlying file system path mounted in the Alluxio namespace as "path". Alluxio objects under "path" are removed from Alluxio, but they still exist in the previously mounted under storage. |
| unpin | unpin "path" | Unpin the given file to allow Alluxio to evict this file again. If the given path is a directory, it recursively unpins all files contained and any new files created within this directory. |
| unsetTtl | unsetTtl | Remove the TTL (time to live) setting from a file. |
How to test it's working
Be sure to have configured correctly the Alluxio interpreter, then open a new paragraph and type one of the above commands.
Below a simple example to show how to interact with Alluxio interpreter. Following steps are performed:
- using sh interpreter a new text file is created on local machine
- using Alluxio interpreter:
- is listed the content of the afs (Alluxio File System) root
- the file previously created is copied to afs
- is listed again the content of the afs root to check the existence of the new copied file
- is showed the content of the copied file (using the tail command)
- the file previously copied to afs is copied to local machine
- using sh interpreter it's checked the existence of the new file copied from Alluxio and its content is showed
