Step 1 - Clone the Falcon repository
$git clone https://git-wip-us.apache.org/repos/asf/falcon.git falcon
Step 2 - Build Falcon
$cd falcon $export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m -noverify" && mvn clean install
It builds and installs the package into the local repository, for use as a dependency in other projects locally.
[optionally -Dhadoop.version=<<hadoop.version>> can be appended to build for a specific version of Hadoop]
NOTE: Falcon drops support for Hadoop-1 and only supports Hadoop-2 from Falcon 0.6 onwards [optionally -Doozie.version=<<oozie version>> can be appended to build with a specific version of Oozie. Oozie versions >= 4 are supported] NOTE: Falcon builds with JDK 1.7/1.8 using -noverify option To compile Falcon with Hive Replication, optionally "-P hadoop-2,hivedr" can be appended. For this Hive >= 1.2.0 and Oozie >= 4.2.0 should be available.
Step 3 - Package and Deploy Falcon
Once the build successfully completes, artifacts can be packaged for deployment using the assembly plugin. The Assembly Plugin for Maven is primarily intended to allow users to aggregate the project output along with its dependencies, modules, site documentation, and other files into a single distributable archive. There are two basic ways in which you can deploy Falcon - Embedded mode(also known as Stand Alone Mode) and Distributed mode. Your next steps will vary based on the mode in which you want to deploy Falcon.
NOTE : Oozie is being extended by Falcon (particularly on el-extensions) and hence the need for Falcon to build & re-package Oozie, so that users of Falcon can work with the right Oozie setup. Though Oozie is packaged by Falcon, it needs to be deployed separately by the administrator and is not auto deployed along with Falcon.
Embedded/Stand Alone Mode
Embedded mode is useful when the Hadoop jobs and relevant data processing involve only one Hadoop cluster. In this mode there is a single Falcon server that contacts the scheduler to schedule jobs on Hadoop. All the process/feed requests like submit, schedule, suspend, kill etc. are sent to this server. For running Falcon in this mode one should use the Falcon which has been built using standalone option. You can find the instructions for Embedded mode setup here.
Distributed Mode
Distributed mode is for multiple (colos) instances of Hadoop clusters, and multiple workflow schedulers to handle them. In this mode Falcon has 2 components: Prism and Server(s). Both Prism and Server(s) have their own their own config locations(startup and runtime properties). In this mode Prism acts as a contact point for Falcon servers. While all commands are available through Prism, only read and instance api's are available through Server. You can find the instructions for Distributed Mode setup here.
Preparing Oozie and Falcon packages for deployment
$cd <<project home>> $src/bin/package.sh <<hadoop-version>> <<oozie-version>> >> ex. src/bin/package.sh 1.1.2 4.0.1 or src/bin/package.sh 0.20.2-cdh3u5 4.0.1 >> ex. src/bin/package.sh 2.5.0 4.0.0 >> Falcon package is available in <<falcon home>>/target/apache-falcon-<<version>>-bin.tar.gz >> Oozie package is available in <<falcon home>>/target/oozie-4.0.1-distro.tar.gz
NOTE: If you have a separate Apache Oozie installation, you will need to follow some additional steps:
- Once you have setup the Falcon Server, copy libraries under {falcon-server-dir}/oozie/libext/ to {oozie-install-dir}/libext.
- Modify Oozie's configuration file. Copy all Falcon related properties from {falcon-server-dir}/oozie/conf/oozie-site.xml to {oozie-install-dir}/conf/oozie-site.xml
- Restart oozie:
- cd {oozie-install-dir}
- sudo -u oozie ./bin/oozie-stop.sh
- sudo -u oozie ./bin/oozie-setup.sh prepare-war
- sudo -u oozie ./bin/oozie-start.sh