V tomto příspěvku ukáži instalaci hadoop ve verzi 2.9. Pokud si budete o hadoopu hledet informace, dávejte si pozor na verze. Mě se třeba stalo, že první výsledek při vyhledávání infomací o HDFS (Hadoop Distributed File System), mě odkázal na verzi 1 a chvíli trvalo, než jsem na to přišel.
Instalace je popsána na stránkách projektu. K aktuální stabilní verzi je zde. Tuto instalaci jsem zkoušel, ale ne příliš úspěšně a skončil jsem s chybou "Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.". Našel jsem ale web data-flair.training, kde mají velmi podrobný a polopatě vysvětlený postup instalace hadoopu (viz zde). Je zde vysvětleno vše od instalace Javy 8, konfigurace ssh až po spuštění hadoopu. Tutoriálu není co vytknout. Já zde ukáži jen závěrečné kroky.
Pokud jste provedli všechna nastavení konfiguračních souborů, je třeba namenode naformátovat (hdfs namenode -format)
vitfo@vitfo-VirtualBox:~/hadoop/hadoop-2.9.0$ hdfs namenode -format 18/01/26 13:18:17 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = vitfo-VirtualBox/127.0.1.1 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 2.9.0 STARTUP_MSG: classpath = /home/vitfo/hadoop/hadoop-2.9.0/etc/hadoop:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/gson-2.2.4.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/log4j-1.2.17.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/stax2-api-3.1.4.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/curator-client-2.7.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/junit-4.11.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/httpclient-4.5.2.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/jetty-sslengine-6.1.26.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/commons-cli-1.2.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/jersey-server-1.9.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/jersey-json-1.9.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/servlet-api-2.5.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/jersey-core-1.9.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/httpcore-4.4.4.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/commons-lang3-3.4.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/activation-1.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/curator-framework-2.7.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/jettison-1.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/hadoop-annotations-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/commons-collections-3.2.2.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/avro-1.7.7.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/woodstox-core-5.0.3.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/zookeeper-3.4.6.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/commons-io-2.4.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/asm-3.2.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/hadoop-auth-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/nimbus-jose-jwt-3.9.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/commons-codec-1.4.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/commons-net-3.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/jetty-6.1.26.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/paranamer-2.3.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/xz-1.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/jcip-annotations-1.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/snappy-java-1.0.5.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/guava-11.0.2.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/jsp-api-2.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/commons-digester-1.8.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/commons-lang-2.6.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/slf4j-api-1.7.25.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/jsch-0.1.54.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/jsr305-3.0.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/json-smart-1.1.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/xmlenc-0.52.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/hadoop-common-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/hadoop-nfs-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/common/hadoop-common-2.9.0-tests.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-core-2.7.8.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/okhttp-2.4.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/okio-1.4.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-databind-2.7.8.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-annotations-2.7.8.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/asm-3.2.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-native-client-2.9.0-tests.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-client-2.9.0-tests.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-native-client-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-2.9.0-tests.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-client-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/gson-2.2.4.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/fst-2.50.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/commons-math3-3.1.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/stax2-api-3.1.4.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/curator-client-2.7.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/httpclient-4.5.2.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/commons-configuration-1.6.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/jetty-sslengine-6.1.26.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/jets3t-0.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/curator-recipes-2.7.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/commons-math-2.2.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/guice-3.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/apacheds-i18n-2.0.0-M15.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/httpcore-4.4.4.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/commons-lang3-3.4.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/commons-beanutils-core-1.8.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/activation-1.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/curator-framework-2.7.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/jettison-1.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/curator-test-2.7.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/api-asn1-api-1.0.0-M20.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/json-io-2.5.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/javax.inject-1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/java-util-1.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/metrics-core-3.0.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/avro-1.7.7.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/api-util-1.0.0-M20.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/woodstox-core-5.0.3.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/java-xmlbuilder-0.4.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/asm-3.2.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/nimbus-jose-jwt-3.9.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/commons-net-3.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/commons-beanutils-1.7.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/jetty-6.1.26.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/paranamer-2.3.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/xz-1.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/jcip-annotations-1.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/snappy-java-1.0.5.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/jsp-api-2.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/commons-digester-1.8.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/jsch-0.1.54.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/json-smart-1.1.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/xmlenc-0.52.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/lib/htrace-core4-4.1.0-incubating.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-api-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-router-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-client-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-registry-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-common-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-common-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/hadoop-annotations-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/avro-1.7.7.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/lib/snappy-java-1.0.5.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.9.0-tests.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.0.jar:/home/vitfo/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.9.0.jar:/contrib/capacity-scheduler/*.jar STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r 756ebc8394e473ac25feac05fa493f6d612e6c50; compiled by 'arsuresh' on 2017-11-13T23:15Z STARTUP_MSG: java = 1.8.0_151 ************************************************************/ 18/01/26 13:18:17 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 18/01/26 13:18:17 INFO namenode.NameNode: createNameNode [-format] Formatting using clusterid: CID-58d73ffb-2447-4142-ab4e-6fe6769518dd 18/01/26 13:18:18 INFO namenode.FSEditLog: Edit logging is async:true 18/01/26 13:18:18 INFO namenode.FSNamesystem: KeyProvider: null 18/01/26 13:18:18 INFO namenode.FSNamesystem: fsLock is fair: true 18/01/26 13:18:18 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false 18/01/26 13:18:18 INFO namenode.FSNamesystem: fsOwner = vitfo (auth:SIMPLE) 18/01/26 13:18:18 INFO namenode.FSNamesystem: supergroup = supergroup 18/01/26 13:18:18 INFO namenode.FSNamesystem: isPermissionEnabled = true 18/01/26 13:18:18 INFO namenode.FSNamesystem: HA Enabled: false 18/01/26 13:18:18 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling 18/01/26 13:18:18 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000 18/01/26 13:18:18 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true 18/01/26 13:18:18 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000 18/01/26 13:18:18 INFO blockmanagement.BlockManager: The block deletion will start around 2018 Jan 26 13:18:18 18/01/26 13:18:18 INFO util.GSet: Computing capacity for map BlocksMap 18/01/26 13:18:18 INFO util.GSet: VM type = 64-bit 18/01/26 13:18:18 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB 18/01/26 13:18:18 INFO util.GSet: capacity = 2^21 = 2097152 entries 18/01/26 13:18:18 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false 18/01/26 13:18:18 WARN conf.Configuration: No unit for dfs.namenode.safemode.extension(30000) assuming MILLISECONDS 18/01/26 13:18:18 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.9990000128746033 18/01/26 13:18:18 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0 18/01/26 13:18:18 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000 18/01/26 13:18:18 INFO blockmanagement.BlockManager: defaultReplication = 1 18/01/26 13:18:18 INFO blockmanagement.BlockManager: maxReplication = 512 18/01/26 13:18:18 INFO blockmanagement.BlockManager: minReplication = 1 18/01/26 13:18:18 INFO blockmanagement.BlockManager: maxReplicationStreams = 2 18/01/26 13:18:18 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000 18/01/26 13:18:18 INFO blockmanagement.BlockManager: encryptDataTransfer = false 18/01/26 13:18:18 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000 18/01/26 13:18:18 INFO namenode.FSNamesystem: Append Enabled: true 18/01/26 13:18:19 INFO util.GSet: Computing capacity for map INodeMap 18/01/26 13:18:19 INFO util.GSet: VM type = 64-bit 18/01/26 13:18:19 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB 18/01/26 13:18:19 INFO util.GSet: capacity = 2^20 = 1048576 entries 18/01/26 13:18:19 INFO namenode.FSDirectory: ACLs enabled? false 18/01/26 13:18:19 INFO namenode.FSDirectory: XAttrs enabled? true 18/01/26 13:18:19 INFO namenode.NameNode: Caching file names occurring more than 10 times 18/01/26 13:18:19 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: falseskipCaptureAccessTimeOnlyChange: false 18/01/26 13:18:19 INFO util.GSet: Computing capacity for map cachedBlocks 18/01/26 13:18:19 INFO util.GSet: VM type = 64-bit 18/01/26 13:18:19 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB 18/01/26 13:18:19 INFO util.GSet: capacity = 2^18 = 262144 entries 18/01/26 13:18:19 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10 18/01/26 13:18:19 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10 18/01/26 13:18:19 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25 18/01/26 13:18:19 INFO namenode.FSNamesystem: Retry cache on namenode is enabled 18/01/26 13:18:19 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis 18/01/26 13:18:19 INFO util.GSet: Computing capacity for map NameNodeRetryCache 18/01/26 13:18:19 INFO util.GSet: VM type = 64-bit 18/01/26 13:18:19 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB 18/01/26 13:18:19 INFO util.GSet: capacity = 2^15 = 32768 entries 18/01/26 13:18:19 INFO namenode.FSImage: Allocated new BlockPoolId: BP-284323615-127.0.1.1-1516969099775 18/01/26 13:18:19 INFO common.Storage: Storage directory /home/vitfo/hadoop/hdata/dfs/name has been successfully formatted. 18/01/26 13:18:19 INFO namenode.FSImageFormatProtobuf: Saving image file /home/vitfo/hadoop/hdata/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression 18/01/26 13:18:20 INFO namenode.FSImageFormatProtobuf: Image file /home/vitfo/hadoop/hdata/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 322 bytes saved in 0 seconds. 18/01/26 13:18:20 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 18/01/26 13:18:20 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at vitfo-VirtualBox/127.0.1.1 ************************************************************/
Vypuštění 🙂 hdfs a yarn démonů.
vitfo@vitfo-VirtualBox:~/hadoop/hadoop-2.9.0$ start-dfs.sh Starting namenodes on [localhost] localhost: starting namenode, logging to /home/vitfo/hadoop/hadoop-2.9.0/logs/hadoop-vitfo-namenode-vitfo-VirtualBox.out localhost: starting datanode, logging to /home/vitfo/hadoop/hadoop-2.9.0/logs/hadoop-vitfo-datanode-vitfo-VirtualBox.out Starting secondary namenodes [0.0.0.0] 0.0.0.0: starting secondarynamenode, logging to /home/vitfo/hadoop/hadoop-2.9.0/logs/hadoop-vitfo-secondarynamenode-vitfo-VirtualBox.out vitfo@vitfo-VirtualBox:~/hadoop/hadoop-2.9.0$ start-yarn.sh starting yarn daemons starting resourcemanager, logging to /home/vitfo/hadoop/hadoop-2.9.0/logs/yarn-vitfo-resourcemanager-vitfo-VirtualBox.out localhost: starting nodemanager, logging to /home/vitfo/hadoop/hadoop-2.9.0/logs/yarn-vitfo-nodemanager-vitfo-VirtualBox.out
Kontrola služeb. Jps je Java Virtual Machine Process Status Tool. Hadoop používá Javu (hadoop je v Javě). Jps zobrazí seznam java procesů uživatele. V pseudo distribuovaném módu (to je ten způsob uvedený výše, kdy vše běží na jediném stroji) běži každý démon ve vlastní JVM a jsp je zobrazí. Je možné použít přepínač -l (dlouhý výpis)
vitfo@vitfo-VirtualBox:~/hadoop/hadoop-2.9.0$ jps 3056 DataNode 3412 ResourceManager 3525 NodeManager 3256 SecondaryNameNode 3834 Jps 2941 NameNode
Hadoop nabízí i webové rozhraní pro zobrazení dat o HDFS na localhost:50070.
Zastavení yarnu a dfs
vitfo@vitfo-VirtualBox:~/hadoop/hadoop-2.9.0$ stop-yarn.sh stopping yarn daemons stopping resourcemanager localhost: stopping nodemanager localhost: nodemanager did not stop gracefully after 5 seconds: killing with kill -9 no proxyserver to stop vitfo@vitfo-VirtualBox:~/hadoop/hadoop-2.9.0$ stop-dfs.sh Stopping namenodes on [localhost] localhost: stopping namenode localhost: stopping datanode Stopping secondary namenodes [0.0.0.0] 0.0.0.0: stopping secondarynamenode