goal
|------ webserver -| |-----hadoop namenode -----|
| 192.168.0.230 -| | hdfs://192.168.0.230:9000 |
* upload file from webserver to hadoop hdfs .
* development language java
* kerberos authenication
ref
https://community.hortonworks.com/articles/56702/a-secure-hdfs-client-example.html
주제 Authentication with Kerberos
구글 검색어 : hadoop hdfs java client example
이해 해야 할 문서
https://community.hortonworks.com/articles/56702/a-secure-hdfs-client-example.html
읽어 봤는데 잘 모르겠다. 이해 할때까지 부족한 부분을 채우자.
<h2>kerberos 란? </h2>
Oracle Solaris 관리: 보안 서비스 - Kerberos 서비스
https://docs.oracle.com/cd/E26925_01/html/E25888/seamtm-1.html#scrolltoc
<h2>hdfs 코드 자체는 간단하다.</h2>
<code>
Configuration conf = new Configuration();
conf.set("fs.defaultFS","hdfs://one.hdp:8020");
FileSystem fs = FileSystem.get(conf);
FileStatus[] fsStatus = fs.listStatus(new Path("/"));
for(int i = 0; i < fsStatus.length; i++){
System.out.println(fsStatus[i].getPath().toString());
}
</code>
하둡도 내가 설치 한게 아니고, conf.set("fs.defaultFS","hdfs://one.hdp:8020"); 여기에서 one.hdp:8020 는 멀로 대체 해야 하나?
하둡 설치 하는 문서를 보면 제일처음 설정하는 파일중에 core-site.xml 이 있다. configuration 노드 아래 property 아래 name 가 fs.default.name 의 value 를
적으면 된다.
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop-1.0.0/hadoop-${user.name}</value>
</property>
</configuration>
http://blog.acronym.co.kr/332
이친구 정리 잘해놨네, 설치부터 이클립스 플러그인 및 ant 이것 저것 나중에 한번 방문 하기로 하고
<section>
<article>
<h2>일단 java에서 kerberos 인증을 하려면 jaas (Java Authentication and Authorization Service) 가 필요하다.</h2>
</article>
GSS-API/Kerberos v5 Authentication 요걸 사용 한다고 써져 있는데, http://docs.oracle.com/javase/jndi/tutorial/ldap/security/gssapi.html 보면 된다.
위치먼저 보니 ldap security gssapi 다.
이 문서에서 말하는 포인트는
To use the GSS-API SASL mechanism, you must do the following.
1.Authenticate to Kerberos.
2.Assume the identity of the authenticated principal.
3.When creating the initial context, set the Context.SECURITY_AUTHENTICATION(in the API reference documentation) environment property to the string "GSSAPI".
In a standalone application, you would have to perform all three steps.
However, in an applet environment or other container environments, the first two steps might be performed for you already
if you are accessing a directory service that uses the same Kerberos service as the container. This section shows how to perform all three steps.
요고다. 코드까지 자세히 설명 하고 있는데, 스탠드얼론으로 만들려면 3가지다 해야 하고, an applet environment or other container environments 이라면 1,2 는 할 필요 없넹,
you are accessing a directory service that uses the same Kerberos service as the container.
웹로직이나 톰캣은 jndi 나 구동하는 유저만으로 될려나?
https://ko.wikipedia.org/wiki/JNDI 보니깐 웹서버에서 먼가 설정만 하면 , 컨테이너 안에서는 그냥 쓸수 있을것 같은 예감이 든다.
jass 관련자료
<ul>
<li> 위키페디아 jaas
https://en.wikipedia.org/wiki/Java_Authentication_and_Authorization_Service
<li> 오라클 jaas 레퍼런스 가이드
http://docs.oracle.com/javase/6/docs/technotes/guides/security/jaas/JAASRefGuide.html</li>
</ul>
대략 감은 잡히는데,
</section>
2.
http://stackoverflow.com/questions/12783968/copying-directory-from-local-system-to-hdfs-java-code
Configuration conf = new Configuration();
conf.addResource(new Path("/home/user/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/home/user/hadoop/conf/hdfs-site.xml"));
FileSystem fs = FileSystem.get(conf);
fs.copyFromLocalFile(new Path("/home/user/directory/"),
new Path("/user/hadoop/dir"));
http://hadoopinrealworld.com/writing-a-file-to-hdfs-java-program/
kerberos 인증을 빼고 업로드 소스만 있는데, 테스트머신에서 계정 자체가 hadoop fs -copyFromLocal 을 실행 할 수 있으면 그냥 된다는 이야기인가?
누군가는 5년전에 끝낸 문제다.
webhdfs-java-client
Hadoop WebHDFS REST API's java client code with kerberos auth.
https://github.com/zxs/webhdfs-java-client
클라우데라 설명 Authenticating Kerberos Principals in Java Code
https://www.cloudera.com/documentation/enterprise/5-2-x/topics/cm_sg_principal_keytab.html#topic_2
ibm 자료
Accessing HDFS and MapReduce with Kerberos authenticatio
https://www.ibm.com/support/knowledgecenter/en/SSPT3X_3.0.0/com.ibm.swg.im.infosphere.biginsights.admin.doc/doc/kerberos_mapreduce.html
Configuring a Java client for Kerberos authentication
https://www.ibm.com/support/knowledgecenter/en/SSAW57_7.0.0/com.ibm.websphere.nd.doc/info/ae/ae/tsec_kerb_auth_client.html
An example of the kinit command is as follows. In this example, a Kerberos ticket-granting ticket (TGT) can be obtained for duke, and the Kerberos credential cache is stored in the default location, c:\Documents and Settings\duke\krb5cc_duke:
kinit duke@JAVA.SUN.COM
https://linuxjunkies.wordpress.com/2011/11/21/a-hdfsclient-for-hadoop-using-the-native-java-api-a-tutorial/
/etc/hadoop/conf/
/etc/hadoop/conf.cloudera.hdfs/core-site.xml
http://stackoverflow.com/questions/23385229/deploying-hdfs-core-site-xml-with-cloudera-manager
Re: What is the Path of hdfs.site.
Configuration conf = new Configuration();conf.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));conf.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
/etc/hadoop/conf/
/etc/hadoop/conf.cloudera.hdfs/core-site.xml
http://stackoverflow.com/questions/23385229/deploying-hdfs-core-site-xml-with-cloudera-manager
Re: What is the Path of hdfs.site.
/etc/hadoop/[service name]/hdfs-site.xml
Example:
/etc/hadoop/conf.cloudera.hdfs1/hdfs-site.xml
core-site.xml on the same path.
Example:
/etc/hadoop/conf.cloudera.hdfs1/hdfs-site.xml
core-site.xml on the same path.
https://community.cloudera.com/t5/Cloudera-Manager-Installation/What-is-the-Path-of-hdfs-site-xml-core-xml/td-p/15180
댓글
댓글 쓰기