Sqoop 将mysql 数据导入到hdfs(import)
1.创建mysql表
CREATE TABLE `sqoop_test` (`id` int(11) DEFAULT NULL,`name` varchar(255) DEFAULT NULL,`age` int(11) DEFAULT NULL) ENGINE=InnoDB DEFAULT CHARSET=latin1
插入数据
2.hive 建表
hive> create external table sqoop_test(id int,name string,age int)> ROW FORMAT DELIMITED> FIELDS TERMINATED BY ','> STORED AS TEXTFILE> location '/user/hive/external/sqoop_test';OKTime taken: 0.145 seconds
3.使用sqoop将mysql数据导入到hdfs
sqoop import --connect jdbc:mysql://localhost:3306/sqooptest --username root --password 123qwe --table sqoop_test --columns id,name,age --fields-terminated-by , --delete-target-dir --target-dir /user/hive/external/sqoop_test/ -m 1
--delete-target-dir:如果目标目录存在则删除。
EFdeMacBook-Pro:bin FengZhen$ sqoop import --connect jdbc:mysql://localhost:3306/sqooptest --username root --password 123qwe --table sqoop_test --columns id,name,age --fields-terminated-by , --delete-target-dir --target-dir /user/hive/external/sqoop_test/ -m 1Warning: /Users/FengZhen/Desktop/Hadoop/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/../hcatalog does not exist! HCatalog jobs will fail.Please set $HCAT_HOME to the root of your HCatalog installation.Warning: /Users/FengZhen/Desktop/Hadoop/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/../accumulo does not exist! Accumulo imports will fail.Please set $ACCUMULO_HOME to the root of your Accumulo installation.SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/Users/FengZhen/Desktop/Hadoop/hadoop-2.8.0/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/Users/FengZhen/Desktop/Hadoop/hbase-1.3.0/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]17/09/13 11:12:19 INFO sqoop.Sqoop: Running Sqoop version: 1.4.617/09/13 11:12:19 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.17/09/13 11:12:19 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.17/09/13 11:12:19 INFO tool.CodeGenTool: Beginning code generation17/09/13 11:12:19 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `sqoop_test` AS t LIMIT 117/09/13 11:12:19 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `sqoop_test` AS t LIMIT 117/09/13 11:12:19 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /Users/FengZhen/Desktop/Hadoop/hadoop-2.8.017/09/13 11:12:21 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-FengZhen/compile/1a0c4154ffefb21d4af720813dd0b3fc/sqoop_test.jar17/09/13 11:12:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable17/09/13 11:12:22 INFO tool.ImportTool: Destination directory /user/hive/external/sqoop_test deleted.17/09/13 11:12:22 WARN manager.MySQLManager: It looks like you are importing from mysql.17/09/13 11:12:22 WARN manager.MySQLManager: This transfer can be faster! Use the --direct17/09/13 11:12:22 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.17/09/13 11:12:22 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)17/09/13 11:12:22 INFO mapreduce.ImportJobBase: Beginning import of sqoop_test17/09/13 11:12:22 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address17/09/13 11:12:22 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar17/09/13 11:12:22 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps17/09/13 11:12:22 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:803217/09/13 11:12:24 INFO db.DBInputFormat: Using read commited transaction isolation17/09/13 11:12:24 INFO mapreduce.JobSubmitter: number of splits:117/09/13 11:12:24 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1505268150495_000817/09/13 11:12:25 INFO impl.YarnClientImpl: Submitted application application_1505268150495_000817/09/13 11:12:25 INFO mapreduce.Job: The url to track the job: http://192.168.1.64:8088/proxy/application_1505268150495_0008/17/09/13 11:12:25 INFO mapreduce.Job: Running job: job_1505268150495_000817/09/13 11:12:35 INFO mapreduce.Job: Job job_1505268150495_0008 running in uber mode : false17/09/13 11:12:35 INFO mapreduce.Job: map 0% reduce 0%17/09/13 11:12:41 INFO mapreduce.Job: map 100% reduce 0%17/09/13 11:12:41 INFO mapreduce.Job: Job job_1505268150495_0008 completed successfully17/09/13 11:12:41 INFO mapreduce.Job: Counters: 30File System CountersFILE: Number of bytes read=0FILE: Number of bytes written=156817FILE: Number of read operations=0FILE: Number of large read operations=0FILE: Number of write operations=0HDFS: Number of bytes read=87HDFS: Number of bytes written=26HDFS: Number of read operations=4HDFS: Number of large read operations=0HDFS: Number of write operations=2Job Counters Launched map tasks=1Other local map tasks=1Total time spent by all maps in occupied slots (ms)=3817Total time spent by all reduces in occupied slots (ms)=0Total time spent by all map tasks (ms)=3817Total vcore-milliseconds taken by all map tasks=3817Total megabyte-milliseconds taken by all map tasks=3908608Map-Reduce FrameworkMap input records=3Map output records=3Input split bytes=87Spilled Records=0Failed Shuffles=0Merged Map outputs=0GC time elapsed (ms)=33CPU time spent (ms)=0Physical memory (bytes) snapshot=0Virtual memory (bytes) snapshot=0Total committed heap usage (bytes)=154140672File Input Format Counters Bytes Read=0File Output Format Counters Bytes Written=2617/09/13 11:12:41 INFO mapreduce.ImportJobBase: Transferred 26 bytes in 18.6372 seconds (1.3951 bytes/sec)17/09/13 11:12:41 INFO mapreduce.ImportJobBase: Retrieved 3 records.
可在hdfs看到传入的数据
EFdeMacBook-Pro:jarfile FengZhen$ hadoop fs -ls /user/hive/external/sqoop_testFound 2 items-rw-r--r-- 1 FengZhen supergroup 0 2017-09-13 11:12 /user/hive/external/sqoop_test/_SUCCESS-rw-r--r-- 1 FengZhen supergroup 26 2017-09-13 11:12 /user/hive/external/sqoop_test/part-m-00000
可在hive中查看数据。
hive> select * from sqoop_test;OK1 fz 133 dx 182 test 13Time taken: 1.756 seconds, Fetched: 3 row(s)