hive - Hadoop - Create external table from multiple directories in HDFS -


i have external table reads data hdfs location (/user/hive/warehouse/tablex) files , created external table in hive.

now, let's assume there's pre-partitioning of data , previous files spitted in several directories specific name convention <dir_name>_<incnumber> e.g.

/user/hive/warehouse/split/   ./dir_1/files...   ./dir_2/files...   ./dir_n/files...

how can create external table keeps track of files in split folder?

do need create external table partitioned on each sub-folder (dir_x)?

also that, needed kind of hive or shell script can create/add partition each sub-directory?

you have create external table partitioned dir_x access files in multiple folders.

create external table sample_table( col1 string,                                     col2 string,                                     col3 string,                                     col4 string) partitioned (dir string) row format delimited fields terminated '\t' stored textfile location '/user/hive/warehouse/split';

then add partition regular partitioned table

alter table sample_table add partition(dir='dir_1') location '/user/hive/warehouse/split/dir_1'; alter table sample_table add partition(dir='dir_2') location '/user/hive/warehouse/split/dir_2';

this approach work. there issue approach. if time in future decide add new folder(e.g. dir_100) hive warehouse path, have drop , recreate sample_table , re add partitions sample_table again using alter table statement. haven't worked hive 10 months now, not sure if there better approach. if not issue, can use approach.


-

Popular posts from this blog

Apache NiFi ExecuteScript: Groovy script to replace Json values via a mapping file -

i working apache nifi 0.5.1 on groovy script replace incoming json values ones contained in mapping file. mapping file looks (it simple .txt): header1;header2;header3 a;some text;a2 i have started following: import groovy.json.jsonbuilder import groovy.json.jsonslurper import java.nio.charset.standardcharsets def flowfile = session.get(); if (flowfile == null) { return; } flowfile = session.write(flowfile, { inputstream, outputstream -> def content = """ { "field1": "a" "field2": "a", "field3": "a" }""" def slurped = new jsonslurper().parsetext(content) def builder = new jsonbuilder(slurped) builder.content.field1 = "a" builder.content.field2 = "some text" builder.content.field3 = "a2" outputstream.write(builder.toprettystring().ge...
Read more

python 3.x - PyQt5 - Signal : pyqtSignal no method connect -

following example in doc found on sourceforge faced error. from pyqt5.qtcore import pyqtsignal, pyqtslot def setsignal(): signal = pyqtsignal() signal.connect(self.myaction) @qtcore.pyqtslot() def myaction(): print("signal triggered") results in attributeerror: 'pyqt5.qtcore.pyqtsignal' object has no attribute 'connect' as mentioned in docs linked, signal needs defined on class level: class foo(qobject): signal = pyqtsignal() def connectsignal(): self.signal.connect(self.myaction) @qtcore.pyqtslot() def myaction(): print("signal triggered")
Read more