Using sorted tables in Hive

hadoop hive

Have you checked out the effect of set hive.enforce.bucketing=true? From http://svn.apache.org/repos/asf/hive/branches/branch-0.7/conf/hive-default.xml

<property>  <name>hive.enforce.sorting</name>  <value>false</value>  <description>Whether sorting is enforced. If true, while inserting into the table, sorting is enforced. </description></property>

You may also find reading the implementation of org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer#genBucketingSortingDest useful:

http://svn.apache.org/repos/asf/hive/branches/branch-0.7/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java

hadoop hive

hive.enforce.bucketing does not do a global sort of the data set. Instead it writes the data sorted within the buckets (in your case 8/partition). Thus it requires a global sort step to satisfy the query you are looking for.

Hope this helps,Nat

hadoop hive

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL

The CLUSTERED BY and SORTED BY creation commands do not affect howdata is inserted into a table – only how it is read. This means thatusers must be careful to insert data correctly by specifying thenumber of reducers to be equal to the number of buckets, and usingCLUSTER BY and SORT BY commands in their query.

Also look at https://cwiki.apache.org/confluence/display/Hive/LanguageManual+SortBy

CodeHunter

Using sorted tables in Hive

Recent Posts

How can I color dots in a xy scatterplot according to column value?

How to update a claim in ASP.NET Identity?

What does {0} mean when initializing an object?

Accessing members of items in a JSONArray with Java

How to log SQL statements in Spring Boot?

Powershell Get-WebSite name parameter is ignored

How to detect scroll to bottom of html element

Java synchronized method

How to test controllers with CodeIgniter?

Detect Visual Composer

Matplotlib: Specify format of floats for tick labels

Rails join a list of strings with commas and "and" before the last