list - pyspark collect_set or collect_list with groupby -


how can use collect_set or collect_list on dataframe after groupby. example: df.groupby('key').collect_set('values'). error: attributeerror: 'groupeddata' object has no attribute 'collect_set'

you need use agg. example:

from pyspark import sparkcontext pyspark.sql import hivecontext pyspark.sql import functions f  sc = sparkcontext("local")  sqlcontext = hivecontext(sc)  df = sqlcontext.createdataframe([     ("a", none, none),     ("a", "code1", none),     ("a", "code2", "name2"), ], ["id", "code", "name"])  df.show()  +---+-----+-----+ | id| code| name| +---+-----+-----+ |  a| null| null| |  a|code1| null| |  a|code2|name2| +---+-----+-----+

note in above have create hivecontext. see https://stackoverflow.com/a/35529093/690430 dealing different spark versions.

(df   .groupby("id")   .agg(f.collect_set("code"),        f.collect_list("name"))   .show())  +---+-----------------+-----------------+ | id|collect_set(code)|collect_list(name)| +---+-----------------+-----------------+ |  a|   [code1, code2]|          [name2]| +---+-----------------+-----------------+

-

Popular posts from this blog

php - How should I create my API for mobile applications (Needs Authentication) -

background so i've been researching quite bit past week api's , have been reading concepts , programming one. currently have website programmed in php using popular framework called laravel. website has user database , users able log dashboard on website, works want website side. now next project of business i'm focusing on creating mobile applications (ios & andriod).what need these mobile applications being able login through application ui (not being redirected site callback url) able view , manage dashboard. the method of authentication , authorization want use application go like client asks user login through ui user enters credentials client sends request login api the api checks if credentials correct api creates token stored in token database linked user id api returns 200 ok json response or this { "token" : "ols25usjiay81hdy81", "expiry" : 3/06/2016 14:00} client remembers token , expiry whenever user/c...
Read more

python 3.x - PyQt5 - Signal : pyqtSignal no method connect -

following example in doc found on sourceforge faced error. from pyqt5.qtcore import pyqtsignal, pyqtslot def setsignal(): signal = pyqtsignal() signal.connect(self.myaction) @qtcore.pyqtslot() def myaction(): print("signal triggered") results in attributeerror: 'pyqt5.qtcore.pyqtsignal' object has no attribute 'connect' as mentioned in docs linked, signal needs defined on class level: class foo(qobject): signal = pyqtsignal() def connectsignal(): self.signal.connect(self.myaction) @qtcore.pyqtslot() def myaction(): print("signal triggered")
Read more