Xtykutl

I am new to pyspark and python. So, please help me with this problem where i need to find max value using fold and by using operator.gt or by writing our own lambda function.

Following code written by me throws error that rdd cannot be indexed. I understood it but how to pass and compare values each value 1,2,0,3 with 0 and find max.
Here 0 is my accumulator value and 1,2,0,3 are current values each time.
I am trying to convert a program written in scala that explained fold to python.
Answer expected : ('d', 3)

from pyspark import SparkContext

from operator import gt



def main():

    sc = SparkContext("local", "test")



    data = sc.parallelize([('a', 1), ('b', 2),('c', 0), ('d', 3)])



    #dummy = ('dummy', 0)



    maxVal = data.fold(0, lambda acc, a : gt(acc, a[1])).collect()



    print(maxVal)





if __name__ == '__main__':

    main()

• 
  

  Use neutral value (a one which can be merge an arbitrary number of times without changing the final result) suitable for a particular operation and matches the type of data (the function should be (T, T) => T or with Python conventions Callable[[T, T], T]). With max by value it makes sense to use float("-Inf") and a dummy key:

  
  
  

  zero = (None, float("-Inf"))
  
  

  
  


• 
  

  To reduce use max with key:

  
  
  

  from functools import partial
  
  from operator import itemgetter
  
  
  
  op = partial(max, key=itemgetter(1))
  
  

  
  

Combined:

data.fold(zero, op)

('d', 3) 

Of course in practice you can just use RDD.max

data.max(key=itemgetter(1))

('d', 3)