Xtykutl

I have a pandas dataframe containing 2 columns and I want to use sklearn TfidfVectorizer for text-classification in one of them. However this column is a list of lists and TFIDF wants raw input as text. In this question they provide a solution in case we have just one list of lists, but I would like to ask how it would be possible to apply this function in every single row of my dataframe, which row contains a list of lists. Thank you in advance.

Input:



0    [[this, is, the], [first, row], [of, dataframe]]

1    [[that, is, the], [second], [row, of, dataframe]]

2    [[etc], [etc, etc]]

Wanted Output:

0    ['this is the', 'first row', 'of dataframe']

1    ['that is the', 'second', 'row of dataframe']

2    ['etc', 'etc etc']

You could use apply:

import pandas as pd



df = pd.DataFrame(data=[[[['this', 'is', 'the'], ['first', 'row'], ['of', 'dataframe']]],

                        [[['that', 'is', 'the'], ['second'], ['row', 'of', 'dataframe']]]],

                  columns=['paragraphs'])





df['result'] = df['paragraphs'].apply(lambda xs: [' '.join(x) for x in xs])

print(df['result'])

Output

0     [this is the, first row, of dataframe]

1    [that is the, second, row of dataframe]

Name: result, dtype: object

Further, if you want to apply the vectorizer in conjunction with the above function you could do something like this:

def vectorize(xs, vectorizer=TfidfVectorizer(min_df=1, stop_words="english")):

    text = [' '.join(x) for x in xs]

    return vectorizer.fit_transform(text)





df['vectors'] = df['paragraphs'].apply(vectorize)

print(df['vectors'].values)