Extracting Unique Dictionary Values

Hello!

I am dealing with a list of dictionary files that contain tokens for keys and vector values, like this:
list = [a.dict, b.dict, ...., n.dict]
where each dictionary is something of this form:
dict = {"token": vector}.
My goal is to use a for loop to iterate through the dictionary files in the list and extract the unique vector values (some tokens have the same value; some don’t), storing those values in a list.

Does anyone have any ideas of how to do this? Would I need to first convert the file list into an array to be able to use np.unique on it? I am slightly lost as to where or how to start…

Any help is appreciated!

Perhaps I misunderstand the question (are these dictionary types or something else involving reading files?) but what’s wrong with simple loops and conditionals? You’ve already mentioned one possible route to do this by extracting first and filtering afterwards, another would be to filter whilst building.

If memory’s not an issue I’d lean towards the first due to the increasing complexity of checks in option 2 but that’s more fiddling with performance than solving the problem. Is there something preventing you from extracting this data in the first place?

Just noticed you caught an answer already on discord. Please feel free to update your post accordingly. Ideally with the solution, even if it’s in a simplified form, so that other users can benefit from it.

Yes!

This is the solution kindly provided by @LinearBlck on discord:

new_list = []
for dictionary in lst:
    for vector in dictionary.values():
        if vector not in new_list:
            new_list.append(vector)

You can find a link to the start of the discussion here