Xtykutl

I have a bunch of manifest/yaml files that may or may not have these key value pair duplicates:

...

app: activity-worker

app: activity-worker

...

I need to search through each of those files and find those duplicates so that I can remove one of them.

Note: I know that to replace a certain string (say, switch service: to app:) in all files of a directory (say, dev) I can run grep -l 'service:' dev/* | xargs sed -i "" 's/service:/app:/g'. I'm looking for a relation between lines.

What you call YAML, is not YAML. The YAML specification
very explicitly states that
keys in a mapping must be unique, and your keys are not:

The content of a mapping node is an unordered set of key: value node
pairs, with the restriction that each of the keys is unique. YAML
places no further restrictions on the nodes. In particular, keys may
be arbitrary nodes, the same node may be used as the value of
several key: value pairs, and a mapping could even contain itself as
a key or a value (directly or indirectly).

On the other hand some libraries have implemented this incorrectly, choosing to overwrite
any previous value associated with a key, with a later value. In your case, since
the values are the same, which value would be taken doesn't really matter.

Also your block style representation is not the only way to represent key-value pairs of a
mapping in "YAML", these duplicates could also be represented in a mapping, as

{...., app: activity-worker, app: activity-worker, .... }

With the two occurences not necessarily being next to each, nor on the same line. The
following is also semantically equivalent "YAML" to your input:

{...., app: activity-worker, app: 

       activity-worker, .... }

If you have such faulty "YAML" files, the best way to clean them up is
using the round-trip capabilities of
ruamel.yaml (disclaimer: I
am the author of that package), and its ability to switch except/warn
on faulty input containing duplicate keys. You can install it for your
Python (virtual environment) using:

pip install ruamel.yaml

Assuming your file is called input.yaml and it contains:

a: 1   # some duplicate keys follow

app: activity-worker

app: activity-worker

b: "abc"

You can run the following one-liner:

python -c "import sys; from ruamel.yaml import YAML; yaml = YAML(); yaml.preserve_quotes=yaml.allow_duplicate_keys=True; yaml.dump(yaml.load(open('input.yaml')), sys.stdout)"

to get:

a: 1   # some duplicate keys follow

app: activity-worker

b: "abc"

and if your input were like:

{a: 1, app: activity-worker, app: 

    activity-worker, b: "abc"}

the output would be:

{a: 1, app: activity-worker, b: "abc"}