Thanks @jbarrow for amazing work!
While working with a pdf form, with default confidence 0.3 for detection, the detector was missing few fields.
So when I reduced the confidence, the detector detected the desired fields, but additionally detected some non-existent fields (likely false positives).
To address this, I have written a streamlit app as a temporary solution. The app allows user to select confidence score and also shows all the detected and editable fields and allow the user to deselect the undesirable fields, before output PDF is generated.
My code is on a fork. https://github.com/suvarnak/commonforms
Please consider adding such functionality. I can raise a PR if required