Feature Request: Ability to adapt to Canadian Bank Statements

[chandu]

Thanks for a wonderful tool.
I was trying to make the command line tool work with the Canadian Credit Card Bank Statements (TD to be specific), but ran into some issues (because of the way the statements are structured) that I could not manage via the config.
I am listing the issues:

1. Ability to skip pages from the statement for parsing (in the TDD CC Statement, the second page doesn't contain any transactions info and the tool was failing with error)
2. Ability to determine if a data row is valid for consumption prior to passing it for further processing like date time conversion, currency conversion etc. In the extracted rows one entry in the PDF was generating two rows as the transaction description had a new line.
3. Ability to inject year for the transaction date, the TD statement only provides month and day and not the year

Not sure if you have time to spare on this library.
Are you open for a discussion on if/how the above could be adapted without breaking/changing the existing tool much?

[marlanperumal]

Hi @chandu

I've just stepped back onto active development of this library after leaving it stagnant for a while. I've made some minor changes to the dev toolchain and some minor bugfixes and am currently working on a frontend gui in the pdf-table-extractor repo that I think you've also seen to enable easier selection of areas columns etc. I'm hoping to have a working version of that ready in the next week or so, after which I'll step back onto this library and starting making some of the config changes to allow for some additional functionality including what you've requested.

In the meantime can you please log separate issues for each of the features you've requested so that they can be marked off one by one.

I'd also be happy for you to implement the features you've requested and submit pull requests. Unfortunately one of my backlog tasks is to create a Contributing reference to set some guideline for how the code should be structured etc. but I haven't gotten around to that yet. Happy to have a bit of back and forth with you in lieu of that so we can land on the right patterns

[chandu]

Thanks @marlanperumal
I will log the issues as mentioned.

I will take a stab at implementing the features as well.
I can assist in the frontend-gui, if need be.

Regarding the back and forth, what would you prefer?
Github issues or any other means (discord etc..)?

[chandu]

Have craeted issues #72 and #73 towards points 1 and 2 from my list.
For the third point, I think it is a verify specific use case and not sure if it should be part of the library.
Needs further discussion on this.

[marlanperumal]

Thanks @chandu

I'll comment on the other 2 issues and we can use this one for general discussion. I think just communicating on the github issues should be fine. If we need to have a detailed. conversation on something specific we can do a quick call if that helps.

On the 3rd point, there was a previous request to have per column date formats, which should also address your requirement (I think most date parsers will insert the current year if it's not explicit in the format string and worst case, you can just post-process the resultant csv to modify the year if required). I'll link that issue

[marlanperumal]

Date format covered in #59

[marlanperumal]

@chandu I've made some major updates to the pdf-table-extractor frontend to mock up most of the functionality required (though it still doesn't actually generate the config file and needs a fair amount of refactoring to be usefully readable). See if you can get that running locally on your machine and have a look a the github project I've invited you with read access to. If you think you'll be able to help to pick up some of the tasks, please let me know and I'll give you write access to the project and I'll flesh out the tasks with specific requirements.

[chandu]

@chandu I've made some major updates to the pdf-table-extractor frontend to mock up most of the functionality required (though it still doesn't actually generate the config file and needs a fair amount of refactoring to be usefully readable). See if you can get that running locally on your machine and have a look a the github project I've invited you with read access to. If you think you'll be able to help to pick up some of the tasks, please let me know and I'll give you write access to the project and I'll flesh out the tasks with specific requirements.

Thanks @marlanperumal I am able to run the app locally. Looks like you have already made major progress on the marlanperumal/pdf-table-extractor#3 of that repo. I guess rest of the tasks are heavily dependent on this being available in develop/any other feature branch to build upon. Let me know if you think otherwise or if you want me to take any of the open tasks in marlanperumal/pdf-table-extractor#3

[chandu]

@marlanperumal Would it be possible to enable discussions for both the repos on github so that we don't need to use issues for a long discussion or an ideating discussion?

[chandu]

@marlanperumal Here is the code I used, for now, to be able to extract the transactions from the TD Credit Card Statements: https://gist.github.com/chandu/5efa1ec607bf928337baa9e4d18049a9 . Sharing it if it could be of any help

[marlanperumal]

@chandu I've enabled Discussions on both repos and given you write access on the project. I'm still getting used to using Github as a primary medium of communication so it might get a bit messy having info split between Issues, Discussions, Projects and PRs but hopefully it'll quickly become apparent what works and what doesn't.

I started a discussion on a new config file format to better deal with the different use cases. I'm not sure if that's overkill but I think I made some bad initial design choices on the layout so now's as good a time as any to make some better informed choices.

My current plan is to

1. Get the pdf-table-extractor frontend fully working against the v1 config file by the end of this week
2. Nail down a v2 config format to proceed with (though I'm sure it'll change a bit before the end)
3. Flesh out the spec for the new features (to enable you and potentially others to contribute)
4. Develop 80% of the new features
5. Update the frontend GUI to work with the new format
6. Put out a release

If you could please take a look at 2 for now and make some comments in the discussion, while I'm busy with 1, then we can potentially do 3 together and divide and conquer the rest after that. Thanks!