Lessons learned from creating a Contact Scraper/Marketing app in Python

Lessons learned from creating a Contact Scraper/Marketing app in Python

Lessons learned from creating a Contact Scraper/Marketing app in Python

Winding down the days until release of the Rancorsoft Contact Scraper application.  It’s been many months since the first iteration of the app, which was originally birthed as a Sikuli browser automation app, and then re-engineered in to a full fledged Python/Kivy UI.  We thought it was a good time to compile and share some lessons learned from developing the app, aimed primarily at fellow Python programmers.

Without further ado, I present to you a short list of some of the things we have learned:

1. Know which library functions are asynchronous

Asynchronous program flows can be hard to debug (especially if you don’t know they are asynchronous, or if you’re new to python, and not yet comfortable a debugger).

 

2. Learn to use a debugger

I personally like “pdb”, which is included with most (or maybe all?) python distributions.  PDB has saved me a ton of time analyzing bugs.  The ability to stop on a line and get the value of any variable is incredibly useful.  Alternatively, you could use an IDE with a debugger like PyCharm.

 

3. Get some code completion in your editor/IDE

I use Notepad++ with NPP-Python http://sourceforge.net/projects/npp-python/ added on top. Good code completion will save you a lot of typing/googling over time.

 

4. Design your classes well

This can be a hard one to follow, since python lets you get really sloppy – if you want to – and still get away with it.  I have seen fully functional python code that rivals the most terse Perl scripts in lack of readability.  Don’t use this as an excuse for getting lazy.
Each class should do one thing, and modularity should be well planned for.  Be prudent in your use of comments, and never assume another coder – or even yourself, in a few months time – will be able to read and immediately understand your clever uses of fancy Python syntactic sugar.
At a certain level of complexity I like to use visual analysis tools like UML Class Diagrams.
Sample UML Class Diagram showing Student and Professor classes inheriting from Person class

Sample UML Class Diagram showing Student and Professor classes inheriting from Person class

 

5. Think through your app deployment ahead of time, if possible

Usually this will mean using PyInstaller, although sometimes it doesn’t – and you opt for an alternative deployment scheme.  Get an expert to help, if possible.  An expert with experience installing on multiple systems will be especially useful, considering some of the difficulties and differences in processes and protocols required for deployment on Windows, Mac, Linux, Android, iOS, etc.

 

6. Matching phone numbers is HARD

This one is relevant mostly to what our app specifically does.  First pass of the implementations was using a Regular Expression.  This worked well for US numbers only, and took a lot of work to get right.  My personal recommendation would be to use  https://github.com/daviddrysdale/python-phonenumbers – which is a Python port of Google’s phone number matching library, or something similar, if your use case allows it.

 

Finally / In conclusion…

Python is a wonderful language to work in, and we are incredibly fortunate to have the cross platform Kivy UI library, and multitudes of other free libraries to read, admire, and use available on the internet.  The capabilities of every Python coder expands every day through the bettering of these tools and techniques.  We would like to thank the Kivy team for their support throughout all of our development, as well as all the people who contribute, discuss, support and admire all of the Python and platform specific libraries available on github and the general internet.

Leave a Reply

Your email address will not be published. Required fields are marked *