| vTiger to Quickbooks Integration Design |
 |
 |
 |
| Written by Springli D Sage |
| Monday, 22 June 2009 16:06 |
|
A new friend from Australia said in a conversation about this project: "One-way communication from vTigerCRM to Quickbooks shouldn't be that bad (to create), however two-way communication would make things much more complex." Another gentleman who I'm only just starting to know from North Carolina asked, "Exactly what is going to sync?" followed by "could you let me know if its going to be a complete integration (which it sounds like it is) ?" This document contains information to address questions and comments like these and hopefully instigates more of them from folks like you... let's get started!
Let's get started...As of late, questions and comments like those posted above about this project have been taking place with various interested parties like yourself. I think its time to publish some more information so all of you are in the loop. After all, you are the customers and you have a right to know. Plus, Early Adopters have a say in the design! This document attempts to shed some light and save all of us time by putting the information out there and hopefully provoking questions and ideas that assist in a better end product! This document attempts to explain the folowing:
-------------------------------------------------------------------------------- Design PhilosophyIt's important to mention that doing consulting work for a client requires accessing the clients needs and following the specifications they put forth so that the work you deliver is and does what was asked. When writing software to sell, it is ultimately the author's responsibility for the final design and technology decisions. Now, as the developer of this project I'm the author. I'm not mentioning this as some sort of power trip fuel; I'm only pointing it out because you should know who is writing the code and what drives the decision making process. The philosophy can be summed up in the quote on page 2 of our About Us page, "Technology is always changing and people just want it to work right.". Below are some general preferences and considerations that influence design decisions. The goal is always to get the best software possible for whatever solution is being sought:
If an application can be written in a way that does not marry it to a specific operating system and/or vendor, the resulting software gives the users more options; a quality which equates to more freedom. Many developers jump on the mainstream band wagon and go for it. I prefer to keep abreast of what's available and employ new technology when its the right tool for the job. Now this is not to say that I am unable to develop in a variety of programming languages and operating systems or that I'm unaware of the value of writing software in technologies that many people understand or can pick up easily. I'm just not the type of developer who drools over every new technology because, frankly, almost everything "new" that comes out is something I've been doing in one form or another for years already. Bouncing from one technology to another every time something new is released doesn't allow you to truly become a craftsman. There is a reason that plumbers, carpenters, and masons go through rigorous training and progress through the ranks of their trade with instruction and lots of repetition along the way. Building platform independent software allows one to concentrate on what the software does and leaves you free from much of the "noise" created from trying to write code for specific platforms and constantly trying to write code in some mainstream-accepted "buzzword" technology.
Binary Executable Applications (get ready for a little tech-speak) Binary executable applications are preferred over scripted and JIT ("Just-In-Time") compiled solutions. A Binary application, pound for pound against an equivalent script, always runs faster because they speak the same language as the microprocessor itself and do not need to perform a bunch of work prior to being able to be executed. Scripted languages like PHP, Perl, Javascript, VBScript first tokenize your application's code to tokens and then interpreted by an actual binary program - this is a waste of CPU clock cycles. Java is the the only one that solves some of this waste by allowing developers to distribute "pre-tokenized" software so that it only needs to be interpreted by a binary runtime - not parsed first. Another benefit is security. When people have access to the source code to a software application, vulnerabilities can be exploited by studying the code. This enables wicked souls to write malicious code to exploit them. Likewise, intellectual property is difficult to protect using these solutions. This even applies to .Net compiled applications because they are actually tokenized like scripted applications into the CLR (Common Language Runtime) which is as easy to reverse engineer as opening a compressed Zip file unless the developer takes the time to obfuscate the code prior to compiling it. .Net applications do not compile code to binary until just before they are run - hence the acronym: JIT. There is definitely a place for scripted applications and .Net and I use them all the time. However for sensitive mission critical processes, I always leans towards a secure and fast binary solution.
The Performance of software applications is, in large part, a direct result of the technologies used as touched on in the Binary Executable Applications section above. There is more to the picture, however. The Scripts vs. Binary comparison concerning performance might not be far off from comparing a Porsche Sports Car against a Formula One Racing Car. Both are fast... but the the Formula One is generally faster. The race car engineers know exactly how to maximize performance based on their experience & available knowledge. Likewise, skilled developers know how to get the most performance out of their technology. To get that same "Formula One" racing performance, I need to consider performance through every step of software development if the result is going to be a winning product. I have noticed that many developers these days just try to get it done, without much regard for performance considerations. Most will say, when approached with concerns about performance speed, "if its slow just get faster hardware"! This type of thinking makes me cringe. Sometimes it is warranted but usually it is just poor software architecture whose poor performance is justified as the fault of the hardware! (twitch twitch) When software is written correctly, buying faster hardware means your software just got a complete speed upgrade. How many times will people upgrade their operating system and software to all the "latest" technologies just to realize that you end up with something that looks different, needs to be learned, is often more steps to accomplish the same thing, and really runs about the same speed as before? Software bloat and sloppy programming practices are like a Formula One Race Car towing a trailer! Performance considerations are always a concern for me and for Jegas, LLC as a whole, exactly as they should be!
Object Oriented Programming Model Discussions about Object Oriented Programming (OOP) are all over the Internet. There are all sorts of buzzwords in use, people having conversations about what should and what should not be coined Object Oriented, etc. For most people, it is very tiring and difficult to sift through. Let me break it down for you. Object Oriented Programming allows you to organize and reuse your code in such a way that it snowballs and helps make creating the software application and maintenance of it faster and easier! Now, OOP is not the answer to everything, and like most technology, there is a time and place for everything. Jegas, LLC is a small company doing its best to grow. Frankly, we don't have time for redundancy, large maintenance issues and sloppy design. We most certainly employ OOP practices in order to make the best use of our time while creating software that performs and is easy to maintain. However, if an OOP solution isn't the best fit for the problem at hand, we use something else!
I'll try to get past the obvious here and move right into my point. Jegas, LLC owns a home grown code source library that consists of over 200,000 lines of source code in languages such as: DiBoL, Javascript, Java, FreePascal, C++, VB.Net, ASP.Net, C#, PHP and Perl. Each and every line was written for a specific business problem needing a solution or some cpu/gpu intensive graphics application for commercial or entertainment use. Whenever this code can be used and is the right solution for the job at hand - we use it! Do I suffer from NIHS (Not invented Here Syndrome)? The answer is NO! I use pre-written code all the time, however - the way I study technology is by writing it myself! Some say I'm a masochist, my wife says I'm an enthusiast- I say its more than a job - its a passion!
Through strict adherence to proven software architecture theory, methodologies and careful use of available technologies, Jegas, LLC creates software that is reliable, performs well and has a surprisingly small footprint. Creating software that is platform independent, when possible, additionally gives users more freedom for deployment options and purchasing decisions concerning operating systems and hardware. The design philosophy is really simple. Build it, and they will come. Build it right, and they will stay!
Jegas, LLC software is always put together with an eye on the future. This simply means that questions are constantly applied to the task at hand. For example: • Is this code reusable? I have been building a modular suite of back-office business and CRM software for years. The original source code for it was created in 2003 and has evolved into the Jegas Application Server code base. It's been built with the design philosophy outlined in this document and has literally been designed to eventually become a CRM of its own that serves as the middleware hub to any and all kinds of information systems. If you look at the SchemaMaster v1.0 product, you'll see it can copy entire databases across database management systems: tables, fields and data. Well, the same basic principles that made this hefty (for one person) project possible are what will make not only the vTigerCRM to Quickbooks integration successful, but also possible future projects that could eventually allow copying, synchronizing, and migrating to and from any of the following: • Microsoft Dynamics CRM Diagram: Future of JAS These goals are admittedly lofty, but that is the future... This isn't some adolescent fantasy- I have made a career of writing software of all kinds with a focus on Middleware because, lets face it, software rarely plays nice and needs major coaxing to communicate successfully. This marketing push by the big guys for SaaS (Software as a Service) for everything under the sun and the promises of the Cloud will have their successes... but you have to ask yourself, do you really want all your data in a stranger's possession all the time? How scary is it that someone else owns all of your business data? The goal here is not about a quick fix. My philosophy is and always has been striving to create software systems which efficiently utilize hardware, are as platform independent as possible and are reliable. I believe customers do not like that "Hotel California" type vendor lock-in where: "you can check out any time you like, but you can never leave." (Thank you, Eagles!) Writing the core business logic, "Business Kernel", in .Net might be great for the Microsoft shops out there, and I know there are plenty, but writing the business logic "kernel" in a manner that creates faster executing binaries and is capable of running on anything from an iPhone to a Playstation seems more viable. This is where the Jegas Application Server code base, again comes into play. They say business is a dog-eat-dog world, and software is often no exception. Allowing customers many options for both "Getting In" and strategies for "Getting Out" is just about unheard of unless it's a competing vendor trying to woo a customer from one product to another. It doesn't have to be that way. Honesty, flexibility and adaptability to customers' needs is what Jegas, LLC is about. This vTigerCRM to Quickbooks integration project is a perfect application for the software code base I've been building!
System Architecture • vTigerCRM to JAS Some Installation Scenario Diagrams: • Scenario #1 - vTigerCRM, Quickbooks Desktop, JAS in Standalone mode (is its own webserver) The vTigerCRM to JAS module's job is to synchronize data from vTigerCRM to the JAS server. Currently polling and event driven solutions are being investigated but you can expect real-time or near real-time data transactions. Note: Development is not finished and it has not yet been determined if the vTigerCRM Webservice module named WebForms will be implemented or whether we will use a direct MySQL approach. Eventually, having both mechanisms for interacting with vTigerCRM may exist, but now - the jury is out as the vTigerCRM WebForms module currently has a security issue we're keeping an eye on. However, if the entire CRM is available through the Webservice module "WebForms", then keeping up with new vTigerCRM versions might be easier to do if we utilize it. It would be easier to maintain one communication protocol than two but the advantages of having two mechanisms need to outweigh the difficulties in maintaining two code bases for this module. The Jegas Application Server (JAS) component is platform independent and can be run as either a standalone web/web service server or be used in CGI mode in conjunction with your existing web server. Note: CGI Mode is also the desired configuration to utilize SSL for communication with vTigerCRM over the internet. Future revisions may have SSL built in.
JAS Database The JAS database contains a customizable storage schema designed to accommodate any type of datamodel. Currently the JAS database model is geared for CRM but adding new tables with regards accounting information is planned for sometime in the life of this project. The JAS storage schema is already married to the JAS code base and has three levels of record-locking which are platform and DBMS independent: Table Level, Record Level, and Field level - with complete user accountaibility. Note: User accountability is not of value in a system to system integration of this type- however the core is in place. What is important is that user accountability is preserved between the two applications when the applications allow for it. It's to soon to say if user accountability can be preserved from vTigerCRM to Quickbooks and back. There are already some administration tools in place that allow dynamically creating data entry forms and search forms for custom tables in the JAS database. To date, the code for this ability resides in the SchemaMaster v1.0 codebase, but will likely be made into a standalone tool, and/or translated to .Net or a more portable web interface.
JAS Webservice The JAS Webservice is the newest component of the JAS code base, and is still in progress. The goal of this piece is to generally allow hosting webservices for JAS' own Application Programming Interface as well as other modules developed by Jegas, LLC for end users. Webservices generally are a xml based, platform and programming language independent way of offering services like a DLL or shared library does in an operating system. You can for example create ONE webservice that provides a list of products and then create a "Consumer Application" for it in one ore more of the following: .Net, Java, PHP and many others. They sort of work like HTTP web forms do... sometimes they just return a page, sometimes the results change based on the data you submitted. The reason I developed this now is that I think this sort of flexibility allows both long term value as well as giving this project a way to cross the gap from the .Net Quickbooks world to the JAS world - where the synchronization and staging area for data reside. I know some people will ask: Why don't you just write it all in .Net - Have some .Net app talk to Quickbooks and the vTigerCRM integration directly? Well, first let me say this is a valid question. I recommend reading the Design Philosophy section for details, but in short: .Net would require the Quickbooks server to not only house all the Quickbooks data and processing, but also all the processing for the vTigerCRM Syncronization. The selected design not only offloads the main processing to a central core that can be installed literally anywhere on any operating system, but also allows the user to decide where to place the work horse. Quickbooks might be running on some old PC that just works fine, but could suffer from the added workload. Perhaps the vTigerCRM is not on the same network and the user doesn't want to open ports on the quickbooks machine unless there is a middleware component they trust. If not, we then have the whole JIT versus binary topic to advance on this matter, the extra work involved to secure intellectual property via Obfuscating code before compiling releases is enough to make an author think twice. Lastly, and most importantly - Modular design. The JAS already is platform independent. Quickbooks can be hosted, and if it is there is a web service component for that as well. With the design we are employing, hosted vTigerCRM and hosted Quickbooks installations can be served by the our vTigerCRM Quickbooks Integration entirely from Linux, Mac or Windows without any .Net dependencies.
.Net Communication Module for Quickbooks This piece is a mouthful but its job can summarized as that of a courier. The logic for this piece, and the code behind it is far from trivial. This code handles all the Quickbooks SDK Calls (which are numerous) as well as the communication of all data to and from the JAS database. It's perhaps too early to say, but I suspect there will be weeks of heads down development doing nothing more than developing this portion of code. Fortunately much of the other necessary code, that which concerns JAS, is already done, works, and is run every day in other applications!
Communication Technologies • HTTP(SSL) - vTigerCRM + Webforms, Hosted Quickbooks, JAS User/Admin Interfaces (Limited SSL Support)
Integration Plans: Full or Partial A new friend from Australia said in a conversation about this project: "One-way communication from vTigerCRM to Quickbooks shouldn't be that bad (to create), however two-way communication would make things much more complex." You know, I couldn't agree more... A one way data migrator would be fairly straight forward however there are reasons why this can cause serious problems in your daily operations! I'm also aware that the scope of the project more than doubles when you decide on a two way integration... and frankly just making this type of system without the design that is already being implemented would make matters even more difficult! Fortunately, there is a strategy that will make this a little easier to pull off. The JAS database will serve as a staging area. This staging area will be populated by the .Net piece and from gathering information from vTigerCRM via Polling. An event driven paradigm, if it can be implemented without impacting the vTigerCRM Source code too much, then this technique will be sought. In short, the strategy is to uniquely identify entire data transactions like a typical transactional database does for each data transaction so that the information is gathered, and flagged as such once it is correctly gathered. Likewise, the entire transaction is released when the data push is done successfully. With the JAS staging area retaining all data transactions, management can monitor data flow, synchronization problems, retry, employ full synchronization tasks, perform integration maintenance, and even configure any applicable business rules for the system so that the system can be tuned to be even more hands off than it might be "out of the box"! You Early Adopters and Beta Testers Out there will be KEY in making this work Great!
International Considerations However, we will concentrate on the each countries with the most Early Adopter licenses first- it's only fair! So far, it's Looking like USA first, Australia second, and Canada in hot pursuit! Purchasing your Early Adopter's license(s) now could sway or completely change this! Each country has its own laws regarding taxes and general accounting reporting requirements etc. Likewise, vTigerCRM has configurable taxes that can be applied, as well as allowing each individual product to have it's own settings as to what taxes should be applied. This makes things a little difficult, but this will be resolved in the admin control panel of the Jegas Application Server User/Admin interface.
Release Dates Now, Jegas, LLC and I personally work hard at keeping our word! So if you feel like this is risky, ok. So far, Jegas, LLC has made all the investments developing, marketing, correspondence and keeping you posted with email and web site changes. If this seems small, let me put it another way: This document alone took a over a day of work! (I'm much faster writing code BTW!) - Last week and this week I'm working on this project full time, and not because I don't have other work to do. I literally have worked out a deal with a client who is paying larger installments on his bill so that I may focus on this integration project. That's two weeks I'm basically coasting, not generating income! I promise I'll keep up the pace if people can step up to the plate... If the web site and articles like these aren't enough to validate that this is for real... I don't know what will. Become an Early Adopter and help move the dates forward! CO-OP Software... It makes sense!
Jegas, LLC- Virtually Everything IT! TM |
| Last Updated on Tuesday, 09 March 2010 14:12 |
