So, your client comes to you with an AI project that has an undefined scope and no out-of-the-box solutions. What do you do first, and how do you do it? Learn how to leverage applied artificial intelligence techniques for custom algorithm development.
What is applied artificial intelligence (AAI)?
First, let's discuss what applied artificial intelligence (AAI) is and is not. Applied artificial intelligence covers applications in management, industry, engineering, administration, education, finance, and many other vertices.
Applied AI is concerned with taking techniques from research or industry and applying them to business problems. Applied AI uses all the techniques we know and love as data science geeks, including deep learning, neural networks, matrices, complex mathematics, among others. These are, again, applied to business problems. It covers evaluations of systems and tools, as well.
What isn't applied artificial intelligence? Applied artificial intelligence is not obscure research. Research and development of artificial intelligence systems can take years, if not decades. Think about the 1928 computer IBM built—its tabulator subtracted, added, and automatically took in card feeds. Computers small enough to fit on a desktop became available in the 1970s. ELIZA, released in 1966, was the first AI chatbot “therapist” and most definitely a research project.
We're walking a fine line and middle path with outside-the-box projects. For these projects, we use Applied AI, when often other solutions will not work. Now that we framed what Applied AI is and is not, let's look at how to use its tools and techniques for challenging projects. There's a three-step procedure for getting the design of algorithms on the right track.
How to design custom algorithms in 3 steps
Read academic journals and patents
I just explained that designing a custom algorithm is applied AI, and we're not performing research. So why read research journals? The reason is that most questions and approaches found in the business environment started in the academic or research and development environment. That means the research we read in journals applies to business problems in the right context. A good rule of thumb is to choose from high-quality journals such as American Computing Machinery or IEEE journals. Access many publications for free using Cornell's on-line library arxiv.org.
Another good source for research journals is Research Gate. Often if you do not find a paper open access, then write to the authors. Most authors are thrilled to see their work applied in real-world situations. They will gladly converse when approached respectfully and with care. Make sure to do your background work before contacting the author. For example, if they have carefully explained a term, their algorithms, or provided open-source code—do not ask them questions covered in those items. Instead, be very specific. Most of the time, you will not need to ask the author.
Let's say you've now narrowed down your chosen techniques to two or three that seem to have good support in research journals. Should you apply the author's code and then call it a day? Should you look for the best library such as Tensorflow, Keras, or PyTorch? Should you start coding the algorithms from scratch?
Make an algorithm architecture diagram to customize for your client’s needs
The second step—which also becomes part of a project's specifications—is to make an architecture diagram of your proposed approach. I suggest making these very detailed but clear enough to discuss with a client.
This part is tricky because putting complex ideas into layperson's terms isn't easy. But this is a step not to be skipped. Why?
The client needs to know what your algorithm will and won't do for their project. The case can be that the algorithm will perform the critical tasks for the project but may not perform less important tasks. If it does not perform the tasks the client wants, then go back to step one.
Research and think through. Make a new architecture diagram. Go over this will the client. Once the client approves, then it's time to go on to step three. That's right, implementation time!
Code and test your custom algorithm
We've discussed Applied AI, scanning the research literature for information and ideas, as well as making architecture diagrams. Now it's time to start coding and testing. This phase breaks down into discrete stages. The first is data collection. Make sure to have the right data in the right format (see this paper on Tidy Data) to test the design of the algorithm. Testing it with the wrong data can give a false sense of accomplishment. The next subphase is data cleaning. The data must be in the right format for ingestion by the algorithm or ensemble. Now you are ready to code up your algorithm. For most projects, an open-source library such as Tensorflow, Keras, Pytorch, or any number of others is sufficient to get started. Sometimes though, you may need to do what I call “coding by hand.” That includes looking at the mathematical details presented in a paper and then code them up yourself in something like NumPy, SciPy, or another low-level library. Sometimes you may need to use another tool. If this takes more time, clear the architecture diagram with your client. If they know that this is the best (sometimes only) approach for the project, it is typically not problematic. However, when you can use open source libraries or the author's code—go for it.
Top Python Libraries for AI:
- Tensorflow
- Keras
- PyTorch
- Pandas
- Scipy
- Numpy
- Theano
- Sci-kit Learn
- Others depending on project needs
Next steps
So we've gone through the three stages of designing a custom algorithm for an "outside the box" project. Learning to leverage Applied AI in your projects might seem like a steep learning curve. However, if you're interested in getting into more complex, sophisticated projects, it's worth the extra effort. There are also many advanced courses you can audit on learning platforms. Starting with a personal project is always beneficial as well. May you go forth and do great!
.svg)