ICYMI: our team gave a NiFi presentation and demonstration at January’s CodeMash conference. The endeavor, put together by collaborators Ryan Hendrickson, Elli Schwarz, and Bill Farmer, emerged from our ideas and innovations program (i2).
We’re pleased to share details on this special project.
What is i2?
Clarity’s i2 program supports employee exploration and innovation providing resources, time, and team participation for concepts that could become realities. Each of us is encouraged to present ideas, receive feedback, work on a white paper or presentation, build a prototype, and see solutions in action.
In that spirit, three of our software engineers and NiFi experts worked together to create a conference session that provided a how-to with examples and demos showing basic capabilities of the data processing and distribution system.
NiFi in a Nutshell
Apache NiFi is an intuitive, scalable, and customizable dataflow system. It comes with an easy-to-use and appealing management UI, a large market of standard processors, and a vibrant Open Source Community supporting it.
“NiFi takes data from one place, processes it, routes it, and puts it somewhere else. It’s a sophisticated platform that allows you to be flexible in how you do that no matter how large
or small your dataflow.” – Ryan Hendrickson
Thanks to the Internet of Things, smart homes and cities, social media communications, and electronic financial transactions, among countless other digital phenomena, data drives almost everything. Remarkably, in one minute, Venmo completes $68,000 in peer-to-peer transactions, people produce 12 million text messages, and The Weather Channel receives 18 million forecast requests.* Demand for the capture, processing, enrichment, and delivery of useful data grows daily — even in the time it takes to read this post.
The Data Game (it’s challenging)
When processing data; its volume, velocity, variety, and value (the four V’s) are all considerations. Of course, software architecture questions must be answered as well: how is the data acquired, processed, stored, and enriched? In addition; how will analytics be run on the data and the results stored, how will analytics be run on the analytic result, and how will the results be served up on a web site? Answers to these questions help determine the dataflow stack. With its scalable and customizable dataflow system and a large market of standard processors, NiFi offers a flexible solution.
Features & Benefits of NiFi
Our team’s CodeMash session was designed to walk attendees through the NiFi basics by showing how to move data across servers and networks using the platform. It covered how to manipulate, enrich, and stream data through custom enrichment processors and in doing so, revealed the features and benefits of NiFi…
The Process is its Own Reward**
The advantages of using NiFi are many:
· Guaranteed Delivery
· Data Buffering with Back Pressure and Pressure Release
· Prioritized Queuing
· Flow Specific QoS (latency vs. throughput, loss tolerance, etc.)
· Visual Command and Control
· Flow Templates
· FlowFile Recovery
· Extensions (Processors and more)
· Classloader Isolation
· Scale Out (Clustering)
· Scale-up & down
· HTTPS & SSL
· Multi-tenant Authorization (User Mgmt)
“We’ve solved some pretty complex data flow and ETL challenges with Nifi. As we demonstrated with our CodeMash presentation, we were able to perform ETL mostly with built-in NiFi capabilities and used one simple custom data processor for a case where we needed a more custom solution.” — Elli Schwarz
To showcase NiFi’s capabilities and ease-of-use, our team ran some relatable demos. Using NiFi and implementing a variety of processors, the team polled reddit for new posts and aggregated subreddit topic names to serve up a simple web site that showed popular subreddits. They also used NiFi to consume tweets that were filtered on #CodeMash to light-up LED lights that were illuminated thanks to a Raspberry Pi that was equipped with MiNiFi.
“We were pleasantly surprised by the questions at the end of the session. People weren’t aware of NiFi’s capabilities.” — Bill Farmer
Clarity’s CodeMash 2019 Team
Ryan, Bill, and Elli collaborate on technical projects every day. Along with others on their team, they find great satisfaction in anticipating and meeting the mind-bending demands of our ever-changing world with custom software solutions. Smart, creative, and diligent, they run on mission and thrive on data science.
Ryan Hendrickson is a senior software engineer and technical project lead. He has been using NiFi for four years.
Bill Farmer is a principal software engineer, technical project lead, and Clarity’s vice president of engineering. He has been using NiFi for four years.
Elli Schwarz is a senior software engineer and lead technical contributor. He is an Elastic and NiFi guru. He pushes NiFi to its limits to solve complex challenges.
What’s Next for our NiFi Experts?
Ryan, Bill, and Elli plan to continue sharing their technical knowledge and insights. Catch them at upcoming tech user groups and conferences — follow us for announcements. They regularly attend Data Works MD Meetups in the Baltimore-Washington region.
If you’d like to request a copy of Clarity’s NiFi presentation from CodeMash 2019, please get in touch with us by email.
More on NiFi
Download NiFi: https://nifi.apache.org/download.html. Notes on how to install and start NiFi are available at https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#how-to-install-and-start-nifi.
We discussed NiFi on The At Scale Podcast. Download it for an interesting talk about the technology — also available on iTunes and other podcasting apps.
Bonus for reading all the way to the end — here’s a glimpse at what happens in an Internet Minute: https://www.visualcapitalist.com/internet-minute-2018/. Multiply these numbers by three…this post is about a 3-minute read!
**Discussing decision-making and fear, Amelia Earhart once said: “The process is its own reward.” source: www.ameliaearhart.com