We’re pleased to announce that three of our technical experts will present a talk and demonstration at the November Data Works MD Meetup. Senior software engineers Bill Farmer, Ryan Hendrickson, and Elli Schwarz spend the workweek collaborating on complex technical challenges — they will bring their dataflow expertise to the Data Works group with their presentation, Connect Data and Devices with Apache NiFi.
The Meetup is scheduled for Thursday, November 7, from 6:30 pm until 8 pm in CACI’s auditorium at 2720 Technology Drive in Annapolis Junction.
Apache NiFi is an intuitive, scalable, and customizable system used to process and distribute data. It comes with an easy-to-use and appealing management UI, a large market of standard processors, and a vibrant Open Source Community supporting it.
About the presentation
An accessible how-to
Clarity’s team aims to provide an accessible how-to that will walk participants through NiFi basics, showing practical examples that are easy to follow. The Meetup group will learn how to use NiFi to move data across servers and networks, as well as how to manipulate, enrich, and stream data using custom processors.
Data manipulation and open-source tools
Specific how-tos will include performing data manipulation using:
a custom Java processor
the JoltTransformData processor
The team will demonstrate open-source tools, such as Jolt, jQ, and JsonPath. The presentation will explain how to prototype a REST service with standard processors. Expect light bulbs to flash from (exciting) things happening in NiFi!
About our presenting engineers
Ryan Hendrickson is a Senior Software Engineer and Clarity’s Director of Innovation. He has 11 years of software engineering experience. Ryan is a Software Project Co-Lead, and he participates in Data Works MD Meetups. He presented at CodeMash in 2019 and will return in 2020 to give a talk about horizontal scaling of graph databases (the NiFi team is also collaborating on this one). Ryan earned an M.S. in Information Systems Engineering from Johns Hopkins University and a B.S. in Software Engineering from Robert Morris University. He has significant experience developing dataflow pipelines, enriching and manipulating streaming data, and in the storage of large dataflows. Driven by an endless desire to “find a way,” Ryan is also an inventor, he holds two U.S. patents. In his free time, he enjoys his young family, playing hockey, and working on his 1966 MGB.
Elli Schwarz is a Senior Software Engineer at Clarity. He has 15 years of experience developing Java applications, creating custom data processing solutions, and applying specialized data models and ontologies to facilitate data exchange. An Apache NiFi enthusiast, he pushes NiFi to its limits to perform complex ETL tasks for his clients. Elli has experience with various NoSQL databases such as MongoDB, ElasticSearch, AllegroGraph, Neo4j, and Redis. He holds a B.S. in Computer Science from the University of Maryland, Baltimore County.
More on Data Works MD
From the website: Data Works MD is a monthly gathering of professionals, students, and enthusiasts living and working in the Maryland area that come together to discuss diverse topics related to data science, data analytics, data products, software engineering, machine learning and other data engineering topics.
Meetups are held locally, usually once a month. Each event includes time to network with other members, sponsors and partners.
iSchool at the Loft: TJ Greenier talks about running a technical consulting firm.
Our work with students at the University of Maryland’s iSchool in College Park continued this spring. TJ Greenier participated in the iSchool’s Capstone Mentoring Panel — a discussion with undergraduate seniors that took place as the semester progressed. In April, we welcomed the students to our Collaboration Loft for two special presentations.
What is the Capstone Project?
During their final year of study in the iSchool’s undergraduate and graduate programs, students must complete a Capstone project. Some of them worked with Clarity on an information management challenge we provided.
At the iSchool’s recent Capstone Mentoring discussion, along with three other panelists, TJ provided details on his background and experience, some of the work opportunities and challenges he has faced, and the management processes he has used. He talked about collaboration, project management, and consulting in technology. TJ gave the students advice about seeking experience through internships, entering the tech workforce, and interviewing for jobs. After the panel presentation, students asked thoughtful questions about careers in technology.
A Field Trip to our Collaboration Loft
In late April, students and staff from the iSchool joined us at our office space to learn more about what it takes to start and run a successful small technology company like Clarity. In addition, Bill Farmer, VP of Engineering, presented our team’s 2019 NiFi CodeMash project and demo. He also provided a machine learning demo that processed images to determine the different types of dogs pictured.
Bill Farmer shares Clarity’s NiFi project & demo.
“The experience was designed to give the students some insight into what a technical small business does — hopefully getting them excited about the types of work opportunities out there in a technical field.” — TJ Greenier
More on UMD iSchool
The University of Maryland College of Information Studies (iSchool) is located in College Park. It was founded in 1965. From the institution’s web site: From labs to libraries, we are combining principles of information science with cutting edge technology to foster access to information, improve information interfaces, and expand how information is used in government, education, business, social media, and more. The College is ranked in the top ten nationally by US News and World Report. Visit ischool.umd.edu for more information.
When we’re not coding and consulting, our team enjoys getting together in pursuit of fun. Check out what we’re up to outside the office:
SNOW is a Weather Phenomenon & a State of Mind
At the start of the new year, we reintroduced a collaborative three-month wellness initiative: SNOW (Start Now On Wellness). The effort is designed to encourage participation in healthy activities for the body, mind, and soul. We share activities, personal progress reports, and more. There is a monthly prize drawing for participants. The biggest reward? — Inspiration from the team.
#Siplocal — Brewery Tour & Tasting
We enjoyed a taste of Ireland and Baltimore at theGuinness Open Gate Brewery this winter. Our guided tour group learned all about how one of the world’s most loved beers is made here and in Dublin, the history of the Baltimore facility (originally a distillery, it was the first to open in the U.S. after Prohibition ended), and more. There were tastings, slow pours, and delicious Maryland seafood appetizers as well.
Start Your Engines — The Clarity Cup Returns
A handful of years ago, we began what became a racetrack tradition, Clarity’s outing to The Autobahn Indoor Speedway in Jessup. In a friendly and fast competition, we race around the track in electric go-karts. After three rounds, one of us is declared the winner of The Clarity Cup. This year’s race is sure to be another thrilling championship. We’re heading back to the racetrack for our fifth annual Clarity Cup in April.
Great Wine & Food in Annapolis
It’s no secret that we’re into teamwork and business partnerships… We also enjoy a perfect pairing. That’s why every May, when the weather is just right, we plan an outing to Annapolis. Employees and their guests enjoy delicious appetizers and wines that complement the food.
#claritygear Photo Contest
We love our Clarity gear. Thealmost-IG-famous #claritywaterbottle kicked things off a few years ago with appearances in our travel photos and sports pics. To share team fun, every summer, we run a contest — employees post photos of Clarity gear on social media and tag them with #claritygear, #claritywaterbottle, #claritymug, or #clarYETI. Each month, May through September, we draw a winner for a gift card.
We recently contributed to a very special Backpack Buddies program through the local charity, Services from the Heart. Clarity donated $100 per employee, along with an additional amount to round our total gift to the organization up to $3,500.
Healthy Meals for Children in Need
Every week, Services from the Heart provides bags with healthy meals and snacks for children in need. The food packs are given to some of the students who receive free or reduced lunch at specific Anne Arundel County schools to sustain them over weekends, holidays, and during school breaks.
Food items that might be included in a bag are raisins, peanut butter, fresh fruit, breakfast bars, and shelf-stable milk.
About $300 pays for a year of food for one child. Our donation is estimated to cover more than six weeks of supplies for 655 food bags.
To have even more impact, we look forward to volunteering with Services from the Heart later this year.
Local Food Insecurity
According to Services From the Heart, there are more than 16,000 children in Anne Arundel County living with food insecurity. The Backpack Buddies program not only provides nutrition and comfort, it also aims to reduce the impact of poverty by helping meet essential needs so that kids can learn and thrive.
From the Heart
Based in Severna Park, near our HQ, Services from the Heart is supported by many local businesses, Girl Scout and Boy Scout troops, and private citizens. We are proud to join such groups in making a difference. The organization uses the Severna Park Community Center as a packing and storage facility. Please visit the charity’s web site to learn more about their important community work.
Inspired to Give More…
In addition to our Services from the Heart contribution, we have participated in a range of charitable golf outings, supported AFCEA Central Maryland’s scholarship fundraising, and sponsored Severna Park High School sports and its celebrated Rock ‘n’ Roll Revival, as well as a ski race with Seven Springs’ ski racing team — WPRC. TJ served on the selection committee for AFCEA’s Central Maryland STEM Kickstarter grant program.
Clarity also donated to a number of nonprofits through our employee Charitable Matching Program. The organizations that benefitted included:
ICYMI: our team gave a NiFi presentation and demonstration at January’s CodeMash conference. The endeavor, put together by collaborators Ryan Hendrickson, Elli Schwarz, and Bill Farmer, emerged from our ideas and innovations program (i2).
We’re pleased to share details on this special project.
What is i2?
Clarity’s i2 program supports employee exploration and innovation providing resources, time, and team participation for concepts that could become realities. Each of us is encouraged to present ideas, receive feedback, work on a white paper or presentation, build a prototype, and see solutions in action.
In that spirit, three of our software engineers and NiFi experts worked together to create a conference session that provided a how-to with examples and demos showing basic capabilities of the data processing and distribution system.
NiFi in a Nutshell
Apache NiFi is an intuitive, scalable, and customizable dataflow system. It comes with an easy-to-use and appealing management UI, a large market of standard processors, and a vibrant Open Source Community supporting it.
“NiFi takes data from one place, processes it, routes it, and puts it somewhere else. It’s a sophisticated platform that allows you to be flexible in how you do that no matter how large
or small your dataflow.” – Ryan Hendrickson
Thanks to the Internet of Things, smart homes and cities, social media communications, and electronic financial transactions, among countless other digital phenomena, data drives almost everything. Remarkably, in one minute, Venmo completes $68,000 in peer-to-peer transactions, people produce 12 million text messages, and The Weather Channel receives 18 million forecast requests.* Demand for the capture, processing, enrichment, and delivery of useful data grows daily — even in the time it takes to read this post.
The Data Game (it’s challenging)
When processing data; its volume, velocity, variety, and value (the four V’s) are all considerations. Of course, software architecture questions must be answered as well: how is the data acquired, processed, stored, and enriched? In addition; how will analytics be run on the data and the results stored, how will analytics be run on the analytic result, and how will the results be served up on a web site? Answers to these questions help determine the dataflow stack. With its scalable and customizable dataflow system and a large market of standard processors, NiFi offers a flexible solution.
Features & Benefits of NiFi
Our team’s CodeMash session was designed to walk attendees through the NiFi basics by showing how to move data across servers and networks using the platform. It covered how to manipulate, enrich, and stream data through custom enrichment processors and in doing so, revealed the features and benefits of NiFi…
The Process is its Own Reward**
The advantages of using NiFi are many:
· Guaranteed Delivery
· Data Buffering with Back Pressure and Pressure Release
· Prioritized Queuing
· Flow Specific QoS (latency vs. throughput, loss tolerance, etc.)
· Visual Command and Control
· Flow Templates
· FlowFile Recovery
· Extensions (Processors and more)
· Classloader Isolation
· Scale Out (Clustering)
· Scale-up & down
· HTTPS & SSL
· Multi-tenant Authorization (User Mgmt)
“We’ve solved some pretty complex data flow and ETL challenges with Nifi. As we demonstrated with our CodeMash presentation, we were able to perform ETL mostly with built-in NiFi capabilities and used one simple custom data processor for a case where we needed a more custom solution.” — Elli Schwarz
To showcase NiFi’s capabilities and ease-of-use, our team ran some relatable demos. Using NiFi and implementing a variety of processors, the team polled reddit for new posts and aggregated subreddit topic names to serve up a simple web site that showed popular subreddits. They also used NiFi to consume tweets that were filtered on #CodeMash to light-up LED lights that were illuminated thanks to a Raspberry Pi that was equipped with MiNiFi.
“We were pleasantly surprised by the questions at the end of the session. People weren’t aware of NiFi’s capabilities.” — Bill Farmer
Clarity’s CodeMash 2019 Team
Ryan, Bill, and Elli collaborate on technical projects every day. Along with others on their team, they find great satisfaction in anticipating and meeting the mind-bending demands of our ever-changing world with custom software solutions. Smart, creative, and diligent, they run on mission and thrive on data science.
Ryan Hendrickson is a senior software engineer and technical project lead. He has been using NiFi for four years.
Bill Farmer is a principal software engineer, technical project lead, and Clarity’s vice president of engineering. He has been using NiFi for four years.
Elli Schwarz is a senior software engineer and lead technical contributor. He is an Elastic and NiFi guru. He pushes NiFi to its limits to solve complex challenges.
What’s Next for our NiFi Experts?
Ryan, Bill, and Elli plan to continue sharing their technical knowledge and insights. Catch them at upcoming tech user groups and conferences — follow us for announcements. They regularly attend Data Works MD Meetups in the Baltimore-Washington region.
If you’d like to request a copy of Clarity’s NiFi presentation from CodeMash 2019, please get in touch with us by email.
We discussed NiFi on The At Scale Podcast. Download it for an interesting talk about the technology — also available on iTunes and other podcasting apps.
January 2019 — we’re partnering with MongoDB to reinvigorate a local mongoDB users group. Our aim is to provide a place for developers to learn more about mongoDB, a non-relational, open source, document-oriented database.
Meetup discussions will focus on features, new releases, data modeling, schema design, and migration to mongoDB from other alternatives. Representatives from MongoDB; Clarity’s Senior Software Engineer and certified mongoDB professional, Mike Keith; and others will provide insights and share their experience.
MongoDB is the leading NoSQL database. Maybe you’d like to learn more about it? Join us! This group will be particularly interesting to anyone facing challenges with MySQL, Postgres, or Oracle options. Our meetup sessions will explore what mongoDB offers — a more flexible and scalable alternative.
Join our team at the November @DataWorksMD Meetup, 11/7, where we'll present on Apache #NiFi. Sign up: http://dataworksmd.org & information on what we'll cover that night: http://claritybizsol.com/data-apache-nifi.
❤️ This year "Hour of Code will highlight how we can all use computer science to make a positive impact on our world, from writing apps to solve a local problem, to using innovative ideas in data science to address global challenges." #CSforGood @codeorg https://link.medium.com/gpS3Ob5lQ0