ICYMI: our team gave a NiFi presentation and demonstration at January’s CodeMash conference. The endeavor, put together by collaborators Ryan Hendrickson, Elli Schwarz, and Bill Farmer, emerged from our ideas and innovations program (i2).
We’re pleased to share details on this special project.
What is i2?
Clarity’s i2 program supports employee exploration and innovation providing resources, time, and team participation for concepts that could become realities. Each of us is encouraged to present ideas, receive feedback, work on a white paper or presentation, build a prototype, and see solutions in action.
In that spirit, three of our software engineers and NiFi experts worked together to create a conference session that provided a how-to with examples and demos showing basic capabilities of the data processing and distribution system.
NiFi in a Nutshell
Apache NiFi is an intuitive, scalable, and customizable dataflow system. It comes with an easy-to-use and appealing management UI, a large market of standard processors, and a vibrant Open Source Community supporting it.
“NiFi takes data from one place, processes it, routes it, and puts it somewhere else. It’s a sophisticated platform that allows you to be flexible in how you do that no matter how large
or small your dataflow.” – Ryan Hendrickson
Thanks to the Internet of Things, smart homes and cities, social media communications, and electronic financial transactions, among countless other digital phenomena, data drives almost everything. Remarkably, in one minute, Venmo completes $68,000 in peer-to-peer transactions, people produce 12 million text messages, and The Weather Channel receives 18 million forecast requests.* Demand for the capture, processing, enrichment, and delivery of useful data grows daily — even in the time it takes to read this post.
The Data Game (it’s challenging)
When processing data; its volume, velocity, variety, and value (the four V’s) are all considerations. Of course, software architecture questions must be answered as well: how is the data acquired, processed, stored, and enriched? In addition; how will analytics be run on the data and the results stored, how will analytics be run on the analytic result, and how will the results be served up on a web site? Answers to these questions help determine the dataflow stack. With its scalable and customizable dataflow system and a large market of standard processors, NiFi offers a flexible solution.
Features & Benefits of NiFi
Our team’s CodeMash session was designed to walk attendees through the NiFi basics by showing how to move data across servers and networks using the platform. It covered how to manipulate, enrich, and stream data through custom enrichment processors and in doing so, revealed the features and benefits of NiFi…
The Process is its Own Reward**
The advantages of using NiFi are many:
· Guaranteed Delivery
· Data Buffering with Back Pressure and Pressure Release
· Prioritized Queuing
· Flow Specific QoS (latency vs. throughput, loss tolerance, etc.)
· Visual Command and Control
· Flow Templates
· FlowFile Recovery
· Extensions (Processors and more)
· Classloader Isolation
· Scale Out (Clustering)
· Scale-up & down
· HTTPS & SSL
· Multi-tenant Authorization (User Mgmt)
“We’ve solved some pretty complex data flow and ETL challenges with Nifi. As we demonstrated with our CodeMash presentation, we were able to perform ETL mostly with built-in NiFi capabilities and used one simple custom data processor for a case where we needed a more custom solution.” — Elli Schwarz
To showcase NiFi’s capabilities and ease-of-use, our team ran some relatable demos. Using NiFi and implementing a variety of processors, the team polled reddit for new posts and aggregated subreddit topic names to serve up a simple web site that showed popular subreddits. They also used NiFi to consume tweets that were filtered on #CodeMash to light-up LED lights that were illuminated thanks to a Raspberry Pi that was equipped with MiNiFi.
“We were pleasantly surprised by the questions at the end of the session. People weren’t aware of NiFi’s capabilities.” — Bill Farmer
Clarity’s CodeMash 2019 Team
Ryan, Bill, and Elli collaborate on technical projects every day. Along with others on their team, they find great satisfaction in anticipating and meeting the mind-bending demands of our ever-changing world with custom software solutions. Smart, creative, and diligent, they run on mission and thrive on data science.
Ryan Hendrickson is a senior software engineer and technical project lead. He has been using NiFi for four years.
Bill Farmer is a principal software engineer, technical project lead, and Clarity’s vice president of engineering. He has been using NiFi for four years.
Elli Schwarz is a senior software engineer and lead technical contributor. He is an Elastic and NiFi guru. He pushes NiFi to its limits to solve complex challenges.
What’s Next for our NiFi Experts?
Ryan, Bill, and Elli plan to continue sharing their technical knowledge and insights. Catch them at upcoming tech user groups and conferences — follow us for announcements. They regularly attend Data Works MD Meetups in the Baltimore-Washington region.
If you’d like to request a copy of Clarity’s NiFi presentation from CodeMash 2019, please get in touch with us by email.
We discussed NiFi on The At Scale Podcast. Download it for an interesting talk about the technology — also available on iTunes and other podcasting apps.
January 2019 — we’re partnering with MongoDB to reinvigorate a local mongoDB users group. Our aim is to provide a place for developers to learn more about mongoDB, a non-relational, open source, document-oriented database.
Meetup discussions will focus on features, new releases, data modeling, schema design, and migration to mongoDB from other alternatives. Representatives from MongoDB; Clarity’s Senior Software Engineer and certified mongoDB professional, Mike Keith; and others will provide insights and share their experience.
MongoDB is the leading NoSQL database. Maybe you’d like to learn more about it? Join us! This group will be particularly interesting to anyone facing challenges with MySQL, Postgres, or Oracle options. Our meetup sessions will explore what mongoDB offers — a more flexible and scalable alternative.
We are collaborating with students at the University of Maryland’s iSchool in College Park. This fall, we outlined a specific business challenge for the students. To address it, they will use their information management skills to develop a Capstone project that provides possible solutions.
What is the Capstone Project?
During their final year of study in the iSchool’s undergraduate and graduate programs, students must complete a Capstone project. They choose from a list of real information management assignments provided by clients that have partnered with the University. Clarity is one of the organizations participating in this unique opportunity.
From the iSchool web site: Capstone projects “allow students to work through real information management challenges, providing them an opportunity to showcase their capabilities, synthesize their knowledge, and further develop their ability to communicate as information management professionals.”
Our Problem — Data Overwhelm
How do we better manage a wealth of proposal and contract information?
There is so much “data” to consider — detailed technical requirements, fluid management and staffing needs, budgets, prior teaming success, not to mention the complex business ecosystems involved.
As data scientists, we recognize the value of this rich information landscape. We’d like to dive into the data and efficiently use what it has to offer. Our primary aim is to navigate and analyze the possibilities of our contract-based business with more clarity and agility. As a result, we’ll make better-informed business development decisions. This capability becomes more critical as Clarity grows.
To solve our challenge, five undergraduate seniors will study specific requirements and research and report on ways to visually capture and analyze the complex and dynamic web of business proposal and contract information. Subsequently, two graduate students will further the project to determine how to develop and/or implement a solution. We are consulting with students in person and via email as they work to investigate, understand, and address our needs.
Next spring, students will present their final projects. As a small business, we see this partnership as an opportunity to gain creative outside input. “I’m excited to see what they come up with,” Clarity’s President, TJ Greenier, explained. “They may have different approaches that we haven’t considered.” At the conclusion of the project, we hope to use Capstone findings to improve our business development process. Our collaboration with students on a real-life business issue offers problem-solving practice and provides meaningful experience.
That’s Not All…
In addition to the Capstone initiative, our management team is conducting mock job interviews with iSchool students. We ask thoughtful questions and provide feedback on responses, promoting effective job seeking.
More on UMD iSchool
The University of Maryland College of Information Studies (iSchool) is located in College Park. It was founded in 1965. From the institution’s web site: From labs to libraries, we are combining principles of information science with cutting edge technology to foster access to information, improve information interfaces, and expand how information is used in government, education, business, social media, and more. The College is ranked in the top ten nationally by US News and World Report. Visit ischool.umd.edufor for more information.
“The work of today is the history of tomorrow, and we are its makers.”
Juliette Gordon Low, Founder, Girl Scouts
Over the last several years, Clarity has grown in big ways. We’ve evolved from a one-person shop in our early start-up days, to a team of many — consulting on a diversity of projects. Our still small-but-mighty team of engineers and technical leads provides insights, solutions, and results every day. We’re exceptionally proud of the innovations we drive and the mission we fulfill.
Women are integral to our growing team. In the last three years, they have made up 33% of our new hires — a number we are working to build upon. In this edition of our Latest News, we’re introducing some of the women at Clarity. Vital to our team, they are advancing all areas of our business. Their tech career journeys speak of creativity, problem-solving skills, a desire to contribute to the big picture, and the determination to never stop learning.
DIANE CRAFT, PRINCIPAL SOFTWARE ENGINEER,
“My best friend and her dad pushed me towards college. I was the first in my family to go.”
Diane enjoys working in a fast-paced environment where deadlines are a way of life. “What we’re doing is really cool and exciting,” she says of her team’s project. With more than 20 years of software development experience, Diane’s ever-growing technical skills include fluency in an impressive array of applications, languages, databases, platforms, frameworks, and standards. As a front-end developer, Diane knows that user experience is critical. She applies her technical skills to ensure smooth and intuitive interfaces and gets great satisfaction receiving positive feedback from clients. Hearing: “I love how this works,” makes Diane most happy about what she does.
It wasn’t always about software development. “I wanted to be a National Geographic photographer,” Diane admits of her childhood dreams. “We had National Geographic subscriptions, so I’d flip through the magazines. I asked my parents for cameras. I knew that’s what I wanted to do.” Before college, aptitude tests led Diane to Computer Science. She had a knack for math and saw the success of her cousins who were working in CS. Diane decided a technical path would be worth pursuing. During her college years, she worked full time as a junior programmer to pay her tuition and fees.
APRIL DEPETRIS, SENIOR DATABASE ENGINEER,
PROFESSIONAL PROBLEM SOLVER
“Because I had good grades, they put me in programming class, and I did well.”
With a can-do attitude, April takes on the critical challenge of administering two databases. She manages these behemoths for growth, query efficiency, changing requirements, backups, and more. Creation, migration, and building make up just part of her technical toolbox. April gains immense satisfaction from solving problems, and she takes on the challenges of database engineering with flexibility and determination every day.
“I like to build stuff,” she explains. As a young girl, April enjoyed art and making things with her hands. She worked on circuit boards, Erector Sets, and Lincoln Logs. When she was scheduled to take an elective high school sewing class, April drew a line, telling her mom she didn’t like sewing at all. Instead, the school cautiously placed her in a basic programming class. She was considered too young to master the material, but April thrived in the world of programming and resolved to pursue more technical courses. At UMBC, Computer Science was a natural fit. “I got my first IBM when I got to college,” April recalls. From there, she didn’t turn back, accepting her first development job when Visa’s Owings Mills office launched its development department.
RACHEL GREENIER, CORPORATE ADMINISTRATOR,
“I use my psychology degree every day for HR and much more.”
Rachel proved herself an essential Clarity resource even before officially signing on in 2011. Throughout the years, she has been willing to pitch in wherever necessary by seeking the training needed to get the job done. Rachel handles Clarity’s human resource matters, health care benefits, security requirements, accounting, purchasing, company outings, and the much-anticipated annual holiday party. As TJ’s closest advisor in this small business adventure, in countless ways, she has served as a guiding light through periods of growth and change. Everyone at Clarity relies on Rachel for help with any number of administrative matters. With ceaseless adaptability and a positive attitude, she takes care of our team so that our technical work stays in focus.
She wanted to be a teacher. A graduate of Frostburg State University, Rachel originally went to college to specialize in Early Childhood Education. Realizing she preferred Psychology, she switched majors and has since enjoyed practicing the skills she learned in earning her degree. Psychology has been fundamental to Rachel’s role at Clarity, in raising her kids, and as a children’s ski coach for more than 20 years.
TERRI HOBBS, SENIOR SOFTWARE ENGINEER,
JHU LECTURER “In the last ten years, my work experience has been very diverse — in addition to what I’d call ‘normal’ software development work, I’ve done some consulting and teaching.”
Terri has more than 20 years of experience in software development and related fields. She has designed and implemented a variety of applications, programmed in a range of in-demand languages, and developed data models for diverse document collections to support information retrieval and presentation. Her management experience includes leading technical staff and delivering products. Terri is a lecturer with the Engineering Program for Professionals at Johns Hopkins University, teaching the online course, “Biological Databases and Database Tools,” and serving as a faculty advisor for student projects. With her development career well established — because the subject matter intrigued her — in recent years, she pursued and earned a MS in Applied Molecular Biology. Terri also holds a MS and a PMC in Computer Science.
Terri encourages people to consider software development even if they don’t have a passion for math. When she chose an area of study in college, Terri became interested in Computer Science. “I wanted to go to a liberal arts college,” she says. “I liked science, but I didn’t like doing labs.” She liked math too, but didn’t want to be a math major. Computer Science had the right mix of topics and skills needed. Since then, Terri’s career has focused primarily on handling data, requiring very little of the math she learned in college.
BETHANY LORENZ, OPERATIONS MANAGER,
OPPORTUNITY FACILITATOR “I have always enjoyed learning. Critical thinking and strong communications have allowed me to excel in operations — I put these skills to use in a variety of business functions.”
Bethany serves as Clarity’s Operations Manager. She works closely with the Leadership Team to ensure that day-to-day activities run smoothly and efficiently. She leads company outreach and tech community involvement; works closely with customers, employees, and candidates to assess industry needs and company opportunities; and helps handle human relations, contractual relationships, recruiting, event planning, and marketing. In a nutshell, Bethany helps run all areas of Clarity’s operations and growth. Her enthusiastic involvement and impact on our success is felt every day as the Leadership Team navigates opportunities and challenges.
Her professional journey began with an interest in international travel. “I wanted to travel the world on other people’s money, so I went into International Business,” Bethany reveals. At a liberal arts college where she focused on business classes, concentrating in Marketing and International Business, Bethany enjoyed the well-rounded nature of her studies. A native of Brooklyn, New York, she launched her career in New York City, working in recruitment and development at an international law firm. When life brought her south to Maryland, Bethany worked at law firms in Washington, DC, before migrating to our region’s burgeoning tech industry.
GINA SHANEY, SOFTWARE ENGINEER,
“I like learning new things and finding solutions.”
Gina has over 20 years of programming and operations support and leadership experience in the commercial sector. For the last eight-plus years, she has worked on government projects in a software engineering and testing capacity. Her technical expertise is always growing, and she has pursued training opportunities and been enthusiastic about picking up new skills along the way. Known for her customer- and mission-focused approach, Gina is quick to credit others and thankful for the teams and leaders she’s worked with. “There have been a few people who have been very helpful in my development growth,” she explains. Excellent communications skills have made Gina a real asset to the teams she has been part of.
Her favorite subject in school was math, and she liked solving problems. But, when she was little, Gina wanted to be a psychiatrist. In the end, she knew that Computer Science was important to our future, so she studied data processing in college before embarking on a career that soon introduced her to programming, analysis, information systems management, and testing. Today, she is enjoying her software development role and still seeking opportunities to learn.
SHERRI TEARMAN, SENIOR TEST ENGINEER,
“Try to listen to your heart. Don’t put too much pressure on yourself at a young age to figure out exactly what you want to do.”
Sherri Tearman has happily dedicated over 20 years of her career to providing support to government customers. She has contributed to a variety of projects and programs, working in functional testing, usability testing, quality control, instructional design, web design, stakeholder outreach, and user and documentation support. As a Senior Test Engineer, her keen attention to detail and unmatched communication skills provide immense value to her team and the users of the software that it is developing.
“I was so excited to be accepted into the School of Engineering at University of Maryland,” Sherri says of her college years. She had taken aptitude tests that steered her towards the field. Less than two years into her coursework, Sherri took a leap of faith and moved on to pursue a degree in Fashion Merchandising and Child Development. Disenchanted by a short career in retail, Sherri looked for opportunities and pivoted to a job with a consulting firm that had IT clients. Before long, she trained and joined one of their technology teams. Today she most enjoys helping people and contributing to projects that are bringing useful tools and products to life.
written by Jane Seiss — she loves working here too
#CodeWithUs — Are you a creative problem solver who loves to learn and contribute to critical projects? We’re hiring qualified technical women and men. To learn more about joining Clarity’s team, visit our Careers page or send your resume to firstname.lastname@example.org.