Realtime Risk Management Using Kafka, Python, and Spark Streaming by Nick Evans

  • Published on

  • View

  • Download


Realtime Risk ManagementU S I N G K A F K A , P Y T H O N , A N D S PA R K S T R E A M I N G2U N D E R W R I T I N G C R E D I T C A R D T R A N S A C T I O N S I S R I S K Y3W E N E E D T O B E Q U I C K AT U N D E R W R I T I N G4W E A L S O N E E D T O AV O I D L O S I N G M O N E Y5Some Numbers$12BnC U M U L AT I V E P R O C E S S E D200k+M E R C H A N T S14kE V E N T S / S E C O N D7R I S K A N A LY S T S6Risk Analysts7W E N E E D E D T O B U I L D S O M E T H I N G T H AT S T O P S T H E H O P E FA C T O R89S PA R K S T R E A M I N G A L L O W S U S T O D O R E A L -T I M E D ATA P R O C E S S I N G . W E C A N D E C I D E W H I C H E V E N T S N E E D A C L O S E R L O O K10Intro to Kafka, Zookeeper, and Spark Streaming11Apache KafkaImage taken from Kafka docs12Apache ZookeeperImage taken from Zookeeper docs13Apache Spark StreamingImage taken from mapr.com14O L D WAY: R E C E I V E R S15KafkaReceiverSpark Enginet0 t1 t2 t3Event Event Event Event Event EventBuild RDD with Events from t0 to t1Build RDD with Events from t1 to t2Process RDD Process RDD16Problems w/ Receivers The only way to get at-least once delivery makes it hard to deploy new code Zookeeper is updated with which offsets to start from when data is received, not when it is processed Were actually duplicating Kafka17N E W WAY: R E C E I V E R L E S S18KafkaSpark Enginet0 t1 t2 t3Event Event Event Event Event EventProcess RDD with Events from t0 to t1Process RDD with Events from t1 to t219General Structure Load Kafka offsets from Zookeeper Tell Spark Streaming to create a DStream that consumes from Kafka, starting at the specified offsets Define your processing step (ie. filter out non-risky events) Define your output step (ie. POST the data to the case management software) Save Kafka offset of most recently processed event to Zookeeper Start your streaming application, and grab some popcorn!20Example Filtering: Risky Productshair extensionspharmacyvaporizergateway cardwifi pineappleiPhoneguccicannabistravel package21Risky Productshair extensionsguccicannabissweet bagnice shoestaylor swift t-shirtRDD for Time 0Filterhair extensionsguccicannabisMap{title: hair exten}{title: gucci}{title: cannab}HTTPS PostCase Management Software2223T I M E - W I N D O W E D A G G R E G AT I O N S24The Future Time-Windowed Functions A necessity for most of the non-trivial jobs Performance Tweaks We havent spent any time on this, so lots of potential for gains Machine Learning We could use the Risk Analyst decisions to build a ML model Improved Monitoring We are only monitoring the basics right now Apache Cassandra Others use it as a fast key/value store for their jobs Improved Receiverless API An API to access Kafka / Zookeeper without hard work25Icon CreditsCredit Card by Rediffusion from the Noun ProjectMoney Bag by icon 54 from the Noun Project26W E R E H I R I N GS H O P I F Y. C O M / C A R E E R S27T H A N K Y O U !@n_e_evans